If you want to see them in action, head over to our Release Radar launchpad on the Dynatrace Playground.

Smartscape gets a unified topology view and ad-hoc filters

In a significant Smartscape update, a new All topology view shows every relationship for a given node in a single graph: the infrastructure stack, communication flows, and relationships such as monitoring, load balancing, routing, and API dependencies. Where the existing Vertical and Horizontal views each focus on a subset of relationships, the All view provides a more comprehensive view of relationships, from any node in any app across the platform.

Two changes make these views faster and more focused:

• The AWS and Kubernetes views now use flat layouts instead of nested ones, bringing the same relevance-based edge fetching and priority-driven node loading used elsewhere in Smartscape for more consistent visibility across your cloud landscape.
• New ad-hoc node and edge filters let you narrow any view by node type, cloud and infrastructure labels, team ownership, environment, and other properties.

Filters work alongside segments and are saved in the URL, so you can bookmark and share a focused view with segment, timeframe, and filters all preserved.

Press Ctrl+F (Cmd+F on Mac) in any Smartscape view to find nodes by name or ID. Matching nodes are highlighted in the graph, and the legend is narrowed to matching entity groups.

For broader context on Smartscape, see The new Dynatrace Smartscape improves operational efficiency across clouds, Kubernetes, infrastructure, and more.

AI Observability gains LLM evaluation, OpenInference, and Python instrumentation

AI applications can fail without obvious indicators — returning responses at normal speed with no errors, while delivering answers that are inaccurate, unsafe, or inconsistent. Traditional performance monitoring misses this entirely.

dt-evals is a new open source CLI that closes that gap. It pulls live gen_ai.* spans directly from your Dynatrace environment. Built-in evaluators use an LLM judge to score real production interactions for faithfulness, hallucination, relevance, toxicity, bias, PII leakage, prompt injection, and drift. The judge writes structured results back to Dynatrace as business events. Evaluation scores sit alongside latency and error metrics in the same dashboards. These scores can trigger alert workflows and gate CI/CD releases based on quality thresholds the same way that performance metrics do. For the thinking behind this approach, see Evaluate LLM and agent quality in Dynatrace AI Observability and LLM evaluations as a foundation for trustworthy agentic AI systems.

Dynatrace OneAgent now automatically instruments Python applications that use AWS Bedrock, OpenAI, Azure OpenAI, and LangChain. Dynatrace captures distributed traces, logs, and AI-related telemetry for supported model interactions — provider, operation, model, duration, token usage, and prompt and completion metadata where available. To capture prompt and completion content, go to OneAgent features and turn on Python OpenAI prompt capture.

The same visibility extends to teams using OpenInference with OpenTelemetry (OTel). Dynatrace ingests OpenInference traces and normalizes them to the same gen_ai.* attribute schema — covering model usage, token consumption, prompts, completions, agents, tools, embeddings, and guardrails — so OTel-instrumented applications get consistent telemetry without switching instrumentation frameworks.

As AI adoption grows, evaluation and instrumentation together turn AI services into observable, governable assets rather than black boxes.

Logs gains pattern analysis, Kubernetes insights, and in-context traces

Log analysis gets three meaningful upgrades.

Log pattern analysis (Preview) lets you aggregate query results in Logs into patterns that cluster similar logs together. You can focus quickly on recurring errors, reduce thousands of similar logs to a handful of patterns, recognize the changing parts of a pattern (and their datatypes), and reuse the generated Dynatrace Pattern Language (DPL) for other queries or in OpenPipeline.

In-context trace details mean that when you investigate a log entry with trace context, you can open the associated trace directly inside Logs. A waterfall icon signals that you stay in context rather than navigating away to Distributed Tracing.

Log insights in ready-made Kubernetes dashboards provide built-in log analytics for clusters, namespace workloads, namespace pods, and node pods. Error log counts appear alongside health metrics, with log level distribution and severity trends below. Direct links to the Logs app ensure that a deeper investigation is only one click away.

Faster service investigation with the Services Explorer Preview

The Services app now includes a visual service map that overlays performance and health indicators on service-to-service relationships and messaging flows. It’s the fastest way to understand blast radius during an incident, providing a single view of topology context, performance signals, and bottlenecks without switching views.

You can also filter services directly by primary Grail fields such as k8s.cluster.name, k8s.namespace.name, aws.region, and azure.location — the same attributes that power segments across Dynatrace. Both capabilities are available in the Explorer Preview view and open for feedback before general availability; see the Community post for details.

New security integrations and a Kubernetes security tab

Threat Observability expands its ingestion options with new integrations. Dynatrace now integrates with Checkmarx for software composition analysis and container security findings, and adds CrowdStrike and Kyverno integrations — pulling detection findings and Kubernetes policy compliance data into Dynatrace as security events. For Kyverno, see Ingest Kyverno compliance findings.

Kubernetes monitoring also gets a dedicated security tab (Kubernetes app version 1.42.0+) that replaces the Vulnerability tab in the Explorer, bringing security context into the same place teams already investigate cluster health.

Runtime Vulnerability Analytics now has a native interface, replacing the legacy management-zone-based monitoring rules with a single consolidated workflow.

One change worth flagging for security teams: ingested security.events must now carry a timestamp within −1h/+10min, tightened from the previous −24h/+10min window. Events with older timestamps are dropped, so please review any pipelines that backfill security events.

Performance, drilldowns, and navigation improvements

Improved discovery of ready-made dashboards. Ready-made dashboards deliver instant insights without requiring complex queries. Finding, installing, configuring, and customizing them is now more straightforward — so new users get value faster and experienced users can build confidently on best-practice templates.

Contents tab added to all extension apps. All extension apps in Dynatrace Hub now include a Contents tab that surfaces the extension’s ready-made dashboards, so you can quickly go from installation to insights.

Session Replay has two improvements:

• Full-screen mode is now available, removing viewport constraints during playback.
• Navigating to a session through Error Inspector now opens Session Replay directly in context, keeping the investigation continuous.

Cleaner Smartscape topology. Inactive Synthetic Locations no longer appear in Smartscape, keeping topology views focused on what’s live.

Smartscape navigation for database tables and indexes. Direct navigation intents let you jump from a database node to its table or index detail view in one click.

Filters stay with you. Automated filtering suggestions scope correctly to OR and AND conditions across all apps. Filter state, search terms, and highlights survive page reloads. HTTP Status Filter selections persist through navigation steps in Distributed Tracing.

DQL durations support decimals. Duration literals (h, m, s, ms, us, ns) now accept decimal numbers — for example, 0.5h or .2m. Note, however, that this doesn’t apply to calendar durations.

More headroom in Distributed Tracing. The log viewer no longer caps at 1,000 entries, with full deduplication across trace and span IDs. Span scan limits are configurable from settings (default 5,000, up to 10,000). Field naming is also cleaned up — Smartscape fields drop the redundant prefix, and classic ME fields are clearly labeled.

More allowlist entries for external requests. You can now add up to 100 allowlist entries, double the previous limit of 50, with existing entries preserved across all environments.

Affected entity names enriched in problem records. A new affected_entity_names array is now populated alongside the existing affected_entity_ids and affected_entity_types arrays, index-aligned across all three.

This brings the 3rd-gen platform to parity with classic problem notifications and enables notification workflows and external integrations to reference entity names without additional lookup. The Problems app v1.27 reached General Availability on June 29.

Proactive Cost Intelligence across your entire stack

Dynatrace now makes it easier to understand costs, act before they spike, and optimize with less effort. New Optimize documentation walks Dynatrace Platform Subscription (DPS) customers through the full journey from understanding to optimizing costs, aligned with the FinOps Foundation framework.

The bigger shift is that Dynatrace Assist can now do the cost analysis work for you, designed to reduce the need for specialist expertise. You can ask it to:

• Understand spikes — “I received a notification that costs have increased. Can you find anything notable?” returns the root cause along with a full drilldown into your billing_usage
• Predict costs — “Based on my log ingest usage over the last 90 days, can you predict my usage for the next 30 days?” returns a capability-level forecast based on actual consumption, useful when onboarding new teams.
• Optimize usage — “Are there any log queries duplicated by multiple users?” surfaces overlapping queries with concrete suggestions to improve them.

For more on building cost discipline into your observability practice, see Driving your FinOps strategy with observability best practices.

Why these changes matter

Taken together, the June releases make everyday investigation work feel less fragmented. You get more context in the places where teams already troubleshoot: a fuller Smartscape view, AI quality signals alongside performance data, log patterns that identify root causes faster, service maps for incident response, and security and cost insights that are easier to act on without switching tools or relying on specialists.

These are the kinds of changes that add up across a week of real work.

Check out all these updates in action on our Release Radar launchpad.

The post Dynatrace Release Radar 06.26 appeared first on Dynatrace news.

Log forwarding includes metrics, spans, and events

Logs are a core pillar of observability, and in many organizations, logs serve at least a dual purpose. They drive day-to-day troubleshooting, root cause analysis, security investigations, and many other use cases.

Logs also need to be available for compliance audits, Business Intelligence (BI) analysis, and traditional Security Information Event Management (SIEM) solutions. The challenge is doing this without relying on bolted-on forwarders or rigid, costly integrations.

With the introduction of the new Dynatrace OpenPipeline capability, we’re providing you with the freedom to precisely define your organization’s forwarding, paired with flexible and unified log collection and processing.

_{Figure 1. OpenPipeline provides rich configuration and customization options for each forwarding setup.}

Update – June 2026

Data egress will be in General Availability starting June 17, 2026.

OpenPipeline now supports forwarding for all data types ingested into Dynatrace, including metrics, spans, and events, giving you a single, unified pipeline for your entire observability dataset. The same principles described below apply across the board: forward before or after processing, export in open formats, and route to the storage or third-party destination of your choice, regardless of signal type.

Read on for the full details of how OpenPipeline forwarding works.

Centralized log collection and processing

OneAgent, Cribl, OpenTelemetry, AWS Firehose, journalD, Logstash, and Fluent Bit … Have you heard of any of these tools? Maybe you’re using all of these at the same time.

With Dynatrace, you have the choice of using one or all of these tools simultaneously. Dynatrace OpenPipeline allows you to process and transform them in the same way, regardless of their source, and to move seamlessly from one shipping method to another. Just as our customer, United Wholesale Mortgages, successfully consolidated their tools while moving logs off Splunk.

Forwarding logs un/processed

Depending on your industry or geography, you might be required to retain logs in an unprocessed state on third-party-managed storage, just as you might have used tape backups in the old days. With Dynatrace OpenPipeline, you can configure a forward-before-process to retain log events in an unaltered state.

Alternatively, you can first send log events to your defined processing pipeline, extract or convert the logs into metrics, and even drop unnecessary details before determining what to forward-after-processing to your object storage.

The last step in the pipeline is to decide whether to route log events to a Dynatrace Grail® bucket for retention or drop them.

Overcome vendor lock-in with precise, open forwarding

While other log management solutions provide proprietary log archive formats, you can overcome these challenges with Dynatrace and log forwarding for data archiving.

The Dynatrace OpenPipeline log forwarding capability delivers log events in a standardized and compressed NDJSON (Newline Delimited JSON) (.JSON.GZ) archive format, for which you can define the details, such as segmentation and filename prefixes.

Residing on destinations like AWS S3 or Azure Blob storage, other 3^rd party solutions such as Microsoft Sentinel, Snowflake, or Tibco Spotfire can easily leverage these archives for use cases not covered by your observability platform.

Cost-effective long-term retention for logs

Dynatrace Grail offers cost-effective log retention for up to 10 years, accessible at any time with the same speed as on day one of ingestion, supporting a daily log ingest volume of 1 PB per tenant.

Many of our customers already retain years’ worth of logs today, always accessible without the headaches of re-ingestion, re-indexing, or storage tiering and scaling concepts.

Still, there are use cases where aged logs become low-value, low-touch data.

You may want to keep certain log events available in Grail for the first 60 or 90 days for troubleshooting purposes, and then store a copy in an archive for years.

NDJSON-exported log archives are an ideal file format for further increasing cost-effectiveness and are ready for handover to other teams or parties for investigation, thanks to the standardized JSON.GZ format. These are especially valuable for use cases like:

• Reviewing access logs during a post-mortem security incident
• Providing transaction logs to fulfill a court order
• File access audit-trial review

Tool consolidation with OpenPipeline and log forwarding

Log archiving presents an opportunity and an integration strategy for solutions that don’t yet integrate seamlessly with Dynatrace.

More importantly, OpenPipeline provides you with processing and transformation capabilities, volume control, and a reliable cadence for transmitting your log events.

You no longer have to rely on custom-defined scripts for the splitting and shipping of your logs. And there are no worries about truncating large log events; Dynatrace supports up to 10 MB.

Forget about all the retransmission headaches with custom log shippers when a connection breaks; all this is baked natively into Dynatrace components such as OneAgent, ActiveGate, and others.

Start leveraging OpenPipeline and log forwarding today and retire your third-party forwarding tool stack.

For complete details, go to Log Forwarding in Dynatrace Documentation.

The post Flexible vendor-agnostic log forwarding with OpenPipeline appeared first on Dynatrace news.

The real OpenTelemetry challenge: When OTel meets scale

Standardizing on OpenTelemetry gives teams flexibility and control in modern cloud native architectures. The instrumentation works. The data flows. The collection is solved. However, as organizations scale out OTel, they encounter the analytics gap- and the cost continues to climb without clear returns. Can your observability platform turn millions of spans and log lines into clear, contextual answers without manual correlation or vendor lock‑in?

While having OTel data is exciting, many teams find themselves asking, ‘What’s the actual payoff?’ Engineers may occasionally explore telemetry data, but without intelligent analytics to connect the dots, the data often remains underutilized. Meanwhile, managers struggle to quantify the value of their investment, especially as costs climb with scale. This is the natural challenge of DIY setups, where teams focus on data collection but rarely think critically about turning that data into actionable insights.

Anyone can analyze a single trace; that’s easy. Deriving answers from millions? That’s the hard part. The promise remains unfulfilled. Until now.

Why OpenTelemetry + Dynatrace changes everything for teams

Dynatrace closes this gap. With Dynatrace, all telemetry signals are combined to give you the insights you need:

• Comprehensive failure analysis across large trace sets
• Response-time insights revealing performance patterns at scale, with logs and exceptions in context
• Deep visibility into database and queuing systems
• AI-powered intelligence delivering contextualized insights
• Enterprise operational controls providing cost allocation, secure data handling, and scalable telemetry management

Your OpenTelemetry data transforms into actionable contextualized answers that help you move faster, ship confidently, and get back to building.

Complete enterprise coverage for OpenTelemetry

Dynatrace brings OpenTelemetry for Enterprise to life through specialized analysis designed for practitioners troubleshooting issues in Kubernetes environments, available in our extended Services app. AI-powered intelligence, including anomaly detection, ensures you get answers faster, without losing context.

Failure analysis: from single traces to mass insights

When your booking service fails, you can see the complete story: failed traces, related log entries, specific database statements, exceptions- all automatically correlated in one view. The Failure Analysis also provides a visual investigation of your OTel data.

Time-based comparisons allow you to overlay current failures against previous windows, instantly identifying regressions- what was stable yesterday and failing today becomes obvious. And there’s more: advanced visualization distinguishes between different failure types and severities, automatically categorizing them so you can prioritize based on actual user impact.

It’s a new, intuitive way to explore data visually with full context- analyzing failure patterns across your entire architecture and understanding how problems flow through distributed systems. Derive answers from millions of spans that individual trace inspection would never reveal.

Response time analysis with full telemetry context

Mass data analysis extends to performance insights. Dynatrace delivers response time analysis and comparisons built for practitioners. The platform allows you to easily compare failures between two time windows to spot exactly when things degraded.

See how response times correlate with database performance, downstream dependencies, like calls or queue interactions, and infrastructure resource utilization.
Dive in visually, explore the correlated context, and understand what’s happening- all in one place.

Database queries: understand service-to-database interactions

Modern services thrive or fail based on their interactions with their databases. Dynatrace provides comprehensive database analysis for OpenTelemetry-instrumented services, showing exactly what your services are doing against your databases.

Get immediate visibility into your most expensive queries across your entire environment- whether it’s Cassandra, SQL, or other databases. Queries are automatically ranked by cumulative duration (query count times query duration), surfacing what’s actually costing you performance.

When troubleshooting an individual service, you immediately see which database calls are problematic. You can also examine patterns across many services: identifying query problems, detecting spikes, or discovering when services start overwhelming databases with inefficient calls. The view aligns with how your architecture actually functions.

Cloud native queuing systems support

Modern applications stream data through Kafka, RabbitMQ, MQTT, and SQS, sending thousands of messages per second through distributed architectures. Dynatrace delivers comprehensive visibility into these message processing interactions with dedicated metrics, dashboarding, and alerting designed specifically for how modern streaming systems actually operate.

See which services are publishing or receiving messages from which queues, with full performance metrics. Advanced filtering lets you explore your entire environment or drill into a specific service’s queue interactions.

Exception analysis: uncover patterns and failures

We’ve only scratched the surface of how service analysis capabilities can make an impact. From the Services app, you can seamlessly navigate to related traces in the Distributed Tracing app, which now includes extended Exception Analysis. This enhancement surfaces exceptions across traces with readable stack traces, aggregated insights, and visual markers to highlight problematic spans.

By analyzing exceptions in context, teams can quickly identify patterns, prioritize fixes, and reduce MTTR. Whether leveraging OneAgent or OpenTelemetry, no critical issue goes unnoticed, providing complete visibility and reliability across modern environments.

AI-powered intelligence: pinpoint the needle in the haystack

Dynatrace AI delivers actionable insights through baselining, anomaly detection, and precise alerting, continuously learning your environment’s behavior. From day one, these capabilities surface meaningful deviations with full context, enabling teams to act quickly and confidently.

By analyzing OpenTelemetry data, you can detect trends, predict potential issues, and get intelligent, context-rich alerts. This ensures teams can focus on what matters most- resolving problems faster and optimizing performance- without manual effort or guesswork.

Enterprise operational controls that scale

Beyond analytics, enterprise teams need operational capabilities that work with OpenTelemetry data:

• Primary fields and tags: Use your existing Kubernetes labels and cloud tags (AWS, Azure) to filter and organize telemetry data. Filter by namespace, cluster, deployment, or custom business dimensions to focus on what matters most.
• Cost allocation: Track and understand costs by subscription, project, or resource group to optimize spending and ensure efficient resource usage.
• Pipeline routing and processing: Route telemetry data to specific pipelines based on cloud provider, region, or cluster. Control how data flows through your observability stack to improve efficiency and ensure compliance.
• Bucket assignment: Assign data storage by environment, account, or custom dimensions. Optimize retention and costs while adapting to operational requirements.
• Security context: Tag data with permissions and access controls, so teams see only the namespaces and services they’re authorized to access.

These aren’t add-ons; they’re core platform capabilities that work identically whether you use OpenTelemetry or OneAgent instrumentation.

The bottom line

Success comes from choosing the analytics platform designed for practitioners in modern environments- one that delivers insights across millions of signals with AI-powered intelligence. Dynatrace meets you where you are with the “Data in Context” advantage: every signal works together with the enterprise capabilities that cloud native environments demand.

The enhanced Services app and the Distributed Tracing app are now available for Dynatrace Platform Subscription (DPS) customers.

Check out the Dynatrace Playground to experience OpenTelemetry for Enterprise firsthand.

Join us at Perform in Las Vegas, January 26-29!

The post Data in context: How Dynatrace solves the OpenTelemetry analytics challenge appeared first on Dynatrace news.

Large enterprises such as Western Union, Vodafone, and United Airlines are ditching legacy log solutions in favor of a single, unified observability platform that delivers real-time insights and scalability at the petabyte level, as you’ll hear firsthand from them at Perform 2026.

In this blog post, we’ll look back at the log-focused Dynatrace product releases of 2025, while keeping in mind the three benefits that customers love most about Dynatrace:

1. Fast log onboarding with unified ingestion from any source
2. It’s easy to get started, yet powerful for your daily work
3. Productivity boosts with Davis AI

Boost productivity with Davis AI

The promise of “Logs in Context” is simple:
Find the right log line at the right time, automatically and powered by AI.

The magic of Dynatrace is not a single feature or hyped AI. It’s the sum of many Dynatrace capabilities that comprise the foundation of the Dynatrace platform: Grail®, Smartscape®, Davis® AI, OpenPipeline®, and many others, that come at no extra cost, providing the automation and assistance you need.

If you aren’t yet using Dynatrace for your logs, stop stitching together clues across tools and say goodbye to manual swivel chair ops:

• Logs in context: The right log lines appear automatically within the workflow or Dynatrace app you’re using. Whether that’s troubleshooting a service, reviewing Kubernetes node health, or investigating performance incidents of Infrastructure or cloud native apps.
• Free of charge: Every in-context query, including surrounding logs, is now zero-rated (non-billable) when you view logs inside these Dynatrace core apps: Clouds, Infrastructure & Operations, Services, and Distributed Traces. While these apps don’t generate query consumption, ingestion and retention consumption are billed individually. We’re delivering the logs you need to take action – instantly, efficiently, and automatically correlated.
• Leverage the power of Dynatrace Davis AI: With Dynatrace, features like “Explain logs” dramatically shorten time to action. Our customers report that their teams can more easily understand the possible causes and impacts of incidents without having to manually search for error codes in logs on Google.
• By leveraging Davis AI, Workflow Automation, and integrations such as our ServiceNow partnership, customers can dramatically reduce the number of incidents; one of our customers reported reducing MTTI by 90%.

AI summaries are available across the Dynatrace platform and MCP server.

With Dynatrace, observability is not limited to cloud native apps. These features work seamlessly across cloud native, on-premises, hybrid, and traditional IT stacks. So, whether you’re on Kubernetes, a Mainframe, or an AWS Lambda function, the experience is the same.

Effortless for everyone, powerful for experts

Once your logs are ingested, you need to be able to understand them. This is where our Logs app shines for both new and expert Dynatrace users.

Pre-defined and admin-curated views boost productivity

Earlier this year, we improved the simplicity of applying complex and advanced queries with new data segmentation and advanced filters.

Using segments, admins and power-users can provide reusable and pre-scoped filters. When paired with dynamic variables, users can easily modify filter conditions.

Simultaneously, we continued enhancing the Logs app to provide advanced click-to-filter capabilities in various areas, like pinning frequent queries and filters:

• Filter field: Suggest attributes, operators, and entities
• Facets: Gain a quick understanding of patterns and groups, or build queries
• Advanced filtering: Intuitive click-to-filter side pane, including JSON-structure log support with nesting

JSON‑structured log handling

Log messages aren’t always clean. A field might be hidden inside a nested message attribute or buried three levels deep in nested JSON.

Dynatrace log handling:

• Detects and normalizes JSON.
• Exposes nested fields in the UI without manual mapping.
• Provides human-readable log messages in the results across all apps that use logs.

This way, you and your users can focus on analysis, not plumbing and normalizing logs.

Correlation at scale

With Traces on Grail, your traces are automatically correlated in context with surfaced logs within the Distributed Tracing app, including associated exceptions.

The value you and your teams gain

If you’re accustomed to working with traces, you can continue using your troubleshooting routine and easily navigate from traces to logs and error exception messages. If you prefer to start your work by focusing on logs, you can achieve the same outcome.

Remember, Logs in context are free with the Distributed Tracing app!

Fast log onboarding with unified ingestion from any source

You want all your logs, and you want them fast. You don’t want to wrestle with YAML files, forward scripts, or configure custom collectors.

Centralized configuration, self-service management, and enabling teams with granular permissions to collect and ingest logs—these are what customers asked for:

• OneAgent + Journald – Enhanced capabilities for automatically capturing logs on Linux machines with a single, centralized, configured agent: Dynatrace OneAgent®. Just deploy and watch the logs magically appear in your tenant.

Kubernetes logging made easy – The Dynatrace Kubernetes Logs Module gives you complete visibility without requiring OneAgent to operate in Full-Stack mode or to configure OTel manually.

• Log Onboarding Wizard – To further simplify the onboarding experience, we’ve introduced a new wizard across several apps. When logs are missing, or you manually launch the wizard, it provides guided steps to onboard your logs, including creating an API key.

Scale that never breaks

You can ingest up to 1 PB of logs per day per tenant, which should eliminate most sizing or scaling headaches. This bandwidth is part of the Dynatrace SaaS magic: Dynatrace Grail stores and processes everything in an indexless manner and using schema on-read. At the same time, OpenPipeline® routes the telemetry according to your rules and requirements defined in the pipelines.

• Thousands of pipelines and self-service: Every pipeline has fine-grained permissions, so teams can create isolated pipelines and self-service onboarding, processing, and routing to buckets for retention.
• 120+ parsing processors – From JSON normalization to custom field extraction, you can assign a processor to a pipeline and let OneAgent do the matching magic for you. Are you using OpenTelemetry or Cribl? Matching conditions or technology attribution offers you the same experience, regardless of the log source.

  

With Dynatrace, your log ingestion stays ahead of your growth curve, no matter how many new sources you add.

Keep your costs predictable

Large enterprises often need to charge back to internal business units. We’ve introduced increased flexibility for existing features related to chargebacks:

• Retain with Included Queries: Configurable on the individual bucket level, and seamlessly combinable with the established usage-based IRQ model.
• Cost Allocation: Attribute your logs, metrics, and traces with business‑unit and product labels. This supports your FinOps efforts, as recently discussed in our blog post, Cost Allocation for Logs.

Best Practices: Not everything we delivered in 2025 was a product enhancement. We’ve also delivered a new best practices section in our product documentation, based on field feedback from pre-sales, post-sales, and support teams.

If you prefer to watch a webinar recording instead of reading, we recorded a video that walks you through all the best practices detailed in this blog post.

Dynatrace YouTube Series  | Optimize your logs: Save money and boost performance

Ready to get started?

Let’s make observability effortless, not overwhelming.

Your team can spend less time chasing logs and more time delivering value. Dive in today and experience the power of an observability platform that was built for the future of IT.

If you’re using Dynatrace SaaS with a DPS contract, all the features mentioned in this blog post are available to you. If you’re not, why not start a free trial today and experience the value yourself?

Resources

Dynatrace University – Free training that covers everything from basic log ingestion to advanced analytics.

Dynatrace Playground – Our free sandbox tenant with sample log files, ready to explore log

State of Log Management 2026 – Download the report to explore benchmark data on how AI workloads are exploding log volume and costs, and why unified observability is now essential for reliable, trustworthy AI.

The post How Dynatrace supercharged log observability in 2025 appeared first on Dynatrace news.

As onboarding and deployment become faster, self-service and automation have become a requirement; more than ever before, costs must remain predictable, attributable, and easy to report.

If you’re working to make log spend visible and fair across teams, you’re not alone—this is a common challenge in modern, and cloud native environments.

Assign cost centers and products

Dynatrace allows precise cost allocation for logs, so you can attribute log ingestion and retention to the right cost centers and cost products. This makes internal showback and chargeback straightforward.

Why this matters

• Cloud-native apps and microservices generate log sources rapidly, while shared platforms can blur ownership. Cost allocation brings clarity.
• Teams want autonomy and instant access, without surprises. Build accountability and trust with simplified and automated cost attribution.
• Service providers require accurate reporting for budgeting, audits, and governance purposes. Cost allocation makes it predictable and repeatable, with more than 60,000 cost allocation combinations out of the box and more available for our large enterprise customers, who are adopting this feature today at scale.

Cost optimization is a team sport

The era of budgets and cost optimization being a concern solely left to finance departments is a thing of the past. Teams are expected to own their budgets. Meaning that cost optimization is not a separate accounting artifact, but rather a shared accountability that each team is expected to contribute to.

Whether your teams offer services, applications, or infrastructure, they will want to leverage logs. Team-level accountability for log management begins with allocating log spend to individual products, owners, or any other method your FinOps practice uses for tracking.

With cost allocation for logs, you can take the non-disruptive route

You can leverage the established and defined annotations and labels of the source, for instance, directly from Kubernetes.

But there might be reasons you want to make that attribution at the processing stage:

• Your source might not be capable of providing annotations and tags.
• You don’t have the resources to configure each source individually to match attribution.
• You might want to take a centralized approach, rather than contacting each team individually.
• Your cost allocation requirements are too complex and require a script or a processing technology.

In Dynatrace OpenPipeline®, you can enrich your logs during processing. What may be tedious manual work elsewhere is now centralized and automated.

Whatever your requirements and expectations are, whether you need simple tagging or complex attribution rules, OpenPipeline is here to help.

Because cost optimization is a team sport, the output aligns perfectly with the most common FinOps formats, providing the exact granularity necessary to support enterprise-wide optimization initiatives.

What’s new with cost allocation

• Billing usage events for logs can now be enriched with dt.cost.costcenter and dt.cost.product.
• Attribution of cost centers and products should best take place at source, but can be dynamically processed with OpenPipeline.
• You can mix and match both attributes or use them individually. This allows you to allocate costs by business unit and product/service, allowing for granular chargeback and showback.

Example: Chargeback and showback with Dynatrace cost attributes

Many organizations use chargebacks to create accountability and transparency for shared costs by charging internal departments for the resources or services they consume, based on actual usage. An effective way to implement this with Dynatrace is to use the dt.cost.costcenter and dt.cost.product attributes together.

Consider a scenario where a central IT team provides observability services to multiple business units, such as Retail, Corporate Banking, and Wealth Management. Each unit runs several applications that generate logs through ingestion channels, such as OneAgent®, Log Ingest API, or OpenTelemetry integrations. To ensure accurate cost attribution, the IT team configures these log sources to automatically enrich each log with the appropriate cost center and product identifiers.

For example, logs generated by the Retail unit’s mobile banking app are enriched with dt.cost.costcenter: retail and dt.cost.product: mobile-app. This dual-tagging approach allows the central IT team to allocate log-related costs to the correct business unit and break down those costs by specific products or services within that unit. When billing usage events are enriched with these attributes, Finance teams can apply direct chargeback methods.

Optimize log spend with granular showback

Using the same attribution, IT teams can generate detailed reports showing how much each cost center is spending on log ingestion and retention, as well as which products drive that spend. These reports can be flexibly incorporated with other costs attributed to the same owners or products, such as query costs, using Lookup data in Grail®.

Now consider that same Retail unit. The granular attribution shows that the mobile app is responsible for 70% of its log spend. The team can now take targeted actions. For instance, it can reduce log verbosity in non-critical flows or adjust retention policies to optimize costs.

Meanwhile, central IT maintains full transparency and control over the shared observability platform. This centrally operated, data-driven chargeback model allows teams to operate autonomously without disruption, while aligning with FinOps principles.

Getting to the numbers

You can report and analyze cost allocation in multiple ways, depending on your audience, business requirements, workflows, and the tools you have.

• Dashboards: Crafting individual dashboards to visualize log ingestion and retention by cost center and product is one of Dynatrace’s key strengths. Individual filters, views, and visuals allow you to slice and dice custom dashboards for your teams.
• Notebooks: Explore and validate enriched billing usage events alongside Grail data for deeper analysis or ad‑hoc investigations. This route allows admins to align consumption data with log query insights of users in a shareable manner, as the results are stored. Non-admin users can view the results this way when the Notebook is shared with them.
• Account Management portal: Create cost management reports to track accrued costs and perform showback/chargeback at scale across business units and products, sent by email and downloadable in CSV format.
• Lookup data: Some organizations may prefer using lookup tables to allocate costs to their owners or products. This is a good fit for customers who already work with organizational structures that link owners with product and their respective cost centers. It can also serve as an additional support to track queries in your environments. Learn more about Lookup data in Grail.

With these views, IT and Finance can align on the same source of truth, driving targeted optimizations such as adjusting log verbosity in non-critical flows or tuning retention policies, while maintaining shared platform governance.

Get started: a guide for cost allocation

Let’s recap the best practices to get started successfully:

1. Inventory your log ingest channels and sources (OneAgent, API, OpenTelemetry, cloud/hyperscaler forwarders, log shippers).
2. Define attributes and assign labels for dt.cost.costcenter and dt.cost.product at the source before ingestion, for example, in Kubernetes, OneAgent, or the API and OpenTelemetry configuration of your apps and services.
3. If updating agents or code changes aren’t feasible, define OpenPipeline rules to enrich during the process.
4. Send sample data, verify attributes configured in Grail, and confirm visibility using the Logs app or your existing dashboards.
5. Iterate by team/product, expand coverage, and standardize reporting in the Account Management portal.

New to Log Management & Analytics in Dynatrace?

Ready to make log costs clear, fair, and easy to report? Define your attributes, turn on enrichment, and give your teams the accountability and insights they need—without slowing them down.

If you’re already using Dynatrace Platform Subscription (DPS), you can instantly get started with logs today! Additional resources and downloads are available in our community examples space on GitHub.

We invite you to explore our Dynatrace Playground tenant at no cost or to start a free trial to ingest your first logs with cost allocation.

The post Cost allocation for logs: Precise, flexible, and non-disruptive appeared first on Dynatrace news.

Logs are essential observability telemetry

Organizations are increasingly challenged to deliver seamless digital services, an essential component for achieving business-critical objectives. These challenges are amplified by complex hybrid cloud environments, where managing diverse technologies across cloud providers and platforms like IBM Z Mainframe becomes particularly demanding.

To address this complexity, it’s vital to unify observability telemetry and its signals across all layers of the application delivery chain, including the mainframe platform, within a single AI-powered observability solution.

Dynatrace enrichment capabilities enhance ingested log records by adding contextual metadata such as trace IDs, span IDs, and process group instance IDs automatically and without any manual tagging efforts. This allows seamless correlation between logs, application performance metrics, and traces. This is essential for accelerating AI-driven root cause analysis and significantly reducing time spent on shortening RCA and MTTx, as many of our customers can confirm.

Simplify and automate z/OS log collection

Dynatrace Log Management & Analytics extends beyond distributed technology stacks to include support for the IBM Z Mainframe platform. It automatically captures and ingests logs from monitored IBM CICS and IBM IMS regions and offers you advanced ingest rules. All collected logs are enriched automatically with topological metadata, allowing seamless mapping to Dynatrace’s topology and entity model for z/OS Hosts (LPARs) and z/OS Processes (regions).

Additionally, logs can be enriched with Trace IDs and Span IDs to precisely correlate each log line with the corresponding CICS or IMS trace or span that generated it.

This dramatically simplifies navigation for any user in your organization. By selecting the log line containing the Trace ID and Span ID, you can directly navigate to the related trace while understanding the load times and delay.

Let’s summarize what we’ve seen so far. Log enrichment significantly enhances and accelerates:

• Correlation with distributed traces, enabling end-to-end visibility across systems.
• Troubleshooting, by linking logs to specific spans or transactions—accelerating troubleshooting.
• Observability for both structured and unstructured log data, ensuring comprehensive insights regardless of log format.

Beyond the agent: Stream mainframe logs to Dynatrace with OpenTelemetry

Dynatrace OneAgent® already supports ingestion of logs out of the box. However, when it comes to mainframe environments, the story is more nuanced.

Why all logs are not created equal

Some logs—especially those on mainframes—are proprietary, customer-specific, and deeply embedded in legacy workflows. And while Dynatrace is constantly adding additional and automated coverage for additional log types, some might never be supported natively by OneAgent, simply because their structure and relevance are unique to each customer.

But that doesn’t mean they’re out of reach.

OpenTelemetry to the rescue

For logs that fall outside OneAgent’s native scope, OpenTelemetry offers a powerful alternative. By deploying an OpenTelemetry Collector, customers can stream log data from their LPARs to a distributed host—preferably to Linux, Windows, or zLinux to save MSU consumption. But even z/OS itself is an option for hosting an OpenTelemetry Collector.

The Collector supports:

• Filelog receiver for arbitrary text files
• Syslog receiver for structured system logs
• Filter processor for preprocessing and enrichment
• Concurrent export to multiple backends, including Dynatrace via OTLP

Getting logs off the mainframe

There are several ways to move logs from LPARs to distributed systems:

• SFTP: A blunt but reliable method
• z/OSMF: Offers REST API access to SMF records
• z/OS Data Gatherer: SMF REST Services
• Custom scripts or processes: Tailored to specific datasets
• Streaming frameworks: Kafka, MQ, or even FTP-to-Collector bridges

The bottom line: Just transfer log data from the mainframe to the host where the OpenTelemetry Collector is located, and it will handle everything for you from there.

There’s no strict requirement to provision a dedicated host to run your OpenTelemetry Collector. If Dynatrace OneAgent is already monitoring one of your LPARs, the ActiveGate hosting the Dynatrace zRemote component is a perfectly suitable environment for the Collector.

Preprocessing and enrichment

Both the Dynatrace Distro and the Contrib Distro of the OpenTelemetry Collector support advanced preprocessing:

• Timestamp normalization (for example, converting z/OS timestamps to Unix time)
• Resource attribute extraction (for example, job name, LPAR ID, subsystem)
• Sensitive data masking and filtering

This ensures that even complex logs are transformed into structured telemetry before reaching the backend.

The cherry on top: Dynatrace OpenPipeline

While OpenTelemetry handles ingestion and transformation, Dynatrace OpenPipeline® adds another layer of intelligence during ingestion and processing:

• Further enrich logs with business context
• Extract metrics, events, and business observability events
• Apply AI-driven baselining and anomaly detection
• Transform, mask, or drop data
• Some of these capabilities overlap with the Collector, giving users flexibility to choose where to apply logic based on performance, cost, and control.

Operlog: A prime candidate

Let’s take Operlog as an example—a system log that many customers are keen to stream. While it’s not a simple text dataset, creative solutions can bridge the gap. If you can extract Operlog records and store them as text on a Linux host, the Collector can ingest them immediately. From there, Dynatrace visualizations and alerting kick in.

Here’s a snapshot of how Operlog looks once streamed and processed in Dynatrace:

Not satisfied with only logs?

You’re not limited to ingesting only proprietary log files via OpenTelemetry.

Do you have access to metrics that are relevant for tracking the health of the subsystems on your mainframe? Would you rather feed in certain data as events instead of logs?

Just as the OpenTelemetry Collector is highly customizable, Dynatrace offers multiple ways of ingesting all these signals.

Conclusion

Mainframe logs may be complex, but they’re not unreachable. With OpenTelemetry and Dynatrace working in tandem, even the most proprietary datasets can be brought into the fold. Whether you’re using OneAgent, OpenTelemetry, or a hybrid approach, the key is creativity—and the right tooling.

What’s next

Dynatrace currently supports CICS MSGUSR and IMS Master Terminal Logs via its z/OS Agents, and remains committed to enhancing these capabilities. This includes exploring support for additional z/OS log types and expanding the scope of information captured through the z/OS Agents.

Log monitoring is also available for Linux on IBM Z and LinuxONE. For more details, see the blog post, Enable full observability for Linux on IBM Z mainframe now with logs.

Get started with Dynatrace log observability

If you’re looking to elevate your end-to-end observability and explore tailored possibilities within your specific z/OS environment, we’d be happy to connect. Reach out to us to request a demo and dive deeper into what Dynatrace can offer.

The post Significantly improve your Mainframe availability by connecting logs with traces appeared first on Dynatrace news.

OpenTelemetry excels at capturing data from any environment and service: traces flowing from microservices, metrics streaming from containers and infrastructure hosts, and logs capturing the application and service lifecycles. OTel does exactly what it was designed to do: standardize telemetry collection. But here’s what OpenTelemetry doesn’t do by design: unified observability that analyzes that data, correlates it across services, or turns it into actionable, intelligent insights.

This is where most organizations hit a wall and require lots of expert knowledge. Organizations today have more observability data than ever before, but somehow less visibility into what’s actually happening in their systems. Raw telemetry data becomes a burden rather than an asset. Engineers spend more time hunting through dashboards than solving actual problems.

This is the “analytics gap” that Dynatrace was built to solve, transforming your telemetry data from scattered signals into unified, AI-powered intelligence.

Why OpenTelemetry + Dynatrace changes everything

Here’s what makes this combination powerful: OpenTelemetry gives you standardized data collection. Dynatrace gives you intelligent analysis that goes far beyond the static dashboards and manual correlation work that other platforms require.

While many observability solutions leave it to you to build custom dashboards and manually connect the dots between your telemetry signals, Dynatrace transforms your OpenTelemetry data into insights that actually drive decisions. Your traces, metrics, and logs aren’t just stored; they’re automatically correlated, analyzed, and contextualized as they flow through Dynatrace OpenPipeline®.

When a trace shows latency spikes, you immediately see related log entries and metric anomalies automatically contextualized and correlated. Lightning-fast queries via Dynatrace Grail® data lakehouse process millions of spans at the speed of thought, making observability accessible to your entire team, not just the experts who know how to build complex queries and visualizations.

The result? Your OpenTelemetry investment becomes a competitive advantage, not just another data collection project that requires a team of dashboard architects to maintain.

Complete OpenTelemetry coverage

Here’s how Dynatrace helps you to get the most out of your telemetry data, without requiring additional agents or complex configurations:

Distributed tracing excellence

Native OpenTelemetry tracing delivers superior span and trace processing with dynamic visualization tools that transform complex distributed architectures into complete end-to-end visibility. But it doesn’t stop there; all your telemetry signals (logs, traces, and metrics) correlate seamlessly, giving you clear, actionable insights within the full context of your traces and services. Get simple answers to advanced questions by expanding your investigations with DQL for powerful analytics, including correlation of logs and traces.

Service monitoring that understands your Architecture

Comprehensive service health monitoring built on OpenTelemetry standards. Dynatrace provides intelligent service analysis, anomaly detection, and visualization that work seamlessly with your OpenTelemetry-instrumented applications. Our service monitoring goes beyond simple health checks.

When issues arise, you see exactly which services are affected and how problems cascade through your architecture, all without manual tagging, configuration, or service discovery setup with YAML files.

Intelligent metrics with full context

You get flexible metric ingestion for custom business metrics and standard application performance indicators. Your metrics connect directly to the services and traces that generated them. But here’s where Dynatrace takes it further: we allow you to unify all your OpenTelemetry signals into comprehensive service intelligence. Instead of analyzing metrics in isolation, you see how they connect to actual service behavior, request flows, and application logs. Every metric becomes part of a complete service story.

Complete log processing

Your OpenTelemetry logs are transformed from noise to narrative. Instead of searching through endless log streams and manually created dashboards, Dynatrace supports a comprehensive log ingestion and analysis pipeline, allowing you to go big with Dynatrace.

Every log event becomes part of a larger story about user journeys, interactions, services, and app behavior, all focused on your desired business outcomes and incident investigations.

Here’s where it gets powerful: you automatically get additional contextual enrichment when you direct all your telemetry signals to Dynatrace. By creating bi-directional relationships between logs and traces, where logs provide context to traces and traces illuminate relevant logs, Dynatrace evolves troubleshooting from detective power-user work into AI-driven, streamlined, and intuitive investigations.

Video: See the full story behind every trace with correlated logs.

Kubernetes native support

For teams running OpenTelemetry in Kubernetes, Dynatrace delivers enterprise-grade support that scales with your cloud native operations. Native Kubernetes handling of spans, metrics, and logs from your Kubernetes OpenTelemetry deployments automatically collects Kubernetes context for automated enrichment: namespace, cluster, and workload relationships, all without any additional instrumentation. Your existing Kubernetes labels, AWS tags, and Azure tags become first-class filtering dimensions for all OpenTelemetry data, enabling automatic cost attribution and comprehensive data permissions using your existing RBAC patterns.

The result is that your OpenTelemetry observability inherits the same operational patterns, security boundaries, and cost structures as your Kubernetes infrastructure.

Why this matters for your team

Every organization adopting OpenTelemetry faces the same challenge: turning data collection into intelligent insights. The engineering teams that succeed are those that choose analytics platforms built specifically for OpenTelemetry data.

Dynatrace transforms your OpenTelemetry investment from a data collection project into a competitive advantage. We meet you where you are. We respect your choice to standardize OpenTelemetry by simplifying its operational complexity and enhancing it with analytics that actually deliver value.

Ready to transform your OpenTelemetry data?

Open standards have clear benefits. Industry standardization makes it easier to make sense of data coming from multiple different sources, whether it’s traces, metrics, logs, or telemetry from third-party tools. Your analytics platform can deliver intelligent insights across your entire technology stack. Your OpenTelemetry investment deserves analytics that reveal its full potential.

• Want to explore specific OpenTelemetry capabilities with Dynatrace? Try them out on the Dynatrace Playground
• Boost your productivity with these quick video guides for service owners working with OpenTelemetry:

Video: Easy access to your OTel Logs and Traces

Video: Analyze Service Failure from OTel Data

Video: Easy access to your OTel & Prometheus Service Metrics

Join us at OpenSource Summit. We’ll be in Amsterdam August 25-27. Stop by our booth to see the magic in action!

The post OpenTelemetry and Dynatrace: Complete unified observability analytics for modern applications appeared first on Dynatrace news.

Enriching observability data with additional context means improved data quality, which leads to better decision-making and faster troubleshooting. Instead of switching to an external data source and searching for a specific identifier across multiple documents, you now gain immediate insights at query time, effectively streamlining your work.

In this blog post, you’ll learn how to ingest lookup data and use it to effortlessly enrich your observability data. Practical use cases outline scenarios in which lookup data improves user workflows and makes root cause analysis and troubleshooting more efficient.

How to ingest lookup data

Lookup data files can be uploaded in formats such as CSV, JSON, or XML. You can upload data files using Workflows, via API, or by creating your own custom app. Once ingested, you can query lookup data just like any other Grail data, using Dynatrace Query Language (DQL) commands like lookup and join, or built-in Dynatrace® Apps like Dashboards, Notebooks, and Security Investigator for exploratory analytics.

In addition to an uploaded data file, you also need to provide a parse pattern written in Dynatrace Pattern Language (DPL) that defines the structure of the lookup data.

Once uploaded, you can access lookup tables via the load command. Be aware that files are organized in a directory-like structure in Grail. To make it easier to find stored files, we’ve introduced autocomplete functionality. Just start typing and jump directly to the respective file.

To learn more about supported file types, available attributes for data ingest, or the structure of parse patterns, please have a look at our documentation.

Practical use cases

Populating lookup data is a fantastic choice for enriching data with additional context in several scenarios:

• Mapping error codes in your logs to readable text for streamlined troubleshooting,
• Enriching IP addresses or IDs with respective account names to convert meaningless identifiers into meaningful qualifiers that speed up triage and root cause analysis.
• Accelerating security investigations with allow lists for security data.

Enrich your data with business context

Imagine that your system’s business-relevant events logged in Grail contain product IDs, and you’d like to enrich the IDs with the vendor’s name and some additional information from an external source.

This can easily be done with lookup tables by ingesting data containing the product and vendor information. In the example below, we use the product ID as the lookup field and enrich the business events with the mapped vendor values from the lookup table.

fetch bizevents
| lookup [ load "/lookups/vendorlist" ],
    sourceField: product.id,
    lookupField: product.id

Improved insights when working with security data

In another use case, imagine a security analyst is tasked with finding suspicious login attempts to your company’s network outside of business hours. Let’s assume corporate policy allows IT engineers to work from home any day, but that is not the case for accountants. The security analyst wants to understand which usernames belong to which role. Doing this manually would mean spending considerable time cross-referencing employees with their respective roles and manually creating filters based on usernames.

Creating a lookup table containing employees’ usernames and roles could significantly streamline this work, allowing the analyst to use external data to filter and summarize more accurate results.

Filtering for malicious IP addresses

Suppose your security analyst has obtained a list of fraudulent IP addresses from a threat intelligence feed that tracks malicious IP activity. These IP addresses are associated with spam, malware, botnets, or other malicious activities that expose your applications to potential threats.

The security analyst can now store this suspicious IP list as lookup data in Grail, update it whenever necessary, use it to detect and flag requests from any listed IP addresses, and leverage the data for further analysis in Security Investigator.

Flag TOR exit nodes

Going a step further, your security analyst can identify, flag, and track requests from TOR networks. TOR is an anonymizer that hides your tracks on the internet. By rerouting your internet activity via at least three other nodes before reaching your website, the TOR network obscures where requests originate, allowing bad actors to hide their identity and explore the internet with malicious intent.

The analyst creates a lookup table and populates it regularly with the latest list of TOR exit nodes. This data is used for further analysis, for example, in Security Investigator to detect login attempts that originate from TOR.

To utilize the Dynatrace® platform’s full power and set the TOR data in context, the analyst automates the fetching, writing, and uploading of the list of IP addresses using Workflows and visualizes the data with Dashboards.

What’s next?

Lookup tables provide a method to efficiently add context to any type of data stored in Grail. They can be used to integrate operational and transactional data, supporting your users in their day-to-day lives.

Stay tuned for further updates, such as improving our existing Snowflake Workflow Connector by adding capabilities to create and manage lookup tables, and using Security Investigator to create new lookup tables or view and filter existing tables.

Are you interested in trying out lookup tables in your own environment? This new capability is available as a public preview for all customers running the latest version of Dynatrace SaaS with an active Dynatrace Platform Subscription (DPS). It is super simple to activate; head over to our documentation to learn how.

The post Enrich your Dynatrace data with the newly introduced lookup tables appeared first on Dynatrace news.

It’s 3:00 AM. Alerts are firing. Something’s broken, latency is spiking, there’s too much noise, and you’re under pressure to find the root cause fast. You need to be able to understand how your system interacts to solve the problem as soon as possible. But systems just keep getting more complex as your organization adds new technologies, AI models, and container-based microservices. That’s why, as complexity scales, so does the need for connected insights. In the world of observability, logs and traces serve distinct but complementary purposes. When used together, they unlock a powerful view into system health, performance, and behavior.

The secret lives of logs and traces

In theory, correlating logs and traces should be straightforward. However, in practice, teams often find themselves context-switching to follow the path of a trace and all the logs involved. To understand why, let’s take a closer look at the roles and responsibilities of logs and traces.

Traces: The big picture view

Traces follow the journey of a request as it moves through various services in a distributed system. They provide end-to-end observability of how different components interact, making them ideal for understanding latency, bottlenecks, and service dependencies. Distributed traces connect events into a cohesive timeline, helping engineers see how one service’s performance affects others.

Traces shine when you’re trying to answer questions like, “Where did this request slow down?” or “Which service caused the failure?”

Logs: The detailed detective work

Logs are detailed, timestamped records of events generated by applications and infrastructure. They’re rich in context, often containing error messages, debug information, and custom outputs that developers write into the code. While traces show the flow, logs show the details. Logs can exist independently of traces and are often the first place developers look when something goes wrong.

Logs shine when you’re trying to answer: “What exactly happened here?”

Don’t forget metrics and other telemetry signals

Although we’re focusing here on logs and traces, metrics and other telemetry data are also essential for observability and deeper context. For more about why it’s important to unify the full spectrum of observability signals, see What is observability and Unified observability: Why storing OpenTelemetry signals in one place matters.

The power of correlating logs and traces from a single, full-context platform

Isolated telemetry signals can lead to blind spots and wasted time searching for answers. Some of the main ways to use logs and traces are to simplify troubleshooting, enhance performance, improve security posture, and meet compliance standards.

When you can correlate logs and traces from a single source of observability data, you eliminate the constant context switching that slows down investigations. Instead of toggling between tracing tools and log viewers, you get a unified view that connects the dots fast.

Correlating logs and traces from a single platform transforms troubleshooting from a fragmented hunt into streamlined analysis, where you spend time solving problems instead of searching for information. Core technologies like Grail®, OneAgent®, and Davis® AI provide the scalable foundation while embracing open-source frameworks like OpenTelemetry for flexibility.

Cracking the case of the failed checkout: Investigating logs and traces

Not every investigation starts the same way. Sometimes a trace gives you the high-level view you need to spot an issue and dive deeper. Other times, a log entry is the first clue that something is off. In the next section, we’ll walk through two examples, one that starts with traces and the other with logs, to show how you can get the answers you need.

Scenario 1: Investigating from traces to logs

While doing some routine monitoring in the Distributed Tracing app, we notice a series of failed requests in our Kubernetes prod namespace. So we filter for unsuccessful transactions to examine them more closely.

One request stands out: “/cart/checkout”. It’s a critical transaction path, and we’re seeing failures.

We dive into the trace waterfall. Just below it, we find the logs tied to each span, giving us deeper insight. That’s where we find the message:

“error: failure to complete the order”

Digging further, another log reveals the root cause: only Visa and Mastercard are accepted, which is in line with our policy, but potentially limiting our business. This raises a new question: how often is this happening?

With a single click, we pivot to the logs app, where we can search for this specific message and quantify how many transactions may have been impacted.

This approach turns scattered signals into a cohesive story, helping us move from surface-level symptoms to actionable insights with speed and precision.

Scenario 2: Investigating from logs to traces

No matter how you start your day, whether you are coming from PagerDuty, Slack or start directly in Dynatrace through one of the many apps like Kubernetes or the Clouds app, you can always see logs in context of your investigation.

In this scenario, we’re investigating this case from another angle, starting with the logs app using the prefiltered segment for the Kubernetes prod namespace. The view is tailored to the services we own. A quick scan reveals something suspicious: numerous errors in some of the log files.

To dig deeper, we navigate in the logs app and use the content filter for “payment” and “error”, and we find several logs with the following message:

Could not charge card for user id = xxxxxxxxxxxxx

But what is causing the failure? We click Show surrounding logs, which reveals all logs associated with the trace ID. Now we can view log messages sequentially as they happened.

Investigating some of the surrounding logs, we see that the user is using a credit card other than Visa or Mastercard, which our organization doesn’t support. Now that we understand why things are failing, let’s investigate further to see if we can optimize this experience.

To understand the full impact, we pivot seamlessly to the trace view. Here, we see the full waterfall breakdown of the request: service calls, timing, and span-level metadata.

One detail stands out: it took 5 seconds for the user to receive the failure message. That’s a long time to wait just to be told their card isn’t supported.

With this insight, we can now make targeted improvements so that the user does not have to wait a long time to understand that their payment method is not supported and deliver a better user experience.

While these examples highlight how seamless navigation between logs and traces accelerates troubleshooting, they’re just one part of the story. With Dynatrace Grail and Notebooks, you can take things a step further by running advanced queries, automating repetitive tasks, and building collaborative, data-rich workflows. These tools empower teams to go beyond reactive troubleshooting and into proactive, scalable observability.

Why seamless navigation between logs and traces matters

Seamless navigation between logs and traces isn’t just a convenience; it’s a game-changer. Whether you start with a trace or a log, the ability to pivot instantly between signals means you spend less time hunting for answers and more time solving problems. It accelerates root cause analysis, improves team collaboration, and gives you the full context needed to act with confidence. This is just one example of how Dynatrace helps you move from fragmented troubleshooting to unified intelligent observability.

Ready to start investigating?

Explore Distributed Tracing and Log Management and Analytics, complete with prepopulated data in the Dynatrace Playground.

The post Logs and traces: Why context is everything for seamless investigations appeared first on Dynatrace news.

Empowering real-time insights through petabyte-scale data processing of all your data

Modern cloud native systems generate massive volumes of telemetry data, spanning logs, metrics, events, and traces. Extracting real-time, actionable insights and enabling meaningful alerting and root cause analysis requires more than just centralized data storage; it requires a scalable data pipeline. Without such a pipeline, even the most advanced analytics tools are constrained by data flow and preparation bottlenecks. To address these challenges, we launched OpenPipeline, our unified data ingest solution for the Dynatrace® platform. OpenPipeline is a customizable, scalable data pipeline for ingesting, transforming, and harmonizing data for reliable analysis and correlation.

With its latest enhancements, OpenPipeline supports real-time data processing even at petabyte scale, by reliably handling high volumes of logs, traces, and other telemetry. This level of scalable pipeline processing is essential for delivering timely insights, enabling Dynatrace to:

• Detect anomalies across distributed systems in real time
• Perform proactive security analytics and real-time vulnerability detection
• Offer real-time insights into your business processes

Scale is not only about the sheer amount of data signals. It’s also represented in the size of individual signals that can be handled. An increasing number of use cases, such as parsing JSON/XML request bodies, capturing detailed audit logs, exploring transactional data dumps, or analyzing input vectors from ML model logs, require the ability to efficiently process large log files. With the release of Dynatrace version 1.311, OpenPipeline now supports log records up to 10 MB in size, empowering you to use Dynatrace for high-volume use cases. For more information, have a look at this Dynatrace support for large log records blog post.

Break silos with unified ingestion across all data types

Alongside scale, OpenPipeline provides unified ingestion across all data types, removing silos, simplifying integration, and ensuring that all data, regardless of format or source, is enriched, contextualized, and processed in the same tool, ready for advanced analytics.

Recent enhancements to data type processing include:

• Spans: You can now configure how spans are processed, including dropping specific fields or entire records, and assign a security context for fine-grained, record-level access control. Spans can also extract metrics and route them into defined target buckets. Full ingest functionality for spans is coming soon.
• RUM data: Ingesting Real User Monitoring (RUM) data, including user events, sessions, and associated metrics, is currently in preview and will soon be made generally available for all Dynatrace SaaS customers.
• Events: OpenPipeline now supports custom processing rules for a broad range of event types, including security events, software development lifecycle (SDLC) events, business events, and Dynatrace internal system-level events.

Simplified pipeline management from setup to optimization

Historically, setting up pipelines for data ingestion involved time-consuming configuration of parsing, field mapping, and transformation logic for each data source. OpenPipeline now streamlines this process with ready-made processor bundles for widely used technologies and formats, accelerating data onboarding and standardizing data handling at scale for ingest sources such as:

• Hyperscaler support, including AWS and Azure
• Web servers like Apache, IIS, JBoss, HAProxy, or Nginx
• Programming languages like .NET, PHP, Java, Python, or NodeJS
• Databases and app frameworks like Elastic or Cassandra

With just a few clicks, you can apply a processor to a source, ensuring all fields are parsed correctly, attributes are renamed, and data structures are accurately standardized. Each bundle includes example records that can be tested interactively, with further customization available to meet specific requirements. This allows your teams to onboard new telemetry streams in minutes instead of hours. We’ll continue to expand our support to cover more technologies in the future. You can also create your own processors to handle any custom format.

Reveal hidden value with data transformation

Upon ingestion, modern observability, security, and business data can be structured in varying, often complex forms. OpenPipeline offers advanced transformation capabilities that allow for easy extraction of relevant data beyond the creation of simple events and metrics:

• Extract values from deeply nested JSON fields. For example, extract the threat identifier from a JSON-formatted security event and convert it into a security metric.
• Create unified metrics from different data types. For example, consolidate error codes from both logs and spans into a single metric for cross-service comparison.

Stay informed: Real-time visibility and alerting for your pipelines

To optimize pipelines, teams need visibility into their performance. OpenPipeline now exposes detailed metrics at key processing stages: ingest, routing, and output, as well as not-stored-records. With these metrics, you can instantly verify any pipeline configuration and detect anomalies early.

These metrics power:

• Real-time microcharts within the OpenPipeline user interface, showing data trends and ratios over the last 30 minutes.
• A ready-made dashboard used to explore daily or weekly summaries, historic volume trends, and detailed routing stats by pipeline or source.
• Smart alerting. Traffic fluctuations on ingest are common, so you can leverage AI-powered dynamic baselining to raise custom alerts and detect anomalies before an issue occurs.
• Automated operations that trigger notifications or perform autonomous remediation steps once an ingest anomaly or traffic volume deviation is detected.

Easy transition to OpenPipeline for existing customers

For Dynatrace SaaS customers using classic pipelines, we’ve simplified the transition to OpenPipeline. Start utilizing OpenPipeline for new data sources while keeping your existing log processing configurations fully operational and uninterrupted. This side-by-side, risk-free approach supports gradual adoption and allows you to modernize your data processing without disrupting ongoing workflows.

Get started today

OpenPipeline is now available to all customers running the latest version of Dynatrace SaaS. Already today, you can:

• Benefit from processing a broad range of data types, including logs, traces, events, and RUM data (currently in preview).
• Take advantage of processing the full payload of large log records, parse embedded XML or JSON structures, parse events or metrics from complex log records, or use them for deep search analytics and forensic use cases—all without splitting individual log lines into separate events. Leverage the built-in processor bundles to get started with popular formats.
• Explore the ready-made OpenPipeline dashboard on the Dynatrace Playground.
• Use real-time pipeline metrics to tune and optimize your configurations.

The post Latest OpenPipeline upgrades simplify high-volume, real-time data processing appeared first on Dynatrace news.

When you’re working with complex, multi-layered data, pinpointing the root cause of an issue or identifying anomalies can feel like trying to find a flickering light in a city skyline. Effective troubleshooting demands a deep understanding of data structures, filters, and query logic. But what happens when you’re staring at a mountain of logs or metrics and don’t even know what to look for? For instance, imagine you’re investigating a sudden spike in error rates across your application. Without knowing which service triggered the cascade or how to filter the noise, you’re left guessing where to begin.

With the introduction of deep search functionality in Grail, we’ve introduced a new way to cut through the noise of complex data. You can now explore data effortlessly without needing to understand every single layer of complexity in your data structure.

Investigating data is a time-consuming process

Difficult data investigations can be taxing. Examining data takes too long, and probing for issues is tricky for those who don’t know the complexities of the data structure. In the query example below, taken before the introduction of DQL search, we attempt to filter within spans where any event sent from an Oracle database contains the phrase “error”.

Search within complex structures

Grail’s unique data warping technology allows for index-free, schema-on-read, high-performance queries, enabling you to flexibly analyze diverse datasets. While Grail has long been capable of dynamically interpreting complex data structures at query time, with the addition of the DQL Search command, you can now deep dive into your metrics, logs, traces, and other data stored in Grail in a way more agile and intuitive way.

Simplified string-based filtering across multiple fields

You can now search across vast and complex data sets without needing to understand the underlying structure. By leveraging string-based search, the system offers intuitive exploration of deeply nested data—no manual filters or query syntax are required. This approach eliminates the traditional barriers of schema knowledge and filter construction, making data investigation accessible to everyone. (See Figure 2 below.)

Surface the needle in the haystack faster

You can search across all fields, not just indexed or predefined ones, making it easy to surface any data without needing to pre-tag or preprocess it. DQL search is optimized to be highly performant, even with terabytes of data.

Search in nested structures

Finding relevant data within nested elements or arrays is cumbersome, requiring a lot of effort to ensure you don’t exclude necessary parts of the data structure. With the new DQL Search command, you can search across complex data structures, including nested elements and arrays, to find exactly what you need.

Imagine a DevOps engineer troubleshooting a failed deployment in a development environment. Suspecting a misconfiguration or runtime error, the engineer needs to find traces containing the term “exception”—buried deep within the span.events array of nested records. The system scans across all nested fields and surfaces relevant spans instantly—saving time, reducing friction, and accelerating root cause analysis.

Now, consider another example with Dynatrace Security Investigator. You’re investigating a security incident in your Amazon cloud environment. You need to track a specific event from your AWS CloudFront logs, and you have the CloudTrail event ID value. You have no idea where to find the event ID. It could be buried in the content, tucked away in header values, or stored in an obscure field. The DQL Search command empowers you to search across all fields instantly—no filters, no field-mapping, just fast, accurate results when you need them the most.

As simple as using a search bar, inside DQL

Experience the flexibility and power of exploratory analytics in Dynatrace without worrying about indexes or schemas. The DQL Search command allows for deep, detailed investigations while maintaining simplicity, making it accessible to a wider range of users.

The post Deep Search: Transform your data exploration and investigation with Dynatrace Grail appeared first on Dynatrace news.

The Dynatrace Log Module for Kubernetes is a fully automated, full-service approach to Kubernetes log management and analytics. Dynatrace is your one-stop shop and go-to solution for all Kubernetes needs.

Logging is integral to Kubernetes monitoring

In the ever-evolving software development landscape, logs have always been—and continue to be—one of the most critical sources of insight. Log data is essential, whether you’re troubleshooting and conducting forensics into past problems, investigating potential security issues, or debugging in real time.

However, the distributed and ephemeral nature of Kubernetes means that logs are scattered across multiple nodes and pods, making it difficult to ensure that all logs are preserved, easy to access, and enriched with necessary context for future analytics.

A robust log collection solution must ensure:

• Central log management. Logs must be centralized, preserved, easily accessible, and connected with other signals from your Kubernetes environments to allow effective monitoring and troubleshooting.
• Context-aware and topology-rich logs. Logs must be enriched with metadata and contextual information, allowing powerful log analytics of all distributed traces, metrics, and events within the Kubernetes topology. This allows analysis of logs from out-of-memory containers or pods with application errors.
• Powerful access controls. The ability to define fine-grained access control for logs based on Kubernetes namespace or cluster level is essential. This ensures that users can only access the necessary logs, helping maintain the highest security, compliance, and operational efficiency.

At Dynatrace, we support whichever integration you use to collect logs from your systems, whether it’s OpenTelemetry, Fluent Bit, or other tools. Moreover, we’re continuously looking to improve our customers’ experience, which is why we’ve enhanced our fully automated, full-service log streaming solution with the Dynatrace Log module.

The updated solution allows you to:

• Stream and analyze logs from your Kubernetes environments without running OneAgent on each Kubernetes node.
• Flexibly choose the level of observability you need. For example, you can start small with only Kubernetes platform monitoring and log analytics and grow into comprehensive observability maturity with distributed tracing, security analytics, real-user monitoring, and more. This flexibility extends to privileges, ensuring you can grant the least possible number of privileges in Kubernetes to get the needed data.
• Get insights into logs from short-lived containers and pods such as InitContainers or Jobs.
• Easily onboard log analytics within the Kubernetes app and control log ingestion and management centrally to ensure an optimal experience. Rather than configuring log collection locally, as you might have done with open source log shippers you’ve used, you can now centrally define and manage which logs Dynatrace collects.
• Effortlessly get logs in context as the Dynatrace Log Module for Kubernetes is fully managed for you; Dynatrace handles all upkeep, configuration, and lifecycle management.

Let’s look at how the new capabilities integrate with the Dynatrace platform.

One platform, complete log value

Not having to lift a finger to stream your logs is awesome, but that’s not the only value the Dynatrace Log Module provides.

Central log management

You have the ability to fully manage the lifecycle and configuration of your Kubernetes log collection within the Dynatrace platform. Gone are the days when you needed to manage the configuration of a log shipper across all your clusters. Dynatrace allows you to do this on one platform. You can centrally manage and control data collection, data masking, data dropping, data transformation, and data retention vs distributed configurations at the source. Additionally, the ability to dynamically define retention periods and compliance settings gives you much more flexibility when working and allows you to get the data you need to resolve issues easily. This ensures the smooth and reliable operation of applications running on Kubernetes.

Troubleshooting and remediation

Logs are critical to understanding and maintaining system health and performance. Dynatrace helps you quickly troubleshoot and easily remediate in a variety of ways, allowing you to:

• Inspect and understand logs from crashing application containers and other workloads in the Dynatrace Kubernetes app.
• Slice and dice log data with traces and Kubernetes topology in Notebooks with DQL.
• Drive evidence-based investigations for security issues and data forensics.
• Easily derive metrics and events from logs for dashboarding and prediction.

• Automate remediations with log data in the Workflows app.
• Dive into log data to explore surrounding logs and patterns using the Dynatrace Logs app.

Cost management and allocation

Dynatrace allows you to manage costs by controlling the amount of ingested logs, the retention period for which logs are kept for analysis, and query timeframes for your analytics. You can also manage ingest and retention by filtering log sources, managing buckets, and picking the right license model for your needs.

Additionally, you can leverage existing labels or annotations to enrich log data with cost-allocation metadata, making it easier to allocate expenses accurately. This allows for precise cross-charging, helping ensure that bills are charged to the correct departments.

Log enrichment

Dynatrace enriches every log line with additional Kubernetes metadata. In addition to cost-allocation use cases, you can enrich logs with security context and other common Kubernetes context metadata.

This allows you to define IAM rules to control access to log data so that only users with specific roles or permissions can query certain logs. This ensures that teams see only the logs they need, helping your organization maintain the highest levels of privacy and security. Further, if you also monitor your applications with Dynatrace, your log lines will be enriched with span and trace IDs for context-rich analytics of log, trace, and metrics data.

Get started

Seamlessly manage your logs with the new Dynatrace Log Module. Get started today to see how Dynatrace can help you with your observability and security needs.

_{Dynatrace and the Dynatrace logo are trademarks of the Dynatrace, Inc. group of companies. All other trademarks are the property of their respective owners.}

The post Kubernetes logging made easy: Comprehensive Kubernetes observability with Dynatrace appeared first on Dynatrace news.

Fast and efficient log analysis is critical in today’s data-driven IT environments. For enterprises managing complex systems and vast datasets using traditional log management tools, finding specific log entries quickly and efficiently can feel like searching for a needle in a haystack.

Dynatrace segments simplify and streamline data organization in large and complex IT environments, providing pre-scoped data without compromising performance.

In addition to adding segments, we’ve overhauled the filter capabilities of the Logs app, introducing a new advanced filter bar.

In this blog post, we’ll explore how these features boost productivity and accelerate access to the right data sets using shortcuts like segments. We’ll also take a closer look at how the improved filter capabilities can be combined to differentiate segments.

What are Dynatrace Segments?

Segments are reusable, pre-defined filter conditions powered by the Dynatrace Query Language (DQL). Segments can implement variables to dynamically provide, for example, a list of entities to users, such as available Kubernetes clusters, for unmatched flexibility and dynamic segmentation.

Segments focus on specific, relevant data sets across apps and data types as they’re available and applied across the Dynatrace platform. Think of segments as a feature that allows you to create views for teams, departments, or application owners—or just for yourself. With segments, you can isolate particular OpenPipeline™ log sources, resource entities, cloud regions, or even certain buckets your developers use.

Segments allow you to provide a platform-wide pre-filtered and restricted data set on which you can execute and refine additional queries using the advanced filter bar.

Here’s what Dynatrace customers love about using segments in their daily work:

• Time savings with reusability
  Segments save you from manually re-applying filters for every new task, or when switching between Dynatrace Apps. For example, a segment for Service Errors in Azure Region can be applied instantly by selecting it from the dropdown.
• Dynamic and flexible conditions
  Segments can leverage variables (e.g., $bucket, $region, $entity_name) to adjust the context dynamically. This eliminates the need to create multiple static segments or filters for varying conditions using the filter bar.

For example, the Service Errors in Azure Region segments can provide a dynamic list of available regions instead of creating multiple fixed region segments.

Unsure if your team would rather use unique IDs or region names? Don’t worry; with the power of the DQL commands like concat you can easily address this for your users with a single line of code.

Simplified collaboration

Individual users and teams can share segments to ensure consistent filtering logic across apps, dashboards, or even business observability use cases. This fosters collaboration and alignment across departments and teams.

Optimized query performance

Segments narrow the available data scope in real time, improving query speed, reducing overhead, and helping to optimize consumption.

Use case from the field: Data segmentation and filtering Kubernetes cluster logs

Imagine your team frequently troubleshoots errors from specific Kubernetes deployments within your production environment. Instead of defining and applying filters manually each time you launch the Logs app, you can now create segments with predefined filters for:

• Environment: Create and define a segment to narrow the scope to your Production environment using specific cloud subscriptions or tags.
• AWS region: Using a variable, you can dynamically limit the scope to AWS regions selected from the dropdown.
• Cluster: Instead of combining the K8s cluster as an additional DQL filter statement within the AWS region segment, you can create another segment for the Kubernetes cluster.

The obvious benefit is that you can reuse the Kubernetes segment and combine and apply it independently or with other segments and filters.

• Log level: When you and your team only want to see status = error messages for the clusters in question, you can add this to the segment’s DQL statement.

It’s easy to apply the status level as a dynamically applied filter using the filter bar or by selecting the status levels within the histogram.

If you want to learn more about the required steps, Dynatrace Documentation provides a detailed step-by-step guide to getting started with segments and logs.

Advanced filter field: Precision meets simplicity

Dynatrace offers an improved filter bar with powerful operators and dynamic suggestions for available entities. The filter field and entity value suggestions allow you to refine your search precisely, building even the most complex queries quickly.

Why Dynatrace customers love the new filter field

Logical operators (AND, OR, NOT)

You can combine multiple conditions with Boolean logic for complex queries.

For example, if you operate Kubernetes clusters in multiple AWS regions, but don’t want to see the US-based regions:
loglevel = "ERROR" AND log.source = "kubernetes" AND aws.region NOT IN ("us-east-1a","us-west-1b")

Dynamic suggestions

The filter bar provides real-time suggestions for entity parameters such as Kubernetes nodes, virtual machine names, or log pipelines provided by OpenPipeline.

These filters can be added by typing or selecting a field and choosing whether to include or exclude it.

Entity searches

The namespaces of Kubernetes clusters and nodes can be quite lengthy. In addition to automatically suggesting results, the filter bar will suggest a list of results while also acting as a wildcard operator, so you can match multiple entities with a single query.

Field-specific operators

You can use advanced matching options, such as for tags:

host.tag in ("production", "critical", "region-x")

Use case from the Field: Filter field in action

Suppose you’re investigating an issue in a Kubernetes cluster in a dev-staging environment. The dev-staging cluster isn’t monitored regularly or included in an existing segment.

Watch this scenario in action.


^{Figure 5. The filter bar provides instant precise suggestions, helping you find relevant entities and resources and boosting productivity.}

Combining filter fields with suggestions of available entities and values saves time and ensures accuracy in even the most detailed analyses.

Segments vs. filters: Finding the perfect balance

While both segments and filter fields help streamline log analysis, they serve distinct purposes:

Segments are usually pre-defined by power users or Dynatrace administrators and shared across teams. While any individual user can build segments, not all users have permission to share segments.

Segments provide reusable, pre-defined views that can be applied ad hoc when running queries in the Logs app. They’re available across all applications on the Dynatrace platform. On the other hand, individual users apply the Filter field on demand when they model queries.

As a best practice, you can combine segments to initially narrow the search scope to a bucket, cloud platform, region, or cluster, and then use the filter bar to run a focused query within that sliced data set.

Conclusion: Simplified, focused, and faster log analysis

Dynatrace segments and advanced filter fields bring unparalleled efficiency to log analysis while enabling an even broader set of users to make meaningful and powerful use of your data.

Segments help you ensure data consistency across teams and departments, all backed by DQL and Grail.

By enabling predefined, reusable views combined with granular real-time queries, these features empower teams to:

• Spend less time configuring filters when they use the Logs app.
• Focus on identifying and solving critical data faster.
• Ensure consistent analysis across use cases and departments.

Ready to transform how you work with log data?

Start using segments and filters in Dynatrace today to experience faster, smarter, and more collaborative log analysis.

For a deeper dive into these features, visit Dynatrace Documentation.

The post Log filtering made easy: Data segmentation and advanced filters in Dynatrace Logs appeared first on Dynatrace news.

Log ingestion can seem daunting when getting started with Dynatrace, especially when staring at an empty screen in the Logs or Clouds apps.

To address any concerns you and your new-to-Dynatrace teams may have regarding log ingestion, we’ve introduced a new and intuitive log onboarding wizard for Dynatrace® Apps.

The newly introduced step-by-step guidance streamlines the process, while quick data flow validation accelerates the onboarding experience even for power users.

This blog post explains how Dynatrace simplifies log ingestion, whether you’re onboarding logs from your infrastructure using OneAgent®, cloud services using log forwarding, or driving open-source standardization leveraging OpenTelemetry (OTel), Fluent Bit, or any other API-based ingestion methods.

Get started with Logs: The OneAgent advantage

For most scenarios, Dynatrace OneAgent is your best friend for getting started with Dynatrace log ingestion. OneAgent is flexible and dynamic, adapting to new requirements by configuring rule sets centrally within the Dynatrace platform.

A cornerstone of Dynatrace monitoring capabilities, OneAgent boosts your log ingestion experience by automatically detecting and tagging logs based on the detected process technology—even for custom-developed applications.

1. Step-by-step setup

The log ingestion wizard guides you through the prerequisites and provides ready-to-use command examples to start the installation process. This ensures that logs flow into Dynatrace within minutes.

This includes the creation of API access tokens, given the required permissions.

The pre-defined monitoring mode settings, for example, Full-Stack, are pre-selected following your platform administrator’s guidelines. Configuration is fully customizable.

2. Automatic log discovery

Once installed in an operating system or Kubernetes cluster host, OneAgent automatically detects logs from your deployment and ingests them into Dynatrace without requiring additional local configurations. During this process, OneAgent detects and links technologies, such as Java, Docker, or Microsoft IIS, for improved parsing and log analysis.

Tagging is also available when using API-based ingestion methods or later within the platform.
In any case, this ensures that your logs are immediately available for quicker filtering and grouping, and the readability of these log lines is improved.

After successfully installing OneAgent, the log ingestion wizard provides a host selector drop-down to validate the data flow.

3. Preconfigured log rules

Dynatrace provides a set of prepared log ingestion rules, so you don’t need to create custom configurations for common technologies and services. You can refine and modify these preconfigured rules to ingest relevant logs, mask sensitive data, and use advanced options—all configured centrally at scale.

While OneAgent automatically detects technologies and tags log lines accordingly, you can also manually configure OpenPipeline™ with processing rules for other log ingestion methods, such as OpenTelemetry (OTel), Fluent Bit, and any other API-based log ingestion method supported by Dynatrace OpenPipeline.

Hyperscalers and cloud platforms: Effortless log integration

Log ingestion is equally straightforward in cloud environments like AWS, Azure, and GCP.

The log ingestion wizard provides step-by-step instructions like you’ve just seen for OneAgent but also highlights the benefits of each integration, supporting you with additional resources to streamline your real-time log forwarding setup.

One of the main differences between OneAgent and public cloud platforms is that cloud platforms offer differing methods of log forwarding, streaming, and collection. Depending on your current setup and needs, you may want to choose one platform or another.

Following the instructions and recommendations ensures that your logs are available within a few minutes, just like with OneAgent, Otel, and other API-based ingestion methods.

Log forwarding and API-based ingestion: OpenTelemetry, Fluent Bit, and APIs

If you’re already using OpenTelemetry, Fluent Bit, or another log-forwarding solution, integrating with Dynatrace is effortless.

The API-based approach is the most flexible. It’s designed for users familiar with open source standardization looking for support for OpenTelemetry semantic, granular, and advanced log pipeline management capabilities paired with token-based API authentication.

In addition, the Dynatrace Log ingestion API supports additional use cases not covered by the aforementioned methods, such as Edge Computing, IoT, or Point-of-Sales (PoS) use cases.

With any or all of these methods simultaneously, you can build and customize your instrumentation precisely how you need it and benefit from Dynatrace enterprise-ready features without sacrificing open source principles. Using Dynatrace, you can maintain control and increase flexibility in your instrumentation choices.

Why log management is easier with Dynatrace

Dynatrace is the ideal enterprise-wide observability and security platform, as it supports your current and future cloud-native log ingestion methods.

The new log onboarding process is designed with simplicity, scalability, and user experience in mind—for novice users and experts. It provides the same accelerated experience regardless of whether you use OTel, OneAgent, or API-based ingestion.

The onboarding process allows you to create required access tokens, reuse existing tokens, and validate the data flow of selected entities, pipelines, and API tokens.

The post Simplify log onboarding: From zero to observability in minutes appeared first on Dynatrace news.

We’re excited to announce that Davis CoPilot Chat is now available across the Dynatrace platform. Davis CoPilot™, launched in October 2024 to support Dynatrace users with access to their data, now extends across the platform, streamlining user onboarding and providing comprehensive support and contextual insights from various Dynatrace® Apps. With the new Davis CoPilot conversational interface, users can leverage natural language to quickly get answers to their questions, making it easier than ever for users to interact with Dynatrace.

Intuitive access to information boosts team productivity

We understand that taking advantage of the numerous features and functionalities offered by platforms like Dynatrace can be challenging. To help you navigate this and boost your efficiency, we’re excited to announce that Davis CoPilot Chat is now generally available (GA). This new feature provides information and guidance exactly when and where you need it, making your Dynatrace experience smoother and more efficient.

Davis CoPilot leverages the power of generative AI to answer your questions through a globally accessible chat interface. We’re proud to say that Davis CoPilot is multilingual: you can ask questions and get answers in many different languages, including French, Spanish, German, Portuguese, Chinese, Japanese, and, of course, English. Davis CoPilot provides immediate, accurate responses, eliminating the need for extensive searches and reducing dependency on support channels. This makes knowledge more readily available and boosts productivity and user experience for both new and experienced users.

Davis CoPilot Chat follows our recent announcement of the general availability of Quick Analysis in Notebooks and Dashboards, which makes data accessible to technical and non-technical users alike. This means you can interact with data stored in the Dynatrace Grail™ data lakehouse just by using natural language.

Simplify onboarding and quickly find what you’re looking for with Davis CoPilot

You can start using the Davis CoPilot conversational interface immediately. Simply enable Davis CoPilot and assign the relevant user permissions, and the Davis CoPilot button will appear in the Dock.

Start a new conversation with Davis CoPilot Chat by selecting it in the Dock or by pressing CTRL/CMD + I and entering your question.

New users can quickly get up to speed with Dynatrace by asking Davis CoPilot for help with basic commands, setup instructions, and troubleshooting tips. This reduces the learning curve and enables new users to become productive faster. The conversational interface provides step-by-step guidance, making the onboarding process smoother and more efficient.

If you’re already familiar with Dynatrace, you can rely on Davis CoPilot to provide detailed explanations for a wide range of expert questions related to exploring new use cases, advanced configuration topics, and building custom apps.

Here are some examples of questions you can ask Davis CoPilot:

• Onboarding: How do we start sending OpenTelemetry data to Dynatrace?
• Understanding Dynatrace: What is the difference between an event and a problem in Dynatrace?
• Exploring Dynatrace solutions: How can we comply with the Digital Operational Resilience Act (DORA) using Dynatrace?
• Configuring your environment: How do I set up an alert based on an anomaly detector?
• Developing custom apps: How can I import external table data and visualize it using the Dynatrace App Toolkit?

Get contextual assistance at the press of a button

Davis CoPilot seamlessly integrates into our use-case-specific Dynatrace Apps, offering you contextual insights and guidance at the press of a button. While we plan to release additional contextual app integrations in the coming months, several will be available a few weeks after launch, allowing Davis CoPilot to provide you with insights into:

• Kubernetes warning signals
• Individual problem details and the relationships between problems
• Database performance optimization

Simplify Kubernetes: Davis CoPilot decodes warning signals

Understanding the background and root cause of warnings often requires in-depth subject matter expertise. That’s why we integrated Davis CoPilot into Kubernetes. Instead of manually looking up error messages, Davis CoPilot translates warning signals into clear, understandable language. In addition, Davis CoPilot offers a list of typical root causes and related remediation steps. This way, newcomers can quickly become proficient, and experts can elevate their expertise to hero status.

Problems demystified: Davis CoPilot provides insights into root causes

In Problems, Davis CoPilot provides clear summaries of problems, their root causes, and the suggested remediation steps. Davis CoPilot explains individual issues in clear language from the problem details page and can perform a comparative analysis when multiple problems are selected from the list view. This helps you identify common root causes and propose corrective steps without relying on a team of experts and waiting for hours for critical insights. If you want to learn more, have a look at Wolfgang Beer’s latest blog post and learn more about recent advancements in the Problems app.

Optimize database performance: Understand query execution plans

Query execution plans provide detailed information on how a database will execute an SQL query. While these provide the raw data on how to improve query performance and reduce resource consumption, they require expert knowledge to read and interpret. Now, in Databases, Davis CoPilot can provide natural language explanations of execution plans, breakdowns of relevant details, and recommendations on how to improve statement performance. This gives non-expert database users, such as developers, the knowledge they need to optimize their application performance and database utilization.

Empower your teams with Davis CoPilot today

The launch of Davis CoPilot Chat marks the second milestone of our journey. We’re committed to continuously enhancing the assistant’s capabilities with upcoming features, including query explanations, workflow actions, and troubleshooting guides.

Get started with Davis CoPilot today and transform how you and your teams interact with Dynatrace:

• Try Davis CoPilot Chat on the Dynatrace Playground
• Activate Davis CoPilot in your environment and grant your users access.

Thanks for joining us on this exciting journey. We look forward to your feedback and to seeing how Davis CoPilot helps your teams achieve their goals.

Davis CoPilot Chat, as well as the Dynatrace Apps integrations mentioned in this blog post, will be available starting with the release of Dynatrace SaaS version 1.307.

The post Davis CoPilot expands: Get answers and insights across the Dynatrace platform appeared first on Dynatrace news.

As cloud complexity increases and security concerns mount, organizations need log analytics to discover and investigate issues and gain critical business intelligence. But exploring the breadth of log analytics scenarios with most log vendors often results in unexpectedly high monthly log bills and aggressive year-over-year costs. To give organizations the freedom to explore log analytics without barriers due to cost concerns, Dynatrace is proud to announce a new Dynatrace Platform Subscription (DPS) pricing model option called Retain with Included Queries.

With this new DPS pricing model option, customers can retain data at a fixed low cost with no additional cost to query for up to 35 days. This model provides a predictable way for customers to manage and analyze logs, drive log management tool consolidation, and reduce costs while gaining maximum value from their log data.

Based on customer feedback, we’re offering the Retain with Included Queries pricing model as an alternative to our existing usage-based plan. Both plans offer the same low ingest price. However, the new all-access plan combines retention and queries into one low price to simplify scoping and budgeting.

Retain with Included-Query pricing simplifies forecasting and annual usage calculation costs. Customers who choose this pricing option get:

• Retention cost: $0.02 per GiB per day
• No cost to query for up to 35 days
• Ingest cost: $0.20 per GiB ingested (no change)

With this approach, the whole team can leverage the power of Grail queries and dashboards without worrying about limiting query usage. Customers can configure the Retain with Included Queries option with retention periods ranging from 10 to 35 days. Customers requiring longer retention periods should opt for our existing usage-based pricing, which supports retention for up to 10 years.

Queries are included

• Predictable pricing: If you know the number of logs you ingest daily, then you’ll know roughly your total annual cost upfront, providing peace of mind and less managerial overhead.
• Simple scoping: Remove the complexity associated with predicting query search volumes. Realize cost savings immediately for high-query usage scenarios. Get started quickly!
• No cost management required: Once your configured retention period ends (a maximum of 35 days), logs are automatically deleted. No oversight is needed over query usage.

Usage-based pricing is still an option

Over time, our existing usage-based pricing is the more cost-optimized option, as you only pay for the queries your users execute, and you benefit from the competitive $0.0007 per GiB per day to retain logs for up to 10 years. Queries are charged at $0.0035 per GiB scanned. Usage-based pricing is ideal for organizations with longer retention requirements and known query patterns. This pricing flexibility allows customers to optimize their log analysis expenses by paying only for what they use.

Cost-efficient:

• Lowest upfront cost
• Charges are strictly based on query execution

Scalability:

• Ideal for businesses with varying query demands
• Adapt dynamically to usage patterns

Retention:

• Supports log storage from 1 day to 10 years
• Optimal for longer-term log analytics needs

Guidance on using both plans

The Retain with Included Queries pricing option is a great way to get started while you learn about your query usage. Dynatrace includes a ready-made cost dashboard that provides insights into query usage and DQL best practices. Once you develop best practices and are confident with your consumption patterns, you can switch to usage-based pricing to maximize the value of your DPS investment.

Innovations on the horizon*

We’re very excited about our new Retain with Included Queries pricing, but we expect to deliver more updates. This pricing model is part of our plan to introduce new features that help customers align the right pricing strategies to their use cases. With these features, customers can easily see, manage, and choose how to align the Retain with Included Queries pricing with the usage-based pricing model.

Customers will soon be able to mix and match log pricing options on a per-bucket basis and provide users with access to both models simultaneously. This flexibility will allow customers to optimize the pricing selection based on the anticipated use case associated with each bucket, yielding even greater savings and value.

Retain with Included Queries: Start here

With the Retain with Included Queries pricing model, Dynatrace now offers a more cost-effective way to get started with log analytics. Drive efficiency and get more value out your logs with this predictable pricing model while you’re building your log analytics practices.

* Disclaimer: This publication may include references to the planned testing, release, and/or availability of Dynatrace products and services. The information provided in this publication is for informational purposes only; its contents are subject to change without notice, and it should not be relied on in making a purchasing decision. The information is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for products remains at the sole discretion of Dynatrace

The post Predictable costs for Log Management & Analytics with new simplified licensing plan appeared first on Dynatrace news.

Complex information systems fail in unexpected ways. That’s why IT teams need both observability tools and an observability platform. To understand the distinction between observability tools and an observability platform, let’s start with some definitions.

What is observability?

Observability gives developers and system operators real-time awareness of a highly distributed system’s current state based on the data it generates. With observability, teams can understand what part of a system is performing poorly and how to correct the problem.

Observability is made up of three key pillars: metrics, logs, and traces.

• Metrics are measures of critical system values, such as CPU utilization or average write latency to persistent storage.
• Logs are files that record events in a system, such as the start of a subprocess or the trapping of an error.
• Traces provide performance data about tasks that are performed by invoking a series of services. They’re particularly important in distributed systems, such as microservices architectures.

Each is useful alone, but integrating all three in context gives you a more comprehensive view of a system’s state. This real-time visibility creates situational awareness and opens the door for IT use cases ranging from DevSecOps to digital experience management.

Teams gain observability from telemetry data sent by endpoints across the environment using instrumentation from a wide variety of tools.

Observability platform vs observability tools: What’s the difference?

Observability tools, such as metrics monitoring, log viewers, and tracing applications, are relatively small in scope. Teams can use them independently to gain insights into single components of larger systems. Unfortunately, they often don’t communicate with each other or offer a single source of truth. This means the teams relying on these tools—teams that should be working together—must make decisions with incomplete data. With limited visibility, teams have a narrow understanding of how those decisions impact other software components and vice-versa.

A platform approach, on the other hand, presents a more effective option for understanding observability as a whole.

What is an observability platform?

An observability platform is a comprehensive toolset designed to provide IT professionals with deep visibility into the performance, health, and behavior of complex systems and applications. Unlike traditional monitoring tools that only track predefined metrics, an observability platform enables you to explore and diagnose unexpected issues across your entire system.

Key features of an observability platform

Log management: Aggregates, processes, and analyzes system logs to identify anomalies and patterns.

Metrics monitoring: Tracks system-wide metrics, such as CPU usage or memory consumption, providing a quantitative view of performance.

Distributed tracing: Captures end-to-end visibility of user requests across services to pinpoint bottlenecks and latency issues.

Dashboards and visualizations: Presents system data in easily digestible charts, graphs, and heatmaps, allowing for quick assessments.

Anomaly detection powered by AI/ML: Utilizes advanced algorithms to identify irregularities in real time and predict potential issues.

By correlating logs, metrics, and traces, observability platforms empower teams to efficiently identify the root causes of issues, reduce downtime, and maintain optimal application performance.

Ultimately, an observability platform isn’t just about monitoring; it’s about gaining actionable insights that help IT professionals make informed decisions to ensure system reliability and scalability. An observability platform that also integrates user experience data and business context into these capabilities provides a real-time advantage that helps teams respond faster and get more done.

The case for an integrated observability platform

As applications have become more complex, observability tools have adapted to meet the needs of developers and DevOps teams. For example, in 2005, Dynatrace introduced a distributed tracing tool that allowed developers to implement local tracing and debugging. This was sufficient for monolithic applications, which were common at the time. But by 2015, it was more common to split up monolithic applications into distributed systems. The key driver behind this change in architecture was the need to release better software faster.

The shift to multicloud microservice-based architectures introduced an unintended but inevitable consequence: operational complexity. Today, developers face the challenge of understanding what happens within a system comprising hundreds or thousands of interdependent services. Observability tools that provide local tracing and debugging are no longer sufficient for either operations or development teams.

Observability platforms provide root-cause analysis

Operations teams need broad, system-wide views and focused, drill-down views into services. This visibility ensures systems function as expected and helps teams understand the conditions that cause a system failure. For example, if the average response time for a service is increasing, the operations team needs to understand the cause. It could be due to a spike in load on the service, which increases system response time.

Adding more compute resources to an application cluster could address the problem, but load spikes are only one possible cause. A database could start executing a storage management process that consumes database server resources. In this case, the best option may be to stop the process and execute it when the system load is low. The key is knowing what is the root cause of the performance issue. This is where an observability platform approach becomes a real advantage.

Observability platforms provide context

The shift to multicloud has increased complexity further, driving the need for an observability platform that can provide visibility into the operational details of distributed systems.

A microscopic view of systems is also particularly valuable to developers. Debugging can require access to low-level details about how an operation works and how it may be causing problems for a downstream service.

For example, an operation may fail 2% of the time it is performed. Developers need to know what distinguished those 2% instances from the 98% that succeed. It could be differences in inputs, such as malformed inputs from another service. It could also be a bug on an infrequently executed logic path in the service.

OpenTelemetry and modern observability tools

Collecting data from observability tools is an important part of what an observability platform does. With the spread of DevOps and microservices, the vast array of possible data formats can be a nightmare for developers and SREs who are just trying to understand the health of an application.

The open-source observability framework, OpenTelemetry, provides a standard for adding observable instrumentation to cloud-native applications. OpenTelemetry provides a standardized method to instrument, generate, collect, and export telemetry data for analysts to understand software performance and behavior.

With unified data collection formats, libraries, and utility tooling, OpenTelemetry makes data more interchangeable and integrable. This streamlining simplifies how teams gather telemetry data, but to make sense of it, teams need a modern observability platform to store the data and to help generate actionable insights from the information.

Such a modern observability platform must also scale to ingest, analyze, and store the increasing volumes of metrics, logs, and trace data. It must provide analysis tools and artificial intelligence to sift through data to identify and integrate what’s most important. This approach helps developers and operations teams understand and act on the state of a complex system.

Observability tools and an observability platform: better together

For observability that scales with cloud-native technologies, organizations need an AI-driven observability platform like Dynatrace. Our distributed tracing technology powered by PurePath 4 automatically captures and analyzes transactions at every tier of an application stack. With no code changes, Dynatrace extends distributed tracing and code-level analysis to OpenTelemetry data, service mesh, and all data from your serverless computing services.

Capturing every process from start to finish and automatically providing insights enables fully integrated, no-silo collaboration across development, operations, and applications teams. PurePath end-to-end tracing contrasts tools that require manual instrumentation and user expertise to understand performance issues.

For development, operations, security, and SRE teams alike, Dynatrace brings automation and answers rather than just raw data in dashboards. To drive better business outcomes with automatic and intelligent analysis, integrate observability tools with an observability platform.

The post Observability platform vs. observability tools appeared first on Dynatrace news.

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. To unlock business value, the data must be:

• Accessible from anywhere. Data has value only when you can access it, no matter where it lies.
• Easy to access. Simplicity accelerates time-to-value and reduces implementation and maintenance costs.
• Real time. Agile business decisions rely on fresh data.
• Precise. Accuracy provides the confidence needed for business automation.
• Contextualized. Metadata enrichment improves collaboration and increases analytic value.

The Dynatrace® platform continues to increase the value of your data—broadening and simplifying real-time access, enriching context, and delivering insightful, AI-augmented analytics. Our Business Observability solution is a prominent beneficiary of this commitment.

Business events: Delivering the best data

It’s been two years since we introduced business events, a special class of events designed to support even the most demanding business use cases. Since then, many of our customers have embraced the opportunity to explore and adopt new business observability use cases. Most of these leverage the unique capability of Dynatrace OneAgent® to extract business data from in-flight application payloads—without writing any code. Other data sources, including APIs and log files—are used to expand access, often to external or proprietary systems. Enhancing access to business data from log files is an important priority, and OpenPipeline makes this a reality.

Dynatrace OpenPipeline is a new stream processing technology that ingests and contextualizes data from any source. You can now use OpenPipeline to extract business events from log files, complementing OneAgent as a primary source of business data. This is especially important when legacy or proprietary systems don’t meet OneAgent’s environmental prerequisites; in these cases, log files are usually the preferred source of business data.

In fact, it’s likely that some of your critical business systems already write business data to log files. For years, logs have been the dominant approach many observability vendors have taken to report business metrics on dashboards. You might still be using one of these tools despite the drawbacks, which include a lack of IT context, constraints on data privacy and data retention, and, as the only convenient source of business data, a limiting lack of breadth. OpenPipeline makes migrating these use cases to Dynatrace easy, helping you overcome these limitations to realize greater value.

Use case categories

It’s helpful to categorize the nearly unlimited use cases covered by Business Observability. Two of the most important categories are:

• Business reporting, analytics, and automation. Track business metrics, key performance indicators (KPIs), and service level objectives (SLOs)—automatically and in context with IT infrastructure and services—to promote collaboration between business and IT teams.
• Business process monitoring and optimization. Monitor and optimize business processes with real-time visibility into process KPIs and detailed analytics for each step to improve customer satisfaction, increase operational efficiency, and reduce cost.

Most of the use cases in these two broad categories benefit from the flexibility that comes from multiple available sources of business data. For example:

• An airline’s reporting and analytics dashboard includes data showing flights, passengers, available seats, passenger load, revenue per passenger, flight crew staffing, arrival delays, and customer satisfaction metrics. The data may come from a mix of systems, including a departure control system (DCS), an airline reservation system (ARS), a legacy inventory control system, and a SaaS-based Voice of the Customer (VoC) solution.
• A financial institution’s loan origination business process includes a series of process milestones, including loan application, credit scoring, application review, contract generation, CRM, and loan funding. All of these steps are critical components of the process, likely to be implemented using different systems. Furthermore, to understand process health, individual steps must be viewed in the context of the entire process. For this, we use Business Flow to track, analyze, and optimize complex business processes, treating the process as an observable IT and business asset.

Use OpenPipeline to extract business events from logs

Using OpenPipeline, you can ingest log data into Dynatrace from a wide range of sources, including OneAgent, Extensions, the Log ingest API, and OpenTelemetry. The pipeline includes a configurable business event processor as part of the data extraction stage; this processor is used to extract and convert relevant business data into business events.

There are many benefits to extracting business events from log files. These include:

• Improved data privacy. Sensitive business data is separated from IT observability data.
• Improved data management. Fine-grained permission and retention policies can be tailored to individual business use cases.
• Reduced storage and query overhead for business use cases. Business events are a small, often negligible subset of log data.
• Simplified and enhanced analytics efficiency. Business events from log files are unified with business events from OneAgent, RUM, and API.

Extracting business events from logs: Configuring OpenPipeline

The OpenPipeline app guides you through the three configuration steps required to extract business events from log files:

1. Identify the source of the log file.
2. Define OpenPipeline’s dynamic routing matching criteria used to route the log lines of interest to a second pipeline.
   
3. Create the target pipeline. Use the Data extraction tab to define the matching condition that creates the business event. You need to include static or dynamic Event type and Event provider fields.
   

Within the target pipeline, you can also define processing rules, extract metrics, set the security context, and define retention periods. Log data is then processed accordingly, stored in Dynatrace Grail™ causational data lakehouse, and available for your Business Observability use cases.

The value is in the data

OpenPipeline broadens Dynatrace’s unified access to the best data to deliver the best value. Now’s the time to see how it can benefit your organization.

For more details about OpenPipeline read this Dynatrace OpenPipeline blog post.

The post OpenPipeline: Simplify access to critical business data appeared first on Dynatrace news.

Considering the latest State of Observability 2024 report, it’s evident that multicloud environments not only come with an explosion of data beyond humans’ ability to manage—it’s also increasingly difficult to ingest, manage, store, and sort through this amount of data. According to 85% of tech leaders, organizations also see a substantial rise in complexity caused by increased tool volume. This results in a siloed view of data, hindering team collaboration, making it more difficult to get to the root cause of a problem, and increasing inefficiencies. Teams need a better way to work together, eliminate silos and spend more time innovating.

Enhanced data collection with the Dynatrace Otel Collector

OpenTelemetry has reached broad recognition as open source standard for collecting and transmitting data, but many organizations struggle to fully leverage its capabilities. To strengthen this open standard further, Dynatrace provides a curated and supported OpenTelemetry collector. With the Dynatrace Otel Collector, teams gain a curated, validated integration framework that simplifies data collection and ensures reliability across diverse data sources from Syslog, fluentd^®, Jaeger™, Prometheus^®, StatsD, and more. You can find the list of use cases here.

Understand your applications with ease

Due to a lack of contextual insights and actionable intelligence, application teams often find themselves overwhelmed by data, unable to quickly identify the root causes of performance issues.

For Operations and SREs, the main task is to reduce downtime and service degradations—that is, to remediate issues as quickly as possible. Application teams, however, need more than surface-level data—they need actionable insights that pinpoint the root cause of issues, enabling faster and more effective problem resolution.

Visualize your application data

In complex application environments, visualizing data effectively is key to uncovering hidden performance trends and gaining actionable insight. Dynatrace provides rich visualizations, empowering application teams to interact with data more intuitively and accelerating their ability to identify and resolve issues.

Quickly achieve immediate value out of Dynatrace—new use cases and persona-driven experiences guarantee out-of-the-box value, easy onboarding, dedicated apps, and ready-made dashboards with rich visualizations. Interact with data intuitively and easily and benefit from immediate, AI-supported insights.

Trace your application

Imagine a microservices architecture with hundreds of dependencies. Without distributed tracing, pinpointing the cause of increased latency could take hours or even days. With Dynatrace, application teams can immediately see which services are experiencing delays, reducing the time to resolution and minimizing the impact on users.

Easily understand opportunities for performance optimization and troubleshooting with the new Dynatrace experience for Distributed Tracing. This experience provides visibility into complex, microservices-based architectures, allowing teams to Identify performance bottlenecks and optimize system reliability:

• Use a histogram view of response time distribution to pinpoint slowdowns
• Take advantage of direct interaction with tables for effortless grouping and filtering
• Streamline the feature flag analysis process
• Simplify testing procedures (for example, A/B testing)

Get access to intuitive troubleshooting:

• Simplified error and response time analysis with histograms covering failures and response times alongside dynamic analysis charts.
• Easy access to exception details in the context of a full trace for exception analysis

Expanded log insights for your applications

Only Dynatrace enables true and unified observability across all ingested logs, including infrastructure components and hyperscalers providing backend services or cloud-native frontend applications.

Dynatrace Grail™ data lakehouse is schema-on-read and indexless, built with scaling in mind. There is no need to think about schema and indexes, re-hydration, or hot/cold storage. This architecture also means you’re not required to determine your log data use cases beforehand or while analyzing logs within the new logs app.

The same is true when it comes to log ingestion. It doesn’t matter whether OneAgent^®, OpenTelemetry, or another method is used. Dynatrace Davis^® AI will process logs automatically, independent of the technique used for ingestion.

Speaking of log ingestion and OneAgent, one of the main differentiators compared to other market participants is the automatic detection and integration of technologies and the collection of the corresponding application or backend service logs. This empowers application teams to gain fast and relevant insights effortlessly, as Dynatrace provides logs in context, with all essential details and unique insights at speed. This eliminates the need for swapping tools or manual log correlation. In contrast, threat hunters, developers, or DevOps on the lookout for such a tool are provided the flexibility to manually analyze logs of all sources with the all-new Dynatrace Logs app. All of this without any complexity of re-hydration or re-ingestion of logs.

The benefits reaped are increased productivity and less likelihood of overlooking relevant log lines during timely investigations, as Davis AI automatically surfaces the pertinent details.

Analyze your data exploratively

Gathering further insights and answers from the treasure trove of data is conveniently achieved by accessing Dynatrace Grail with Notebooks, Davis AI, and data in context for advanced, exploratory analytics. Increase productivity and start automating your work with all related data in context.

Avoid flying blind by adopting software development lifecycle events

With the need for increased innovation frequency, having a clear view of the entire software development lifecycle (SDLC) is critical. The SDLC has similarities to the DevSecOps loop as it outlines the various phases a product idea undergoes before it’s released into production. While the lifecycle starts with a ticket specifying a new product idea, an actual code change often triggers various automated tasks kicking off the next phase. These automation tasks process the code change to build a deployable and create new artifacts accompanying the change throughout the delivery process.

Dynatrace defines the semantics of essential data points and stores them in a normalized manner. For instance, the Git commit, ticket ID, artifact version, release version, and deployment stage are data points that require special attention. With the semantics of this metadata, well-defined, stable applications, robust automation, and frictionless collaboration can be achieved.

Try out OpenTelemetry traces and log data analysis yourself

The capabilities highlighted in this blog post will be available in Dynatrace SaaS environments in the coming weeks.

Collaborating with your peers based on your software development lifecycle and all data in context has never been easier.

The post Analyze OpenTelemetry traces and log data at scale to optimize application performance appeared first on Dynatrace news.

The new Dynatrace Logs app, fully powered by Grail™ data lakehouse, significantly enhances the experience for novice and seasoned users. Logs delivers unparalleled simplicity with powerful, sharable views and insights to address these needs.

Enhanced log ingestion and seamless out-of-the-box integration

The Logs app is supported by comprehensive log ingestion capabilities provided by the Dynatrace platform and Dynatrace Grail while interwoven with other use-case-specific Dynatrace Apps:

While this is just an overview of the commonly used ingest routes and methods you can leverage, the Dynatrace Hub offers the complete set, currently with 500 supported technologies, 150 additional public extensions, and 65 Dynatrace Apps.

Harness log source sprawl

It’s common for large enterprises to provide hundreds of applications to employees and to host more than a dozen critical line-of-business applications. “Digital workers are now demanding IT support to be more proactive,” is a quote from last year’s Gartner Survey

Understandably, a higher number of log sources and exponentially more log lines would overwhelm any DevOps, SRE, or Software Developer working with traditional log monitoring solutions. The Dynatrace logs in context and surrounding logs features ensure that Dynatrace users are provided with a view that’s tailored to their needs and supports their daily tasks.

The Logs app enables Dynatrace users, whether novice or experienced power users, to rapidly filter and investigate results manually within the app or experience a seamless drill-down handover from use-case-specific apps.

Whether it’s cloud applications, infrastructure, or even security events, this capability accelerates time to value by surfacing logs that provide the crucial context of what occurred just before an error line was logged. For example, if one of your customers unexpectedly uploaded a 1 GB file instead of a 1 MB file, was there an error with the buffer overflowing, or was the network stack unable to handle the unexpected load? Even more importantly, how was the error handled, and did the process end successfully for the customer? With Dynatrace and Surrounding logs, answers to such questions are never more than a click away, with no need to write complex queries or patterns.

Simplicity for novice and power users

For users who seek quick access to relevant logs without the need to write complex queries, easy filtering capabilities are available from within individual log line details or by adding and selecting fields in the menu bar.

For those who aspire to become power users, the new in-app DQL editor (Dynatrace Query Language) translates manually selected filters into the DQL code executed in the backend.

Power users can edit and enhance DQL queries to update results, or they can start directly within the DQL editor view. This allows deep-dive analysis when manual investigation or complex queries are required.

As the screenshot above shows, you can transition back to the filter selection menu bar by selecting Back to previous filters or by sharing the query and results with other teams.

This ensures a smooth user experience for DevOps engineers and SREs, whether they prefer intuitive click-and-filter workflows or fine-grained control through DQL.

Shortening time to value and increasing impact

With the general availability (GA) of the Logs app, another enhancement was introduced that shortens the time to value and increases the positive impact made within organizations.

Filtered log views that were customized by click-to-filter or by leveraging the DQL editor can now be shared using direct links—simply copy and paste the link displayed in your browser’s address bar.

This enables you to

• bookmark and share reoccurring or complex queries.
• share situation or incident-specific views across teams.
• provision new Dynatrace users with relevant queries
• share timeframe-specific and pre-filtered views in a support case

See more with less

During the preview release of the Logs app, many customers provided us with valuable feedback that we incorporated into the design of the app.

User requests, such as the newly added line wrap feature, increase the readability of results and eliminate the extra clicks required to open a detail view or load the results within a notebook. This shortens the time it takes to investigate results and take action.

Speaking of doing more with less, more was released simultaneously with the GA of the Logs app:

• Increased performance of query results returning
• Live search within query results
• Live sorting within query results
• custom column arrangement
• enhanced open-with capabilities to open queries in other platform apps,
  or natively within the Logs app (for example, with Notebooks)

Why Dynatrace

The Dynatrace advantage lies in its ability to set logs in direct context, either automated by Davis® AI, in the context of specific team needs within an application, or as part of the broader observability scope provided by the Logs app itself.

Only Dynatrace provides a comprehensive and accessible log management and analytics experience, helping teams resolve issues faster without compromising on depth. Only Dynatrace Grail is schema-on-read and indexless, built with scaling in mind and built for exabyte scale, leveraging massively parallel processing. With Dynatrace, there is no need to think about schema and indexes, re-hydration, or hot/cold storage concepts.

This architecture also means you’re not required to determine your log data use cases beforehand or while analyzing logs within the new Logs app.

This includes logs collected from your organization’s Hypervisors, Linux and Windows servers, cloud-native logs from Azure, AWS, GCP, or Oracle, alongside the networking signals from Cisco, Juniper, NetScaler, or F5 appliances—to name a few examples.

What’s next

It’s simple, fast, and easy to ingest and gain multi-purpose value from logs—all without the need to be an expert or learn a complex query language first.

Learn how Dynatrace can address your specific needs with a custom live demo. Our observability experts will walk you through our solutions and show you how to deliver excellent customer experiences, foster your application security, and simplify IT operations.

If you’re not yet a Dynatrace customer, start your 15-day free trial.

The post Simplicity meets power: Introducing the all-new Dynatrace Logs app appeared first on Dynatrace news.

Dynatrace enables various teams, such as developers, threat hunters, business analysts, and DevOps, to effortlessly consume advanced log insights within a single platform. Dynatrace Grail™ and Davis^® AI act as the foundation, eliminating the need for manual log correlation or analysis while enabling you to take proactive action.

Dynatrace unified observability and security help enterprises, like our customer, BMO, save time and money, fostering collaboration across business, development, and operations teams.

In this blog post, we’ll provide an overview of the following log-related topics:

• Logs in the context of applications
• Easy yet powerful access to any log with the all-new Logs app
• How logs are ingested
• Dynatrace Application Security with logs
• Dynatrace Davis CoPilot™ integration and AI-powered Application Security
• Schemaless: Instantly gain business event insights from logs

Simplicity is key to success

As IT responsibilities shift left and expand, such as when developers take on application security duties, simplicity in toolsets becomes essential. Existing siloed tools lead to inefficient workflows, fragmented data, and increased troubleshooting times.

Tool consolidation is becoming a priority for C-level decision-makers in 2025. Enterprises are turning to Dynatrace for its unified observability approach for cloud-native, on-premises, and hybrid resources.

Rather than relying on disparate tools for each environment and team, Dynatrace integrates all data into one cohesive platform. Davis AI contextually aligns all relevant data points—such as logs, traces, and metrics—enabling teams to act quickly and accurately while still providing power users with the flexibility and depth they desire and need.

Logs in context with applications

Applications are provided within the Dynatrace platform to address the various needs of different teams and specific use cases. DevOps teams operating, maintaining, and troubleshooting Azure, AWS, GCP, or other cloud environments are provided with an app focused on their daily routines and tasks.

For instance, in a Kubernetes environment, if an application fails, logs in context not only highlight the error alongside corresponding log entries but also provide correlated logs from surrounding services and infrastructure components. This shortens root cause analysis dramatically, as explained in our recent blog post Full Kubernetes logging in context from Fluent Bit to Dynatrace.

The show surrounding logs function provides Dynatrace users with the ability to dive deeper and surface context-specific log lines of the components and services linked to the problem—all without a single line of code or complex query language knowledge. This is explained in detail in our blog post, Unlock log analytics: Seamless insights without writing queries.

For advanced analysis, there is a direct “open with” path, allowing you to load the current view in Notebooks for manual analysis, the Logs app, or other apps capable of visualizing context-specific log lines.

While the way that logs in context are interconnected and provided by Dynatrace is unique, as also Gartner^® recognized for the 14^th consecutive time in their Magic Quadrant™, there are use cases where this is not sufficient and raw access to the logs is required.

Easy yet powerful access to any log with the all-new Logs app

Developers love Dynatrace, without a doubt, and are one of the many teams next to DevOps or SREs making use of our all-new Logs app. The reasons are easy to find, looking at the latest improvements that went live along with the general availability of the Logs app.

In our product news blog post, Simplicity meets power: Introducing the all-new Dynatrace Logs app, we examine these features, which make life easy for new Dynatrace users, along with the newly introduced DQL Editor for power users.

Directly from individual log line results, you can filter simply by selecting corresponding items in the details pane or by loading the surrounding logs when selecting Show surrounding logs.

Keep in mind that Dynatrace Grail is schema-on-read and indexless, built with scaling in mind. There is no need to think about schema and indexes, re-hydration, or hot/cold storage. This architecture also means you are not required to determine your log data use cases beforehand or while analyzing logs within the new logs app.

How logs are ingested

Dynatrace offers OpenPipeline to ingest, process, and persist any data from any source at any scale. OpenPipeline ensures data security and privacy—data is collected and processed securely and compliantly, with high-performance filtering, masking, routing, and encryption—and contextualizes incoming data in real time. Using patent-pending high ingest stream-processing technologies, OpenPipeline currently optimizes data for Dynatrace analytics and AI at 0.5 Petabyte per day and tenant; this will soon increase to one Petabyte per day and tenant.

OpenPipeline’s high-performance filtering and preprocessing provide full ingest and storage control for the Dynatrace platform. As a result, dedicated data pipeline tools are unnecessary for preprocessing data before ingestion.

Dynatrace meets your teams where they are, with your preferred ingest routes and methods—be they Fluent Bit, OpenTelemetry, SysLog, or automatic log collection leveraging OneAgent, to name a few options.

If your team deploys applications cloud-natively, we meet you there, too, as we recently covered in our blog post, Dynatrace log management innovations: Syslog, AWS Firehose. We covered in rich detail how Dynatrace supports log ingestion for cloud-native workloads and simplified log ingestion also for hybrid environments.

In any case, at the heart of the Dynatrace Platform, Grail enables contextual analytics across unified observability, security, and business data. Grail is built for exabyte scale and leverages massively parallel processing (MPP) as well as advanced automated cold/hot data management to ensure that data remains fully accessible at all times, with zero latency, and full hydration.

With no index or schema boundaries in place, paired with long-term data retention ranging from 1 day up to 10 years, you can leverage metrics, logs, and traces for a variety of additional use cases, such as business analytics or security analytics.

Dynatrace Application Security with logs

While most enterprises have Application Firewalls (AppFW), Intrusion Detection Solutions (IDS), and Static Code Analysis (SCA) in place for applications that will be deployed to production, it’s still relevant to understand if anomalies occur during runtime. Monitoring known vulnerabilities within the service hosting the application itself is just another puzzle piece to be considered for full end-to-end observability.

Instead of relying on static patterns, Dynatrace Causal AI understands the desired outcome of the triggered action and the context of the environment hosting the service. For example, deleting the database is not an expected outcome when the function provided is to update a user profile.

With these sophisticated security analytics, simple use cases such as authentication failure anomalies or password spraying attacks, along with more technical HTTP RST statistics, can be visualized in simple and sharable views in Security Investigator, leveraging logs.

Advanced analytics are not limited to use-case-specific apps. Dynatrace offers Notebooks and Dashboards to build views and reports—without the need to write a single line of code or Dynatrace Query Language, all supported by Dynatrace Davis CoPilot™ integration.

Dynatrace Davis CoPilot integration and AI-powered Application Security

Davis CoPilot™ assists occasionally visiting Dynatrace operators throughout the platform in a variety of applications, including Dynatrace Notebooks.

With natural language input in the example displayed in the screenshot below, “Show me the most recurring log lines and add a column with the log source and AWS region,” Davis CoPilot will evaluate the input, translate it into a corresponding DQL Query, and fetch the results accordingly.

Guardrails are in place and can be altered to prevent a high volume of unwanted material from being scanned or returned to Notebooks.

In the same way, Davis CoPilot can recommend remediation strategies and simplify security analysis across all data by translating natural language into the Dynatrace Query Language (DQL) to drive attack protection, security investigations, and forensics.

As mentioned, when ingesting relevant logs into Grail, including firewall and authentication gateway logs, Davis AI can provide end-to-end security insights. At the same time,

Davis CoPilot is integrated into the Dynatrace platform as an intelligent assistant to ease the effort of configuring and finding relevant details and options. By simply asking CoPilot the question stated above, you’re provided with the required configuration steps and product documentation references.

Schemaless: Instantly gain business event insights from logs

A unique picture can be drawn when logs from business-critical LoB applications are collected, which is an underestimated value that can be gained when using Dynatrace.

There are many customer examples and use cases, like room bookings in hotel portals, shopping cart statistics from online shops, or customer metrics from finance platforms, where customers can gain additional business value by translating logs to metrics within Dynatrace.

A pipeline health dashboard showing growth, paired with security-related information, is just one of the many examples of what you can build, either on your own or with the help of the Dynatrace Services team.

In our blog post, Leverage logs for an end-to-end view of your business processes via Dynatrace OpenPipeline, we demonstrated the retail company dashboard example above at a higher granularity, including the necessary steps for JSON log ingestion via Dynatrace OpenPipeline.

Conclusion

It’s simple, fast, and easy to ingest and gain multi-purpose value from logs—all without the need to be an expert or the requirement to first learn a complex query language.

Davis AI and CoPilot make it easy for casual contributors and power users to gain meaningful insights and build notebooks or dashboards while simultaneously increasing application security and reliability.

Learn how Dynatrace can address your specific needs with a custom live demo. Our observability experts will walk you through our solutions and show you how to deliver excellent customer experiences, foster your application security, and simplify IT operations.

If you’re not yet a Dynatrace customer, start your 15-day free trial.

The post Unlock the power of contextual log analytics appeared first on Dynatrace news.

Unrealized optimization potential of business processes due to monitoring gaps

Imagine a retail company facing gaps in its business process monitoring due to disparate data sources. Due to separated systems that handle different parts of the process, the view of the process is fragmented. On top of that, the data sources are inconsistent. While some data comes from modern systems with APIs, other data stems from older systems that generate log files, and some data originates from external vendors. This inconsistency complicates uniform gathering and data analysis, resulting in incomplete or inaccurate insights. This scenario is shared among large organizations that rely on multiple internal and external systems for data collection.

Get business process observability across data silos with Dynatrace OpenPipeline

Traditional observability platforms often focus on technical metrics and logs, which are essential for troubleshooting, but they don’t take business value into consideration. Dynatrace OpenPipeline goes beyond this by offering the possibility of extracting business-relevant data from logs and using them to monitor end-to-end business observability.

In our retail company example, older systems are involved in shipping the order. They only produce logs that have not yet been monitored from a business perspective, although they contain valuable business information.

By leveraging Dynatrace OpenPipeline, the retail company can integrate data from all sources across the whole process, including Dynatrace OneAgent^®, logs, and external business tools. This approach ensures that the company can easily track the entire journey from online order to its successful delivery.

Benefits of capturing business events

Logs often contain valuable insights into your business; however, this information can be difficult to process, particularly as you probably only need data from some specific log lines.

Dynatrace OpenPipeline extracts this business information from logs and stores it as a separate data type, so-called business events. This has several advantages:

• Separation of concerns: Logs are often used for technical troubleshooting. Extracting business events allows for a clear separation between technical and business-relevant data.
• Access control: Different teams can access different types of data. For example, support teams might access logs for troubleshooting, while business teams access business events for analytics.
• Ease of access: Having business events in a uniform format simplifies querying and visualization, making it easier to analyze and derive insights.

How to find valuable business information in logs

In our example of a retail company, we need to extract business information from shipping logs to know if our order has already been shipped. See a typical log file with shipping details below.

From this log file, we can identify and capture the following business information:

1. Correlation ID – The Correlation ID is part of the log payload and identifies this event in the context of a business process.

2. Message – The message is also part of the log payload and contains information on which part of the business process it represents.

3. Context – The context contains information on which IT system handled this part of the business process.

4. Status—The status information tells us if something was successful or if we hit an issue.

Use OpenPipeline to identify relevant log events

To turn this information into a business event for analytics, we utilize the capabilities of Dynatrace OpenPipeline to identify relevant log events, parse the data, and create a business event out of it. The approach is the following:

1. Go to the OpenPipeline app (available by default in your Dynatrace tenant)

1. Create a route that specifies which logs should be processed. For example, you can only process log events coming from a specific ingest source or containing a specific phrase in its log message.
2. Define a pipeline that will process these logs.
3. Configure the data extraction rules within the pipeline. This involves:

1. • Extracting correlation IDs: Identify and extract correlation IDs from the log content. Rename these IDs to make them uniform with other events (for example, rename “order ID”).
   • Extracting the message: The relevant message will be extracted from the log content and used as part of the business event.
   • Parse out contextual information such as the context (which is the related host) or status information in our case.
   • Defining event types: Specify the event type using the extracted message and provider information. For example, you might name the provider retail.logs for consistency.

The extracted information is re-ingested into the pipeline as a new business event. The event is then stored in Grail™ datalake house and can be used for further analysis or process visualization.

Do you want to dig deeper into extracting the data? Watch the dedicated Dynatrace Lab episode with Andreas Grabner and Alistair Emslie for a step-by-step guide on how this is done:

Turn your business data into tangible value

The extracted business data enables you to optimize your processes and gain valuable new insights. You can easily create a dashboard like the one below: it provides the example retail company with real-time data and KPIs, such as the success rate of shipping orders. Unlike traditional dashboards focusing on specific applications or technical aspects, this dashboard tracks the performance of the complete business journey with business KPIs and metrics, utilizing the information extracted from the log files.

Of course, you can also leverage the full power of the Dynatrace platform, like AI-powered monitoring of your business KPIs through Davis^® AI. You can also visualize the end-to-end process flow in our Business Flow app, which enables you to identify delays, errors, and exceptions, providing insights into IT and business-related issues.

Learn more about the capabilities of Dynatrace OpenPipeline

If you’re struggling with gaps in your business process monitoring caused by disparate data sources and outdated systems, consider how Dynatrace can transform your approach.

The post Leverage logs for an end-to-end view of your business processes via Dynatrace OpenPipeline appeared first on Dynatrace news.

In today’s data-driven world, businesses across various industry verticals increasingly leverage the Internet of Things (IoT) to drive efficiency and innovation. IoT is transforming how industries operate and make decisions, from agriculture to mining, energy utilities, and traffic management.

Agricultural businesses use IoT sensors to automate irrigation systems, while mining and water supply organizations traditionally rely on SCADA to optimize and monitor water distribution, quality, and consumption.

Mining and public transportation organizations commonly rely on IoT to monitor vehicle status and performance and ensure fuel efficiency and operational safety.

As businesses increasingly embrace these technologies, integrating IoT metrics with advanced observability solutions like Dynatrace becomes essential to gaining additional business value through end-to-end observability.

Dynatrace offers a feature-rich agent, Dynatrace OneAgent^®, and an agentless open source approach perfectly tailored for edge-IoT use cases, leveraging OpenTelemetry. Both methods allow you to ingest and process raw data and metrics.

In the following sections, we’ll explore how aircraft monitoring fits into this broader IoT landscape and how to implement OpenTelemetry and Dynatrace to enhance the monitoring and management capabilities of IoT data in aviation.

Get started with IoT: Aircraft monitoring

Have you ever wondered how services like flightradar24 or FlightAware collect aircraft location details and report them live?

Each aircraft equipped with an Automatic Dependent Surveillance-Broadcast (ADS-B) transponder broadcasts ADS-B data packets at a frequency of 1090 MHz that include GPS-derived position, altitude, velocity, and other information.

These broadcasts occur periodically and are received by ground stations and other aircraft equipped with ADS-B receivers. They enable real-time tracking and enhanced situational awareness for air traffic control and collision avoidance systems.

The ADS-B protocol differs significantly from web technologies. While web technologies use the internet and HTTP for data transfer, enabling communication between web clients and servers, ADS-B relies on radio frequency broadcasts for direct, real-time communication. ADS-B transmits structured data packets containing specific aviation-related information such as position and velocity. Unlike web technologies, which support a wide range of applications from webpage serving to API interactions, ADS-B is designed explicitly for real-time physical tracking and monitoring in aviation—just like any other IoT monitoring solution in the earlier mentioned verticals.

The hardware

We’ll delve into collecting ADS-B data with a Raspberry Pi, equipped with a software-defined radio receiver (SDR) acting as our IoT device, which is a RTL2832/R820T2 based dongle, running an ADS-B decoder software (dump1090).

To transport our ADS-B application’s JSON log files into our Dynatrace tenant, we’ll leverage an agentless approach using open source software (OpenTelemetry).

The data format

The aircraft.json file in /run/dump1090-mutability/ contains a timestamp (now), the processed Mode S messages (messages), and an array of JSON objects for all known aircraft metrics, such as identifiers, position, altitude, speed, and signal power.

Critical data includes the aircraft’s ICAO identifier, squawk code, flight callsign, position coordinates, altitude, speed, and the time since the last message was received.
This information is essential for later advanced analytics and aircraft tracking.

Locations of critical data

• Aircraft data: /run/dump1090-mutability/aircraft.json
• History data: /run/dump1090-mutability/history.json

View the raw data

The data in dump1090, once received, can be viewed through the application user interface, accessible at: http://<your-raspberry-pi-IP-address>/dump1090/gmap.html

Why observability and data ingestion?

In our example, the ADS-B application provides an excellent visual representation for short-term live monitoring purposes. IoT devices are not meant to store data for longer periods. This is why the log files’ default location is set to /run within the device—a temporary location.

By additionally sending this data to Dynatrace, you can store data for longer periods, from 1 day up to 10 years, while gaining enhanced insights as these metrics can be correlated with other business processes.

If your IoT devices hold business-relevant metrics that can provide additional value when combined with other business processes, you should store them in Dynatrace Grail™, as demonstrated in this blog post.

Ingest decoded ADS-B data into Dynatrace

The specific log files created by the ADS-B software contain aircraft data (aircraft.json) and historical metrics (history.json). They provide detailed information that, when sent to Dynatrace, enables data analytics and improved decision-making capabilities.

The data in aircraft.json can be ingested into Dynatrace using the Dynatrace OpenTelemetry Collector, our officially supported OpenTelemetry Collector.

After extracting the collector, we simply update its configuration, the otel-collector-config.yaml file, to send the aircraft.json live feed to your tenant’s Dynatrace OpenTelemetry endpoint.

receivers:
  filelog:
    include: [/run/dump1090-mutability/aircraft.json]
    start_at: beginning
    include_file_path: true
    multiline:
      line_start_pattern: '^\{'

processors:
  attributes:
    actions:
      - key: log.source
        value: opentelemetry-iot-dump1090-collector
        action: upsert
      - key: iot-device
        value: rpi24-adsb-receiver
        action: upsert
      - key: loglevel
        value: INFO
        action: upsert
      - key: device.address
        value: IWJHUD768H
        action: upsert

exporters:
  otlphttp/dynatrace:
    endpoint: "https://<your-tenant-id>.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlphttp/dynatrace]
      processors: [attributes]
  telemetry:
    logs:
      level: "info"
    metrics:
      address: "0.0.0.0:8998"

Note the OpenTelemetry processor log attributes log.source and opentelemetry-iot-dump1090-collector. We’ll use these values later to filter the log data and focus on the log events from this specific IoT device and application log.

View, query, and analyze ingested IoT data

The ingested data in Dynatrace Notebooks shows that most critical information like flight callsign, location (latitude/longitude), altitude, vertical rate, speed, and aircraft category are stored in JSON format. Sample JSON data is shown below:

Dynatrace Notebooks and Dashboards allow you to analyze and visualize the ingested data. We’ll begin querying the JSON data and start extracting critical information.

Parsing JSON data is an amazingly simple and fast process, as DQL (Dynatrace Query Language) commands like expand and fieldsFlatten place all the above data into the appropriate columns and fields:

With the data now available, we’ll again use DQL to analyze and summarize various metrics further, such as the total number of unique aircraft monitored and the highest altitude recorded.

Some outputs can be displayed as single values, categorical charts, line charts, or other dashboard elements, as shown below.

Some example queries that help populate this data are shown below:

DQL query to find the aircraft traveling at the highest altitude at a given time

fetch logs, from:now() - 7d //fetches data from the past 7 days
| filter matchesValue(log.source, "opentelemetry-iot-dump1090-collector") // Filter ADS-B logs ingested by the opentelemetry collector on our IoT device
| parse content, "JSON:json_content"
| fieldsAdd aircraft=json_content[aircraft]
| expand aircraft
| fieldsFlatten aircraft
| filter isNotNull(aircraft.flight)
| filter isNotNull(aircraft.altitude)
| fields aircraft.flight, aircraft.altitude
| sort aircraft.altitude desc
| limit 1
| fields aircraft.altitude // or | fields aircraft.flight

DQL query to find out the aircraft type based on the aircraft category codes that are transmitted via ADS-B

fetch logs // fetching logs without any limitation
| filter matchesValue(log.source, "opentelemetry-iot-dump1090-collector") // Filter ADS-B logs ingested by the opentelemetry collector on our IoT device
| parse content, "JSON:json_content"
| fieldsAdd aircraft=json_content[aircraft]
| expand aircraft
| fieldsFlatten aircraft
| filter isNotNull(aircraft.flight)
| filter isNotNull(aircraft.category)
| fields aircraft.flight, aircraft.category
| sort aircraft.category desc
| fields aircraft.category

 | fieldsAdd aircraft.type = if(matchesValue(aircraft.category,"A1"), "light",
 else:if(matchesValue(aircraft.category,"A2"), "medium",
 else:if(matchesValue(aircraft.category,"A3"), "medium",
 else:if(matchesValue(aircraft.category,"A0"), "heavy",
 else:if(matchesValue(aircraft.category,"A5"), "heavy",
 else:if(matchesValue(aircraft.category,"B4"), "small",
 else:if(matchesValue(aircraft.category,"A7"), "rotorcraft")))))))
 | summarize count(), by:{aircraft.type}

DQL Query to count the number of flights tracked per day, each hour over two months

fetch logs, from:now() - 60d // fetching the past two month of logs
| filter matchesValue(log.source, "opentelemetry-iot-dump1090-collector") // Filter ADS-B logs ingested by the opentelemetry collector on our IoT device
| parse content, "JSON:json_content"
| fieldsAdd aircraft=json_content[aircraft]
| expand aircraft
| fieldsFlatten aircraft
| filter isNotNull(aircraft.flight) AND  isNotNull(aircraft.altitude)
| filterOut aircraft.altitude == "ground"
| fields aircraft.flight, timestamp
| summarize no_of_flights = countDistinct(aircraft.flight), by:{bin(timestamp, 1h), alias:timestamp}
| fields timestamp,`Flight Count` = no_of_flights

Like the above queries, we can extract critical aircraft statistics and data to report them on a dashboard like the one seen below. Some of the statistics that were captured and useful for various other verticals are:

• Total unique flights tracked over the selected time frame
• Time since the aircraft was last seen
• Callsign of the aircraft with the lowest altitude
• Flight tracked per day and hour

(Interested in using this dashboard in your Dynatrace environment? Download the JSON data)

Often, we need to examine data related to a specific aircraft in depth. Adding the aircraft callsign as a variable can achieve this.

In the dashboard below, we dynamically use a variable resulting from a DQL query. This variable is then used as a filter in other dashboard tiles, restricting the view to the specific aircraft the user selects.

(Interested in using this dashboard in your Dynatrace environment? Download the JSON data)

Advanced mathematics and data science in Dynatrace

Looking at the raw data we have just seen, how are we able to tell the distance to an aircraft from a given point? Additionally, how can we determine the distance of the aircraft to the destination airport?

This information is not directly available from the raw ingested logs and metrics but is calculated using the aircraft’s latitude and longitude within Dynatrace. In the previous dashboard, you might have noticed a tile titled “Distance.” That tile’s value value was calculated using DQL.

To provide such insights, we need to apply mathematical formulas to calculate the distance between an aircraft and an airport. We use the Haversine formula to calculate the distance as we already have the aircraft’s latitude and longitude through the ingested logs.

The Haversine formula is an equation used in navigation to calculate the shortest distance between two points on the surface of a sphere, given their longitudes and latitudes. It accounts for the Earth’s curvature and is helpful in determining great-circle distances between two locations.

Distance calculations using trigonometric functions

Dynatrace supports advanced analytics and mathematical functions such as trigonometric, allowing us to accurately compute the distance from the aircraft to the airport on demand.

Applying this formula in DQL provides us with the distance from the Aircraft to the airport. Here is an example of the formula written in DQL:

The trigonometric calculations used above in DQL are shown here:

// Haversine formula calculates the distance between two points on the Earth's surface given their latitude and longitude.
| fieldsAdd a = sin(toDouble(dLat) / 2) * sin(toDouble(dLat) / 2) + cos(toDouble(myLatRad)) * cos(toDouble(aircraft.lat2rad)) * sin(toDouble(dLong)/2) * sin(toDouble(dLong) /2)
| fieldsAdd C = 2 * atan2(sqrt(a), sqrt(1-a))
| fieldsAdd distance = (earthRadius * C)/1000 // Divide by 1000 to convert from mts to kms

The DQL queries above demonstrate the powerful capabilities of Dynatrace in solving complex business problems encountered in the field.

Similar mathematical calculations can be utilized in other industries, such as:

• Warehouse location planning: Combine sales and shipping metrics to determine optimal locations for new warehouses based on distance to the customer.
• Emergency services: Calculate the nearest hospitals or medical facilities for ambulances and emergency responders to ensure quick response times.
• Agriculture field management: Calculate distances between different plots of land to optimize the use of machinery and labor.
• Mining: Leverage real-time distance monitoring of mining trucks and heavy machinery for operational safety purposes.

Monitor IoT devices and environmental data

IoT devices are often deployed outdoors, making monitoring their network signal strength and quality crucial, considering typical IoT networks such as LoRa, BLE, cell signal, and health metrics such as temperature or other environmental data, rather than just CPU and memory utilization.

To monitor such advanced metrics, you can leverage Dynatrace custom metrics functionality. A script can periodically collect additional data and send it to the Dynatrace metric ingestion endpoint.

Conclusion

IoT devices are now common in industrial, agricultural, space, utilities, hospitals, mining, and several other industries. Extracting critical business data from these devices in real time is now possible using OpenTelemetry and the powerful analytical capabilities of Dynatrace.

By integrating ADS-B airplane data and using trigonometric calculations, we’ve provided advanced analytics for actionable insights while visualizing them within dynamic dashboards.

This blog post provides real-world use cases and demonstrates how Dynatrace can extend existing real-time IoT monitoring systems by leveraging OpenTelemetry. It also highlights the value of IoT monitoring across various industries and provides a hands-on guide to using DQL for advanced analytics, empowering you to extract deeper insights from your data.

Start leveraging Dynatrace for your IoT- and edge-computing needs today.

The post Advanced analytics: Leverage edge IoT data with OpenTelemetry and Dynatrace appeared first on Dynatrace news.

Complex syslog ecosystems can be challenging

Monitoring devices and applications that provide output via the syslog protocol is a must-have for many organizations. The key to success is making data in this complex ecosystem actionable, as many types of syslog producers exist. These include traditional on-premises network devices and servers for infrastructure applications like databases, websites, or email. You also might be required to capture syslog messages from cloud services on AWS, Azure, and Google Cloud related to resource provisioning, scaling, and security events.

Syslog’s unique nature is also often a challenge. While newer syslog daemons have implemented support for TLS encryption, you can still encounter unauthenticated and plain-text message exchanges. A local endpoint in a protected network or DMZ is required to capture these messages.

Finally, adding additional components on the edge to filter and transform syslog messages (for example, Dynatrace OpenTelemetry distribution) isn’t always possible due to architectural reasons or because it adds unnecessary complexity and cost of ownership when scaling your business.

The ultimate challenge lies in making data from syslog-supported log sources actionable. Without seeing syslog data in the context of your infrastructure, metrics, and transaction traces, you’re slowed down by manual work with siloed data. Without syslog in your observability platform, you miss out on automation, insight into service level objectives, faster mean time to repair, increased security, and resiliency.

Secure and trusted local endpoint now collects syslog data

Dynatrace now makes integrating syslog data into an AI-powered observability platform for cloud and hybrid deployments easy, simple, and secure. After ingesting syslog data safely via Environment ActiveGate, you automatically make it actionable in the right context of your infrastructure and cloud services. At the same time, you can offload the overhead and component upkeep burden to a known and trusted observability platform.

With Davis® AI automatic detection of problems and degradations in your services, you can use syslog data automatically in the context of the correct infrastructure component to fix problems faster with logs and eliminate manual correlation and guesswork. This speeds up your teams’ mean time to identify (MTTI) issues and repair (MTTR), increasing business resiliency to disruptions.

One change to send syslog to Dynatrace

You can now use the syslog ingestion endpoint on Dynatrace Environment ActiveGate for performant network and system monitoring. ActiveGate establishes Dynatrace presence in your local network and serves as a known, trusted, supported technology component with lifecycle management that greatly reduces your maintenance effort. ActiveGate also optimizes traffic volume in your network and serves as a secure relay layer in protected networks and DMZs.

To enable syslog collection on an ActiveGate host, one change to extensionsuser.conf is required. No restart is necessary, and the endpoint is ready on standard ports (514 for UDP and 601 for TCP) to collect and forward logs to Dynatrace.

Change this setting in the ActivateGate extensionsuser.conf file:

#Syslog configuration
Syslogenabled=true

To complete the integration, you need to configure your syslog producer to send data to the Environment ActiveGate IP and port.

Optionally, you can adjust syslog collection beyond the default ports and configuration settings. For example, you might want to enable and provide a certificate for the TLS protocol for secure message exchange. ActiveGate uses an embedded Dynatrace OpenTelemetry Collector instance, which allows you to configure a syslog receiver according to your needs.

What’s next

See these related resources for complete details about Dynatrace syslog support:

• Syslog ingestion is available with Dynatrace Environment ActiveGate version 1.295.
• Read syslog ingestion documentation with configuration samples.
• Setting up your first Environment ActiveGate? See installation documentation for setup.
• For alternative syslog monitoring scenarios, explore syslog (via OpenTelemetry Collector) and syslog (via Fluentd) in Dynatrace Hub.

Coming soon

• We continue to improve syslog integration with easier failover configuration and support for monitoring configuration health.
• Need to send syslog directly over HTTPS? More mature syslog producers support this, and we continue working on providing the required authentication layer.

The post Observe syslog with Dynatrace ActiveGate, a secure, trusted edge component appeared first on Dynatrace news.