You deploy a small change to a Node.js service. All tests pass. You top off your coffee. Fifteen minutes later, alerts start firing.

Your once-snappy API endpoint is crawling. Dashboards light up. Slack fills with “Anyone seeing latency?” messages. You open the logs and start the familiar dance: grep, trace, redeploy, repeat.

We’ve all been there. The hardest part isn’t fixing the bug — it’s finding it.

Why we lose hours chasing logs and people

As developers, we have telemetry overload — yet still lack the one piece of context we need: What line of code introduced the regression?

Modern cloud-native systems aren’t just applications anymore, they’re distributed systems. When something has already happened, the first question isn’t what happened, it’s where?

And in distributed systems, “where” can mean:

• Any container replica
• Any downstream API call
• Any async job or queue consumer
• Any recent deployment or configuration change

Each service has its own logs, metrics, and traces — often split across different tools and dashboards. Context lives in Slack threads. Deployment history lives somewhere else. Institutional knowledge lives in someone’s head.

So, you correlate timestamps manually. You pivot across dashboards. You ask in Slack.

This isn’t debugging — it’s archaeology.

Finding the source means pulling in Dev, Ops, SRE, and application owners — trying to piece together the context of both code and infrastructure.

In other words, is it the code — or is it the node?

When you need code-level insight

Now imagine a different outcome. You immediately see which function in your code triggered the latency spike — without redeploying, adding debug logs, or trying to reproduce the issue locally.

That’s the promise of code-level observability.

Instead of stitching together logs, metrics, and traces across multiple tools, runtime instrumentation automatically captures what executed, where time was spent, and how requests flowed across services. You move from guessing to knowing.

Manual instrumentation is brittle. It adds overhead, introduces blind spots, and slows down delivery. As builders, you want to focus on the business logic and functionality, not maintaining observability integration.

That’s why platforms like Dynatrace automatically instrument the runtime and correlate traces, logs, and method-level execution into a unified view. With OneAgent, OpenTelemetry support, and open APIs, teams get deep visibility without manual instrumentation or lock-in.

In the Distributed Tracing view, logs, traces, and code-level analysis are correlated into a single problem context. Developers can see what failed, where latency was introduced, and which code path was responsible — without losing focus or switching tools.

Real-world example: Tracing a bad commit in a Node.js microservice

You’ve got a simple Node.js microservice that fetches orders from multiple downstream APIs and returns them to checkout.

// routes/server.js
router.get('/api/orders', async(req, res) = >{
  const start = performance.now(); // better precision than Date.now()
  try {
    const orders = await fetchOrders();
    const durationMs = performance.now() - start;​ // ← capture request duration for monitoring​

    // You would normally send this to your metrics system (Prometheus, StatsD, OpenTelemetry...)
    console.log({
      endpoint: '/api/orders',
      orderCount: orders.length,
      durationMs: Math.round(durationMs),
      durationHuman: `$ {
        durationMs.toFixed(1)
      }
      ms`
    });
    res.json(orders);
  } catch(err) {
    // duration even on error is useful for observability
    const durationMs = performance.now() - start;
    console.error({
      endpoint: '/api/orders',
      error: err.message,
      durationMs: Math.round(durationMs)
    });
    res.status(500).json({
      error: 'Failed to fetch orders'
    });
  }
});

After a recent commit, latency climbs to 600ms and error rates spike. Logs show scattered warnings. Locally, everything looks fine. Under light load, tests still pass.

This is the kind of regression that eats up entire afternoons.

The data you need is split across:

• Application logs
• Distributed traces
• Kubernetes events
• Deployment history

What the trace reveals

Opening the distributed trace view in Dynatrace (PurePath®) you can inspect the full execution path of a single request — across services, methods, and downstream dependencies.

You immediately see:

• Failed /api/checkout requests
• Elevated response times
• A downstream gRPC calls consuming the majority of request time

From logs alone, you only see where errors surfaced:

callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:32)
Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:193)
Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:361)
Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:324)
<anonymous> (/app/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129)
AsyncLocalStorage.run (node:internal/async_local_storage/async_hooks:91)
process.processTicksAndRejections (node:internal/process/task_queues:85)
ServiceClientImpl.makeUnaryRequest (/app/node_modules/@grpc/grpc-js/build/src/client.js:161)
ServiceClientImpl.<anonymous> (/app/node_modules/@grpc/grpc-js/build/src/make-client.js:105)
AsyncLocalStorage.run (node:internal/async_local_storage/async_hooks:91)
<anonymous> (/app/node_modules/@opentelemetry/instrumentation-grpc/build/src/clientUtils.js:130)
<anonymous> (/app/node_modules/@opentelemetry/instrumentation-grpc/build/src/instrumentation.js:215)
AsyncLocalStorage.run (node:internal/async_local_storage/async_hooks:91)
AsyncLocalStorageContextManager.with (/app/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33) 

The logs tell you where the exception bubbled up. It does not tell you where the regression started. The trace timeline adds the additional context.

The root cause

Method-level timing shows that fetchOrders() is now executing downstream lookups sequentially instead of in parallel.

// routes/server.js
router.get('/api/orders', async(req, res) = >{
  const start = Date.now();
  const orders = await fetchOrders();
  const duration = Date.now() - start;
  console.log(`Request completed in $ {
    duration
  }
  ms`);
  res.json(orders);
});

Under low traffic, this isn’t obvious.
Under load, latency compounds.

Now our dev team can update the code to fix the issue:

// fixed version of fetchOrders
async 
function fetchOrders()  {
  // run order detail lookups in parallel
  const results  =  await Promise.all(orderIds.map(getOrderDetail));
  return results;
}

The regression only surfaced once concurrency increased— something unit testing didn’t expose.

By correlating runtime traces with deployment metadata, you can see exactly when the behavior changed and which commit introduced it.

How you can cut MTTR from hours to minutes

Depending upon the scenario, we often get judged not by how fast we ship — but by how fast we recover. Mean time to repair (MTTR) is what builds (or breaks) trust between Developers and SREs.

That’s why instrumentation and observability need to be placed before code reaches production — not added after something breaks.

Here’s what happens when you move from manual debugging to connected code-level insight:

With distributed tracing in Dynatrace, every transaction is captured end-to-end, from your Node.js route to downstream APIs and databases, all correlated with the build and deployment that introduced it.

Instead of debating whether it’s the code or the node, both developers and SREs see the same execution path, the same timeline, and the same root cause.

Enabling code-level analysis in your stack

You don’t need to refactor your application or add manual instrumentation to anything. Here’s how most teams get started:

1. Deploy Dynatrace OneAgent on your service host or Kubernetes cluster.
2. Enable Node.js auto-instrumentation in enabled (no code changes required.
3. Integrate CI/CD metadata so commits, build IDs, and deployment versions appear alongside traces.
4. Inspect live traces directly in your IDE (VS Code or IntelliJ) using the Dynatrace plugin — jump from failing span to the exact method instantly.

Within minutes, your next deployment is automatically traced end-to-end — from the first line of code to the live transaction in development through testing and into production.

Less time in tools. More time in flow.

Every debugging session risks pulling you out of flow.

Code-level observability shortens incident response, reduces cognitive load, and replaces guesswork with execution context. Developers gain confidence in what they ship. SREs see fewer escalations and cleaner hand-offs because the root cause is already visible.

Less troubleshooting. More building. That’s the real win.

Explore how Dynatrace connects commits, traces, and runtime behavior — so you can move from incident to resolution without leaving your workflow.

code. observe. deliver.

The post Use code-level analysis to cut MTTR before you push to production appeared first on Dynatrace news.

An exception does not always mean that something in your systems failed, but they can indicate current performance problems or issues that could cause failures in the future. For developers, investigating exceptions has traditionally been slow and tedious, requiring you to juggle different tools to assess the impact of an exception.

The new Exception Analysis in the Dynatrace Distributed Tracing app is designed to make this part of your job easier. It offers a new way to spot, group, and understand exceptions across the full trace, with readable stack traces, aggregated insights, and visual markers to highlight problematic spans. In minutes, it can help take you from “something looks odd,” to “this is actually important; it happens 1,000 times an hour; and here is the root cause.”

In this article, we’ll dive into a demo of exception analysis and explore how to investigate and analyze the exceptions over time.

But first, let’s take a closer look at why we care about exceptions in the first place.

Why investigate exceptions?

Exceptions are often the real reason a trace fails or slows down. When you scroll through spans trying to spot the problem, you may find a slow call—but the real issue is an exception somewhere else.

If you do find an exception, you still must figure out:

• Which span threw it?
• Is it happening all the time or just once?
• Is it the same error repeating?
• Which service is triggering it?

This is why Exception Analysis comes in handy. Having an aggregated view of all exceptions over a specific time frame helps you to see and investigate them clearly. It allows you to quickly answer questions like: What broke or slowed down? Where? How often? What is the impact?

Even if an exception hasn’t broken anything yet, issues like unsupported versions of dependencies, deprecated functions, etc. could cause headaches further down the line. An aggregated view helps you see clearly over time which exceptions caused specific failures, and by investigating exceptions that are “only” slowing the system down, you can proactively prevent major incidents in the future and spot slowdowns that would otherwise go undetected.

An example of exception analysis investigation

Let’s take a look at the new Exception Analysis perspective in the Distributed Tracing app. It offers a streamlined way to analyze exceptions over time and understand if they are directly related to failed requests of your services. This also allows users to understand technical exceptions and errors early before they lead to failures or outages later.

We’ll start at the time window of the last 24 hours to get an overview of what is happening in the environment lately. A PHP exception catches our eye. It looks a little bit awkward, and we see it happens quite often, but it’s not really leading to failing errors.

The interesting information here is that it seems like there is a deprecated function used within the application. In the future, this could lead to breaks and real failures when this function is removed or cannot be used in a way as it is used right now.

For developers, Dynatrace provides immediate access to related stack traces with all frames, making it easy to identify where and why an exception occurs. You can also view relevant logs and requests in context at the same location.

Back to our investigations, we also see that this exception is obviously happening on a specific PHP service on the endpoint /like. From there, we can also jump to specific instances of this exception by clicking on View Trace.

We’re now at the Explorer perspective, getting it filtered down to the specific endpoint on the specific service that we care about. And we also see the exception analysis across the trace. We see all exceptions that happen in the context of this trace, together with the trace.

We also see another reason why we had this high number of exceptions. It seems it is happening three times for this specific transaction. We now have a single occurrence of this exception, with a view of the exception message and the exception stack trace.

It also works the other way around. If you have a specific trace and you want to understand whether there’s more examples of a specific exception you can always drill up to the exception analysis to take a closer look.

For example, we can now examine this exception specifically, see only its occurrences, and identify exactly where it happens repeatedly over time.

Investigating a failure caused by an exception

Sometimes an exception isn’t just a slowdown—it’s a real failure. When that happens, Dynatrace links the exception directly to the underlying failure and guides you to a detailed failure analysis, including the full stack trace captured in the context of the failed service call.

Key takeaways about exception analysis

Exception analysis enables developers to gain insights into exceptions and their behavior within the environment and their code so you can resolve issues more quickly.

And, Exceptions Analysis also works for OpenTelemetry. By following semantic conventions, any exceptions in OpenTelemetry spans will appear in Exceptions Analysis.

Top 3 takeaways from this demo:

• Exceptions are often the real reason a trace fails or slows down—and now you know how to view them easily with Dynatrace.
• Dynatrace shows exceptions across the whole trace, not just inside one span, and it supports both OpenTelemetry and OneAgent collection.
• An exception does not always mean that something failed, therefore you can quickly determine whether something broke or slowed down and if so where and what the impact is.

To learn more, check out the following resources.

• Documentation
• Failure detection example
• Data in context: How Dynatrace solves the OpenTelemetry analytics challenge

The post Why exceptions matter: Elevating your distributed tracing investigations appeared first on Dynatrace news.

The real OpenTelemetry challenge: When OTel meets scale

Standardizing on OpenTelemetry gives teams flexibility and control in modern cloud native architectures. The instrumentation works. The data flows. The collection is solved. However, as organizations scale out OTel, they encounter the analytics gap- and the cost continues to climb without clear returns. Can your observability platform turn millions of spans and log lines into clear, contextual answers without manual correlation or vendor lock‑in?

While having OTel data is exciting, many teams find themselves asking, ‘What’s the actual payoff?’ Engineers may occasionally explore telemetry data, but without intelligent analytics to connect the dots, the data often remains underutilized. Meanwhile, managers struggle to quantify the value of their investment, especially as costs climb with scale. This is the natural challenge of DIY setups, where teams focus on data collection but rarely think critically about turning that data into actionable insights.

Anyone can analyze a single trace; that’s easy. Deriving answers from millions? That’s the hard part. The promise remains unfulfilled. Until now.

Why OpenTelemetry + Dynatrace changes everything for teams

Dynatrace closes this gap. With Dynatrace, all telemetry signals are combined to give you the insights you need:

• Comprehensive failure analysis across large trace sets
• Response-time insights revealing performance patterns at scale, with logs and exceptions in context
• Deep visibility into database and queuing systems
• AI-powered intelligence delivering contextualized insights
• Enterprise operational controls providing cost allocation, secure data handling, and scalable telemetry management

Your OpenTelemetry data transforms into actionable contextualized answers that help you move faster, ship confidently, and get back to building.

Complete enterprise coverage for OpenTelemetry

Dynatrace brings OpenTelemetry for Enterprise to life through specialized analysis designed for practitioners troubleshooting issues in Kubernetes environments, available in our extended Services app. AI-powered intelligence, including anomaly detection, ensures you get answers faster, without losing context.

Failure analysis: from single traces to mass insights

When your booking service fails, you can see the complete story: failed traces, related log entries, specific database statements, exceptions- all automatically correlated in one view. The Failure Analysis also provides a visual investigation of your OTel data.

Time-based comparisons allow you to overlay current failures against previous windows, instantly identifying regressions- what was stable yesterday and failing today becomes obvious. And there’s more: advanced visualization distinguishes between different failure types and severities, automatically categorizing them so you can prioritize based on actual user impact.

It’s a new, intuitive way to explore data visually with full context- analyzing failure patterns across your entire architecture and understanding how problems flow through distributed systems. Derive answers from millions of spans that individual trace inspection would never reveal.

Response time analysis with full telemetry context

Mass data analysis extends to performance insights. Dynatrace delivers response time analysis and comparisons built for practitioners. The platform allows you to easily compare failures between two time windows to spot exactly when things degraded.

See how response times correlate with database performance, downstream dependencies, like calls or queue interactions, and infrastructure resource utilization.
Dive in visually, explore the correlated context, and understand what’s happening- all in one place.

Database queries: understand service-to-database interactions

Modern services thrive or fail based on their interactions with their databases. Dynatrace provides comprehensive database analysis for OpenTelemetry-instrumented services, showing exactly what your services are doing against your databases.

Get immediate visibility into your most expensive queries across your entire environment- whether it’s Cassandra, SQL, or other databases. Queries are automatically ranked by cumulative duration (query count times query duration), surfacing what’s actually costing you performance.

When troubleshooting an individual service, you immediately see which database calls are problematic. You can also examine patterns across many services: identifying query problems, detecting spikes, or discovering when services start overwhelming databases with inefficient calls. The view aligns with how your architecture actually functions.

Cloud native queuing systems support

Modern applications stream data through Kafka, RabbitMQ, MQTT, and SQS, sending thousands of messages per second through distributed architectures. Dynatrace delivers comprehensive visibility into these message processing interactions with dedicated metrics, dashboarding, and alerting designed specifically for how modern streaming systems actually operate.

See which services are publishing or receiving messages from which queues, with full performance metrics. Advanced filtering lets you explore your entire environment or drill into a specific service’s queue interactions.

Exception analysis: uncover patterns and failures

We’ve only scratched the surface of how service analysis capabilities can make an impact. From the Services app, you can seamlessly navigate to related traces in the Distributed Tracing app, which now includes extended Exception Analysis. This enhancement surfaces exceptions across traces with readable stack traces, aggregated insights, and visual markers to highlight problematic spans.

By analyzing exceptions in context, teams can quickly identify patterns, prioritize fixes, and reduce MTTR. Whether leveraging OneAgent or OpenTelemetry, no critical issue goes unnoticed, providing complete visibility and reliability across modern environments.

AI-powered intelligence: pinpoint the needle in the haystack

Dynatrace AI delivers actionable insights through baselining, anomaly detection, and precise alerting, continuously learning your environment’s behavior. From day one, these capabilities surface meaningful deviations with full context, enabling teams to act quickly and confidently.

By analyzing OpenTelemetry data, you can detect trends, predict potential issues, and get intelligent, context-rich alerts. This ensures teams can focus on what matters most- resolving problems faster and optimizing performance- without manual effort or guesswork.

Enterprise operational controls that scale

Beyond analytics, enterprise teams need operational capabilities that work with OpenTelemetry data:

• Primary fields and tags: Use your existing Kubernetes labels and cloud tags (AWS, Azure) to filter and organize telemetry data. Filter by namespace, cluster, deployment, or custom business dimensions to focus on what matters most.
• Cost allocation: Track and understand costs by subscription, project, or resource group to optimize spending and ensure efficient resource usage.
• Pipeline routing and processing: Route telemetry data to specific pipelines based on cloud provider, region, or cluster. Control how data flows through your observability stack to improve efficiency and ensure compliance.
• Bucket assignment: Assign data storage by environment, account, or custom dimensions. Optimize retention and costs while adapting to operational requirements.
• Security context: Tag data with permissions and access controls, so teams see only the namespaces and services they’re authorized to access.

These aren’t add-ons; they’re core platform capabilities that work identically whether you use OpenTelemetry or OneAgent instrumentation.

The bottom line

Success comes from choosing the analytics platform designed for practitioners in modern environments- one that delivers insights across millions of signals with AI-powered intelligence. Dynatrace meets you where you are with the “Data in Context” advantage: every signal works together with the enterprise capabilities that cloud native environments demand.

The enhanced Services app and the Distributed Tracing app are now available for Dynatrace Platform Subscription (DPS) customers.

Check out the Dynatrace Playground to experience OpenTelemetry for Enterprise firsthand.

Join us at Perform in Las Vegas, January 26-29!

The post Data in context: How Dynatrace solves the OpenTelemetry analytics challenge appeared first on Dynatrace news.

OpenTelemetry excels at capturing data from any environment and service: traces flowing from microservices, metrics streaming from containers and infrastructure hosts, and logs capturing the application and service lifecycles. OTel does exactly what it was designed to do: standardize telemetry collection. But here’s what OpenTelemetry doesn’t do by design: unified observability that analyzes that data, correlates it across services, or turns it into actionable, intelligent insights.

This is where most organizations hit a wall and require lots of expert knowledge. Organizations today have more observability data than ever before, but somehow less visibility into what’s actually happening in their systems. Raw telemetry data becomes a burden rather than an asset. Engineers spend more time hunting through dashboards than solving actual problems.

This is the “analytics gap” that Dynatrace was built to solve, transforming your telemetry data from scattered signals into unified, AI-powered intelligence.

Why OpenTelemetry + Dynatrace changes everything

Here’s what makes this combination powerful: OpenTelemetry gives you standardized data collection. Dynatrace gives you intelligent analysis that goes far beyond the static dashboards and manual correlation work that other platforms require.

While many observability solutions leave it to you to build custom dashboards and manually connect the dots between your telemetry signals, Dynatrace transforms your OpenTelemetry data into insights that actually drive decisions. Your traces, metrics, and logs aren’t just stored; they’re automatically correlated, analyzed, and contextualized as they flow through Dynatrace OpenPipeline®.

When a trace shows latency spikes, you immediately see related log entries and metric anomalies automatically contextualized and correlated. Lightning-fast queries via Dynatrace Grail® data lakehouse process millions of spans at the speed of thought, making observability accessible to your entire team, not just the experts who know how to build complex queries and visualizations.

The result? Your OpenTelemetry investment becomes a competitive advantage, not just another data collection project that requires a team of dashboard architects to maintain.

Complete OpenTelemetry coverage

Here’s how Dynatrace helps you to get the most out of your telemetry data, without requiring additional agents or complex configurations:

Distributed tracing excellence

Native OpenTelemetry tracing delivers superior span and trace processing with dynamic visualization tools that transform complex distributed architectures into complete end-to-end visibility. But it doesn’t stop there; all your telemetry signals (logs, traces, and metrics) correlate seamlessly, giving you clear, actionable insights within the full context of your traces and services. Get simple answers to advanced questions by expanding your investigations with DQL for powerful analytics, including correlation of logs and traces.

Service monitoring that understands your Architecture

Comprehensive service health monitoring built on OpenTelemetry standards. Dynatrace provides intelligent service analysis, anomaly detection, and visualization that work seamlessly with your OpenTelemetry-instrumented applications. Our service monitoring goes beyond simple health checks.

When issues arise, you see exactly which services are affected and how problems cascade through your architecture, all without manual tagging, configuration, or service discovery setup with YAML files.

Intelligent metrics with full context

You get flexible metric ingestion for custom business metrics and standard application performance indicators. Your metrics connect directly to the services and traces that generated them. But here’s where Dynatrace takes it further: we allow you to unify all your OpenTelemetry signals into comprehensive service intelligence. Instead of analyzing metrics in isolation, you see how they connect to actual service behavior, request flows, and application logs. Every metric becomes part of a complete service story.

Complete log processing

Your OpenTelemetry logs are transformed from noise to narrative. Instead of searching through endless log streams and manually created dashboards, Dynatrace supports a comprehensive log ingestion and analysis pipeline, allowing you to go big with Dynatrace.

Every log event becomes part of a larger story about user journeys, interactions, services, and app behavior, all focused on your desired business outcomes and incident investigations.

Here’s where it gets powerful: you automatically get additional contextual enrichment when you direct all your telemetry signals to Dynatrace. By creating bi-directional relationships between logs and traces, where logs provide context to traces and traces illuminate relevant logs, Dynatrace evolves troubleshooting from detective power-user work into AI-driven, streamlined, and intuitive investigations.

Video: See the full story behind every trace with correlated logs.

Kubernetes native support

For teams running OpenTelemetry in Kubernetes, Dynatrace delivers enterprise-grade support that scales with your cloud native operations. Native Kubernetes handling of spans, metrics, and logs from your Kubernetes OpenTelemetry deployments automatically collects Kubernetes context for automated enrichment: namespace, cluster, and workload relationships, all without any additional instrumentation. Your existing Kubernetes labels, AWS tags, and Azure tags become first-class filtering dimensions for all OpenTelemetry data, enabling automatic cost attribution and comprehensive data permissions using your existing RBAC patterns.

The result is that your OpenTelemetry observability inherits the same operational patterns, security boundaries, and cost structures as your Kubernetes infrastructure.

Why this matters for your team

Every organization adopting OpenTelemetry faces the same challenge: turning data collection into intelligent insights. The engineering teams that succeed are those that choose analytics platforms built specifically for OpenTelemetry data.

Dynatrace transforms your OpenTelemetry investment from a data collection project into a competitive advantage. We meet you where you are. We respect your choice to standardize OpenTelemetry by simplifying its operational complexity and enhancing it with analytics that actually deliver value.

Ready to transform your OpenTelemetry data?

Open standards have clear benefits. Industry standardization makes it easier to make sense of data coming from multiple different sources, whether it’s traces, metrics, logs, or telemetry from third-party tools. Your analytics platform can deliver intelligent insights across your entire technology stack. Your OpenTelemetry investment deserves analytics that reveal its full potential.

• Want to explore specific OpenTelemetry capabilities with Dynatrace? Try them out on the Dynatrace Playground
• Boost your productivity with these quick video guides for service owners working with OpenTelemetry:

Video: Easy access to your OTel Logs and Traces

Video: Analyze Service Failure from OTel Data

Video: Easy access to your OTel & Prometheus Service Metrics

Join us at OpenSource Summit. We’ll be in Amsterdam August 25-27. Stop by our booth to see the magic in action!

The post OpenTelemetry and Dynatrace: Complete unified observability analytics for modern applications appeared first on Dynatrace news.

It’s 3:00 AM. Alerts are firing. Something’s broken, latency is spiking, there’s too much noise, and you’re under pressure to find the root cause fast. You need to be able to understand how your system interacts to solve the problem as soon as possible. But systems just keep getting more complex as your organization adds new technologies, AI models, and container-based microservices. That’s why, as complexity scales, so does the need for connected insights. In the world of observability, logs and traces serve distinct but complementary purposes. When used together, they unlock a powerful view into system health, performance, and behavior.

The secret lives of logs and traces

In theory, correlating logs and traces should be straightforward. However, in practice, teams often find themselves context-switching to follow the path of a trace and all the logs involved. To understand why, let’s take a closer look at the roles and responsibilities of logs and traces.

Traces: The big picture view

Traces follow the journey of a request as it moves through various services in a distributed system. They provide end-to-end observability of how different components interact, making them ideal for understanding latency, bottlenecks, and service dependencies. Distributed traces connect events into a cohesive timeline, helping engineers see how one service’s performance affects others.

Traces shine when you’re trying to answer questions like, “Where did this request slow down?” or “Which service caused the failure?”

Logs: The detailed detective work

Logs are detailed, timestamped records of events generated by applications and infrastructure. They’re rich in context, often containing error messages, debug information, and custom outputs that developers write into the code. While traces show the flow, logs show the details. Logs can exist independently of traces and are often the first place developers look when something goes wrong.

Logs shine when you’re trying to answer: “What exactly happened here?”

Don’t forget metrics and other telemetry signals

Although we’re focusing here on logs and traces, metrics and other telemetry data are also essential for observability and deeper context. For more about why it’s important to unify the full spectrum of observability signals, see What is observability and Unified observability: Why storing OpenTelemetry signals in one place matters.

The power of correlating logs and traces from a single, full-context platform

Isolated telemetry signals can lead to blind spots and wasted time searching for answers. Some of the main ways to use logs and traces are to simplify troubleshooting, enhance performance, improve security posture, and meet compliance standards.

When you can correlate logs and traces from a single source of observability data, you eliminate the constant context switching that slows down investigations. Instead of toggling between tracing tools and log viewers, you get a unified view that connects the dots fast.

Correlating logs and traces from a single platform transforms troubleshooting from a fragmented hunt into streamlined analysis, where you spend time solving problems instead of searching for information. Core technologies like Grail®, OneAgent®, and Davis® AI provide the scalable foundation while embracing open-source frameworks like OpenTelemetry for flexibility.

Cracking the case of the failed checkout: Investigating logs and traces

Not every investigation starts the same way. Sometimes a trace gives you the high-level view you need to spot an issue and dive deeper. Other times, a log entry is the first clue that something is off. In the next section, we’ll walk through two examples, one that starts with traces and the other with logs, to show how you can get the answers you need.

Scenario 1: Investigating from traces to logs

While doing some routine monitoring in the Distributed Tracing app, we notice a series of failed requests in our Kubernetes prod namespace. So we filter for unsuccessful transactions to examine them more closely.

One request stands out: “/cart/checkout”. It’s a critical transaction path, and we’re seeing failures.

We dive into the trace waterfall. Just below it, we find the logs tied to each span, giving us deeper insight. That’s where we find the message:

“error: failure to complete the order”

Digging further, another log reveals the root cause: only Visa and Mastercard are accepted, which is in line with our policy, but potentially limiting our business. This raises a new question: how often is this happening?

With a single click, we pivot to the logs app, where we can search for this specific message and quantify how many transactions may have been impacted.

This approach turns scattered signals into a cohesive story, helping us move from surface-level symptoms to actionable insights with speed and precision.

Scenario 2: Investigating from logs to traces

No matter how you start your day, whether you are coming from PagerDuty, Slack or start directly in Dynatrace through one of the many apps like Kubernetes or the Clouds app, you can always see logs in context of your investigation.

In this scenario, we’re investigating this case from another angle, starting with the logs app using the prefiltered segment for the Kubernetes prod namespace. The view is tailored to the services we own. A quick scan reveals something suspicious: numerous errors in some of the log files.

To dig deeper, we navigate in the logs app and use the content filter for “payment” and “error”, and we find several logs with the following message:

Could not charge card for user id = xxxxxxxxxxxxx

But what is causing the failure? We click Show surrounding logs, which reveals all logs associated with the trace ID. Now we can view log messages sequentially as they happened.

Investigating some of the surrounding logs, we see that the user is using a credit card other than Visa or Mastercard, which our organization doesn’t support. Now that we understand why things are failing, let’s investigate further to see if we can optimize this experience.

To understand the full impact, we pivot seamlessly to the trace view. Here, we see the full waterfall breakdown of the request: service calls, timing, and span-level metadata.

One detail stands out: it took 5 seconds for the user to receive the failure message. That’s a long time to wait just to be told their card isn’t supported.

With this insight, we can now make targeted improvements so that the user does not have to wait a long time to understand that their payment method is not supported and deliver a better user experience.

While these examples highlight how seamless navigation between logs and traces accelerates troubleshooting, they’re just one part of the story. With Dynatrace Grail and Notebooks, you can take things a step further by running advanced queries, automating repetitive tasks, and building collaborative, data-rich workflows. These tools empower teams to go beyond reactive troubleshooting and into proactive, scalable observability.

Why seamless navigation between logs and traces matters

Seamless navigation between logs and traces isn’t just a convenience; it’s a game-changer. Whether you start with a trace or a log, the ability to pivot instantly between signals means you spend less time hunting for answers and more time solving problems. It accelerates root cause analysis, improves team collaboration, and gives you the full context needed to act with confidence. This is just one example of how Dynatrace helps you move from fragmented troubleshooting to unified intelligent observability.

Ready to start investigating?

Explore Distributed Tracing and Log Management and Analytics, complete with prepopulated data in the Dynatrace Playground.

The post Logs and traces: Why context is everything for seamless investigations appeared first on Dynatrace news.

In observability’s early days, we often talked about the “three pillars.” That is, traces, logs, and metrics, which gave us the information to make our systems observable. The problem with referring to these three signals as “pillars” is that it implies they’re siloed and therefore independent of each other, when in fact, the exact opposite is true.

To harness the true power of observability, you need to treat these signals not as pillars, but as three strands that make up a braid, as OpenTelemetry (OTel) co-founder Ted Young so aptly put it. While observability as a whole also encompasses user behavior and security data among other signals, the main OpenTelemetry signals–traces, logs, and metrics–each serve a different and important purpose and contribute to the observability story, giving us the full picture of what’s happening in our systems. We achieve even greater value from our OpenTelemetry signals through context, the glue that connects related information, revealing patterns and relationships, and making raw data more meaningful and actionable.

Stronger together!

And yet, many organizations practicing observability still tend to send different OpenTelemetry signals to different backends for storage and analysis. The problem with this separation is two-fold. First, you’re not storing all of the signals in one place. This siloes the data, making it impossible to correlate and get insights from the data. Second, you’re having to go back and forth between different tools to look at your signals, try to correlate them, and understand what’s going on. How can you effectively analyze all of your telemetry data if it’s not all stored in the same place?

This problem is further amplified when you consider how some organizations send telemetry data from different applications to different vendors. For example, an organization might have App A send metrics to SaaS Tool X, and traces and logs to SaaS Tool Y. App B sends traces to SaaS Tool J, logs to SaaS Tool L, and metrics to self-hosted Tool M. Let’s not forget the teams that go rogue and decide to do their own thing. See that tower under Bob’s desk? It’s running a whole suite of self-hosted open-source observability tools, and App C is sending its telemetry signals there.

Moving from “swivel chair” observability to “unified observability”

To borrow a term coined by my husband, these types of organizations are practicing “swivel chair” observability, and to be honest, calling it “observability” at this point is being very generous. You don’t have a braid or even pillars. You have islands of pillars.

To leverage the true power of observability, we need a single pane of glass that provides “unified observability.” One single platform for storing, viewing, correlating, and analyzing your telemetry signals.

This enables teams to improve data analysis, streamline workflows, and innovate. It also allows them to focus on delivering better results with less effort. At the same time, organizations get the support they need to take on whatever challenges are thrown at them.

Remember, though, that observability must go beyond tooling if it’s to be truly effective and sustainable. Observability must be treated as a team sport, where everyone in an organization plays a role in making systems observable. This should be complemented by enterprise oversight to guide tooling choices, patterns, and best practices.

Unified observability with Dynatrace

By using the Dynatrace observability and security platform as your OpenTelemetry analytics backend, however, you can connect all these details in context. Dynatrace stores all data in Grail™, a unified and purpose-built data lakehouse optimized for storing and analyzing not just traces, logs, and metrics, but also security, business events, and RUM/behavioral data.

Consider some examples.

Distributed tracing

The first example shows Dynatrace Distributed Tracing. Note how we can see a distributed trace and its associated logs and metrics in Dynatrace. Together, these pieces of information all help paint a picture of what is happening in our system.

Searching logs

The next example shows how you can use Dynatrace Query Language (DQL) to search for logs with a specific error message and to join them to the traces associated with those logs.

Investigating problems

Finally, you can use the Dynatrace Problems app for root cause analysis, showing a problem that was detected, the impacted entities, and its associated metrics and logs. And for those who only occasionally look at observability data, Dynatrace also provides Davis CoPilot™ using natural language queries to quickly explain what this data means, and propose concrete remediation steps.

All of this is possible because of the integrated storage approach of Grail.

By storing all OpenTelemetry signals in context under one roof, users can ask meaningful questions, get useful answers, and act effectively on what they learn. That is, Dynatrace makes unified observability possible.

Unified observability leads to better outcomes

Unifying OpenTelemetry signals in a single analytics backend helps you gain critical context that identifies patterns and relationships among all observability signals. This context helps you spot the meaning in raw data, making it easier and faster to anticipate problems, take action, and automate responses.

To learn more about how Dynatrace augments, amplifies, and accelerates actionable answers from OpenTelemetry, check out these resources.

• Dynatrace Loves OpenTelemetry
• How to send OpenTelemetry data to Dynatrace
• Distributed tracing with Dynatrace just got even better
• Distributed tracing best practices for the software development lifecycle
• How to detect bad patterns in logs and traces

Also check out our playlist featuring the video series, Dynatrace Can Do THAT with OpenTelemetry?, in which my teammate Andi Grabner teaches me all sorts of cool things that Dynatrace can do with OpenTelemetry data ingested into Dynatrace using the OTel Collector.

The post Unified observability: Why storing OpenTelemetry signals in one place matters appeared first on Dynatrace news.

For developers, the ultimate win is creating impactful features, shipping them quickly, and confidently implementing fixes when needed. Distributed tracing plays a critical role in achieving these goals. As a core component of observability, it provides the insights developers need to write better code, collaborate efficiently, and increase application reliability.

Developers know how challenging it can be to follow traces from beginning to end across multiple entities and data providers. Harder still is following those traces in context with metrics, logging data, security details, and real user experience data. Certain tools may provide visibility into small segments or isolated details. But developers often have to piece these clues together manually, which makes it difficult to extract meaningful and reliable results.

Dynatrace delivers distributed tracing with capabilities designed to streamline workflows. From understanding the impact of code changes to quickly resolving issues flagged in production, here’s how distributed tracing can help you increase developer velocity and deliver meaningful innovation throughout the software development lifecycle (SDLC).

Distributed tracing best practices key takeaways:

1. Visualize and understand your system. Distributed tracing plays a key role in uncovering the topology, interactions, and dependencies of your services.
2. Analyze and fix problems fast. Combine distributed tracing with AI analysis, context-driven querying, and collaboration tools.
3. Build better software with telemetry data in context. Unify telemetry data
4. Automate to ship better code faster. Use distributed tracing to automate and streamline workflows and set up intelligent alerts.

Visualize and understand your system

Before implementing changes or designing new features, it’s essential to have a clear understanding of your system’s current state. Without a clear picture of your current system, any change risks overlooking critical dependencies and misalignment with your architecture, whether in the cloud or on-premises. Distributed tracing plays a key role in uncovering the topology, interactions, and dependencies of your services, giving developers the visibility they need to make informed decisions.

Dynatrace Smartscape® technology automatically maps horizontal service relationships, offering real-time visualization of service connections and dependencies. This end-to-end view helps developers quickly understand how components interact within the overall system.

For a more granular understanding, inspecting individual traces allows you to focus on specific workflows or problematic behaviors. This depth of insight can help you plan better and identify challenges early, reducing surprises during later stages of development.

Trace example: Verifying service for “platinum” loyalty users

Say you’re a developer on an app called “AuthenticationService.” You can select a trace that lets you visualize its path through your system and examine each span’s properties to ensure “Platinum” loyalty users are being served effectively. While understanding the state of your system is important at each stage of the SDLC, it’s often one of the first steps in planning, requirements gathering, and designing changes.

In Dynatrace, you would open the Distributed Tracing app and enter a filter for the request attribute that indicates loyalty status. This filter returns a list of traces you can inspect individually to understand any bottlenecks or issues that could be impacting this specific subset of customers.

Analyze and fix problems fast using distributed tracing for developers

Identifying root causes in complex microservice-based architectures can be tough, but Dynatrace Davis® AI analyzes dependencies in real time, helping you quickly and confidently resolve issues.

But what if you want to visualize data outside of an individual problem occurrence? Some issues occur repeatedly in specific areas of a system, while others arise from application changes. Problems often span multiple teams, requiring collaboration to resolve effectively.

For broader insights, the Dynatrace Distributed Tracing app provides flexible access to raw trace data. You can filter by namespace, release version, or database calls, then Dynatrace automatically generates a DQL (Dynatrace Query Language) query so you can view results in notebooks or pin visuals to dashboards, which enable you to collaborate seamlessly with other teams. This approach is particularly useful during testing, deployment, and maintenance, where visibility into system behavior is critical.

This example shows a plot generated by a DQL query that indicates the number of database calls an application makes and how they perform over time.

Let’s assume you’re also interested in how your database queries are performing. To get an instant performance summary, you can easily add a table to your dashboard that summarizes the count, average, and percentiles of the durations of your database spans.

Build better software with telemetry data in context

Developers juggle many responsibilities, from testing and deploying code to maintaining stability. However, fragmented workflows and the endless search for the right information can often slow progress and create frustration, making it harder to meet deadlines and maintain a focus on innovation.

As a developer, how can you get the information you need without wasting time chasing down data and trying to understand all its upstream and downstream effects? The answer: distributed traces and telemetry data in context.

Flexible access to unified telemetry data is a game-changer for developers. With Dynatrace, you can seamlessly query multiple telemetry data sets and visualize the results in ways that align with your specific needs. This flexibility empowers you to connect the dots across complex workflows, analyze incidents within their broader context, and make data-driven decisions that support automation and enhance software quality.

The example below brings together spans and logs. Dynatrace automatically enriches every log with trace and span IDs, making it simple to track correlations among related data. With Dynatrace Query Language (DQL), you can dynamically combine spans and related log content on-the-fly.

To understand the impact of an error, you can chart which services and endpoints are emitting this log error.

As a developer, you can unlock deeper insights, linking operations to specific errors and enabling you to assess the scope and impact of incidents accurately. This gives you and your teams the ability to move beyond reactive problem-solving into proactive, intelligent development.

Use distributed tracing for developers to automate and ship better code faster

Automation is key to accelerating development while maintaining quality. By streamlining workflows, automating repetitive tasks, and setting up intelligent alerts, teams can proactively address issues and minimize manual effort.

For example, as a developer, you can use trace data to establish quality gates or set alerts for specific exceptions, ensuring you can quickly flag and resolve any recurring problems. This approach boosts efficiency and ensures that teams can consistently deliver high-performing, reliable code to production.

Assume you’ve been working on a bug that appeared as an exception in your traces. You want to check if the bug has been fixed and set up an alert to get a reminder if this particular bug comes back.

Let’s start by creating a metric based on this particular exception in the Dynatrace stream-processing data ingestion technology, OpenPipeline™. By creating a metric at ingestion using OpenPipeline’s centralized data handling and advanced data processing capabilities, you can continually monitor for evidence of the exception recurring.

You can use this extracted metric to visualize its frequency over time in a dashboard.

Finally, once you’ve set up a metric, you can use it to create an alert and send a notification to your system of choice (such as Slack, Teams, ServiceNow, and so on).

Increase reliability and automate more with distributed tracing best practices

As development responsibilities continue to expand in today’s complex environments, distributed tracing has emerged as a critical tool for developers throughout the entire software development lifecycle.

Applyin these distributed tracing best practices helps developers visualize systems, resolve issues quickly, and streamline workflows. By providing deep insights and context of upstream and downstream events, traces and spans give teams the data they need to automate tasks and deliver more reliable, higher-quality software faster and with greater confidence. The result allows you to focus on what matters most: creating impactful features.

Start integrating distributed tracing into one part of your workflow today and see how quickly it becomes indispensable to your development practice.

The post Distributed tracing best practices for the software development lifecycle appeared first on Dynatrace news.

Get ready to experience a whole new world of limitless tracing power. With our latest enhancements, we’re transforming the way you work with trace data. The Dynatrace® platform now enables comprehensive data exploration and interactive analytics across data sets (trace, logs, events, and metrics)—empowering you to solve complex use cases, handle any observability scenario, and gain unprecedented visibility into your systems. Whether you’re using OpenTelemetry or OneAgent, operating in the cloud or on-premises—we’ve got you covered.

Introducing a new era of distributed tracing with advanced analytics

In today’s complex systems landscape, understanding the root causes of issues can be daunting, especially as applications scale and OpenTelemetry adds complexity.

Davis® AI automatically pinpoints root causes, offering immediate answers. For deeper exploration, our Distributed Tracing app empowers you to analyze raw trace data and uncover insights, whether troubleshooting errors, optimizing performance, or discovering the “unknown unknowns.”

But why stop there? Building on this solid foundation, we’re thrilled to announce two powerful platform enhancements. Say hello to advanced trace analytics and new data storage and capture options. These game-changing features elevate your data interactions, opening up vast possibilities for advanced queries and efficient data management tailored to your needs.

Site reliability engineers, performance architects, and developers can now leverage dynamic analysis tools like dashboards and workflows to explore trends, automate processes, and maintain control at an unprecedented level. Additionally, these queries serve as excellent starting points for more complex data explorations with Notebooks.

Get ready to maximize the full potential of your trace data—unlock deeper insights and automate like never before, all within a single platform.

Level up your analytics game: Enhanced team collaboration and advanced data insights

With traces now stored in Dynatrace Grail™, our scalable data lakehouse, you can unlock powerful new analytics capabilities, handle massive volumes of data, and run complex queries seamlessly. Combining traces with logs, metrics, Kubernetes events, and telemetry attributes gives you a complete, contextual view of your environment for unmatched end-to-end observability.

Unlock deeper insights

Using Dynatrace Query Language (DQL), you can extract game-changing insights from raw span data with precision. Use these queries to start more complex data exploration with Notebooks. This enables you to uncover hidden patterns, discover unknown unknowns, and make confident, data-driven decisions. These powerful insights can easily be transformed into interactive dashboards.

Example: Exception analysis

Understanding patterns, especially regarding exceptions, is no easy feat. However, you can begin unlocking additional insights using the Distributed Tracing app. For example, you can filter to understand endpoint performance where exception messages contain the string, access denied.

Once filtered, you can easily open a notebook with a pre-populated DQL query. You can then add additional details or modify the query as needed. Combine multiple findings in a notebook or dashboard to share your analysis with your team, allowing them to see these focused updates live in real time.

Achieve superior analytics

Transform trace data into intelligent insights by combining trace data with logs, metrics, and events. This combination allows you to enrich all data and get details in context. Use this intelligent data to see what matters to you most in real time by creating interactive dashboards and driving better decision-making. This gives you the power to break down silos, spark collaboration, and extract actionable insights with ease. It democratizes access to critical data, ensuring all teams can leverage the same reliable insights to drive impactful outcomes.

Example: Combine trace data with logs

A common scenario is understanding which frontend API requests have log messages on the backend indicative of a specific problem. Let’s look at an example where the log message contains timeout, and we want to understand the response time of traces in the context of these messages.

By linking trace data with logs, you can query across spans and related log messages. You can also summarize with DQL to understand how often a specific pattern occurs. To visualize span duration, use p99 to see the slowest percentile of span response times and then navigate directly to the traces.

Automate with Dynatrace OpenPipeline

With Dynatrace, you can create custom metrics from trace data using Dynatrace OpenPipeline™, unlocking powerful new automation capabilities. It’s now possible to create metrics on OpenTelemetry and OneAgent spans with any available attribute, giving you the power to define operational, request, and method-level metrics.

OpenPipeline provides the flexibility to build metrics tailored to your specific needs, enabling you to integrate metrics seamlessly with advanced Dynatrace automation features, such as AutomationEngine and SRE Guardian. You can streamline workflows, intelligently automate repetitive tasks, proactively resolve issues, and spend more time innovating with automation, ensuring that only reliable, high-performing code reaches production.

We’ve only scratched the surface of scenarios where advanced analytics can make an impact—the possibilities are virtually endless. By combining advanced trace analysis, intuitive query capabilities, and seamless automation, your team can enable sharper analysis, streamline workflows, and foster innovation. This powerful approach ensures you can focus on delivering better outcomes with greater efficiency, empowering your organization to tackle complex issues with precision and agility, ultimately bringing unprecedented value.

Extended trace retention: Retain data longer when it matters

Need to analyze trends over the long term or adhere to compliance requirements? With extended trace retention, you can store trace data for up to 10 years. This feature ensures your organization is well-equipped for trend analysis and detailed post-mortem reviews—all while meeting regulatory requirements.

Retention policies are fully configurable in OpenPipeline. You can share data in buckets with varying retention times depending on the use case, allowing you to target specific applications or error-prone services for longer storage. This precision reduces storage costs while ensuring you retain the data that matters most.

Extended trace ingest

You can now customize trace ingestion rates to meet your specific needs. While most trace data is already ingested at high coverage rates, this option gives you more granular control over your trace volume. Ingest as much data as you want, above and beyond what is already included in your Dynatrace license.

Maximize the value of your OpenTelemetry data

At Dynatrace, we love OpenTelemetry. We champion open source innovation and recognize OpenTelemetry’s influence in setting observability standards (we’re also a top contributor). That’s why our advanced capabilities were designed from the ground up with OpenTelemetry at its core. We built our entire new tracing experience on OpenTelemetry semantic conventions and expanded from there. Now, OpenTelemetry users can troubleshoot and analyze while leveraging  OTel standards. This commitment empowers you to simplify complexity and innovate faster by extracting maximum value from your data, regardless of origin.

Experience the future of Distributed Tracing

At Dynatrace, we believe that observability should be effortless and completely on your terms. With our latest advancements, we’re helping you manage complexity, innovate faster, and push boundaries. Our solution adapts seamlessly to your ecosystem, whether you use OpenTelemetry or run cloud-native or on-premises workloads. Built to handle enterprise scale, the Dynatrace platform processes massive volumes of data in real time while unifying insights across all teams in your organization.

We’re rolling out this functionality to existing Dynatrace Platform Subscription (DPS) customers. Elevate your observability journey with these new possibilities—tailored to your needs, your way.

If you’re not a DPS customer, you can try out the new Distributed Tracing experience with prepopulated data on the Dynatrace Playground.

If you’re new to Dynatrace and want to try out the new Distributed Tracing experience with your own data, check out our free trial

Take the leap today and discover how Dynatrace can revolutionize your approach to observability.

The post Distributed tracing with Dynatrace just got even better appeared first on Dynatrace news.

Organizations are depending more on distributed architectures to provide application services. This trend is prompting advances in both observability and monitoring. But exactly what are the differences between observability vs. monitoring?

Understanding when something goes wrong along the application delivery chain is essential so you can identify the root cause and correct it before it impacts your business. Monitoring and observability provide a two-pronged approach. Monitoring supplies situational awareness, and observability helps pinpoint what’s happening and what to do about it.

To better understand of observability vs. monitoring, we’ll explore the differences between the two. Then we’ll look at how you can best utilize both to improve business outcomes.

Monitoring vs. observability

First, let’s define what we mean by observability and monitoring.

What is meant by monitoring?

By textbook definition, monitoring is the process of collecting, analyzing, and using information to track a program’s progress toward reaching its objectives and to guide management decisions. Monitoring focuses on watching specific metrics. Logging provides additional data but is typically viewed in isolation of a broader system context.

What is meant by observability?

Observability is the ability to understand a system’s internal state by analyzing the data it generates, such as logs, metrics, and traces. Observability helps teams analyze what’s happening in context across multicloud environments so you can detect and resolve the underlying causes of issues.

What is the difference between observability and monitoring?

Monitoring is capturing and displaying data, whereas observability can discern system health by analyzing its inputs and outputs. For example, we can actively watch a single metric for changes that indicate a problem — this is monitoring. A system is observable if it emits useful data about its internal state, which is crucial for determining the root cause.

What are the similarities between observability and monitoring?

Observability and monitoring are closely related concepts in systems and software engineering. Both aim to provide insights into the health, performance, and behavior of a system. They utilize data collection, analysis, and visualization techniques to enable proactive detection and troubleshooting of issues. Ultimately, they empower engineers to ensure system reliability, performance optimization, and efficient resource utilization.

Between observability and monitoring, which is better?

So how do you know which model is best for your environments?

Monitoring typically provides a limited view of system data focused on individual metrics. This approach is sufficient when systems failure modes are well understood. Because monitoring tends to focus on key indicators such as utilization rates and throughput, monitoring indicates overall system performance. For example, when monitoring a database, you’ll want to know about any latency when writing data to a disk or average query response time. Experienced database administrators learn to spot patterns that can lead to common problems. Examples include a spike in memory utilization, a decrease in cache hit ratio, or an increase in CPU utilization. These issues may indicate a poorly written query that needs to be terminated and investigated.

Conventional database performance analysis is simple compared to diagnosing microservice architectures with multiple components and an array of dependencies. Monitoring is helpful when we understand how systems fail, but as applications become more complex, so do their failure modes. It is often not possible to predict how distributed applications will fail. By making a system observable, you can understand the internal state of the system and from that, you can determine what is not working correctly and why.

However, correlations between a few metrics often do not diagnose incidents in modern applications. Instead, these modern, complex applications require more visibility into the state of systems, and you can accomplish this using a combination of observability and more powerful monitoring tools.

The “three pillars” of observability and beyond

As mentioned earlier, traditionally, observability is understanding what’s happening inside a system from its logs, metrics, and traces. Modern observability includes these three original pillars along with user experience and security. Systems are observable when they generate and readily expose the type of data that enables you to evaluate the state of the system. Here’s a closer look at logs, metrics, distributed traces, user experience, and security.

• Logs include application- and system-specific data that details the operations and flow of control within a system. Log entries describe events, such as starting a process, handling an error, or simply completing some part of a workload. Logging complements metrics by providing context for the state of an application when metrics are captured. For example, log messages might indicate a large percentage of errors in a particular API function. At the same time, metrics on a dashboard are showing resource exhaustion issues, such as a lack of available memory. Metrics may be the first sign of a problem, but logs can provide details about what is contributing to the problem and how it impacts operations.
• Metrics in this context are sets of measurements taken over time, and there are a few types:
  • Gauge metrics measure a value at a specific point in time, such as the CPU utilization rate at the time of measurement.
  • Delta metrics capture differences between previous and current measurements, such as a change in throughput since the last measurement.
  • Cumulative metrics capture changes over time — for example, the number of errors returned by an API function call in the last hour.
• Distributed tracing is the third pillar of observability and provides insights into the performance of operations across microservices. An application may depend on multiple services, each with its own set of metrics and logs. Distributed tracing is observing requests as they move through distributed cloud environments. In these complex systems, traces highlight any problems that can happen with the relationships among services.
• User experience considers how users interact with the front end; understanding where time is spent and which actions are critical helps prioritize and identify users’ needs. This is essential when the goal is to deliver an exceptional customer experience. This important piece of the puzzle takes into consideration things like revenue, conversions, and customer engagement. All of these are important inputs to get a full understanding of the application landscape.
• Security is an essential component in understanding the internal state of a system. Organizations are shifting away from siloed security teams and taking a DevSecOps approach. This includes security at each stage of the SDLC. So should it be in observability, where security is one element that affects the health, performance, and customer experience of an application.

True observability, however, relies on more data than just key indicators.

Why monitoring and observability need a next-gen approach

When trying to effectively monitor, manage, and improve complex microservices-based applications, observability and monitoring are both vital. Monitoring and observability represent a continuum from basic telemetry of single servers to profound insights about complete applications and dependencies.

Many organizations start with monitoring and realize these tools lack contextual insights. Context is critical to understanding why problems exist and how they impact the business. Organizations look to observability to provide the data they need for contextual analysis. Understanding the problem means they can understand the root cause and its effects.

DevOps practitioners need help to maintain highly available and scalable applications. That’s because these complex, interdependent systems behave in unpredictable ways and issues originate from sources that are often not apparent. Practices and tools that worked when we built monolithic applications simply can’t handle the level of data distributed environments generate. They don’t ingest enough data or provide enough insight into the state of applications to understand how to correct problems quickly. Luckily, some tools and practices address these challenges.

An automatic and intelligent approach to monitoring and observability

An advanced software intelligence solution like Dynatrace automatically collects and analyzes highly scalable data to make sense of these sprawling multicloud environments. Dynatrace’s causal AI engine, Davis, sifts through massive volumes of disparate, high-velocity data streams, and analyzes them through a unified interface. This single source of truth tears down information silos that traditionally separate teams that perform different functions on many application components. This centralized, automatic approach eliminates the need for manual diagnostics. It also provides paths to remediation to keep the technology users rely on functioning smoothly.

The post Observability vs. monitoring: What’s the difference? appeared first on Dynatrace news.

In recent years, more and more large enterprises have embraced microservices-based architectures that run across clouds and geographies to deliver increased agility, improved performance, scale, and reliability. This approach has accelerated innovation and created new opportunities for distributed development and support models.

As companies have modernized, they have embraced new IT business models and faced tightening regulatory requirements. Specifically, the heightened awareness of data residency and sovereignty in many regions of the world has resulted in the requirement to maintain separate data centers with local data footprints across the globe. Furthermore, global enterprises, which typically have multiple business units and divisions, often require separating applications and services.

Dynatrace is proudly committed to providing users with an integrated observability platform that provides true end-to-end monitoring and analysis. Simplifying complexity and delivering not just more data but answers has been our credo from the beginning.

Tracking a transaction from start to finish is critical for end-to-end visibility and is a relatively simple endeavor when applications are contained in a unified observability system. However, when applications span monitoring environments, end-to-end tracing becomes much more difficult because each environment only sees a piece of the complete transaction.

With millions of requests an hour processed, and some requests going into other environments, tracing a single transaction can be like finding a needle in a haystack.

Easy end-to-end distributed tracing through multiple environments

Seeing is believing, so let’s look at an example: a company that runs a travel platform is migrating its services to the cloud. They’re pursuing a hybrid cloud strategy where the front end runs on a hyperscaler cloud provider. The middleware and back end will continue to be managed internally to keep customer data in the local region. Multiple Dynatrace environments are deployed to ensure data residency.

Just before the start of the travel season, special attention is paid to the performance indicators, and an increased response time is noticed. The following distributed trace shows an end-to-end view through three Dynatrace environments from the front end (cloud hosted), to middleware, and the back end (hosted on-premises ). We can see an issue at the back end, but we can go further. A direct link to the Dynatrace back-end environment lets us jump right to the point where we can identify the problem.

Setup is fast and uncomplicated

Connecting multiple Dynatrace environments takes less than a minute. Just enter a unique environment name, the tenant URL of the Dynatrace environment, and a secure token. Be sure to enable the cross-environment feature in your OneAgent settings (Settings > Integration > Remote environments). That’s all there is to it.

How can I get this feature?

This new capability was released in Dynatrace SaaS version 1.248, Dynatrace Managed version 1.250, and OneAgent version 1.247. Please see Dynatrace Documentation to learn how to enable this capability and connect your environments.

The post Break down the barriers to end-to-end monitoring with Dynatrace appeared first on Dynatrace news.

There’s no lack of metrics, logs, traces, or events when monitoring your Kubernetes (K8s) workloads. But there is a lack of time for DevOps, SRE, and developers to analyze all this data to identify whether there’s a user impacting problem and if so – what the root cause is to fix it fast.

At Dynatrace we’re lucky to have Dynatrace monitor our workloads running on K8s. One of those workloads is Keptn, a CNCF project Dynatrace is contributing to, that we use internally for different SLO-driven automation use cases.

Dynatrace Davis, our deterministic AI, recently notified our teams about a problem in one of our Keptn instances we just recently spun up to demo our automated performance analysis capabilities orchestrated by Keptn. I was pulled into that troubleshooting call and started taking notes and screenshots so I can share how easy it is to troubleshoot the Kubernetes workload with our engineers and you – our readers – on this blog post.

It started with the Problem card Davis opened because of a 33% increase in failure rate in the workload called mongodb-datastore.

This mongodb-datastore provides several internal API endpoints to fetch/update data in the actual MongoDB instance. What’s great for our engineers who are responsible to operate Keptn is that alerting happens automatically, without relying on us to define custom thresholds. This is all thanks to Dynatrace’s automatic adaptive baselining.

How the baselining identified the problem can be easily seen with a single click from the problem to the service response time overview as shown next:

What’s even better than Davis detecting the increase in failure rate is that Davis automatically points to the root cause, which Dynatrace picked up from automatically captured container logs. A single click brought us to the Log screen – automatically filtered to the logs captured in that mongodb-datastore during that timeframe:

If you take a closer look at the screenshot above it’s easy to spot the root cause; it was an unhandled error condition in the code that was waiting and processing feedback from the MongoDB instance. The problem was also reported back to the Keptn team via GitHub issue mongodb-datastore: Panics, meaning the team could not only detect the issue fast but also had everything they needed to react fast and immediately fix the problem.

Dynatrace has even more details for the development teams

Just from spending two minutes looking at the data Davis put in front of us, we knew the Impact and Root Cause of the high error rate (including the line of code). I went ahead and took additional screenshots that I sent over to the engineers on top of the direct links to the Dynatrace screens so that they can do their own analysis. Screenshots are however always great as I think they are interesting for the engineers and make my life in writing those blogs easier 😊

One of those screenshots is from one of the PurePaths (=Distributed Traces) that captured the problem. Not only do we have the detailed log, but we also know the API endpoint was the HTTP GET /event.

There’s so much more Dynatrace provides than what’s shown here, but I wanted to keep this blog short and sweet and focus on that one story.

If you’re interested in learning more, I recommend you check out these articles:

• Why DevOps love Dynatrace CodeLab Tutorials by Sergio Hinojosa
• Engineering Blogs from Thomas Schütz on Infrastructure and App deployment automation and Redesigning Microservice deployment strategies

As always – these blog posts wouldn’t be possible if my colleagues wouldn’t share those stories. Thanks a lot Sergio Hinojosa and Maria Rolbiecka for your hard work on keeping this Keptn instance up and running! And a big Thank You to Florian Bacher and Bernd Warmuth from the Keptn development team who went ahead and fixed that issue within a day – now that’s a fast turnaround 😊

The post Kubernetes workload troubleshooting with metrics, logs, and traces appeared first on Dynatrace news.

Distributed Traces are “the source of truth” for developers and architects as they capture the true end-to-end execution path for each individual request processed by your applications and services. When running unit or API tests in a local dev environment, IDE (Integrated Development Environment) and test tool integrations with observability platforms make it easy for engineers to run distributed trace analysis on the distributed traces they just generated. Answering questions like “Does my code correctly call the new backend service version for my specific use case?” becomes easy as the outgoing call with request and response details will be shown in the distributed trace.

The following screenshot shows a distributed trace in Dynatrace with detailed version information of every service call involved. In the rest of the blog, you will learn more about how to capture version information and use it to answer your version-specific questions:

If you’re not already capturing distributed traces in your development environments, I hope this blog enables you to make the case for it.

From a handful to millions of distributed traces requires automation

When distributed traces are captured in shared testing or production environments, where hundreds of services/microservices are deployed in one or multiple versions, you potentially end up with millions of captured traces in a short time span. If you then figure out how to automate the analysis of those traces you can empower DevOps and SRE teams as they need answers to questions such as:

• “Which versions of our services are currently processing our critical transactions?”
• “How does an overloaded backend service impact the SLOs of the frontend service?”
• “Which frontend services are responsible for the changed traffic behavior on the backend services?”
• “Is there a different behavior between two versions of a service? If so – shall we stop the rollout into production?”

I personally keep hearing those questions more frequently these days, which is somewhat worrying. Many members in our Dynatrace community are moving towards k8s and microservices which allows DevOps and SREs to leverage zero-downtime update strategies (also known as Progressive Delivery), such as Blue/Green, Rolling Updates, Canary Deployments or Feature Flags more easily. To answer version-specific questions like the one above it’s not only necessary to capture version information on every distributed trace but you must also automate the analysis as no one can dig through millions of traces manually to end up with answers that lead to better delivery and release decisions.

The good news is that Dynatrace PurePath, our leading automated distributed trace technology for the past 15+ years, is version aware by default meaning that it automatically captures the version information on every PurePath and provides automated analysis options to answer those version specific questions. But it’s not just the raw data that matters – it’s what Dynatrace does with the data, which I’ll delve into now. , which I’ll delve into now.

To learn more about Dynatrace’s version aware analysis capabilities I invited Thomas Rothschaedl, Product Manager at Dynatrace, to my latest Performance Clinic where he explained how version aware PurePaths are captured, how Dynatrace provides real-time release overview, and how Dynatrace provides automated answers to DevOps & SRE based on version-aware PurePath data.

While I encourage you to watch the full 30 minutes recording on YouTube or Dynatrace University I captured the key learnings in the remainder of this blog:

Dynatrace real-time release and version overview

Thomas started of reminded me about the Dynatrace Releases screen, which gives our users a live overview of all deployed releases in every monitored environment, even providing release lifecycle events (deployment, tests, quality gate, promote, rollouts, rollbacks, problems, etc.) as well as direct access to any open development or support tickets:

If you’d like to learn more about the releases overview make sure to watch my Performance Clinic on Risk-Free Delivery with Dynatrace Cloud Automation Release Management.

Analyzing rolling version updates through multi-dimensional analysis

At Dynatrace we’re proud to use Dynatrace on Dynatrace which allowed Thomas to show an internal example of analyzing rolling software updates. The screenshot below shows a 72-hour analysis window of the rolling updates we do in our end-to-end testing environment. Here, we continuously roll out the latest Dynatrace versions that come out of our build system. The environment is constantly under load and it’s therefore great to see how the rolling update is truly and smoothly updating from one version to the next:

Analyzing request count (=throughput) is just one option, as you can see in the next section.

Automatic regression detection across versions

While the above example nicely shows the rolling updates that happened during constant load on the system, it also only focuses on throughput. What’s more interesting is if you switch to a different metric that can be extracted from PurePaths such as “Number of Exceptions Thrown”, “Number of Database Calls Made”, “Number of Database Rows Fetched”, or “Time Spent in I/O”.

Thomas demonstrated this in the performance clinic I mentioned above, where he wanted to know if any of our builds introduced a regression that caused more exceptions to be thrown. Exceptions are by default captured by Dynatrace OneAgent as part of the version aware PurePath. As you can see from the below screenshot, he immediately found a regression that was introduced in one of the builds that were rolled out earlier that day:

This is the true power of analyzing a massive amount of PurePaths in an automated way. No one would be able to identify those problems quickly by manually digging through millions of PurePaths. That’s why Dynatrace’s automation is valued by our users as it finds these issues automatically without any manual effort. But there’s more than what Thomas showed us.

Diagnostics, metrics, dashboards, and alerting

In the 30 minutes I had, Thomas walked me through the use cases he additionally demonstrated how to:

• Drill to the offending line of code of the exception regression
• Create metrics, put them on a dashboard and roll those out across all teams
• Get alerted on version-specific anomalies

If you want to see all these demos, then check out the Performance Clinic recording. The live demo piece starts at the 15:35 timestamp.

Make better release decisions through version aware distributed traces

Whether you use Dynatrace or any other tool to capture distributed traces, it should be clear that you must make sure to capture version information on each trace and have a way to analyze large volumes of distributed traces to make better release and deployment decisions. If you don’t yet have a distributed tracing option, or if your current tooling doesn’t support what Thomas has shown in his demo, then feel free to sign up for a Dynatrace trial and try it out yourself.

To end this, I’d like to say THANK YOU Thomas for your great preparation of the Performance Clinic content. You did an amazing job in showing the value of the latest capabilities in Dynatrace. I also want to say THANK YOU to Dynatrace engineering, which has not only built a great platform but also uses Dynatrace on Dynatrace and with that, makes our demos and storytelling even easier.

The post How to automate version aware distributed trace analysis appeared first on Dynatrace news.

Today’s highly complex microservices environments require IT operations and SRE teams to observe each service and transaction end-to-end. Any observability blind spot imposes a risk, potentially increasing the time it takes to identify the root cause of a problem and address it.

With the extremely successful OpenTelemetry project (to which Dynatrace actively contributes) having recently reached a key milestone with the OpenTelemetry Tracing Specification version 1.0, cloud service providers are now starting to add OpenTelemetry instrumentation as an out-of-the-box feature. Additionally, more development teams are exploring custom instrumentation of domain- and project-specific diagnostic points using OpenTelemetry.

At Dynatrace, we strive to provide enterprise-grade observability regardless of the data source. As such, we recently opened up our platform for metric ingestion and log monitoring and built integrations for key formats in those spaces.

Announcing OpenTelemetry trace ingest

Today, we’re happy to announce that with the new OpenTelemetry trace ingest capability, we now also provide a standard way of ingesting distributed traces into Dynatrace—regardless of the data source.

This enables you to leverage OpenTelemetry data to gain full visibility in the following scenarios:

• If you have complex cloud-native monitoring environments that involve a fully managed cloud service that only exposes OTLP traces as spans but doesn’t allow for OneAgent deployment
• If your teams are using OpenTelemetry custom instrumentation to enrich monitoring data with project-specific details (for example, to add business data or capture developer-specific diagnostics points), and you want to retain their instrumentation investments
• If your organization uses technologies, including programming languages, that aren’t natively covered by OneAgent, but which do emit OpenTelemetry trace data

Now let’s take a closer look at the first two of these scenarios using a concrete example.

Gain full observability in heterogeneous cloud-native environments and reduce resolution times

TL;DR summary

Say, for example, that during the last couple of hours, the cart-checkout service of your Online Boutique has failed 45% of the time, causing immense user frustration. This service relies on an externally hosted third-party currency-conversion service. OpenTelemetry trace data emitted by this third-party service is ingested into Dynatrace and combined with OneAgent data for PurePath analysis. With this approach you can quickly see that too many requests to the service is the reason why every other request is being dropped. This is the root cause of the service failures and user frustration.

Detailed use case

Say you’re running the Online Boutique, a cloud-native microservices demo application that allows users to browse items, add them to a shopping cart, and purchase them. However, you notice that 100% of users are frustrated, with the Apdex rating down to zero. So, you start your investigation at the front-end server service, which shows an increased failure rate with every second request failing for a couple of hours. When investigating the failure rate, you notice that the cart-checkout service fails 45% of the time. To investigate further, you drill down to the cart-checkout transactions and the actual PurePath, which includes seamlessly integrated OpenTelemetry spans.

When selecting the cart-checkout service in PurePath view, you see that the service fails with an internal server error. And within the PurePath, Dynatrace also highlights that the call to the third-party currency-conversion service is failing.

This third-party currency-conversion service is hosted externally, and although no OneAgent is deployed there, you can still seamlessly integrate OpenTelemetry trace data emitted by this third-party service by leveraging the new trace ingest API endpoint.

The span attributes emitted for the conversionRequest include the HTTP 400 status code, which tells us that because of too many requests to the service, throttling has kicked in, causing the service failures and user frustration.

This insight into the root cause of the issue allows you to work out a mitigation plan for the service failures by, for example, reaching out to the cloud provider that hosts the third-party service.

However, you’ll also want to reach out to those frustrated users who haven’t been able to successfully place an order in the Online Boutique. With additional user details added via OpenTelemetry custom instrumentation—picked up by OneAgent this time—you can take further action and offer a voucher to those customers for their next order, for example.

This example shows how Dynatrace and OpenTelemetry work better together helping you to achieve observability at scale with automated instrumentation, continuous topology mapping, and causation-based AI.

Meet data privacy requirements and control the scope of stored span data

Dynatrace automatically captures all OpenTelemetry span and resource attributes. However, as span and resource attributes can potentially include sensitive personal information, personally identifiable information (PII), and other sensitive data, Dynatrace only stores span and resource attribute values if you explicitly specify related attribute keys. This enables you to maximize the data protection features that are provided by Dynatrace.

How to get started

Beginning with Dynatrace version 1.222, trace ingest is generally available, and further information on trace ingest can be found in Dynatrace Documentation.

Please share your feedback with us

We’ve set up a feedback channel in the Dynatrace Community. Please use this channel to share your thoughts about the new trace ingest capability. Your feedback is highly appreciated.

What’s next

We’re constantly improving trace ingest. Stay tuned for further updates.

The post Dynatrace adds OpenTelemetry trace ingest for open, automatic, and intelligent observability appeared first on Dynatrace news.

IT, DevOps, and SRE teams seeking to know the health of their apps and services have always faced obstacles that can drain productivity, stifle collaboration, ratchet up the time to resolution, and limit the effectiveness of their collaboration with other parts of the business. As organizations update their IT environments with the latest cloud-native technologies and architectures, teams need to weigh the effectiveness of traditional monitoring vs. modern, observability-based solutions to decide how to solve their existing challenges amid the growing complexity of their dynamic, multicloud environments.
Legacy approaches to monitoring give your team piles of siloed data, but sometimes provide only fragments of insight into specific layers of the technology stack without context. They often require painstaking manual processes to piece together an accurate picture and pinpoint the source of a problem.
An approach based on observability, automation, and AI, on the other hand, enables you to know precisely what is happening within your environment based on contextualized insights derived from billions of interdependencies among apps, services, and infrastructure. Dashboards and visualizations display precise real-time answers prioritized by business impact, cutting through alert noise so you can prioritize the most important issues first.
Despite its obvious advantages, there are several misconceptions that keep businesses from implementing a modern AI-driven observability platform. To set the record straight, here are five myths about AI-based observability along with some key points on its full capabilities and benefits.

Myth 1 – Enterprise-scale observability is too difficult to accomplish

Although many IT and DevOps teams would love to be able to immediately identify the root cause of application, infrastructure, and user experience issues, they may be skeptical that this goal is even possible to accomplish. After all, they’re often using multiple monitoring tools to keep track of increasingly complex environments, and most of the time, these tools don’t talk to one another. Or they may think that observability is just a fancier word for traditional monitoring.

Fortunately, automatic and intelligent observability is easily achievable using AI and offers benefits that extend beyond IT. With a single source of all internal and external observability data, IT teams can zero in on performance issues across the full software stack, from cloud infrastructure to Kubernetes orchestration, to end-user apps, and see how services interact with each other in real-time. DevOps can view the health of all systems with greater clarity and gain a window into the user experience. Meanwhile, business metrics tie everything together with KPIs that automatically connect digital performance with business outcomes. In addition to the benefits of intelligent automation, having a single source of observability data for all teams also means organizations can streamline operations and tool sets, easing the burden on all teams involved and improving cross-team collaboration.

Myth 2 – AI-based observability won’t help

On average, organizations are using 10 monitoring solutions across their technology stacks despite having full observability into just 11% of their application and infrastructure environments. If digital teams already have multiple monitoring tools in place, they may think an AI-based observability solution won’t help. But if you asked them if they need to eliminate blind spots across the scale, complexity, and frequent changes of multicloud environments, pinpoint root causes among trillions of dependencies, and reduce noise to prioritize only what matters, they would probably answer with a resounding “yes.”
An automatic and intelligent observability platform with AI at the core delivers on all three of those goals, empowering organizations to realize the benefits of cloud-native technologies while cutting through their complexity. With a single AI-based solution, DevOps teams can streamline complex environments, automate their work, and release better software more quickly.

Myth 3 – Users won’t know the difference

When you rely only on siloed telemetry from back-end applications to make sure your front-end applications, microservices, containers, and infrastructure work as intended, it’s hard to get insight into the actual user experience. A solution that offers true AI-driven observability, by contrast, can show you precisely how your applications and infrastructure are performing in real-time. In fact, you can instantly access a session playback recording and see exactly what the user was doing and right where they ran into a problem. From there, you can quickly resolve the issue and deliver a smooth customer experience that keeps revenue flowing.

Myth 4 – An observability solution can’t handle multicloud complexity

Even IT, DevOps, and SRE professionals who think observability is technically achievable may still assume that only applications that run on a single cloud platform is within its scope, resulting in only a partial view of their environment. But an AI-powered observability solution that uses continuous automation is ideally suited to monitor multicloud environments, enabling businesses to successfully manage cloud complexity and keep pace with rapidly accelerating digital transformation.
Intelligent observability can handle the full scale, volume, and diversity of business data across an organization’s multicloud and hybrid cloud environments, including Kubernetes, serverless apps, and open-source technologies. Deterministic AI can identify the relationships and dependencies among apps, services, and infrastructure based on an automated, interactive, real-time map of the environment. With this capability, a solution can deliver precise, context-driven insight to just about any question, workload, or scenario.

Myth 5 – AI-based observability requires too much administrative overhead

Manually instrumenting applications to get observability data means constantly rewriting, retesting, and redeploying each application—and then you have to define performance thresholds for all your infrastructure components and services before monitoring observability data can even begin.
IT teams simply don’t have time for this. As it is, 70% of CIOs say their teams are forced to spend too much time doing manual tasks that could be automated if only they had the means.
Fortunately, there’s a better way. An AI-driven observability solution can automatically set performance parameters derived from a past baseline of normal behavior, which frees your team from the laborious task of having to manually configure each setting. And the best solutions require just a few clicks to deploy a single self-updating agent. With so much maintenance work out of the way, you can focus on innovating and optimizing applications that improve business outcomes and user experiences.

Innovate faster with an all-in-one observability solution

IT and DevOps teams are under pressure to deliver rapid business innovation within increasingly complex environments, all while running on lean resources. Traditional monitoring and siloed observability tools put the burden on these teams to manually troubleshoot performance issues, whereas an AI-driven approach simply presents them with the answers they need. By taking advantage of the full capabilities and benefits of an all-in-one, automatic, and intelligent observability platform, organizations can continuously deliver a compelling user experience that delivers results.

The post 5 myths about observability appeared first on Dynatrace news.

Cloud-native technologies and microservice architectures have shifted technical complexity from the source code of services to the interconnections between services. For example, to address challenges like asynchronous communications or security and isolation in microservice architectures, organizations often introduce third-party libraries and frameworks like Hazelcast IMDG. However, such heterogeneity of interconnected software services can lead to visibility gaps in end-to-end traces, which create blind spots and make it difficult for organizations to keep their software services running and their customers happy.

Dynatrace is fully committed to the OpenTelemetry community and to the seamless integration of OpenTelemetry data, including ingestion of custom metrics, into the Dynatrace open analytics platform. We also want to provide best-in-class end-to-end observability insights to our customers who rely on third-party libraries and frameworks that are equipped with OpenTracing pre-instrumentation.

Therefore, we’re happy to announce support for OpenTracing data that’s emitted by auto- and custom-instrumentation of Java source code with Dynatrace PurePath 4, our distributed tracing and code-level analysis technology. OneAgent enriches OpenTracing data with additional information, for example, trace-level data like method hotspots and CPU analysis, and enables full end-to-end view of transactions in hybrid deployments where not all services have OpenTracing enabled.

Observability for heterogeneous cloud-native technologies is key

Let’s consider the business challenges of an online shop that is powered by a microservice architecture where several instances of each microservice run, including the shopping cart service, to ensure the highest possible availability. To address potentially high numbers of requests during online shopping events like Singles Day or Black Friday, it’s crucial that this online shop have a memory storage strategy that allows for speed, scaling, and resilience of all microservices, especially the shopping cart service. To address these types of challenges, organizations typically introduce third-party libraries and frameworks like Hazelcast IMDG.

Since “hope is not a strategy” when it comes to running software services, you need to eliminate bad monitoring and instead establish an observability strategy for your services, as well as for involved third-party libraries and frameworks, that provides actionable answers instead of just more data.

Announcing seamless integration of OpenTracing data into Dynatrace PurePath 4

Dynatrace provides information on every request, through every single microservice or serverless function, ultimately eliminating existing blind spots or visibility gaps in end-to-end traces. Support of W3C Trace Context ensures end-to-end visibility, even when services from other teams (which may be monitored with other solutions) are called in the middle of transactions.

The seamless integration of OpenTracing traces that are emitted by third party libraries and frameworks like Hazelcast into PurePath 4 enables powerful analytics, including:

• Out-of-the-box service hotspot analysis
• Deep-code execution details
• Always-on profiling in transaction context

On top of that, no manual configuration is required for tapping into OpenTracing pre-instrumentation and using captured trace data for the elimination of blind spots. Dynatrace OneAgent, our continuous auto-discovery and auto-instrumentation technology, instruments the OpenTracing API (more specifically, the OpenTracing GlobalTracer component) and ensures seamless integration of OpenTracing trace data into PurePath 4.

With Dynatrace OneAgent you also benefit from support for traffic routing and traffic control. OneAgent implements network zones to create traffic routing rules and limit cross data-center traffic. Dynatrace OneAgent also has built-in Adaptive Traffic Management to ensure high-fidelity data capture while keeping network traffic low.

Upgrade OpenTracing instrumentation with high-fidelity data provided by OneAgent

We’ve set up a sample Java application that integrates Hazelcast to demonstrate the seamless integration of OpenTracing trace data into Dynatrace PurePath 4. The sample, a Spring app powered by two Hazelcast client instances, allows for the manipulation of a map of key-value pairs. These put and get commands are called by CronJobs every minute. The sample app relies on two Hazelcast cluster nodes. The complete setup runs as Kubernetes pods.

We’ve instrumented the sample application using OpenTracing GlobalTracer and we’ve enhanced HazelcastInstance with TracingHazelcastInstance as shown in the code snippets below.

public Tracer initTracer() {
   return GlobalTracer.get();
}

HazelcastInstance hazelcastInstance;
public HazelcastInstance getTracingHazelcast() {
     if (hazelcastInstance instanceof TracingHazelcastInstance) {
        return hazelcastInstance;
    }
     hazelcastInstance = new TracingHazelcastInstance(
            HazelcastClient.newHazelcastClient(),
                false);
    return hazelcastInstance;
}

ConcurrentMap<String, String> map = getTracingHazelcast().getMap("map");
map.put(key, value);

Once OneAgent OpenTracing support for Java is enabled (at Settings > Server-side service monitoring > Deep monitoring), OpenTracing span data captured by OneAgent is seamlessly integrated into PurePath 4. You get code-level insights into application code without code changes.

Note: This capability is limited to OpenTracing version 0.33.0 and requires OneAgent version 1.205 and Dynatrace version 1.207.

Additionally, you benefit from topological details that are related to the service at hand. For instance, you can easily investigate and navigate to the process or host where the application is running and you’ll be notified of any related problems by our Davis® AI causation engine. Dynatrace OneAgent also collects additional metadata, including span attributes, while ensuring the protection of sensitive data. You can easily control the visibility of personal data with Dynatrace data-privacy masking capabilities.

These Dynatrace capabilities, among others, enable enterprises to deliver superior user experience and resolve problems rapidly.

Try it out today

Are you new to Dynatrace? Start your free trial today—tame blind spots and visibility gaps with end-to-end observability with seamless integration of OpenTracing data.

Already a Dynatrace customer? You can start benefiting from the seamless integration of OpenTracing Java into PurePath today!

Your feedback is highly appreciated

Don’t hesitate to reach out to us with your thoughts about our latest extensions to PurePath 4 as well as the seamless integration of OpenTracing data. Please share your feedback with us at Dynatrace Community.

What’s next?

Stay tuned for upcoming announcements around OpenTracing and OpenTelemetry. For example, we’re broadening our OpenTracing capabilities toward leveraging span attributes within Dynatrace for the analysis of problematic requests.

Furthermore, the marvelous druids of the Dynatrace Go team are currently finishing up OneAgent support for for the capture of OpenTelemetry trace data. This will unlock extended end-to-end traceability for OpenTelemetry pre- and custom-instrumentation.

The post Unlock end-to-end observability insights with Dynatrace PurePath 4 seamless integration of OpenTracing for Java appeared first on Dynatrace news.

What is distributed tracing?

Distributed tracing is used to understand control flow within distributed systems. Especially in dynamic microservices architectures, distributed tracing is an essential component of efficient monitoring, application optimization, debugging, and troubleshooting.

Basically, it’s all about seeing how different services are connected and how your requests flow through those different services. It allows for the finding of cause-effect relationships between events, for instance, identifying which user action in a browser caused a failure in the business logic layer.

Distributed tracing—done completely automatically—has been a core component in Dynatrace from the very beginning. The value of Davis®, the Dynatrace AI causation engine, is built upon the quality of the data we collect. As such, understanding all the interdependencies among services through end-to-end tracing is key to us.

Dynatrace drives a new set of W3C standards to tackle known limitations

There are some known limitations to distributed tracing as it exists right now. Up to now, APM vendors and open source tools used their own defined HTTP headers (for example, x-dynatrace at Dynatrace) for distributed tracing. There were situations where end-to-end visibility would break, for example, when middleware (such as an API gateway) didn’t automatically forward the custom HTTP headers to a called service.

For the last year, my colleagues Alois Reitbauer (chair of the W3C Distributed Tracing Working Group), Christoph Neumüller, and Daniel Khan have been working with the community (open source tool providers and commercial vendors) on a set of standards called the W3C Trace Context specification that defines a unified approach to context and event correlation required for distributed tracing. Trace Context is now a candidate recommendation from the W3C, and we expect cloud vendor services and framework developers to comply with this standard in the future. This will result in fewer broken transactions and, therefore, more end-to-end visibility for our users.

Without W3C Trace Context

APM vendors and open source tools use their own defined HTTP headers for context propagation. These headers aren’t always transported by third-party components such as middleware, which can result in broken transactions.

With W3C Trace Context

While Dynatrace is the first vendor supporting W3C Trace Context, other vendors will have to follow and also open source tools will support the same standardized set of HTTP headers (trace parent and trace state) as defined in the Trace Context specification. Middleware and frameworks will then be empowered to forward the headers standardized by the W3C and propagate them to outgoing calls, thereby avoiding the breakage of end-to-end monitoring of individual requests.

If you’re interested in learning more about the concepts of distributed tracing, context propagation, and the related challenges, take a look at this excellent article by Alois Reitbauer.

Introducing W3C Trace Context support in Dynatrace – get even more precise answers

We’re happy to announce the Early Adopter Release of W3C Trace Context support in Dynatrace. Openness is in the DNA of our software intelligence platform. Supporting the W3C Trace Context will provide significant value to our customers as Davis, our AI-engine, provides even more precise answers with the improved end-to-end visibility, not just more data as traditional monitoring solutions.

End-to-end tracing through cloud services

Along with Microsoft, Google, and others, Dynatrace is a co-editor of the W3C Trace Context standard. Dynatrace is committed to implementing Trace Context into its services. Microsoft has already introduced Trace Context support in some of their services, including .NET Azure Functions, API Management, and IoT Hub.

Up to now, when a request went through an unmonitored .NET Azure Function, the transaction would break and Dynatrace only detected the outgoing web request to the Azure Function:

With Trace Context support in Dynatrace, end-to-end tracing will work out of the box, with no configuration required. The unmonitored Azure Function will be displayed as a proxy between the monitored services Producer and Receiver:

To install OneAgent on Azure and start monitoring Azure Functions, check out our Azure Site Extension for installing OneAgent on Azure Functions.

Interoperability with OpenTracing and OpenCensus

As the popularity of microservices architecture increases, many more teams are getting involved with the delivery of a single product feature. What if a team doesn’t yet use Dynatrace but has manually added instrumentation to their code using an SDK such OpenTracing or OpenCensus and they use an open source tool to help debug code across microservices, such as Jaeger?

Up to now, transactions would stop with the boundary of environment A.

With Trace Context support, you will no longer lose end-to-end visibility even if an unmonitored service from another team is being called in the middle of your transaction, as the same trace ID (contained in the trace parent header) will be shared by all services, provided the library that the developers are using already supports the new W3C Trace Context standard.

If Dynatrace detects in environment A that the unmonitored services in environment B are the root cause of an issue, you will be able to look in the debugging tool used by the developers in environment B for more information about that exact transaction simply by using the trace ID displayed in the PurePath details of environment A:

And of course, if the developers want to look at the transaction in Dynatrace, they can use the trace ID as a filter in PurePath view:

Trace Context support will enable teams to easily migrate to Dynatrace without breaking end-to-end visibility.

How can I get this feature?

• W3C Trace Context support is available as an Early Adopter Release (no registration required) beginning with Dynatrace version 1.180.
• You can enable this support in the Dynatrace web UI at Settings > Server-side service monitoring > Deep monitoring > Distributed tracing.

What’s next

Trace Context support is a first step that will enable us to support more and more distributed tracing use cases, such as:

• Support for Open Telemetry, the new, unified set of libraries and specifications for observability telemetry resulting from the merging of the OpenTracing and OpenCensus projects as an alternative to our OneAgent SDK
• Increased end-to-end tracing in Kubernetes and Cloud Foundry environments with support for Envoy/Istio.
• Correlation of transactions and logs
• Support for tracing through multiple Dynatrace environments

The post Distributed tracing with W3C Trace Context for improved end-to-end visibility appeared first on Dynatrace news.