This is the second post in our series on LLM evaluations. In the companion post, Evaluate LLM and agent quality in Dynatrace AI Observability with dt-evals, we showed you how to run online evaluations against real GenAI prompt traces and bring quality scores into Dynatrace AI Observability alongside latency, cost, and errors. This post steps back to the fundamentals: what evaluations are, how they work, and the methods teams use to measure AI quality.

Just as traditional software relies on testing frameworks to ensure reliability, AI systems require robust evaluation frameworks to measure the quality, accuracy, and safety of model outputs. Evals are the primary mechanism by which teams build trust in, iterate on, and responsibly deploy AI systems. Without them, organizations may risk deploying systems that produce unreliable answers, hallucinate facts, or quietly degrade in performance over time.

Key takeaways

• Evaluations are how teams move from “the LLM feels right” to “we can prove the LLM works.”
• There is no single best evaluation method. The right approach depends on what you’re measuring and why.
• LLM-as-a-Judge is one powerful tool within the broader evaluation ecosystem, not synonymous with evals as a whole.
• Online and offline evaluations serve complementary roles: offline for development, online for production monitoring.
• A mature evaluation strategy combines code-based, model-based, and human-based methods.
• Evals should be treated as living artifacts — maintained, versioned, and improved over time like any other engineering asset.

Why LLM evaluation is fundamentally different from traditional testing

Traditional software produces deterministic outputs — the same input consistently returns the same result, making pass/fail testing straightforward. LLMs are probabilistic systems: the same prompt can produce different responses depending on context, temperature, and model behavior. This variability makes conventional testing methods insufficient.

Instead of verifying a single correct output, teams must evaluate across multiple dimensions simultaneously:

• Correctness— does the response answer the question accurately?
• Relevance — is the output aligned with the user’s intent?
• Faithfulness — is the response grounded in source data, not invented?
• Safety and bias — does the output comply with organizational policies?

This transforms evaluation from simple pass/fail checks into continuous measurement of AI quality.

The hallucination problem

The most well-known consequence of probabilistic generation is hallucination — when a model produces plausible-sounding but factually incorrect information. This happens because LLMs predict likely word sequences rather than verify facts, which enables powerful reasoning but introduces serious risk in enterprise environments where accuracy is critical.

Addressing this requires evaluation frameworks that track signals like factual accuracy, semantic similarity, groundedness in source data, and consistency across responses. These metrics transform subjective quality judgments into measurable, improvable signals.

What is an LLM evaluation?

An LLM evaluation is a systematic process of testing a model or AI-powered system to determine whether it meets a defined standard of quality. That standard could be factual accuracy, helpfulness, safety, tone, latency, cost-efficiency, or any other measurable dimension that matters to the application.

Evaluations translate vague product goals (“the assistant should be helpful and safe”) into concrete, repeatable measurements. They allow teams to:

• Catch regressions when a model is updated, or a prompt is changed.
• Compare candidates — different models, prompt versions, or retrieval strategies — objectively.
• Build accountability by producing evidence that a system behaves as intended.
• Accelerate iteration by giving developers fast, structured feedback loops.

Evals exist on a spectrum of formality, from a small hand-curated test set run locally, to a large, automated pipeline running thousands of test cases in CI/CD on every deployment.

How do LLM evaluations operate?

At their core, evaluations follow a consistent pattern regardless of their complexity:

1. Define the task and success criteria. What should the LLM model do, and how will you know when it does it correctly? This is the hardest and most important step.
2. Assemble a dataset. A set of inputs (prompts, user messages, documents) paired with expected outputs or grading rubrics. Datasets can be human-curated, synthetically generated, or sampled from production traffic.
3. Run inference. Pass the inputs through the system under test and collect outputs.
4. Score the outputs. Apply a scoring method — a function, a model, or a human — to assess how well each output meets the success criteria.
5. Aggregate and analyze. Roll up scores into metrics (accuracy, pass rate, average score), visualize distributions, and compare against baselines or previous runs.
6. Act on results. Use the findings to accept or reject a change, file a bug, update a prompt, or trigger retraining.

This loop can run manually during development, automatically in CI/CD pipelines, or continuously against live production traffic.

What’s the difference between LLM evaluations and LLM-as-a-Judge?

This is one of the most common points of confusion in the space.

LLM evaluations are the broader discipline — the full process described above. They encompass everything from how you define success to how you collect test data to how you score outputs to how you act on results.

LLM-as-a-Judge is one specific scoring method that can be used within an evaluation pipeline. It involves using a language model (often a strong general-purpose model like GPT-5 or Claude Sonnet 4.6) to automatically assess the quality of another model’s outputs.

Think of it this way: evaluations are the framework, and LLM-as-a-Judge is one type of grader you can plug into that framework — alongside code-based graders, human graders, or embedding-based similarity checks.

• LLM-as-a-judge handles open-ended, subjective dimensions (tone, creativity, helpfulness) that are hard to capture in code.
• It scales to large datasets without human effort.
• It can be surprisingly well-calibrated when prompts and rubrics are carefully designed.

Limitations

• Inherent biases of the LLM model used to judge (verbosity bias, position bias, self-preference).
• Requires prompt engineering and validation to ensure the judge is grading what you intend.
• Adds cost and latency to the evaluation pipeline.
• Not appropriate for tasks with clear ground-truth answers where code-based checks suffice.

Code-based evaluations

Code-based evaluations use deterministic functions — written in Python or any language — to score model outputs. No secondary LLM model is involved.

How it works

You write a function that takes the model output as input and returns a score. The function might check for exact string matches, run regex patterns, execute generated code and test it, parse JSON and validate its structure, call an external API to verify a fact, or compare numerical results.

Common patterns

• Exact match — does the output equal the expected answer?
• Contains / regex match — does the output include a required phrase or follow a required format?
• Execution-based — for code generation tasks, run the output and check whether tests pass.
• Structured output validation — parse JSON/XML outputs and verify schema and values.
• Tool call verification — for agentic tasks, did the model call the right tool with the right parameters?

Strengths

• Fully deterministic and reproducible.
• Fast and cheap to run at scale.
• Easy to understand, debug, and audit.
• No dependence on a secondary model’s judgment.

Limitations

• Cannot handle open-ended or subjective quality dimensions.
• Requires knowing the exact expected output or a verifiable property of the output.
• Brittle for tasks where there are many valid correct outputs (for example, summarization, creative writing).

Code-based LLM evals are the first tool to reach for whenever a task has a clear, verifiable answer. They form the backbone of any reliable eval suite.

Online vs. offline evaluations

These two modes are not competing approaches — they’re complementary phases of a complete evaluation strategy.

Offline evaluations

Offline evals run against a static, pre-collected dataset before a system reaches production. They’re the evaluation equivalent of unit and integration tests in software development.

• When: During development, before deploying a new model, prompt, or retrieval change.
• Dataset: Curated, labeled, or synthetically generated. Often maintained in version control.
• Latency: Can run in batch; speed is less critical.
• Use cases: Regression testing, model comparison, prompt optimization, safety red-teaming, fine-tune evaluation.

Key advantage: Full control over the test distribution and ground-truth labels.

Key limitation: The dataset may not reflect real user behavior or the long tail of production inputs.

Online Evaluations

Online evals run against live production traffic in real time or near real time. They observe what is actually happening when real users interact with the system.

• When: Continuously, in production.
• Dataset: Real user inputs — unlabeled, unpredictable, and representative.
• Latency: Must be fast or asynchronous to avoid slowing down user-facing requests.
• Use cases: Production monitoring, anomaly detection, drift detection, A/B testing, continuous quality assurance.

Key advantage: Captures real-world usage patterns, prompts, and failure modes from production traffic, giving teams the most representative signal for monitoring AI quality over time.

Key limitation: No pre-defined labels; scoring must rely on heuristics, implicit signals (thumbs up/down, re-prompts), or async LLM-as-a-Judge pipelines.

Get started with LLM evaluations today

The field of LLM evals is evolving rapidly. As enterprises deploy increasingly autonomous AI systems, evaluation can play an important role in improving AI accuracy, reliability, and safety.

Here are the trends worth watching and investing in:

1. Evaluation-driven development. Treat evals as a first-class engineering artifact. Write eval cases before building features, maintain them in version control, and integrate them into CI/CD pipelines — mirroring test-driven development practices from software engineering.
2. Agentic and multi-step evaluation. As AI systems move from single-turn Q&A to multi-step agents that use tools and maintain state, evaluations must evolve to assess full trajectories rather than just individual outputs. This includes evaluating tool use, planning quality, error recovery, and task completion over long horizons.
3. Adversarial and safety evals. Red-teaming — probing a system for failures, biases, and unsafe behaviors — is becoming a standard part of the eval lifecycle, especially as regulatory requirements around AI safety mature.
4. Human-in-the-loop calibration. Even automated eval pipelines benefit from periodic human review to catch drift in what the judge model or scoring function is measuring. Building lightweight human-annotation workflows alongside automated evaluations yields a more reliable signal over time.
5. Standardization and benchmarking. The industry is moving toward shared benchmarks and eval frameworks (for example, HELM, MMLU, LMSYS Chatbot Arena, OpenAI Evals) that allow apples-to-apples comparisons across models. Building internal evals that complement these public benchmarks will be an increasingly important capability for any team deploying LLMs.
6. Cost-aware evaluation. As evals scale, cost becomes a real constraint. Emerging approaches include training lightweight specialized judge models, using embedding-based similarity as a cheap first filter, and intelligently sampling which examples need expensive LLM-as-a-Judge scoring.

Organizations that invest early in robust evaluation frameworks and combine them with AI observability will be positioned to scale AI safely across their operations.

Ready to put this into practice?

See our companion blog post, Evaluate LLM and agent quality in Dynatrace AI Observability with dt-evals, to learn how dt-evals lets you run LLM-as-a-judge evaluations on real GenAI traces and turn AI quality into a queryable, trendable, and alertable signal inside Dynatrace AI Observability.

Because in the end, AI systems are only as trustworthy as the processes used to evaluate them.

The post Beyond LLM-as-a-judge: Establishing LLM evaluations as a foundation for trustworthy agentic AI systems appeared first on Dynatrace news.

For AI systems, reliability is defined by response quality, factual grounding, data security, and usability — and those signals need to live alongside the same observability data teams already trust to monitor performance and availability.

When evaluation scores are isolated in notebooks, spreadsheets, standalone tools, or CI logs, they’re hard to operationalize. By bringing AI quality metrics into Dynatrace AI Observability—next to latency, cost, errors, traces, and user behavior—teams can connect poor responses and hallucinations directly to the prompts, models, retrieval contexts, tool calls, services, and traces that produced them.

What is dt-evals?

dt-evals is an open source CLI for evaluating LLM and agent quality from real GenAI traces, agentic interactions. Teams can run online evaluations against live or recent interactions, score outputs with an LLM judge, and send structured results back to Dynatrace AI Observability so quality becomes visible, queryable, trendable, and actionable.

A minor prompt edit, model change, or retrieval update to an AI application can improve one behavior while quietly breaking another. The challenge to tracking down where and why these systems break is that evaluation results are often maintained outside the operational workflow, making it difficult to connect a low score to the exact trace, prompt, model version, retrieval context, tool call, or service that produced the unwanted behavior.

Dynatrace AI Observability closes this loop. With dt-evals and the AI Observability Evaluation Preview teams can pull recent gen_ai.*  spans, score real interactions with an LLM judge, and write structured evaluation results back as business events. These scores can be viewed with the originating trace, queried for custom analysis, trended in dashboards, and used to trigger alerts or workflow-driven remediation.

A failing faithfulness score is no longer just a number in a report. It’s now an operational signal.

What are LLM evaluations?

An LLM evaluation system scores an AI response against a range of quality and safety dimensions. Common examples include whether the answer is relevant to the question, faithful to the provided context, free of hallucinations, safe for users, complete enough to be useful, and resistant to prompt-injection attempts.

LLM evaluations are typically applied in two modes:

Offline evaluations run before release against a fixed test set or curated trace dataset. These are used to compare a proposed prompt, model, retriever, or agent-tool change against a known baseline before shipping. For example, replay 500 representative support questions in CI and block the release if faithfulness drops below the configured threshold.

Online evaluations run after deployment against sampled production or user traffic. Use online evaluations to detect regressions caused by live inputs, changing retrieval results, tool behavior, traffic mix, or model drift. For example, evaluate 10% of support-agent traces from the last hour and alert the team if hallucination failures exceed the configured window.

With dt-evals, you can run evaluations from the command line, use them in CI/CD, or schedule them to detect quality regressions autonomously after deployment as a post-processing quality gate for your AI agents and LLM output.

Run evaluations from the command line

dt-evals is an open source evaluation toolkit for teams that want to bring their own data, judge provider, and evaluation logic while keeping traces, scores, dashboards, and alerts connected.

Install the CLI:

npm install -g @dynatrace-oss/dt-evals

Or run it directly with npx:

npx @dynatrace-oss/dt-evals <command>

A typical first run has three steps:

1. Configure your environment and judge provider (Bring Your Own AI API key):

dt-evals configure

2. Verify your local setup and connection:

dt-evals doctor

3. Run evaluations on recent GenAI traces:

dt-evals run --since 1h --sample 10

This command evaluates traces from the last hour and samples 10% of them. In other words, dt-evals evaluates roughly one out of every ten matching traces, including the prompt and completion messages associated with each selected trace.

During configuration, you provide the connection to your Dynatrace environment and the credentials for the LLM judge provider you want to use. dt-evals does not require teams to send evaluations through a fixed provider. You bring your own judge credentials and control where evaluation execution happens.

For CI/CD use cases, run in CI mode:

dt-evals run --since 6h –ci

In CI mode, dt-evals emits machine-readable output and can fail the pipeline when a configured threshold is breached. This makes quality checks part of the same delivery process used for prompt changes, model upgrades, retrieval updates, and agent releases.

Bring your own LLM judge provider

The “LLM-as-judge” evaluation approach involves using an AI model to score another model or agent response. The LLM judge needs to come from an AI provider your team trusts and has approved for the type of data being evaluated.

dt-evals supports common LLM judge AI models and inference providers, including OpenAI, Anthropic, Google/Vertex/Gemini, AWS Bedrock, and Azure OpenAI. Depending on the package and configuration path you use. Teams provide their own credentials, choose the judge model, and can tune execution settings such as thresholds and concurrency.

This matters for both governance and cost control. Teams can decide which LLM provider is assigned to evaluate which traffic, how many judge calls run in parallel, and where evaluation results are stored.

Which quality and safety dimensions are evaluated by dt-evals?

dt-evals supports built-in LLM-as-Judge evaluators for a range of quality and safety dimensions, including:

• Relevance: Does the response answer the user’s question?
• Faithfulness: Is the response supported by the provided context?
• Hallucination: Does the response invent facts that are not present in the available context?
• Answer completeness: Does the response fully address the user’s request?
• Context relevance: Is the retrieved or supplied context useful for answering the question?
• Factual accuracy: Does the response match an expected or known-correct answer?
• Summarization quality: Does the summary preserve the important information?
• Conciseness: Is the response direct with no unnecessary detail?
• Fluency: Is the response clear and readable?
• Toxicity: Does the response contain harmful or abusive content?
• Bias: Does the response show unfair or inappropriate bias?
• PII leakage: Does the response expose sensitive personal information?
• Prompt injection: Did the input or response show signs of instruction manipulation?
• User frustration: Does the interaction suggest the user is blocked or dissatisfied?
• Drift: Are scores changing meaningfully compared with prior behavior?

A Retrieval Augmented Generation (RAG) application might focus on faithfulness, hallucination, context relevance, and answer completeness. A customer-facing support agent might focus on relevance, fluency, bias, toxicity, and prompt-injection risk. An internal assistant might add custom checks for tone, policy compliance, or whether the answer includes required next steps.

Add custom evaluations

Built-in metrics are useful, but most production AI systems also need checks that are specific to the business, domain, or workflow.

Custom evaluations let teams define their own judge prompts, scoring rules, labels, and thresholds. For example, a support team can create a custom evaluator that checks whether an answer includes a required troubleshooting step before recommending escalation. A financial services team can check whether responses include the required disclaimers. A platform team can check whether an agent uses the correct tool before answering.

The critical point is that custom evaluators run through the same pipeline as built-in evaluators. They can produce the same structured results, appear alongside other scores, and be used in dashboards, alerts, and release checks.

A typical configuration defines the target service, judge provider, sampling strategy, enabled metrics, and thresholds:

schemaVersion: 1
name: support-agent-prod

dynatrace:
  environmentUrl: https://your-env.apps.dynatrace.com
  platformToken: dt0s16.xxxxx

judge:
  provider: openai
  model: gpt-5.5

scope:
  service: support-agent
  since: 1h
  sampling:
    strategy: random
    percent: 10

metrics:
  enabled:
    - faithfulness
    - hallucination
    - relevance
    - drift

alerts:
  thresholds:
    faithfulness: 0.7
    relevance: 0.7

Evaluation results in the AI Observability app

Evaluation results appear directly in the AI Observability app, so teams don’t have to jump between a trace view, an eval report, and a separate dashboard to understand what happened.

In the Prompts view, teams can filter for prompts with evaluation scores and inspect row-level verdicts. Score badges such as relevance, fluency, bias, faithfulness, or toxicity make response quality easy to scan without opening every trace.

This is useful when triaging a regression. Instead of starting with a generic failure count, teams can quickly see which prompts failed, which evaluator failed them, and whether the issue is isolated or widespread.

From an individual prompt or trace, the Evaluations tab shows run-level details, including the evaluation name, score, provider, judge model, method, and supporting metadata.

That detail matters because a failed score is only useful if teams can explain it. Engineers and evaluation owners can move from a low score to the exact prompt, response, trace, model, evaluator, and rationale that produced it.

Query, trend, and alert on evaluation scores

Because dt-evals writes results back as structured events, evaluation scores can be analyzed with the rest of your telemetry.

Teams can ask questions such as:

• Which evaluator has the lowest average score?
• Which services are producing the most failed evaluations?
• Did quality drop after a model or prompt change?
• Are hallucinations increasing over time?
• Is quality improving at the cost of latency or token usage?

For example, to get average score by evaluator you could write this query:

fetch bizevents
| filter event.type == "gen_ai.evaluation.result"
| summarize avg_score = avg(gen_ai.evaluation.score.value),
    by: { gen_ai.evaluation.name }
| sort avg_score asc 

Failed evaluations by service and metric can be determined with this query:

fetch bizevents
| filter event.type == "gen_ai.evaluation.result"
| filter gen_ai.evaluation.score.label == "fail"
| summarize failures = count(),
    by: { dt.service.name, gen_ai.evaluation.name }
| sort failures desc 

Trending is where evaluation data becomes more useful than a point-in-time report. A single failed score can show an issue. A trend can show whether quality is drifting slowly, whether a release caused a sudden drop, or whether a fix actually improved behavior over time.

On the AI Evaluation & LLM App Performance dashboard, teams can track trends in quality score, pass rate, failed evaluations, drift detections, evaluator health, run cadence, and pass/fail volume over time.

How to turn quality regressions into alerts

Evaluation results can also drive alerts. For example, a support agent team may want to notify the AI team when hallucinations appear in production, or when faithfulness drops for more than a few minutes.

name: support-agent-prod

alerts:
  notifications:
    - name: hallucination-detected
      metric: hallucination
      condition: count > 0
      window: 5m
      channel:
        type: slack
        connection: ai-observability-slack
        channel: "#ai-alerts"

    - name: faithfulness-regression
      metric: faithfulness
      condition: fail_rate > 10%
      window: 15m
      channel:
        type: email
        connection: ai-team-email
        to: [ai-team@example.com] 

Deploy the alerts with:

dt-evals alerts list ./support-agent-prod.yaml
dt-evals alerts apply ./support-agent-prod.yaml

Once applied, Dynatrace runs these checks continuously as Workflows. If hallucinations appear in the last five minutes, the team gets a Slack alert. If more than 10% of faithfulness checks fail over 15 minutes, the AI team receives an email. This turns LLM quality from something teams inspect manually into something Dynatrace can monitor and route automatically.

For continuous alerting, evaluation runs need to happen continuously or on a schedule. You can run dt-evals in CI for release checks (see our example here), run it manually during investigation, or deploy a scheduled runner for ongoing production evaluation. An alert is only as fresh as the evaluation results it carries.

Once configured, quality signals can be routed to the teams that need to act. If hallucinations appear in the last five minutes, the team can receive a Slack alert. If more than 10% of faithfulness checks fail over 15 minutes, the AI team can receive an email. This turns LLM quality from something teams inspect manually into something they can monitor and route automatically.

Close the loop in the AI software delivery lifecycle

Evaluation gates are most useful when they meet developers where they already work. Because dt-evals writes evaluation results back into the observability data layer, those results are not limited to dashboards or post-release reviews. They can be queried, inspected, and acted on from development workflows, CI/CD pipelines, and agentic coding environments.

For example, a team can run dt-evals after any change to a prompt, model, retriever, or agent tool, and then use dtctl (Dynatrace CLI tool for AI Agents) to query the resulting evaluation data, inspect related traces, review dashboards, or validate whether a release threshold was met. In an AI-assisted workflow, tools such as Claude Code, Cursor, GitHub Copilot, or an internal agent harness can leverage MCP or CLI access to bring that same observability context into the developer’s daily workflow.

That closes the loop of the AI software delivery lifecycle: teams can evaluate behavior, control rollout decisions, remediate regressions, and feed production learning back into the next development cycle. Quality signals are no longer in a separate report; they’ve become a part of how AI software is built, shipped, and operated.

Bring evaluations into the release process

Evaluation support is not just for inspection after something breaks. It can also help prevent regressions before they reach users.

This makes AI quality part of the release process. Teams can gate changes based on relevance, faithfulness, hallucination risk, prompt-injection risk, toxicity, or custom metrics, rather than relying solely on latency and error rate.

What makes this meaningful is that it’s the same pipeline teams already run. AI quality gates sit alongside the latency, error rate, and SLO gates teams have been using for years. There’s no second CI system, no second platform to learn, no second dashboard to monitor. Quality becomes one more dimension of the release decision, gated the same way performance is gated, by the same platform, in the same pipeline.

Coming next

The current experience makes evaluation results visible and actionable inside Dynatrace AI Observability. Next, the focus is on making evaluation workflows easier to run at scale and easier to compare across changes.

Planned improvements include targeted and bulk trace evaluations, custom evaluation libraries, evaluator versioning and lineage, baseline comparisons, experiment views, native quality gates, and deeper visibility into online evaluations.

These capabilities will help teams compare prompt and model variants, understand quality versus cost and latency tradeoffs, and detect sustained quality regressions before they affect more users.

Start today

To get started, check out the Git repository. You’ll need:

• Node.js 20 or later
• A Dynatrace environment with GenAI spans and the AI Observability app installed

• Credentials for the judge provider you want to use
• A service, trace sample, or CI workflow you want to evaluate

Then, install the CLI:

npm install -g @dynatrace-oss/dt-evals

Configure your service and judge provider:

dt-evals configure

Run your first evaluation:

dt-evals run --since 1h --sample 10

With Dynatrace AI Observability and dt-evals, teams can bring LLM and agent evaluations into the operational loop, where they can trace behavior, score outputs, trend results, alert on regressions, and gate releases before silent failures reach production.

The post Evaluate LLM and agent quality in Dynatrace AI Observability with dt-evals appeared first on Dynatrace news.

Dynatrace helps you answer these questions by extending AI observability for a new wave of coding agents, including Claude Code, Google Gemini CLI, OpenAI Codex CLI, OpenCode, and GitHub Copilot SDK. Together, these integrations give engineering leaders, platform teams, and developers a consistent way to understand agent activity, token consumption, costs, tool behavior, and runtime impact: without forcing teams to stitch together fragmented telemetry across terminals, SDKs, dashboards, and development workflows. Dynatrace public AI agent instrumentation examples on GitHub demonstrate how to provide industry leading observability that drives performance, cost efficiency, and governance across complex, distributed AI-driven systems—all through a unified Dynatrace platform experience that developers can access directly via MCP without leaving their IDE.

From agent activity to engineering insight

As organizations adopt multiple coding agents, new adoption challenges emerge. One team might use Claude Code in the terminal, another may build internal tools with GitHub Copilot SDK, while others experiment with Gemini CLI or Codex CLI. Platform teams want visibility into usage, availability, and costs. Engineering leaders want to know whether agents improve delivery. Security and governance teams want confidence that all prompts, tool usage, and actions can be monitored appropriately.

Dynatrace provides a practical answer to these challenges: a single observability layer for agile development workflows. For agents that emit OpenTelemetry directly, such as Claude Code, Gemini CLI, and Codex CLI, Dynatrace can ingest telemetry related to sessions, tokens, costs, tool executions, errors, and performance. For GitHub Copilot workflows, Dynatrace adds production context, software delivery automation, and GitHub-based integrations that connect agent activity to real engineering workflows.

The payoff is clear. Developers gain visibility into how agents behave in real work. Platform teams can track adoption, usage trends, and cost signals. Engineering leaders can correlate agent activity with commits, pull requests, and delivery outcomes. And with an MCP-enabled production context, teams can connect coding-agent actions to what is happening in production.

“Before we instrumented Claude Code, we had no easy way to break down how our engineers actually used AI, which models, for what tasks, and at what cost. Now we can pinpoint inefficient model use and guide usage toward better cost-performance tradeoffs.”
— Markus Heimbach, Senior Director Software Development

Multiple coding agent experiences, one observability strategy

Each coding agent has a different operating model, which is why a common observability layer matters.

Claude Code

Claude Code already supports built-in OpenTelemetry, making it easy to send metrics and logs to Dynatrace with no code changes. Teams can track sessions, tokens, costs, tool activity, API health, and engineering output such as commits and pull requests. Logs, dashboards, and alerts help teams investigate failures, spot latency spikes, and catch unusual spend or error patterns early.

Gemini CLI

Gemini CLI includes OpenTelemetry-based observability and preconfigured dashboards, making it a strong fit for Dynatrace AI observability. Teams can correlate agent activity with broader platform signals and move quickly from raw telemetry to action. This includes debugging failed runs, identifying slow or error-prone tool calls, and alerting on cost or reliability regressions.

Codex CLI

Codex CLI supports opt-in OpenTelemetry monitoring, giving teams a path to audit usage and strengthen governance across CLI, IDE, and app experiences. With Dynatrace, logs and traces help investigate request flows, delays, and failures across agent workflows. Alerts can flag degraded reliability, unexpected behavior, or rising token consumption before they become larger issues.

GitHub Copilot SDK

GitHub Copilot SDK lets teams embed agentic workflows directly into applications, while Dynatrace adds live observability and security context. This matters because embedded agents become part of real engineering and production-adjacent workflows. Dynatrace helps trace execution paths, use logs for debugging and auditability, and set alerts for failures, latency, or policy-relevant events.

OpenCode

OpenCode is a terminal-based AI coding agent that helps developers work through coding tasks directly from the command line. Because OpenCode ships with native OpenTelemetry support, teams can route telemetry to Dynatrace without code changes by setting standard OTLP environment variables. With Dynatrace, teams can track LLM call volume, session activity, tool usage, request latency, and workflow behavior across real developer sessions. Traces help teams inspect LLM requests, tool executions, session lifecycle events, message processing, file snapshots, or diff operations.

Across all operating models, the value is the same: one strategy for monitoring adoption and impact, understanding costs, logging and tracing agent activity, alerting on reliability issues, and debugging real-world workflows as coding agents scale across the enterprise.

Why this matters now

Teams are no longer asking whether coding agents are useful. They’re asking how to drive adoption, scale them safely, govern them consistently, and prove their impact. That requires visibility into usage, cost, reliability, and engineering outcomes across teams and tools. Dynatrace helps organizations make that shift with the observability and production context needed to expand coding-agent adoption with confidence.

The coding-agent market is moving fast. Claude Code, GitHub Copilot SDK, Google Gemini CLI, and OpenAI Codex CLI each represent a different path toward agentic software delivery, from terminal-based workflows to embedded SDKs and governed local execution. At the same time, Dynatrace has been expanding its developer-facing AI surface with the Dynatrace MCP Server, GitHub Copilot integrations, and AI observability capabilities built to connect agent behavior with real production systems. The timing matters because teams are no longer evaluating whether coding agents are useful. They’re deciding how to drive adoption, scale up usage safely, govern usage consistently, and measure real impact.

Ready to see AI coding agents through a Dynatrace lens?

With Dynatrace, teams can understand adoption, spend, reliability, tool behavior, and engineering outcomes in one place, while giving agents access to the live production context they need to make better decisions.

Whether your developers are working in Claude Code, building on GitHub Copilot SDK, experimenting with Gemini CLI, or adopting Codex CLI, Dynatrace helps bring observability, governance, and production awareness into the heart of agentic software delivery.

Public examples already demonstrate this approach for Claude Code, and the broader Dynatrace MCP and AI observability ecosystem provides the foundation to extend the same value across the next generation of coding agents.

Ready to learn more?

In our Git repository, you’ll find step-by-step examples for supported coding-agent workflows, including how to configure OpenTelemetry export, send telemetry data to Dynatrace, and use the provided dashboards to analyze the activity of your AI coding agents.

The post Dynatrace expands AI Coding Agent monitoring for Claude Code, Google Gemini CLI, Codex CLI, OpenCode, and GitHub Copilot SDK appeared first on Dynatrace news.

As agentic AI becomes mission-critical, systems that reason, act, and self-optimize introduce new operational challenges. Their dynamic and non-deterministic behavior makes them difficult to debug, they can drive unexpected cost spikes, and they inherently lack the auditability required for reliable, enterprise-grade use. Today, we’re excited to announce expanded support for leading agentic frameworks and protocols, along with a new dedicated AI Observability app. With this support, you can build, run, and debug agentic AI applications with confidence across AWS, Azure, and Google Cloud.

What’s new: Broader agentic technology support

Dynatrace supports a broad and rapidly growing ecosystem of agentic AI frameworks and protocols, unifying telemetry from these frameworks via OpenTelemetry and OpenLLMetry into a single, correlated observability model, delivering end‑to‑end visibility across clouds, models, tools, and agents from one platform.

• Amazon Bedrock AgentCore – Dynatrace offers observability for Amazon Bedrock AgentCore agents by collecting metrics such as token usage, model behavior, latency, and errors. This integration provides unified tracing, cost, performance, and guardrail monitoring, along with ready-made dashboards and intelligent anomaly detection and forecasting, helping teams quickly and effectively monitor, troubleshoot, and optimize complex autonomous agent workflows.
• Amazon Bedrock Strands – Dynatrace supports the Amazon Bedrock Strands Agents SDK, enabling comprehensive visibility into agentic AI systems. By instrumenting Strands-based AI agents with Dynatrace, organizations can monitor agent behavior, tool usage, and dependencies end to end. This helps ensure performance, reliability, and operational insight across distributed environments, supporting the confident development and operation of agentic AI use cases such as chatbots, recommendation systems, and autonomous workflows.
• LangChain Agents – Dynatrace provides observability for applications built with the LangChain framework, enabling the monitoring of performance, cost, and reliability of Large Language Model (LLM) applications and agents.
• Google Agent Development Kit (ADK) – Dynatrace provides observability for applications built with the Google Agent Development Kit (ADK), enabling visibility into agent execution, dependencies, and performance. This helps teams understand runtime behavior and maintain reliability as agent-based applications
• OpenAI Agents SDK – Dynatrace provides observability for observing applications built with the OpenAI Agents SDK, enabling monitoring of agent workflows, model interactions, latency, and errors. This supports improved operational insight, troubleshooting, and performance optimization for agentic AI applications.
• MCP AI Agent–  Dynatrace provides deep visibility into AI agents communicating via the Model Context Protocol (MCP). By observing both AI agents and MCP servers, organizations gain end-to-end insight into execution flows through tracing, enabling data-driven decisions, performance and cost optimization, and governance for complex agent workflows.

This agentic coverage is on top of the 40+ LLM technologies that Dynatrace already supports, including OpenAI, Amazon Bedrock, Google Gemini and Vertex, Anthropic, LangChain, NVIDIA, and more.

We’re working closely across AWS, Microsoft Azure, and Google Cloud ecosystems to ensure you have consistent, enterprise‑grade observability for your multi‑AI and multi‑cloud applications.

See it in action in the new AI Observability experience

The AI Observability app is now Generally Available, delivering a purpose-built experience for observing AI workloads end-to-end from agents and LLMs to orchestration layers, emerging protocols, and tools. It gives engineering teams deep, production-ready visibility into how AI systems behave in real time, allowing them to validate changes faster, reduce risk, and confidently ship AI-powered features at scale.

Unlike generic observability views, the AI Observability app is designed specifically for agentic and LLM-driven systems, making it easy to understand complex multi-step interactions, reason about cost and performance trade-offs, and troubleshoot issues across models, tools, and dependencies.

Key capabilities

• End‑to‑end observability for agentic AI
  • Monitor agent interactions, tool usage, dependencies, latency, and reliability
  • Track token consumption, cost trends, and caching impact
• Tracing and debugging for complex flows
  • Follow prompts, tool calls, and model invocations from the initial request to the final response
  • Jump from high‑level health to prompt‑level traces in a couple of clicks
• Actionable insights at scale
  • Rapid A/B testing across model and prompt variants for faster validation
  • Identify bottlenecks and optimize resource utilization with ready‑made dashboards and drill‑downs
• Security, privacy, and governance
  • Enterprise‑grade controls, auditability, and policy‑aligned routing
  • Guardrail outcomes (for example, toxicity, PII, or denied topics) are surfaced so you can monitor behavior and trends. (Note that guardrail enforcement occurs at the model/provider; Dynatrace captures and visualizes provider‑reported outcomes.)

Who this solution is for and why it matters

The Dynatrace AI Observability solution is for enterprise teams, including developers, DevOps, SREs, and business leaders who need deep, real-time insights into their cloud native  AI-powered applications and customer experience in a single unified view.

Who benefits the most from this solution?

• AI Engineering and Data Science: This group includes practitioners who develop and optimize models. They use LLM observability to track metrics related to model performance, such as identifying hallucinations and biases, validating changes, and improving prompt engineering practices.
• Software Developers: These individuals benefit from observability by gaining insights into application-level performance, which helps them debug and improve overall code quality. Observability tools allow for faster iteration in development cycles.
• Site Reliability Engineers (SRE): These teams ensure the reliability and performance of AI applications in production environments. They use observability to identify system-level bottlenecks and failures, and to respond swiftly to operational challenges.
• Application Security Teams: Although not traditionally the primary users, security teams can leverage AI observability to identify and mitigate emerging threats specific to AI applications, such as prompt-injection attacks and data leaks.
• Compliance and Governance Teams: Responsible for ensuring adherence to regulatory requirements and internal policies, these teams rely on observability to audit model behavior and to identify potential biases or harmful outputs.

What’s next: Agent topology view with Smartscape

We’re committed to further enhancing these capabilities. As agentic systems evolve into distributed networks of models, tools, and decisions, observability must move beyond traces and metrics. Our next focus is the Agentic Topology View, bringing Smartscape-grade visualization to agent execution flows so teams can see how agents interact, invoke tools, propagate errors, and improve performance end to end.

This agentic topology becomes the foundation for a deeper developer experience by connecting production telemetry with prompt management and evaluation workflows. By unifying agent topology, prompt lifecycle, and LLM-as-judge scoring in a single system, we’re helping teams systematically improve the reliability, performance, and quality of agentic AI at enterprise scale.

Get started today

• Install the AI Observability app from Dynatrace Hub
• Instrument your agents via OpenTelemetry/OpenLLMetry
• Start tracing prompts, tool calls, and model invocations—and turn insights into action

The post Announcing agentic framework support and General Availability of the Dynatrace AI Observability app appeared first on Dynatrace news.

Teams can transition from setup to insights in minutes using a lightweight OTLP configuration and ready-made dashboards.

Agentic observability is evolving

Agentic AI systems are quickly moving from proof-of-concept to production, giving customers the ability to automate complex workflows, invoke a variety of different tools and APIs, and coordinate tasks across multiple services. However, traditional monitoring overlooks critical AI-specific signals, such as token consumption, model behavior, and guardrail outcomes. Teams struggle to trace non-linear agent flows, establish baselines for dynamic systems, and maintain predictable costs as usage scales. Without purpose-built observability, organizations risk degraded experiences, higher costs, and compliance gaps as agent complexity grows.

As agentic AI moves from pilot programs to production, organizations are automating complex, cross-system workflows with Amazon Bedrock AgentCore. However, most monitoring stacks weren’t designed for emergent, tool-driven behaviors and, therefore, leave blind spots around correctness, safety, and cost. Teams struggle to trace non-linear flows, establish baselines for dynamic systems, build agentic workflows, and keep token-driven spend under control as usage scales.

The observability gap in AI agent deployments

While AI agents offer significant benefits, including improved employee productivity, increased efficiency, and competitive advantage, among others, an observability gap remains, creating the following challenges:

• Complex multi-step workflows
  AI agents run non-linear, multi-system sequences with inter-agent dependencies, making data flow and responsibility hard to trace. This obscures where time is spent and who is responsible for failures in the chain.
• Limitations of traditional metrics
  Basic operational metrics often overlook AI reasoning errors and quality issues that don’t significantly affect CPU or p95 latency. Without AI-specific telemetry, subtle degradations often slip through.
• Continuous underlying agent and LLM model version changes
  Your system might be robust today, but upstream model and version updates can alter behavior, latency, and costs, forcing continuous adaptation to prevent regressions and incidents. Proactive detection of model-induced changes is crucial to maintaining stable quality and safety over time.
• Scalability and quality challenges
  As deployments grow, telemetry volume and coordination overhead surge while token usage and API calls remain untracked. This breaks cost predictability and quality control, leading to issues such as hallucinations and model drift. Multi-agent logic evolves constantly, so “normal” is a moving target. Baselines drift, complicating anomaly detection and root-cause analysis.

Without addressing these challenges, organizations face risks, from degraded user experiences and spiraling costs to compliance violations and reputational damage.

New enhancements for teams building with Amazon Bedrock AgentCore

The new Dynatrace AI Observability app embeds Amazon Bedrock AgentCore observability into a dedicated end-to-end experience, featuring out-of-the-box analytics, auto-instrumentation, targeted GenAI metrics, debugging flows, and ready-made dashboards to address all observability gaps in agent deployments. Support is available for over 20 technologies, including Amazon Bedrock, OpenAI, Gemini/Vertex, Anthropic, and LangChain.

These enhancements enable teams to take advantage of the following benefits:

• End-to-end distributed tracing
  Trace every interaction from user prompt to model reasoning to tool calls, so you can pinpoint bottlenecks, errors, or costly loops in seconds. Filter by model, provider, token usage, latency, and more to accelerate root-cause analysis.
• Enriched GenAI telemetry data, out of the box
  Each LLM and tool invocation emits spans with prompts, completions, token counts (for both prompts and completions), finish reasons, model IDs, latency, and errors, utilizing GenAI semantic attributes. Orchestration layers (for example, actions, HTTP durations, and step names) are captured for the complete workflow context.
• Cost, performance, and safety insights
  Use intelligent forecasting to detect cost and performance anomalies in token consumption and latency. Monitor guardrails for toxicity, PII, and denied topics to build trust and meet compliance requirements.
• Simple OTLP setup, fast time to value
  AgentCore already emits telemetry; simply register the OpenTelemetry export to Dynatrace once. Use your Dynatrace OTLP endpoint and token, and you’re streaming signals into the Dynatrace Grail® data lakehouse with no code rewrites. Ready-made dashboards for Amazon Bedrock let you verify ingestion and gain instant insights.

What’s next

We’re investing in a deeper Amazon Bedrock model and provider insights, expanded guardrail analytics, and additional automation so you can attach remediation playbooks to cost or safety anomalies.

Additionally, we’ll introduce a new agent visualization and topology experience that visualizes your AgentCore agents, LLM services, tool backends, and dependencies, allowing you to understand real-time topology and data flows across the entire stack.

Navigate from the topology map to traces to follow agent behavior step-by-step across services, protocols, and external calls, pinpointing hotspots, ownership, and blast radius more quickly.

Expect tighter integrations with popular orchestration frameworks and more dashboards for common agent patterns, such as retrieval, multi-agent collaboration, and tool-heavy workflows.

Get started with Dynatrace AI Observability for Amazon Bedrock AgentCore agents

Ready to learn more? Have a look at our GitHub repository.

Start instrumenting your agents today. Open the Amazon Bedrock AI Observability dashboard in Dynatrace to verify telemetry and begin your analysis.

The post Announcing Amazon Bedrock AgentCore Agent Observability appeared first on Dynatrace news.

• Your AI investments can’t reach their potential without effective AI governance.
• AI governance is a challenge that demands unprecedented agility, proactive measures, and comprehensive oversight to manage complexity.
• With Dynatrace, you’re prepared for whatever comes next.

Stay compliant and build trust in your AI systems

AI regulation is tightening, and non-compliance is becoming a huge risk to broader, production-scale AI adoption. Penalties are only part of the impact: reputational damage, customer mistrust, and stalled innovation can cripple even forward-looking organizations. That’s why we’re introducing data governance and audit trails for AI observability: a scalable way to manage, monitor, and secure the AI data lifecycle with end-to-end lineage, retention controls, and evidentiary records of model and user interactions.

Our platform helps you turn governance into a competitive advantage. Built-in audit support helps with emerging regulations like the EU AI Act, and alignment to industry standard frameworks such as NIST AI and ISO/IEC 42001:2023.

The hidden challenges of AI data governance

The complexity of compliance

AI regulations are becoming stricter, and new regulations are on the horizon. Organizations must maintain detailed records of AI activities for years, ensure transparency of data and processes, and align retention policies with legal requirements. These measures are imperative for trust and safety, but they introduce significant challenges. For instance, AI-related events are often scattered across multiple systems, applications, and teams, complicating efforts to create a unified audit trail. Default retention periods can fall short of regulatory needs, and manual governance processes are error-prone and infeasible at scale.

The risk of non-compliance

Failing to meet regulatory standards risks hefty fines and penalties, but the market consequences, reputational damage, and loss of customer trust are even worse. Without the right tools, organizations will struggle to manage the growing complexity of AI data governance and reap the full benefits of AI investments.

Introducing Dynatrace data governance and audit trails

Dynatrace has a long history of empowering organizations to tackle complex challenges with AI-driven solutions. Building on this expertise, we’re introducing a new set of capabilities designed to simplify compliance, enhance transparency, and streamline data management. With Dynatrace, you can:

• Automatically retain AI-related events for up to 10 years in Grail®, our secure data lakehouse.
• Monitor and capture events from platforms like Amazon Bedrock, tracking everything from model deployments to fine-tuning activities.
• Leverage OpenTelemetry to collect real-time traces and metrics of AI workloads, along with every AI user interaction, giving you a complete picture of your AI ecosystem.

Close the compliance gap with embedded oversight

What sets Dynatrace apart is seamless integration with your existing workflows. With OpenPipeline® on Grail, you can route AI-related events to custom storage buckets with extended retention, automatically, and without forcing teams to change tools or processes. This allows long-term auditability and helps meet sector-specific compliance requirements that might require special retention and auditability measures.
Once configured, Dynatrace can automatically route and store events, creating a reliable and transparent audit trail. This helps to reduce fragmentation, tool sprawl, and manual effort traditionally associated with data governance.

Imagine being able to trace every user interaction, model training session, or deployment event with just a few clicks. Dynatrace makes this possible by consolidating fragmented data into a single, coherent view. Whether you’re responding to a regulatory inquiry or optimizing your AI models, you’ll have the insights you need, when you need them.

Simplified and instant data filtering with Dynatrace segments

Not all audit data carries the same compliance weight. For global enterprises with complex IT environments, the ability to instantly filter data by precise criteria is essential for accelerating compliance across diverse regulations, from strict local regulatory transparency obligations to lighter regimes elsewhere.

Dynatrace segments make it simple to break down and filter data to match your analysis needs and regulatory requirements:

• Targeted compliance views: Instantly filter audit data by region, environment, platform, model, or custom criteria to align with diverse regulatory requirements.
• Dynamic adaptability: Segments automatically update, for example, when new LLM models or environments are introduced, minimizing manual maintenance and keeping governance current.
• Reusable assets: Leverage a single dashboard or notebook across multiple use cases by simply applying different segments, reducing duplication of effort.
• Noise reduction: Exclude irrelevant data such as development or test logs to keep compliance and observability focused on what truly matters.
• Custom team-context: Provide different teams (for example, compliance, data science, operations) with clear, filtered views of their audit data, ensuring ownership and audit-readiness across departments.

From observability to trusted automation

The future of AI governance lies in proactive, automated solutions that not only meet today’s regulations but also anticipate tomorrow’s challenges. With Dynatrace, you’re not just complying—you’re building a foundation of trust and reliability that scales with your business. By capturing and integrating AI events into a unified platform, Dynatrace transforms compliance from a burden into a strategic advantage.

Get started today

Ready to simplify your AI data governance?

Try it out yourself on the Dynatrace playground. Or, learn how to configure AI governance in our documentation.

Read more

• Part one of the Rise of Agentic AI blog series covers the fundamentals of AI agents, models, and emerging communication standards such as Agent2Agent (A2A) and MCP.
• Part two of the Rise of Agentic AI blog series explores AI agent observability and monitoring, A2A and MCP communications, and how to scale and monitor Amazon Bedrock Agents.
• Part three explains how to monitor Amazon Bedrock Agents and how observability optimizes AI agents at scale.
• Part four covers full-stack observability for AI with NVIDIA Blackwell and NVIDIA NIM.
• Part five demonstrates how to build a simple agentic application using the OpenAI Agents SDK and instrument the data with Dynatrace.
• Part six explores AI Model Versioning and A/B testing for smarter LLM services.

The post The rise of agentic AI part 7: introducing data governance and audit trails for AI services appeared first on Dynatrace news.

Debug, optimize, and secure your AI models with confidence

As agentic AI applications and systems gain traction, delivering reliable, high‑performing LLMs and agents becomes challenging due to heterogeneous stacks, non‑deterministic behavior, and cost sensitivity across multi‑cloud runtimes. Reliable delivery and deployment to production requires end-to-end telemetry across the full chain:
UI/services → orchestration/agents (LangChain, LlamaIndex, MCP/A2A) → RAG pipeline (embedding + vector DB) → model gateway (OpenAI, Azure/OpenAI, Bedrock, Gemini, Mistral, DeepSeek) → GPU/infra. To support deterministic rollouts and continuous model improvement, teams need standardized tracing/metrics, guardrail signal capture, and automated cost and performance governance.

The hidden challenges of AI model management

The invisible bottlenecks

AI models, especially LLMs, are prone to issues like hallucinations, degraded performance, and incorrect outputs. Debugging these problems is often like finding a needle in a haystack. Existing tools fall short in providing a unified view to compare prompts, datasets, or model versions, making it hard to identify regressions or improvements.

Navigating the risks of rapid model evolution

The impact of deprecation and automatic upgrades on cost, performance, and quality

The rapid pace of innovation in the AI space means that providers like OpenAI and Anthropic frequently release new versions of their models, such as ChatGPT 5 or Anthropic Opus 4.1.

While these updates often promise better performance and new capabilities, they can also introduce significant risks for your AI services:

• Deprecation of older versions: Providers may discontinue support for older models, forcing you to adopt newer versions without sufficient time to test their impact.
• Automatic upgrades: Many AI providers automatically update their underlying models, which can lead to unexpected changes in behavior, degraded performance, or even broken workflows.
• Compatibility issues: Changes in model behavior, such as output format or token usage, can disrupt your application’s functionality, requiring adjustments to prompts, configurations, or integrations.

Tracking token usage and managing costs is another uphill battle. Add to this the risk of prompt injection attacks and data leaks, and it’s clear that traditional methods are no longer sufficient

The new AI Model Versioning and A/B testing

Ship better models with confidence. In a single view, compare models and versions to validate improvements and spot bottlenecks across latency, reliability, token usage, cost, and output quality, then drill into prompt-level differences to confirm why a variant wins. When something breaks, follow the request end to end with distributed tracing: from input through orchestration steps and model calls to completion, so you can pinpoint exactly where an error or slowdown originated.

Compare models and versions: Detect bottlenecks and validate improvements in a single view.

Trace prompt failures: Debug errors from input to output with our Distributed Tracing solution.

Monitor costs and token usage: Gain real-time insights into token consumption and cost implications.

Detect security and guardrail risks: Identify and alert on vulnerabilities like prompt injection attacks, toxic responses, or captured PII.

Attach your own attributes like user session, feedback, or dataset ID for additional debugging information.

How it works

With AI Model Versioning, you can track metadata such as model version, dataset ID, and hyperparameters.

A/B testing lets you expose different user segments to model variations, providing data-driven insights into performance metrics like accuracy and cost.

Instrument in minutes: Use the supported OpenTelemetry-based SDK to instrument your service to capture prompts, completions, token usage, errors, and guardrail signals.
You can also enrich spans with attributes like model.version, dataset.id, user/session, and feedback for deeper analysis. (You can read more about this here.)

Start analyzing out of the box: Once data is flowing, the AI Observability app provides ready-made dashboards and distributed tracing so you can compare models/versions, monitor costs and tokens, and debug prompt failures end to end. No extra setup is required; you can try it out on the Dynatrace Playground right now.

By combining observability, AI-driven insights, and organizational knowledge, we’re enabling systems that don’t just react but learn and adapt. Each critical issue or incident you resolve fuels a living knowledge base, paving the way for proactive incident prevention through alerting.

What’s next?

We’re committed to enhancing these capabilities further. Upcoming updates will include a dedicated app experience for multi-model and multi-cloud setups, advanced visualization tools, enhanced security features, intelligent forecasting, and alerting for cost/performance and guardrail optimization.

Get started today

Ready to revolutionize your AI services? Here’s how:

1. Sign up for a free trial.
2. Install the AI Observability app.
3. Explore the AI Model Versioning ready-made dashboard, or check it out on our playground

Together, let’s build smarter, more reliable AI systems.

Read more

• Part one of the Rise of Agentic AI blog series covers the fundamentals of AI agents, models, and emerging communication standards such as Agent2Agent (A2A) and MCP.
• Part two of the Rise of Agentic AI blog series explores AI agent observability and monitoring, A2A and MCP communications, and how to scale and monitor Amazon Bedrock Agents.
• Part three explains how to monitor Amazon Bedrock Agents and how observability optimizes AI agents at scale.
• Part four covers full-stack observability for AI with NVIDIA Blackwell and NVIDIA NIM.
• Part five demonstrates how to build a simple agentic application using the OpenAI Agents SDK and instrument the data with Dynatrace.
• Part seven introduces data governance and audit trails for AI services.

The post The rise of agentic AI part 6: Introducing AI Model Versioning and A/B testing for smarter LLM services appeared first on Dynatrace news.

Recently, OpenAI released a customer service agents demo built using the OpenAI Agents Python SDK that showcases an example multi-agent system at work. With the OpenAI Agents SDK, you can build agentic AI applications with the help of agents, handoffs, guardrails, tools (built-in and custom), and built-in tracing. These capabilities support the core pattern of knowledge, reasoning, and actioning as the foundation for scalable and trustworthy automation, first introduced by Dynatrace CTO Bernd Greifeneder.

In this blog post, we explain how to build an OpenAI agents SDK-based agentic application and instrument the agents and app with AI-powered observability from Dynatrace. Dynatrace can help you see agent executions, tool usages, and prompt flows from initial request to final response for quick root cause analysis and troubleshooting.

To illustrate the capabilities of the OpenAI Agents SDK and agent framework with Azure OpenAI on Azure AI Foundry, we have built our multi-agent solution using the OpenAI customer service agents demo mentioned above as a reference and modified it for our use cases.

About our sample agentic AI application

Our multi-agent system enables users to research, summarize and translate across a range of topics and content. The system consists of four agents:

• Welcome Agent: Engages the user, reasons with Azure OpenAI to analyze the prompt, identifies the intent, and passes it to the right agent to start processing.
• Researcher Agent: Searches the web and analyzes the results using OpenAI.
• Summarizer Agent: Summarizes content, including search results, text, PDF, CSV, and more, using Anthropic Claude.
• Translator agent: Translates queries and inputs into any user-requested language using OpenAI.

Next, we want the multi-agent system to perform two distinct scenarios:

1. Context history: In a specified chat session, the entire chat history and context is available for the duration of the session, while the individual prompts might be handed off to different agents for processing.
2. Composite queries: The app orchestrates multiple different agents for different purposes, such as Research, Translate, Summary, and Welcome, so users can engage to process a composite prompt with multiple sub-queries.

Understanding multi-agent frameworks and handoff workflows

There are some key differences between the agent frameworks. Unlike the A2A protocol, the OpenAI framework does not explicitly have a central registry for agents. Instead, OpenAI agents use the concept of “handoffs” orchestrated by the OpenAI Agent Framework.

OpenAI framework agent handoffs

While orchestrator-led coordination offers a more deterministic and structured workflow, agent-to-agent handoffs provide significant advantages in adaptability and modularity. These handoffs enable agents to collaborate dynamically, making it possible to handle complex, multi-step queries with greater flexibility. This approach focuses on a more decentralized and scalable system, allowing agents to specialize and respond to changing requirements in real-time.

Here are two example scenarios to illustrate the agent-to-agent collaboration in chat sessions, with context, as well as delivering multi-agent query processing.

Show the user prompts for a composite query and multi-agent workflow

For example: “Research Michael Jordan, then summarize in 40 words or less, and then translate to French.”

Multi-agent processing for CSV files uploaded

This example includes sample customer data to showcase multi-agent workflow processes with context and history in the chat session.

Overall, the agent-to-agent handoffs worked well (and with context) during all the session runs. Tracing and debugging can be achieved by instrumenting the SDK with OpenTelemetry and sending the data to Dynatrace’s built-in AI Observability solution for Azure OpenAI. You can easily capture the multi-agent workflow for a given prompt on the Azure AI Foundry platform dashboard. Find the code examples in our GitHub repository.

Set up tracing using Python

Using Python, you can set up the tracing by changing a few simple lines of code in your agent framework and core component:

from traceloop.sdk import Traceloop Traceloop.init( app_name="openai-cs-agents", api_endpoint="https://wkf10640.live.dynatrace.com/api/v2/otlp", disable_batch=True, headers=headers, should_enrich_metrics=True, ) 
with tracer.start_as_current_span(name="update_seat", kind=trace.SpanKind.INTERNAL) as span: 
    context.context.confirmation_number = confirmation_number 
    context.context.seat_number = new_seat 
    assert context.context.flight_number is not None, "Flight number is required" 
    return f"Updated seat to {new_seat} for confirmation number {confirmation_number}"

You can see the results right away in distributed tracing:

OpenAI orchestration

Within the OpenAI framework, there are two approaches to orchestrating agents:

1. Allow the LLM to make decisions: Use the intelligence of an LLM to plan, reason, and decide what steps to take.
2. Orchestrate with code: Use code to determine the flow of agents.

Overall, the OpenAI Agents SDK is comprehensive and easy to get running with some minor code changes, this time with OpenAI’s Codex assistant.

Multiple frameworks and toolkits are quickly ramping up to make multi-agent systems a reality. We foresee this space evolving and innovating rapidly.

The evolution of multi-agent systems

As agentic AI continues to advance, multi-agent applications are poised to play a transformative role in reshaping how applications operate. These systems enable dynamic, context-aware collaboration between specialized agents, empowering businesses to tackle increasingly complex workflows. From helping with automation, orchestrating large-scale data analysis, multi-agent systems will unlock new levels of efficiency, scalability, and innovation.

Tools like the OpenAI Agents SDK on Azure AI Foundry and Azure AI Studio are at the forefront of this evolution. By providing built-in capabilities such as agent handoffs, guardrails, and tracing, the SDK simplifies the development and monitoring of multi-agent workflows. These features make it easier for organizations to deploy responsible, secure, and robust AI systems and also ensure transparency and trustworthiness in their operations. These are key factors for widespread adoption.

Looking ahead, we can expect rapid innovation in this space. Emerging standards like MCP, A2A protocols, and frameworks such as OpenAI Agents are creating a vibrant ecosystem for multi-agent interoperability. The focus will likely shift toward even more intelligent and reliable orchestration, where agents autonomously plan, reason, and adapt to dynamic environments.

AI Observability for agentic AI applications

To keep pace with these advancements, we believe that observability must evolve in lockstep to ensure transparency across heterogeneous agent ecosystems. Advancements in observability tools, such as the Dynatrace AI Observability solution, are essential to help create more reliable and scalable AI frameworks at the enterprise level.

The future of multi-agent systems holds immense potential, with the OpenAI SDK marking the starting point. We’re just at the beginning of what’s possible. As this technology evolves, it will gradually become more stable and reliable, ultimately transforming the way we approach automation, collaboration, and AI-powered problem-solving across industries.

Check out our GitHub repo for detailed code examples for OpenAI Agents, AWS Strands, Google ADK, and start building your own AI Observability solutions today.

Read more

• Part one of the Rise of Agentic AI blog series covers the fundamentals of AI agents, models, and emerging communication standards such as Agent2Agent (A2A) and MCP.
• Part two explores how monitoring A2A and MCP communications results in better, more effective agentic AI. This blog post covers AI agent observability and monitoring, and how to scale and monitor Amazon Bedrock Agents.
• Part three explains how to monitor Amazon Bedrock Agents and how observability optimizes AI agents at scale.
• Part four covers full-stack observability for AI with NVIDIA Blackwell and NVIDIA NIM.
• Part six explores AI Model Versioning and A/B testing for smarter LLM services.
• Part seven introduces data governance and audit trails for AI services.

Together, these capabilities make it possible to achieve robust, scalable observability in agentic AI environments so teams can build reliable and trustworthy applications and services.

The post The rise of agentic AI part 5: Developing and monitoring multi-agent applications with OpenAI Agents SDK on Azure AI Foundry appeared first on Dynatrace news.

The next big wave in artificial intelligence is agentic AI, which harnesses autonomous agents to perform tasks by reasoning, learning, and adapting to changing circumstances. The success and efficiency of agentic AI systems depend on how well these AI agents communicate. Facilitating this communication requires monitoring AI agents and their underlying communication protocols, such as Model Context Protocol (MCP).

In this blog post, we explain how to monitor Amazon Bedrock Agents and how observability optimizes AI agents at scale.

Key takeaways

• Effective cross-agent communication requires standardized telemetry. For foundational model-building platforms like Amazon Bedrock, OpenTelemetry-based solutions provide standardization and instrumentation for tracing and logging to debug at scale.
• End-to-end observability is a key best practice for monitoring agentic AI. AI agent observability best practices include using GenAI semantic conventions with traditional logs, traces, and instrumentation.
• Observability helps deliver effective agentic AI results in the context of the whole stack. AI agent observability and Amazon Bedrock Agents monitoring help deliver better performance, ensure compliance, and provide detailed debugging tools.

Cross-agent communication requires standardized telemetry

Given the non-deterministic nature of large language models (LLMs) and dynamic cross-agent communication, organizations need standardized telemetry. OpenTelemetry-based GenAI semantic convention libraries are emerging to unify logging, metrics, and tracing in multi-agent ecosystems. Likewise, these standardized instrumentation libraries let you collect and analyze data from each step in an agent’s decision or communication chain on Dynatrace. Observability of each step lets you monitor the communications among your agents and evaluate their health and performance, regulatory compliance, and debugging.

Scale and monitor Amazon Bedrock Agents with Dynatrace

Amazon Bedrock Agents provide an easy way to build and scale generative AI applications with foundation models.

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models from leading AI companies, such as AI21 Labs, Anthropic, Cohere, Luma, Meta, Mistral AI, poolside, and Stability AI—or from Amazon’s own model, Amazon Nova—all through a single API. In addition, Amazon Bedrock Agents also provide the broad set of capabilities teams need to build generative AI applications with security, privacy, and responsible AI best practices.

Dynatrace provides an AI-powered, unified observability and security solution for tracking and revealing the full context of used technologies and service interaction topology. Using Dynatrace for AI agent monitoring and MCP monitoring, teams can analyze security vulnerabilities and observe metrics, traces, logs, and business events in real time—automatically and securely.

“With the rise of agents, the need for deep visibility and real-time insights is more essential than ever. Through this partnership, AWS and Dynatrace are uniquely positioned to deliver performance, cost, and quality insights alongside robust compliance monitoring—empowering customers to innovate with confidence.”
– Atul Deo, Director of Amazon Bedrock

Best practices for Agent-to-Agent (A2A) and MCP monitoring

As with hybrid and cloud-based environments, context-based observability of AI agents and models is essential for efficient and healthy outcomes. Here are some best practices:

• Adopt common semantic conventions. Standardize metrics and trace attributes—for example, gen_ai.agent.operation.name and gen_ai.agent.name—across different frameworks.
• Use logs and traces for Amazon Bedrock Agents. Log critical task lifecycle events—capability discovery, artifact creation, agent collaboration steps, API calls—so teams can replay and debug complex interactions and detect hallucinations.
• Instrument thoroughly. Bake observability into agent frameworks using external OpenTelemetry libraries or by manually instrumenting calls. Ensure each agent’s start, stop, and reasoning steps, like tools, knowledge base, and guardrails, are captured consistently.
• Secure communication. Enforce enterprise-grade authentication and authorization within agent-to-agent traffic. Use well-defined protocols like A2A to avoid unauthorized data exposure.
• Continuous feedback. Feed observability insights into iterative retraining or fine-tuning for improved agent reliability.

With Amazon Bedrock and the Dynatrace AI Observability solution, you can cover the following use cases for agent observability:

Monitor AI agent service health and performance

• Detect bottlenecks by tracking real-time metrics, including request counts, durations, and error rates.
• Manage service costs with automated cost calculations for each request.
• Stay on track with service-level objectives (SLOs).

Monitor guardrails to ensure compliance

• Monitor your safeguards customized to application requirements and responsible AI policies.
• Validate toxicity, filtered content, and denied topics to ensure compliance.
• Prevent leaks of personally identifiable information (PII).
• Prevent quality degradation by validating models and usage patterns in real time.

End-to-end tracing and debugging

• Achieve complete visibility of prompt flows, from initial request to final response, for faster root cause analysis.
• Capture detailed debug data to troubleshoot issues in complex pipelines.
• Streamline workflows with granular tracing of LLM prompts, including response latency and model-level metrics.
• Resolve issues more quickly by pinpointing exact problem areas in prompts, tokens, or system integrations.

Future of AI agent observability and Amazon Bedrock Agents monitoring

We expect to see deeper integrations between agent orchestration protocols (A2A, MCP) and open observability frameworks, delivering end-to-end visibility from data ingestion to cross-agent collaboration. As standards converge, organizations will rapidly compose advanced AI solutions while retaining full transparency and control, paving the way for even greater scalability, resilience, and confidence in autonomous agents.

Read more

• Part one of the Rise of Agentic AI blog series covers the fundamentals of AI agents, models, and emerging communication standards such as Agent2Agent (A2A) and MCP.
• Part two explores AI agent observability and monitoring, A2A and MCP communications, and how to scale and monitor Amazon Bedrock Agents.
• Part four covers full-stack observability for AI with NVIDIA Blackwell and NVIDIA NIM.
• Part five demonstrates how to build a simple agentic application using the OpenAI Agents SDK and instrument the data with Dynatrace.
• Part six explores AI Model Versioning and A/B testing for smarter LLM services.
• Part seven introduces data governance and audit trails for AI services.

The post The rise of agentic AI part 3: Amazon Bedrock Agents monitoring and how observability optimizes AI agents at scale appeared first on Dynatrace news.

Agentic AI is revolutionizing—and dramatically accelerating—how engineering teams respond to incidents and debug use cases. By monitoring how AI agents communicate using standards such as Model Context Protocol (MCP), performance engineers can gain deep insights into how systems and AI models themselves are performing. And crucially, automate incident responses using coding assistants.

The next step in this scenario is to employ some MCP best practices, such as helping full-stack engineers to debug code at runtime without disrupting operations. Fortunately, Dynatrace has the answer with Live Debugger. First debuted earlier in 2025, Live Debugger gives developers instant access to code-level troubleshooting data in any environment, including production, and is now generally available.

With all these capabilities coalesced in an integrated development environment (IDE), developers can interact with AI using natural language to massively accelerate issue resolution.

Key takeaways:

• Unify AI debugging resources in one place. Debugging agentic AI workflows starts with combining Dynatrace features with custom MCPs in an IDE.
• Leverage an AI coding assistant to add context. An AI coding assistant, such as Cline, helps add vital context and improve prompt engineering using natural language queries.
• Use best practices to configure MCPs and Cline. Combining Cline with MCPs and on-the-fly debugging reduces guesswork and manual overhead.

MCP best practices TELUS uses to accelerate incident response

As an early adopter of Dynatrace Live Debugger, TELUS harnessed the power of Davis® AI, MCP, and real-time debug data to gain a deep understanding of their code, enabling every engineer to ask questions in natural language about their code behavior at runtime.

At a recent Dynatrace Guild meeting, TELUS SREs Dana Harrison and Cheng Li demonstrated how they’re combining Dynatrace features like Live Debugger with custom MCPs for the ultimate purpose-built troubleshooting setup in their Visual Studio Code IDE. By integrating multiple resources in one place, the TELUS team can blend next-generation debugging with agentic AI workflows. This integration reduces developer context-switching and enables developers to use straightforward, natural-language prompts.

With a fully instrumented environment that monitors every service end-to-end, the TELUS team leverages Dynatrace Live Debugger for Node.js and Java microservices to monitor running code in real time to reduce mean time to resolution. They initially tested this approach in non-production but are expanding into production with strict auditing and data masking.

The team also uses MCP in tandem with Cline AI, a coding assistant capable of gathering logs, metrics, and trace data through natural language commands. By unifying multiple data sources under MCP, TELUS avoids manually juggling different dashboards and tools. Developers can simply interact with the AI tools, which fetch relevant context on demand, even correlating Live Debugger snapshots with real-time logs. As a result, the entire debugging and incident investigation process remains inside the IDE, which significantly streamlines performance engineering and accelerates issue resolution.

How MCPs empower AI agents for debugging use cases

As an open standard, MCP connects AI agents to relevant data sources, such as repositories, tools, or external APIs. Instead of bespoke integrations for each data silo, MCP provides a universal interface to connect multiple relevant sources to feed the right context to the models and agents. This interface simplifies how agents access relevant context, leading to better task outcomes, execution, and more consistent performance across complex environments. For reference, you can try out the Dynatrace MCP server on GitHub.

MCP best practices: How Cline AI helps to bring in context

Cline AI is a coding assistant in Visual Studio Code that leverages multiple MCP servers to streamline data retrieval from various back-end systems. With plain-language queries (“Search for a specific service,” “Fetch logs from GKE,” or “Investigate errors”), developers can prompt Cline AI to automatically contact the relevant MCP server.

By becoming better at prompt engineering and handling the underlying API calls and correlating responses from Dynatrace MCP, native Kubernetes logs, or even third-party services, Cline AI provides an all-in-one investigation workflow right in the editor. This means developers don’t have to switch among multiple tools or memorize specialized APIs; they simply ask Cline AI what they need in natural language, and the MCP layer manages the rest.

Cline supports MCP-client features, such as dynamic tool discovery, prompt reusability, and adaptive resource access. It also enables powerful features like custom instruction, cline-rule, and memory banks.

TELUS’ best practices for configuring MCPs and Cline

The TELUS team followed these MCP best practices while configuring Cline.

Install Cline AI as VS Code extension

First, the TELUS team installed Cline AI as a VS Code extension, pointing it to their AI proxy, Fuel iX. Fuel iX lets them choose among various large language models based on the user’s requirements and desires.

Use careful prompt engineering and Cline rules to direct specific MCP tools

Cline AI can then interpret a developer’s natural-language prompt—such as “Investigate errors in our Node.js service”, “What is this service about”—and select the appropriate MCP endpoint(s) to gather data. Directing to specific MCP tools for more accurate results can be fine-tuned through careful prompt engineering, creating detailed .clinerules file definitions, and directives built into the MCPs themselves – including predicted requests and response formatting.

Configure each MCP server for a specific data domain

Each MCP server at TELUS is responsible for interacting with a specific data domain: for instance, they have a Dynatrace MCP server to fetch metrics and trace data, a GCP Logs MCP server to pull container and other forwarded logs, and additional servers for services like JIRA issues or PagerDuty.

Rely on centralized agentic AI commands, not vendor-specific APIs

By standardizing how these data sources are exposed, TELUS ensures that Cline AI never has to manage direct integrations or vendor-specific APIs. Instead, the assistant simply issues commands to the MCP servers, which internally handle authentication, query templates, and data normalization. This architecture not only allows TELUS to maintain clear boundaries between data retrieval logic and AI-driven workflows, but also accelerates developer onboarding, making it simple for any team member to debug or investigate issues from within VS Code by asking Cline AI, rather than switching between specialized tools or dashboards.

​​Use MCPs as a standardized endpoint

​In this context, MCP servers act as standardized endpoints, each responsible for a particular data source or application. By relying on Cline for coding assistance, the TELUS team can type a natural-language command (“Investigate issues in our Node.js service”), and Cline will automatically query Dynatrace Grail or GKE logs using the relevant MCP server.

This approach centralizes the complexities of data retrieval in one place and lets the AI assistant produce a consolidated summary or recommended fix, all within the IDE.

Step-by-step guide to set up Cline with Dynatrace

1. Install Cline through the VS Code Marketplace.
   
2. Configure Cline with the required credentials, usually an API endpoint and key.
   
3. Install your MCP. This example uses an internal package, but there are currently many MCPs published for installation.
   
4. Validate that the Dynatrace MCP server is up and running.
   
5. Ask Cline your questions. The agent formats them according to the needs of the MCP(s) you installed.
   
6. Example response of a production authorization service problem pulled and summarized from Dynatrace.
   

The objective of this combined setup (Live Debugger + MCP + Cline) is fast and intelligent troubleshooting. Instead of juggling multiple dashboards and CLI tools, an engineer can see code snapshots, error traces, logs, and even recently created JIRA tickets within one session. As mentioned, MCP standardizes queries across diverse systems, so the AI agents can correlate all relevant information, such as referencing a single trace ID to pull logs from GKE, configuration details from the Kubernetes cluster, or known issues in PagerDuty. This AI agent-powered workflow saves time and reduces context switching.

The TELUS team found that combining these insights with on-the-fly debugging significantly reduces the guesswork and manual overhead that typically slows resolution times.

The road ahead: Developer-first observability and IDE-centric debugging at TELUS with Dynatrace

Although TELUS currently uses Live Debugger in mostly non-production environments, they plan to enable it in production using strict governance. Their upcoming strategy involves role-based permissions for debugging sessions, mandatory security training on data privacy, and well-defined data retention policies to keep snapshots transient.

When combined with MCP’s universal integration points, like Dynatrace MCP, developers can quickly drill down into real production issues, see the exact variables at fault, and either push a fix or revert a misconfiguration with minimal downtime.

Looking ahead, the TELUS team plans to integrate Davis Copilot APIs to further simplify natural language interactions with Dynatrace.

In practice, this would allow an AI assistant to automatically translate human-readable queries into Dynatrace Query Language (DQL), making it even easier to retrieve the right metrics, traces, or logs. By layering Davis CoPilot™ on top of their existing MCP approach, TELUS aims to reduce manual DQL writing while offering developers a powerful yet streamlined way to perform more advanced data analysis through simple, intuitive prompts.

Read more

• Part one of the Rise of Agentic AI blog series covers the fundamentals of AI agents, models, and emerging communication standards such as Agent2Agent (A2A) and MCP.
• Part three explains how to monitor Amazon Bedrock Agents and how observability optimizes AI agents at scale.
• Part four covers full-stack observability for AI with NVIDIA Blackwell and NVIDIA NIM.
• Part five demonstrates how to build a simple agentic application using the OpenAI Agents SDK and instrument the data with Dynatrace.
• Part six explores AI Model Versioning and A/B testing for smarter LLM services.
• Part seven introduces data governance and audit trails for AI services.

More MCP best practices and resources for Dynatrace Live Debugger

• Dynatrace MCP on GitHub
• Dynatrace Live Debugger on Dynatrace Hub
• Dynatrace for Developers Visual Studio plugin on Microsoft Marketplace
• Dynatrace for Developers product page
• Dynatrace Automation Guild on the Dynatrace Community
• 5 powerful use cases beyond debugging for Dynatrace Live Debugger
• Debug complex performance issues in production with ease with Live Debugger, Open Pipeline, and OpenTelemetry

The post The rise of agentic AI part 2: Scaling MCP best practices for seamless developers’ experience in the IDE with Cline appeared first on Dynatrace news.

By now, everyone is aware of generative AI fueled by large language models (LLMs) and generative pre-trained transformers (GPTs). The next level of innovation is agentic AI and the autonomous AI agents that drive it. Using Model Context Protocol (MCP) to facilitate agent-to-agent communication, these systems are revolutionizing how enterprises automate tasks and orchestrate complex workflows.

Powered by LLMs, vector databases, retrieval augmented generation (RAG) pipelines and additional tools, these AI agents are expanding extensively, giving rise to multi-agent systems, cross-agent protocols, and context-sharing standards. But these autonomous agents also introduce new challenges in monitoring, debugging, and security.

We’ll examine in detail the fundamentals of AI agents, models, and the emerging standards that help them communicate, like Agent2Agent (A2A) and Model Context Protocol (MCP).

Key takeaways:

• Autonomous AI agents are the backbone of agentic AI. These services combine to deliver adaptable automated tasks.
• AI agents depend on LLMs and orchestration logic. These technologies maintain the agent’s state, session memory, context, and reasoning strategies.
• Agents depend on protocols, such as A2A and MCP, to effectively communicate. Models and agents need these protocols to manage multi-agent communication.

What is agentic AI?

Agentic AI is an artificial intelligence system made up of independent agents that can take initiative and perform sequences of actions to complete tasks by reasoning, learning, and adapting to changing circumstances.

Dynatrace Chief Technologist Alois Reitbauer described agentic AI this way:

“It’s really delegating a task to software the way you would delegate it to a human. Say if you wanted to do travel booking, give it some complexity and freedom and some decision points it can make. Like, I have to go to Vegas, I need a hotel, I need a couple of good restaurants to go to, we’re going to be 50 people, fix it with my schedule.” – Alois Reitbauer in The New Stack

Agentic AI systems rely on AI agents to perform the tasks that lead to the desired outcome.

What are AI agents?

An AI agent is a self-directed autonomous application that harnesses large language model (LLM) reasoning, tool usage, and context-awareness from numerous data sources to carry out tasks.

Agents can think and act independently without outside intervention. Agents can think through chain-of-thought, plan, execute (Reason+Act=ReAct), and refine their actions as needed. Businesses are looking into adopting these autonomous agents for applications such as customer service automation, supply-chain optimization, and content generation.

How do AI agents operate?

AI agents operate similarly to a Michelin-starred chef in a busy kitchen: They continuously gather information, plan, execute, and adjust to reach their desired end goal.

In the chef analogy, the cook surveys orders and available ingredients, decides on a suitable recipe, and then refines the approach based on feedback or resource constraints.

Agents do the same thing in a computational context. Specifically, they observe the world (for example, a user request or a set of data), perform internal reasoning about the best course of action, then carry out the steps needed to fulfill the request. This cycle allows them to respond adaptively to changing conditions, much as a chef would substitute ingredients or modify a dish mid-preparation.

Underpinning this iterative loop is the orchestration layer, which maintains the agent’s state, session memory, and reasoning strategies (such as ReAct, Chain-of-Thought, or Tree-of-Thoughts). Large language models (such as OpenAI’s GPT, Anthropic Claude, Google Gemini, Amazon Nova) provide the core reasoning capability for the agent. The model “thinks” about the user’s query. But the agent gains its power by incorporating additional frameworks or tools that can fetch external information or execute actions in the real world. One way to fetch and provide tools and information is through a unified protocol called Model Context Protocol (MCP).

Additionally, the orchestration layer ensures that multiple rounds of reasoning, tool usage, and tool outputs are all tracked and synthesized before the agent returns a final response to the user. Agents follow these steps in a structured way, so they can produce more accurate, context-rich answers and easily manage complex tasks.

What is the difference between models and agents?

A model (like a large language model) simply generates outputs based on its training data and the given prompt, typically without any built-in mechanism for session memory, external actions, or complex decision loops and validations.

An agent, on the other hand, includes the model but goes further. It maintains a stateful process (managing multi-turn conversations and thought processes), uses external tools to gather fresh data or perform actions, and follows a defined orchestration logic (such as ReAct and chain-of-thought). Thus, while a model is a core reasoning component, an agent adds the surrounding structure and capabilities needed for autonomous, goal-directed behavior.

What is Agent2Agent (A2A)? How multiple agents communicate with each other

As enterprises slowly adopt multiple specialized agents, interoperability of these services becomes crucial to create reliable experiences. To achieve this, A2A from Google helps to create an open protocol that enables agents—regardless of vendor or framework—to securely exchange information, coordinate actions, and integrate capabilities. By specifying tasks, capabilities, and artifacts in a standardized JSON-based lifecycle model, A2A fosters multi-agent collaboration across otherwise siloed systems.

A2A protocol enables agents to share updates and delegate tasks without overhead. However, direct communication between agents only solves half the problem: These agents also need relevant, up-to-date data and context to drive decisions and be equipped with the right toolset to execute actions.

Without a unified method for accessing diverse data sources, even the most capable multi-agent ecosystem remains limited in scope. The open-source project Model Context Protocol (MCP) fills this gap.

What is Model Context Protocol? How MCPs empower agents

As an open standard, the Model Context Protocol (MCP) connects AI agents to relevant data sources, such as repositories, tools, or external APIs. Instead of the above mentioned integrations for each data silo, MCP provides a universal interface like USB-C to connect multiple relevant sources to feed the right context to the models and agents. This universality simplifies how agents access relevant context, leading to better task outcomes, execution and more consistent performance across complex environments. For managing complex tasks like the ones highlighted above, the Dynatrace MCP server on GitHub helps to get real-time end-to-end observability and MCP data into your daily workflow.

What’s next: Monitoring A2A and MCP for better agentic AI

As these technologies evolve, we can expect deeper integrations between agent orchestration protocols (A2A and MCP) and open observability frameworks, delivering end-to-end visibility from data ingestion to cross-agent collaboration. Likewise, as standards converge, organizations will rapidly compose advanced AI solutions while retaining full transparency and control, paving the way for even greater scalability, resilience, and confidence in autonomous agents.

Read more

• Part two of the Rise of Agentic AI blog series explores AI agent observability and monitoring, A2A and MCP communications, and how to scale and monitor Amazon Bedrock Agents.
• Part three explains how to monitor Amazon Bedrock Agents and how observability optimizes AI agents at scale.
• Part four covers full-stack observability for AI with NVIDIA Blackwell and NVIDIA NIM.
• Part five demonstrates how to build a simple agentic application using the OpenAI Agents SDK and instrument the data with Dynatrace.
• Part six explores AI Model Versioning and A/B testing for smarter LLM services.
• Part seven introduces data governance and audit trails for AI services.

The post The rise of agentic AI part 1: Understanding MCP, A2A, and the future of automation appeared first on Dynatrace news.

Every software development team grappling with Generative AI (GenAI) and LLM-based applications knows the challenge: how to observe, monitor, and secure production-level workloads at scale. Traditional debugging approaches, logs, and occasional remote breakpoint instrumentation can’t easily keep pace with cloud-native AI deployments, where performance, compliance, and costs are all on the line. How can you gain insights that drive innovation and reliability in AI initiatives without breaking the bank?
Dynatrace helps enhance your AI strategy with practical, actionable knowledge to maximize benefits while managing costs effectively.

Amazon Bedrock, equipped with Dynatrace Davis® AI and LLM observability, gives you end-to-end insight into the Generative AI stack, from code-level visibility and performance metrics to GenAI-specific guardrails.

Developers deserve a frictionless troubleshooting experience and fast access to real-time data—no more guesswork or costly redeployments. Here’s how Dynatrace, combined with Amazon Bedrock, arms teams with instant intelligence from dev to production, helping to accelerate innovation while keeping performance, costs, and compliance in check.

Introducing Amazon Bedrock and Dynatrace Observability

Amazon Bedrock is a serverless service for building and scaling Generative AI applications easily with foundation models (FM). It provides an easy way to select, integrate, and customize foundation models with enterprise data using techniques like retrieval-augmented generation (RAG), fine-tuning, or continued pre-training.

Dynatrace is an all-in-one observability platform that automatically collects production insights, traces, logs, metrics, and real-time application data at scale.  With powerful Davis AI engine Dynatrace notifies teams about production-level issues before they disrupt users, helps predict resource usage,costs, and performance issues, and delivers guardrails that protect data and maintain compliance.

Together, Amazon Bedrock and Dynatrace provide an end-to-end observability solution for AI applications:

• Predictive operations: Proactive usage and cost forecasting to reduce unexpected operational expenses and token usage.
• Production performance monitoring: Service uptime, service health, CPU, GPU, memory, token usage, and real-time cost and performance metrics.
• Guardrail analysis: Detect hallucinations, track prompt injections, mitigate PII leakage, and ensure brand-safe outputs.
• Full-stack tracing: Track each user request across multiple FMs, vector databases, orchestrators (LangChain), and custom business logic.
• Compliance: Document all inputs and outputs, maintaining full data lineage from prompt to response to build a clear audit trail and ensure compliance with regulatory standards.

^{Figure 1. Video overview of Amazon Bedrock dashboard with Dynatrace AI and LLM Observability solution.}

How it works

Dynatrace seamlessly instruments your LLM-based workloads using Traceloop OpenLLMetry, which augments standard OpenTelemetry data with AI-specific KPIs (for example, token usage, prompt length, and model version).

Combined with Amazon Bedrock, you can:

• Spin up your AI model on Amazon Bedrock—choose from providers like AI21, Anthropic, Cohere, Stability AI, Mistral AI, Meta, or Amazon’s own Nova/Titan foundation models.
• Automatically instrument your application with OpenTelemetry.
• Configure OpenLLMetry to capture specialized LLM details as spans and metrics, like model name, completion time, token count, token cost, and prompt text.
• Send unified data to Dynatrace for analysis alongside your logs, metrics, and traces.

Behind the scenes, Dynatrace merges the standard telemetry with these advanced AI attributes, surfaces them in real-time dashboards, and applies AI-driven analytics to discover anomalies, forecast usage costs, and diagnose root causes.

How to set up and instrument your data with OpenLLMetry

Traceloop OpenLLMetry is an open source extension that standardizes LLM and Generative AI data collection. By layering on top of OpenTelemetry standards, OpenLLMetry captures the critical metrics you can’t get by default—like the number of tokens, model temperature, or guardrail triggers.

Here’s how to set it up for Amazon Bedrock:

1. Install OpenLLMetry in your Python or Node.js environment:

 pip install traceloop-sdk

from traceloop.sdk import Traceloop

headers = {

'Authorization': f"Api-Token {environ.get('DYNATRACE_TEAM_KEY')}"

}

Traceloop.init(

app_name=environ.get('DYNATRACE_APP_NAME'),

api_endpoint=environ.get('DYNATRACE_URL'),

headers=headers

)

2. Configure environment variables to send data to Dynatrace via your ingest token:

 DYNATRACE_URL =https://123abcde.live.dynatrace.com/api/v2/otlp 

DYNATRACE_TEAM_KEY=dt0.....

3. You can optionally add OpenLLMetry decorators or instrumentation to your LLM calls (for example, with LangChain or direct Bedrock SDK calls).

When your application queries Amazon Bedrock, OpenLLMetry automatically captures:

• Prompt tokens vs. completion tokens
• Finish reason (did the LLM stop due to a user request, or was the max token limit reached?)
• Model type (which Amazon foundation model or third-party model is used?)
• Performance: Response time, throughput, and error rate
  • Guardrail activations: Toxicity, PII, denied topics, and hallucinations
  • System, prompt, and completion messages and roles

This data is instantly correlated in Dynatrace so you can visualize or alert on critical thresholds (for example, if your average token usage spikes or your overall cost forecast grows beyond budget).

How to debug incorrect responses in production

Let’s walk through a real-world scenario:

Your production travel agent application—powered by Amazon Bedrock and Dynatrace—gives users incorrect travel recommendations. Perhaps it suggests flights or hotels that don’t exist or mixes up time zones. This isn’t just a minor inconvenience; it jeopardizes user experience and can directly impact revenue and trust.

Here’s how Dynatrace helps you trace and resolve the issue quickly:

Proactive alerting with Davis AI

You receive an alert from Dynatrace Davis AI anomaly detection indicating incorrect system behavior. There might be a spike in “incorrect itinerary” complaints or conversation outcomes flagged as “nonsensical.” Davis AI correlates the unusual LLM responses with application telemetry and usage patterns, so you immediately know something is off in the recommendation flow.

Full-stack end-to-end tracing

In Dynatrace Distributed Tracing, you see the entire transaction trace for the affected user session. This includes front-end requests, back-end aggregator logic, calls to Amazon Bedrock, and any vector database lookups performed for retrieval-augmented generation (RAG). Rather than sifting through multiple logs, you have a single timeline that reveals exactly where the LLM call returned unexpected data.

Inspecting the GenAI model details

By drilling down into the span data enriched by OpenLLMetry, you can see:

• Prompt and completion text and tokens used.
• The specific foundation model version (for example, anthropic.claude-v1 or amazon.nova).
• Temperature setting and max token limits.
• Any error codes or guardrail triggers.

This clarity helps you pinpoint if the model produces off-base recommendations because of a misaligned temperature, an out-of-date context, or a mismatch in user inputs.

Root cause analysis

With Dynatrace, you quickly correlate the LLM anomaly to a specific function in your microservice code. You discover that an external data source used for itinerary validation had missing or stale updates, causing the LLM prompt to reference invalid flights. You’ve found the “why” without manually spelunking logs in disparate systems.

Resolving and validating

A fix might involve updating your data pipeline or refining the prompt logic. You can deploy the change and watch in near real-time as Dynatrace collects new traces and logs. Davis AI recognizes that the anomaly is cleared, confirming that your fix resolved the incorrect responses—no guesswork required.

You can find the code example for our travel agent application here for review, and the dashboard on our Dynatrace Playground instance.

Summary

By integrating Amazon Bedrock with Dynatrace end-to-end observability, you not only catch issues early but also trace them across your entire AI stack to the root cause. Building or scaling Generative AI applications with Amazon Bedrock requires robust insights into your environment—from model usage and performance metrics to cost forecasts and guardrail efficacy.

Dynatrace helps you scale with:

• Complete end-to-end tracing across your services, external data pipelines, and LLM calls.
• Predictive analytics that forecast AI resource usage and cost trends, letting you proactively manage budgets.
• Unified dashboards that bring performance, cost, code-level data, logs, metrics, and audit events together.
• Compliance and governance that integrate security checks, data masking, and guardrail analysis.

Whether you’re a developer racing to put your latest AI-powered application or a new feature into production or an SRE ensuring your system meets enterprise-grade SLAs, Dynatrace and Amazon Bedrock help you to create frictionless AI applications, focusing on performance and observability—at any scale, in production, with no downtime.

Useful resources

• Dynatrace Amazon Bedrock instrumentation documentation
• Supported GenAI technologies
• Dynatrace Playground demo
• Video webinar: AI and LLM Observability with Dynatrace

The post Deliver secure, safe, and trustworthy GenAI applications with Amazon Bedrock and Dynatrace appeared first on Dynatrace news.

Traditional monitoring approaches often require manual scripting and integration to get alerted about production-threatening issues in pre-production environments.
Dynatrace Simple Workflows make this process automatic and frictionless—there is no additional cost for workflows. By offering a single trigger and a single task, you can rapidly set up an action (like Slack or email notifications) to detect specific exceptions in your services. This means fewer surprises when deploying to production and more time spent delivering valuable features.

Why manual alerting falls short

As your product and deployments scale horizontally and vertically, the sheer volume of data makes it impossible for teams to catch every error quickly using manual processes. Your teams want to iterate rapidly but face multiple hurdles:

• Increased complexity: Microservices and container-based apps generate massive logs and metrics.
• Unstructured overview: Manually scanning logs or waiting for someone to notice an error in staging is time-consuming.
• Siloed tooling: Stitching multiple tools and scripts together can create friction and decrease your deployment velocity.

As a result, minor issues in staging can slip into production, leading to potential downtime and a poor user experience. Developers need a way to quickly set up alerts for targeted pre-production exceptions without incurring steep costs or heavy overhead.

Introducing Dynatrace Simple Workflows for early alerting

Dynatrace Simple Workflows helps your team overcome the challenges of manual processes by offering easy, no-cost automation for single-step tasks.

Go to Workflows and start creating a new workflow. By default, the Simple Workflow will be selected to give you the most cost-effective experience.

You can select any trigger that’s available for standard workflows, including schedules, problem triggers, customer event triggers, or on-demand triggers.

After you’ve selected + Workflow, you can name your workflow and select a specific trigger.

Here, you can select a specific event or a timed trigger like a cronjob. Additionally, with an on-demand trigger, you can trigger the workflow through REST API from your CI/CD, scripts, or any other automation flows.

After specifying your trigger, you can select a specific action. This action will be executed each time an event occurs, a timer kicks in, or an on-demand request triggers the action. You can easily send Slack/email messages to your teams, create JIRA issues, or trigger PagerDuty alerts. 

Safeguard your pre-production environments.

Imagine you’re running a busy e-commerce platform composed of multiple containerized microservices running on Kubernetes. Each microservice is responsible for a specific and critical business function—managing user sessions, processing payment and authentication, and tracking inventory.

To orchestrate the different logging services, you use Fluent Bit to forward these logs to your centralized logging system, like Dynatrace.

Recently, in production, your team encountered an out-of-memory error exception that killed all the processes for Fluent Bit. This caused you to lose complete visibility of your containers’ logs, performance, and error data, and you could not tell if the system was down or not.

kubernetes.event.message": "Memory cgroup out of memory: Killed process 1146778 (fluent-bit) total-vm:952432kB, anon-rss:303872kB, file-rss:1108kB, shmem-rss:0kB, UID:1000 pgtables:1424kB oom_score_adj:-997",
kubernetes.event.reason": "OOMKilling",

The error message indicates that the Linux cgroup assigned to your Fluent Bit container has reached its memory limit. Because Kubernetes enforces memory constraints for each pod, the kernel OOM (OutOfMemory) killer forcibly terminates the Fluent Bit process once memory usage exceeds the threshold set in the container/pod specification.

You have fixed the error, and as a follow-up to your RCA documentation, you want to ensure you’re alerted of these errors early on staging, so you want to set up a quick alert on staging for your feature branch.

For this example, we go to Simple Workflows and select Trigger > Davis® event trigger to find these out-of-memory errors. You can learn more about event triggers in Dynatrace Documentation.

When Davis AI event triggers find these, we will set a simple Slack or Email notification for our team so the team can ensure the fix is working and other future changes are caught early.
If you ever need more advanced capabilities—like custom code, multi-step logic, or conditional tasks—you can seamlessly upgrade to standard workflows with just a few clicks.

This ensures you’re never locked into an overly complex or expensive solution before you’re ready. It’s as simple as that!

Deep dive into Simple Workflows

How Simple Workflows work:

• Simple Workflows provide core Dynatrace automation capabilities
• You can use any trigger type, problem-event triggers, schedule-based triggers, or external webhooks.
• They trigger a single, out-of-the-box action: for example, Sending a Slack Notification, Creating a Jira Issue, Sending an Email, or Executing an HTTP Request.
• Unlike multi-step workflows, these don’t consume workflow hours.

Limitations:

• You can’t add more than one task.
• JavaScript actions, task conditions, and options are not allowed.

What’s next for Workflows?

Looking ahead, Dynatrace aims to expand the usability of Simple Workflows with more out-of-the-box actions and deeper integrations.

Both simple and standard workflows will benefit from future improvements like:

• A growing catalog of out-of-the-box actions
• Introduction of draft workflows, enabling you to work on your workflows without impact until you’re satisfied with your changes
• Watch Workflows: Get notifications whenever a workflow fails, is updated, or throttled.

For now, Simple Workflows delivers a fast way for developers and SREs to receive automated alerts—exactly when needed—at no extra cost.

Ready to try Simple Workflows?

Check out the Simple Workflows documentation to explore more use cases. For advanced functionality—like multi-step actions or running JavaScript—it’s free to upgrade to a standard workflow and take advantage of the full Dynatrace AutomationEngine capabilities.

With Simple Workflows, you can quickly eliminate manual monitoring steps, reduce time-to-detection, and keep staging errors where they belong—out of production.

The post Create simple workflows to automate alerts during development appeared first on Dynatrace news.