If you’ve already wired an AI coding assistant up to Dynatrace, through the MCP server, the Dynatrace CLI (dtctl), or a custom agent you built yourself, you’ve seen your agent have difficulty interpreting data or calling for fields that don’t exist. This makes sense, your agent may be making assumptions based upon training that isn’t relevant. It lacks the skills to understand how to get the best value from Dynatrace. That is where Dynatrace for AI fills the gap.

What are agent skills?

Agent skills are an open format for packaging domain knowledge that AI agents can load on demand. A skill is a folder containing a SKILL.md file with focused instructions, examples, and optional reference material. Compatible agents, such as Claude Code, GitHub Copilot, Cursor, Cline, or others, discover installed skills and load the full content only when it’s relevant to the task at hand.

The net effect: you can install dozens of skills without bloating an agent’s context window. Agents pull in exactly what’s relevant when it’s relevant, and ignore the rest.

Built for agents working with Dynatrace

Dynatrace for AI is a curated set of skills that give an agent the three things it needs to efficiently do real work on Dynatrace:

• Access to Dynatrace data and insights: through DQL queries against Grail®, Smartscape® dependency graph, or problem records.
• Dynatrace expertise: the syntax rules, entity-model distinctions, and query patterns that separate a working query from one that looks correct but returns nothing.
• Task-level starting points: ready-made prompt templates for common engineering workflows, so teams don’t have to invent the approach from scratch.

Skills don’t connect to Dynatrace directly. You have to pair them with the MCP server or dtctl to perform live queries and initiate actions. Together, they turn an agent with generic observability intuition into one that easily extracts value from Dynatrace.

Complement your agent with domain expertise

The first release of Dynatrace for AI agent skills is focused on the workflows that engineering teams run every day:

• DQL fundamentals: covering the pipeline model, core data objects, and when to use fetch, timeseries, or smartscapeNodes to prevent failures that typically come from models trained on generic query-language data.
• Observability across the stack: services, traces, logs, frontends, and problems, each covering the entity model, key fields, and query patterns that make answers correct rather than merely plausible.
• Infrastructure and cloud: covering Kubernetes, AWS, and hosts.
• Platform tasks worth delegating: providing programmatic creation of dashboards and notebooks

Prompt templates for common workflows

Alongside the skills, the repo hosts a small set of prompt templates you can use as structured starting points to invoke the right skills for specific tasks. These save teams from having to design their approach from scratch and make outcomes more consistent across agents and users.

Current templates include:

• Performance regression: walks the agent through comparing RED metrics before and after a deployment, correlating any regression with distributed traces, and summarizing the root cause.
• Daily standup: pulls the last 24 hours of problems, deployment activity, and notable anomalies for a team’s services, so anyone can walk into a standup with the relevant production context already framed.
• Troubleshoot a problem: takes a problem ID and guides the agent through root-cause analysis, including affected entities, correlated events, relevant logs and traces, and creates a structured summary for the incident channel.

These are a starting point, not a ceiling, designed to be forked and shaped to your team’s on-call runbooks.

What Dynatrace for AI is and what it isn’t

Skills and prompts are a knowledge and workflow layer. They don’t connect to your Dynatrace environment, define what actions your agent can take, or set guardrails. That’s the job of the tool you pair them with and your Dynatrace permission model.

The quality of what your agent can produce also depends on the entities your environment is instrumented to capture. Skills help agents ask better questions of data, but they don’t control what data is collected.

Think of this skill as onboarding a smart new hire who already knows software, but needs to learn your platform. The skills are the platform user guide; your observability data is the work itself.

Get started

It’s super simple to install the skills and prompts in one go. Just run:

npx skills add dynatrace/dynatrace-for-ai

…or activate the skills as a Claude Code plugin:

claude plugin marketplace add dynatrace/dynatrace-for-ai
claude plugin install dynatrace@dynatrace-for-ai

Make sure your agent can reach Dynatrace, then try a real agent-skill task. A few good example starting prompts:

• “Compare the error rate of the checkout service over the last hour vs the same hour yesterday.”
• “Are any pods in the production namespace restarting or getting OOM-killed right now?”
• “Use the performance-regression prompt to check the deployment I just shipped.”

The difference in output quality is immediate: fewer corrections, cleaner queries, and answers that accurately reflect how Dynatrace continuously models your environment in real-time.

The Dynatrace for AI project is open source and actively developed. Issues, discussions, and pull requests are all welcome, especially from teams running agent skills against real workloads. We’d love to hear from you.

The post Dynatrace for AI: Teach your AI coding agent how to use Dynatrace appeared first on Dynatrace news.

From generic AI to task‑focused operational agents

Dynatrace ready‑made agents are purpose‑built capabilities that apply Dynatrace intelligence to specific, recurring operational tasks. Each agent focuses on a clearly defined problem, such as explaining why a service is slow, summarizing unusual behavior in an environment, or helping you understand what changed and why it matters. These agents are designed to take a question or a signal based on the exact data that is in your environment and organization and turn it into a useful answer you can act on.

Because Dynatrace agents are ready‑made, there is no need to define prompts, train models, or design behavior from scratch. Each agent already knows:

• What type of input to expect,
• Which Dynatrace signals and context it should use,
• And what output types are most useful for each addressed problem type.

All available ready-made Dynatrace agents can be found in Dynatrace Hub.

Trigger agent actions with Dynatrace Workflows and the Dynatrace MCP Server

Ready‑made agents can be triggered automatically as part of Dynatrace Workflows or available wherever you already work via the Dynatrace MCP Server.

Using agents in Dynatrace Workflows

Dynatrace Workflows lets you run agents in response to events or on a schedule. Instead of manually asking questions about potential problems and remediation steps, the workflow autonomously responds to changes in your environment.

For example, the Kubernetes Troubleshooting Agent runs nine parallel queries for data enrichment, and Dynatrace Intelligence turns all the information into a structured diagnosis. Customize the agents to your needs, including instructions for human approval steps and automated remediation.

The fastest way to get started is with Dynatrace ready-made agentic workflow templates, currently available in a preview release. Instead of building from scratch, you get proven automations that summarize issues, suggest remediation, and deliver insights directly to the tools your teams already use.

Power users can go further by building their own agentic workflows that combine Dynatrace Intelligence actions with any trigger, data source, or integration in Workflows. Use cases range from auto-scaling Kubernetes clusters based on Dynatrace Intelligence forecasts to generating query-cost-optimization recommendations for stakeholders, to virtually any other automation your environment requires.

Using agents through the Dynatrace MCP Server

The Dynatrace MCP Server makes the agents available outside the Dynatrace web UI, without requiring you to deploy or operate any additional infrastructure. You can connect Dynatrace to any MCP‑compatible client in minutes, with no server to install, host, or maintain.

Through the tools exposed by the MCP Server, you can use natural language to query data in Grail®, check system health, and get problem analyses and remediation recommendations. This brings Dynatrace directly into the tools you already use, such as your IDE, Claude Code and Cowork, Microsoft Copilot, Slack, or automation platforms like n8n. The Dynatrace MCP Server also powers integrations with systems like Azure SRE, AWS DevOps, GitHub Copilot, Atlassian Rovo Ops, Amazon Q, and others.

This means agents are no longer tied to a single interface. You can ask Dynatrace questions and get grounded, production‑ready answers wherever you work, using the same agents and intelligence that power Assist and workflows.

Dynatrace Assist: a simple way to test ready-made agents

The quickest way to use a ready‑made agent and see how it works before you start creating a workflow is with . Dynatrace Assist lets you ask questions about your environment using natural language, without switching tools or setting anything up.

A simple way to start is with a real problem you already have. For example, when a service becomes slow, open Assist and ask a question such as “Summarize the open problems and highlight those that need immediate attention.” Assist interprets the question, evaluates the environment you’re working in, and pulls together relevant data and context using Dynatrace Intelligence. Instead of manually navigating metrics, traces, logs, and dependencies, you get an explanation grounded in what is actually happening in your system.

Continuing your conversation with Assist, you can refine the question or follow suggested drill‑downs. Assist supports this as a single flow, helping you move from an initial question to deeper analysis and, where applicable, to next steps. You’re not configuring an agent or defining behavior. You’re simply asking a question and letting Dynatrace coordinate the right intelligence and ready‑made agents behind the scenes.

This makes Assist your lowest‑friction entry point for using Dynatrace agents. You get a concrete result quickly, using the same data and context you already rely on in your daily work.

What’s next?

If you haven’t already, open Dynatrace Playground, or your Dynatrace tenant, and ask Dynatrace Assist a question to see the ready-made agents in action.

The post Dynatrace AI agents begin working for you on day one, and are built to grow with you appeared first on Dynatrace news.

What is dtctl?

dtctl (short for “Dynatrace control”) is the open-source CLI for the Dynatrace platform; it’s kubectl-inspired, terminal-native, and designed for both AI agents and humans. Platform engineers, SREs, and developers use it to manage workflows, dashboards, queries, and settings from the command line.

With dtctl, AI agents use the same command line interface and commands that your platform engineers, SREs, and developers use to autonomously manage workflows, dashboards, queries, and settings.

Three ways to connect agents to Dynatrace

Dynatrace offers three access patterns for AI agents and automation, all governed by the same IAM permission system. Regardless of which path you choose, an agent can access only the data and operations permitted by its authentication scopes. The right choice depends on your use case; in practice, most teams use more than one approach.

• MCP (Model Context Protocol): A standardized, schema-driven interface in which every tool call is declared upfront and validated automatically. Agents get structured, predictable interactions while platform teams get strict control over which tools are exposed and a full audit trail of every call.
• API: Access to the full platform surface with maximum flexibility. The API is ideal when you need endpoints that higher-level tools can’t cover. The tradeoff: you build and maintain everything yourself, from endpoint selection to token lifecycle, pagination, and rate-limit retries.
• dtctl (CLI): A terminal-native and composable command line interface that’s built for execution speed, with minimal setup: Just run a command, inspect the result, adjust, and repeat. This offers a natural fit for tight iteration loops, scripting, and agents that need to get things done without overhead.

Why is CLI gaining traction for agent workflows?

CLIs have been the primary interface for automation since the early days of Unix, and for good reason. Output is structured, syntax is predictable, and there’s no protocol overhead. MCP mirrors how humans interact with tools and includes discovery, negotiation, and structured handshakes. This is valuable when you need it; however, every schema negotiation round-trip adds tokens and latency to the agent’s context window. CLI skips that entirely and cuts straight to execution.

Use the same CLI tool and commands for human engineers, scripts, and AI agents

Whether you’re a platform engineer or an AI agent, dtctl gives you the same powerful Dynatrace interface. Like kubectl or git, dtctl follows a simple verb-noun syntax: Just state what it is you want to do, then state what you want to act on. What makes dtctl stand out is that it’s designed to support human engineers and AI agents equally.

• A single interface for everything. Workflows, dashboards, notebooks, queries, SLOs, Dynatrace Intelligence, and more, all accessible through the same consistent set of commands. There’s no need to stitch together multiple API endpoints or learn different tools for different resources.
• Built for AI agents. dtctl lets agents discover all available commands at runtime, no documentation needed, no upfront configuration. When running inside an AI agent, dtctl automatically switches to structured output that agents can parse and act on, including follow-up suggestions and error context.
• Built for humans, too. Use tab-autocomplete resource shortcuts like db for dashboards and wf for workflows, –mine to filter your own resources, and an edit command that opens YAML in your $EDITOR and uploads on save. Because dtctl follows familiar command-line patterns, experienced users move fast from day one.
• Managing multiple environments is simple. Switch contexts between dev, staging, and production with a single command. Authenticate via SSO or API token, run dtctl doctor to verify the setup, and you’re ready to go.

For all technical details and the full command reference, visit the dtctl repository on GitHub.

An AI agent modifies a Dynatrace workflow end-to-end

What makes AI agents truly useful is their ability to close the loop: They can discover data, make changes, verify results, and fix what’s broken all without handing control back to a human operator.

Here’s what such a scenario looks like using dtctl. In this example, a Dynatrace workflow queries the number of Kubernetes pods and sends an email report. The goal is to enhance the query so that it lists every pod with its respective node tolerations, making it a cross-entity query that explores the data model, identifies the correct relationships, and iterates until the output is correct.

Using GitHub Copilot in VS Code, the agent works through the full cycle autonomously:

• Discover: Explores the Dynatrace data model to find the right entities and relationships: how pods connect to nodes and where tolerations are stored.
• Iterate: Refines the query step by step until the output matches the goal, then updates the workflow and rebuilds the email report.
• Apply and run: Pushes the updated workflow to Dynatrace and executes it.
• Verify: Checks whether the workflow ran successfully and produced the expected results.
• Fix: If something fails, it reads the error, adjusts the query, and tries again.

The human operator defines only the intent, and the agent handles the rest. Watch this full video walkthrough to see it in action.

Create and modify dashboards without leaving your code editor

One concrete example of what dtctl enables is dashboard creation and management directly from the terminal. Because Dynatrace dashboards are structured data, they can be version-controlled, templated, and automated just like any other code artifact.

To illustrate this, the OpenClaw Gateway Monitoring dashboard shown below was created end‑to‑end in just a few minutes. A developer used GitHub Copilot in VS Code to create an AI observability dashboard similar to others, tailored specifically for monitoring OpenClaw. The agent then pulled existing Dynatrace AI observability dashboards as templates, adapted the layouts and queries to OpenClaw’s monitoring needs, and deployed the results using dtctl; all this was managed by the developer without leaving their IDE.

For day-to-day management, the workflow remains the same, whether executed by an agent or a human: just pull a dashboard, adjust queries or filters, preview the changes, and save the dashbaord. Coding agents can be instructed to update queries across multiple dashboards with a single prompt. And it takes just a single command to promote dashboards from dev to production, or to roll them back instantly if something breaks.

You can take dtctl even further: wire dashboard definitions into your CI/CD pipeline so that, as a service evolves, its dashboards evolve automatically.

Try out dtctl today

dtctl is fully open source and available at dynatrace-oss/dtctl on GitHub, with documentation, skills, and examples to get you started.

Our dtctl Quick Start Guide walks you through the complete setup in under five minutes:

1. Connect dtctl to your environment.
2. Run your first query.
3. Pull a dashboard.
4. Modify the dashboard queries.
5. Save your dashboard.

From there, install the agent skill to teach GitHub Copilot, Claude Code, or Cursor how to operate your Dynatrace environment.

The project is in active development. If you hit a bug or have a use case to share, open a GitHub issue or start a discussion, and help us shape the roadmap.

The post dtctl: The Dynatrace observability CLI that’s built for AI agents and humans appeared first on Dynatrace news.

Imagine commissioning le Carré and Fleming to build your perfect undercover agent: quietly embedded in the system you’re watching. You hand in your mission brief, which includes the target, objective, and behaviors to track. Your agent observes without drawing attention, reporting insights back to you. On cue, the information flow you’ve carefully orchestrated turns signals into actionable intelligence that helps pre-empt risk.

Dynatrace doesn’t write spy fiction. However, even better, Dynatrace now lets you write your own smart agentic workflows that deliver intelligent reports and react to changes in your environment based on your objectives.

Adding generative AI to your workflow

Using the power of gen AI, Davis CoPilot® transforms your workflows into agentic instructions. Davis CoPilot lets you explore data using conversational language, translating complex data and queries into summaries, and provides intelligent recommendations across Dynatrace.

Integrated into Dynatrace Workflows, Davis CoPilot is your toolkit for building smart automations, bringing the power of generative AI into your mission-critical workflows.

By embedding Davis CoPilot in your workflows, you can associate any automation with any number of conversational automations. Your workflows can even perform actions autonomously when combined with precise Davis® AI forecasting, for instance, scaling resources based on forecast demands.

In real time, these workflows monitor live data, summarize critical issues, and identify remediation paths or emerging threats. When scheduled, these smart workflows help you outsource routine tasks, such as alerting stakeholders of costly queries.

Let’s look at some examples of how these smart workflows can help you in your daily work.

Build agentic workflows that respond to critical events

Proactive guiding through complex problem remediation

Let’s assume you want to build an automation that cuts through alert noise and analyzes a problem as it occurs, guiding you through the remediation. When a new problem is detected, Davis CoPilot extracts the problem details, summarizes the situation, and provides tailored remediation guidance. By embedding it into a smart workflow, you can select your preferred automation to automatically syndicate this information, populate a ticket in ServiceNow or Jira, or post it to a dedicated Slack channel.

See how you can set up a workflow automation that automatically sends summaries and remediation guidance when a new problem is detected.

Monitor emerging threats to help you orchestrate a response

Next, you can build an agentic workflow that helps you monitor emerging threats and assess their risk to your environment as vulnerabilities are detected in your tenant. In plain language, you instruct your agent to extract IOCs, query security events in your environment, and correlate them with observability data in your environment. Information provided by the external threat feed is automatically matched against the live context in your tenant. The agent has now collected all the necessary information and provides a reliable risk assessment, along with a plan to orchestrate a response, directly in your Slack channel, ensuring around-the-clock visibility and a rapid response.

Write the future: Build an agentic workflow that autonomously auto-scales your resources

Dynatrace helps you build agents that reason autonomously. The key is to deliver data as precise as Dynatrace forecast capabilities. In this example, we linked the power of Davis AI to forecast demand, with generative AI and GitHub automations. Davis AI predicts the number of resources the hyperscaler infrastructure will need based on forecasted demand. When Davis AI notices a scaling need, Davis CoPilot interprets the data and autonomously edits the manifest using the GitHub automation. Giving you one end-to-end workflow that automatically scales resources up or down based on forecasted needs. To see this in action, watch how this workflow autonomously edits a manifest based on Davis AI suggestions to auto-scale a Kubernetes cluster.

Schedule agentic workflows to optimize routine tasks

Do you feel like sleeping in a little later? Maybe stretching your lunch break a little longer? Running that extra hill without sacrificing your productivity? Scheduling Davis CoPilot into your smart workflow is a great way to automate recurring tasks and save time.

Build an automation that predicts resource consumption

A recurring challenge for SREs is analyzing the full environment to predict future bottlenecks or over-resourcing and continuously translating the data to update stakeholders. Even with great observability in place, you need to ensure that you interpret the data and make timely decisions to inform future provisioning.

By combining Davis AI forecasting automation with Davis CoPilot, you can build an agent that answers key questions, such as which workloads are most resource-intensive, which resources show the most variance, and which require frequent scaling. This automation is capable of highly reliable forecasts, even when data points are limited. The automation interprets the data and emails actionable recommendations directly to you and anyone else who needs to stay informed.

To see this in action, watch the section of this video that explores predicting resource consumption.

Smart workflows that optimize query costs

Scheduling tasks can even help you keep costs lean and efficient. For admins or budget owners, staying within financial limits while maintaining performance is a constant challenge. In this example, we built a smart workflow that identifies the top 20 most expensive queries from the last 24 hours. Davis CoPilot analyzes each query and sends optimization recommendations directly to the query authors via email.

To implement this yourself, tailored to the most expensive queries executed on your tenant, go to our documentation

Conclusion: Adapt your workflows to any stage of your automation journey

These are just a few examples; the applications for it are endless. We’ve designed this workflow action to cater to your organization’s automation appetite. You may want to transform how you keep business stakeholders informed about what’s happening in your environment, leveraging Dynatrace’s highly accurate insights, which are translated into plain language and actionable next steps.

Alternatively, you may be ready to transition towards autonomous operations, where automation not only supports but also acts in a controlled and reliable manner. Davis CoPilot embedded into your workflows opens the door to your agentic journey.

Start your agentic journey and join the Davis CoPilot for Workflows Preview

Davis CoPilot for Workflows is available as a Preview. Sign up now and see how generative intelligence embedded into your workflows transforms your automation. Today, it helps you react faster, optimize more effectively, and collaborate seamlessly. Tomorrow, it will go even further: anticipating needs, orchestrating actions, and enabling truly autonomous reasoning.

The post Write the future: Create your own agentic workflows appeared first on Dynatrace news.

Enterprise automation rarely fails because teams lack ideas. It fails because scaling automation safely is challenging, and conventional workflows often remain static after they ship, while systems and conditions continue to evolve.

The new update to Dynatrace Workflows helps teams accelerate their innovation loop and run automation like production software. Five improvements make it easier to iterate and operate at scale: workflow drafts, sub-workflows, approval requests, persistent execution data, and real-time notifications. Together, they support safer change, reusable standards, built-in governance, and data-driven refinement.

The innovation loop in enterprise automation

The innovation loop is a repeatable cycle for quickly and safely improving enterprise automation that treats workflows as living systems that evolve with your environment, self-improving as systems, teams, and requirements change, rather than remaining static:

• Teams start by designing or updating a workflow and validating it safely before it goes live.
• Next, they standardize the parts that work into reusable building blocks so future workflows are faster to create and more consistent.
• Then they add governance where it matters, such as approval checkpoints for high-impact actions, so speed does not bypass policy.
• After execution, they observe what actually happened using execution records and signals, including errors, latency, and outcomes, to understand reliability and impact.
• Finally, they refine the workflow based on those findings by tuning logic, tightening controls, improving reuse, and adjusting notifications.

The result is a compounding impact: faster delivery over time, fewer incidents at scale, automation that integrates changing circumstances, and greater organizational trust, as transparency, auditability, and tight feedback loops back improvements.

How Dynatrace Workflows accelerates the innovation loop

Dynatrace Workflows helps you accelerate your innovation loop with powerful new features designed for adaptability, collaboration, transparency, and control. Each feature is designed to empower teams to safely iterate on automation in dynamic enterprise environments.

Faster iteration with safer testing and experimentation with workflow drafts

The new workflow drafts feature allows teams to create, edit, and refine workflows in a draft state, allowing them to validate logic and configuration without triggering live automation. Workflows that remain in draft mode don’t consume workflow hours.

Workflow drafts support the first step of the innovation loop: iterate safely. Teams can make changes, review them with peers, and confirm the workflow is ready before anything runs in production.

Key benefits

• Reduce risk by testing configuration changes without production impact.
• Allow faster iteration and cleaner change management.
• Improve collaboration during the design and review stages.
• Control costs with included workflow-hour consumption for draft-only workflows.

Standardize reusable blocks for reliability and speed with sub-workflows

Sub-workflows allow a workflow to be executed as a task inside another workflow, so teams can package repeated logic into reusable sub-workflows and compose larger automations from consistent building blocks.

This supports the “standardize” step of the innovation loop. When teams standardize and reuse proven components, automation becomes more consistent, easier to maintain, and quicker to expand across teams and environments.

Key benefits

• Enhance reliability and consistency by reusing proven workflow components.
• Speed up delivery by composing workflows from standard blocks.
• Simplify maintenance by updating shared logic in a single location.
• Keep complex automations understandable by breaking them down into smaller, manageable units.

Example use cases

• Notification workflows: Automatically send notifications based on incident severity or timing.
• Data processing workflows: Run filter/transform/store stages as distinct sub-workflows, processing large datasets in stages.
• Incident response workflows: Automate repetitive tasks as reusable steps, such as creating tickets, logging incidents, notifying stakeholders, and triggering remediation steps.

Supervised human-in-the-loop control where it matters with approval requests

With Dynatrace Workflows, you can now insert an approval request into a workflow and pause execution until a human reviews and approves the next step.

This feature supports the “governance” step in the innovation loop. Approvals turn your workflows into supervised autonomous operations. This allows the automation of more processes while ensuring strict compliance with enterprise requirements for governance, risk management, and accountability, particularly for high-impact actions.

Key benefits

• Add explicit human control at high-risk decision points.
• Reduce errors in sensitive operations (changes, remediations, escalations).
• Ensure compliance with organizational standards.
• Increase trust in automation across security, ops, and platform teams.
• Minimize operational risks.

End-to-end visibility with an audit trail and persistent execution data

Dynatrace Workflows now persists workflow execution data in DQL. All workflow activities are recorded as system events in the dt.system.events table, creating a queryable execution record for tasks, states, and outcomes as a comprehensive audit trail.

Persisting execution data supports the “observe” step of the innovation loop. When workflow execution is captured as data, teams can troubleshoot failures, provide transparency, prove what happened, and improve automation based on evidence rather than anecdotes.

Key benefits

• Provide an audit trail for enterprise traceability and review.
• Monitor workflow health and reliability through execution outcomes.
• Speed up troubleshooting by pinpointing where failures occur.
• Optimize automation using DQL-driven actionable insights (error rates, trends, hotspots, and more)

Example: Workflow health overview

This example demonstrates how to use execution data to support users in identifying those workflows that need improvement.

The following DQL query…

fetch dt.system.events
| filter event.kind == "WORKFLOW_EVENT" and event.provider == "AUTOMATION_ENGINE" and event.type =="TASK_EXECUTION"
| filter dt.automation_engine.state == "ERROR"
| summarize count = count() , by:{dt.automation_engine.workflow.title}

…counts task execution errors per workflow. Using a honeycomb visualization, it displays error distribution by workflow and provides visibility into those workflows that fail too often.

Instant alerts for failures and workflow changes with real-time notifications

Easily set up and customize real-time notifications by email for all workflow errors or changes made by others, ensuring you’re always informed.

These real-time notifications support the “respond and refine” step of the innovation loop. Proactive alerting allows you to manage workflows effectively, minimize disruptions, and maintain seamless operations.

Key benefits

• Get immediate visibility into failures to address potential issues before they escalate.
• Detect workflow changes early to reduce drift and surprises.
• Improve operational coordination across teams that share workflows
• Ensure uninterrupted operations with timely, actionable alerts.

Empower your team: Build smarter, safer, and scalable workflows today

Scaling automation today is hard, changes are risky to test, automation becomes outdated and buggy as soon as one part of the organization makes changes, and when something inevitably breaks, you’re left hunting for answers across scattered logs and screenshots. But it doesn’t need to be like this.

Are you ready to make things easier, run your workflows like production software, and accelerate your innovation loop? Here’s how you get started:

• Explore our documentation to learn more about these features.
• Join the Dynatrace Community to connect with other users and share your insights in the Workflow and AutomationEngine feedback channel.

The post Reliable enterprise automation at scale: Accelerate the innovation loop with Dynatrace Workflows appeared first on Dynatrace news.

Streamlining site reliability at scale can be daunting, particularly with large-scale AWS environments and architecture that rely on hundreds—or even thousands—of Amazon EC2 instances. However, you can simplify the process by automating guardians in the Site Reliability Guardian (SRG) to trigger whenever there are AWS tag changes, helping teams improve compliance and effectively manage system performance.

This step-by-step guide will show you how to configure your architecture to trigger guardians whenever EC2 tags are updated. Note that EC2 is an example; this guide can be made to work generically for tag changes on any AWS resource. By the end of this guide, you’ll be ready to automate guardians at scale and optimize Amazon EC2 management with ease.

Why automate guardians for AWS tag changes?

Before diving into the technical setup, here’s why automating guardians whenever EC2 tags change is beneficial for your organization:

• Greater efficiency: Automatically triggering guardians removes the need for manual intervention, saving time for DevOps or site reliability engineering (SRE) teams and allowing for more efficient resource management at scale.
• Better compliance: Automating guardians ensures critical policies and checks are consistently applied after changes across your architecture, improving security and compliance efforts.
• Cost optimization: Immediate responses to tag changes lead to informed decisions about scaling, shutting down unused instances, or fine-tuning resource efficiency.
• Proactive site reliability: Automated guardians can monitor the four golden signals, enabling proactive reliability measures.

Now, let’s get started with the setup!

Step 1: Create an API token

First, create an API token to integrate AWS services with Dynatrace for guardian automation.

1. Log into your Dynatrace tenant

Log in to your Dynatrace tenant and note the first part of the URL (for instance, “abc12345”), which is your tenant ID.

2. Access token settings

Press Ctrl + K or CMD + K and search for “Access Tokens” within Dynatrace.

3. Generate a new token

Create a new access token and assign it “bizevents.ingest” permissions.

4. Save the token

Copy and securely store the token, which looks like “dt0c01.*****.*****”. You’ll use this later during configuration.

Step 2: Create the EventBridge connection

Amazon EventBridge acts as the bridge between AWS and Dynatrace. Here’s how to set it up:

1. Navigate to Amazon EventBridge

Log in to your AWS Management Console and go to EventBridge > Connections.

2. Recreate the cURL command

You can use this cURL command as a reference to establish your connection:

curl -X POST \
'https://abc12345.live.dynatrace.com/api/v2/bizevents/ingest' \
-H 'Authorization: Api-Token dt0c01.*****.*****' \
-H 'Content-Type: application/cloudevent+json' \
-d '{…}'

3. Set the Authorization Method

Create a new EventBridge connection with the Authorization Method set to “API Key” and use the API token from Step 1 as the value (i.e., “Api-Token dt0c01.*****.****”).

Step 3: Define and configure EventBridge rules

EventBridge rules define the exact conditions for triggering guardians:

1. Specify the input template:

Create an input template to modify your event data:

{
  "specversion": "1.0",
  "id": "<id>",
  "source": "aws.<source>",
  "type": "ec2.tag.change",
  "time": "<time>",
  "aws.region": "<region>",
  "aws.eventbridge.rule.arn": "<aws.events.rule-arn>",
  "aws.resources": <resources>,
  "data": <detail>
}

2. Set the event pattern

Create a rule in EventBridge with the following event pattern:

{
  "source": ["aws.tag"],
  "detail-type": ["Tag Change on Resource"],
  "detail": {
    "service": ["ec2"],
    "resource-type": ["instance"]
  }
}

3. Apply targets and permissions

Assign targets and permissions to ensure successful data ingestion into Dynatrace. Use an IAM role to permit EventBridge to call your API destination.

The input transformer should be set as follows:

{"detail":"$.detail","id":"$.id","region":"$.region","resources":"$.resources","source":"$.source","time":"$.time"}

Step 4: Test tag changes on Amazon EC2 instances

To validate your configuration, perform the following:

1. Change a tag

Modify a tag by going to your Amazon EC2 instances in the AWS Management Console. For instance, update the “Environment” tag with a new value.

2. Verify event logging

Check the EventBridge console to ensure your tag change triggered the appropriate event.

3. Confirm data in Dynatrace

Within Dynatrace, press CMD/Ctrl + K and search for “Notebooks.” Create a new notebook and run the following query:

fetch bizevents | filter event.type == "ec2.tag.change"

If the query returns results, your configuration is working correctly.

Step 5: Set up the guardian

1. Create a new guardian

In Dynatrace, search for “Site Reliability Guardian” (`CMD/Ctrl + K`) and create a new guardian. For best practices, use the “Four Golden Signals” template.

2. Automate the workflow

Either on the overview page showing all guardians or on the analysis page of a selected guardian, click the Automate button. This will generate a workflow that triggers the guardian based on incoming bizevents (Business events). Configure the event type as `bizevent` and set the filter query to:

event.type == "ec2.tag.change"

3. Add a pause

To allow your systems to stabilize before triggering the guardian, add a “wait before” step. For example, set a delay of 600 seconds (10 minutes).

Example timeline:

• 06:59: Tag changed on EC2 instance.
• 07:00: EventBridge triggers the workflow.
• 07:10: Guardian is executed after 10-minute pause.

4. Save the workflow

Save your final workflow to activate the automation.

Step 6: Validate and monitor the setup

Perform end-to-end validation by changing an EC2 tag again. Confirm the following:

• The tag change event reaches Dynatrace.
• The workflow triggers the guardian.
• The guardian results appear in Dynatrace (e.g., heatmaps or relevant logs).

Run the following query in Dynatrace for additional monitoring:

fetch bizevents | filter event.type == "ec2.tag.change"

You should see log entries confirming the successful execution of your guardian process.

Achieve more with Site Reliability Guardian

In this blog, we highlighted the significant benefits of automating Site Reliability Guardian  triggers for Amazon EC2 changes. With automation, SRG helps engineering teams achieve efficiency, improved compliance, and cost optimization.

The post Enhance efficiency and compliance with automated AWS tag change triggers: A step-by-step guide appeared first on Dynatrace news.

Traditional monitoring approaches often require manual scripting and integration to get alerted about production-threatening issues in pre-production environments.
Dynatrace Simple Workflows make this process automatic and frictionless—there is no additional cost for workflows. By offering a single trigger and a single task, you can rapidly set up an action (like Slack or email notifications) to detect specific exceptions in your services. This means fewer surprises when deploying to production and more time spent delivering valuable features.

Why manual alerting falls short

As your product and deployments scale horizontally and vertically, the sheer volume of data makes it impossible for teams to catch every error quickly using manual processes. Your teams want to iterate rapidly but face multiple hurdles:

• Increased complexity: Microservices and container-based apps generate massive logs and metrics.
• Unstructured overview: Manually scanning logs or waiting for someone to notice an error in staging is time-consuming.
• Siloed tooling: Stitching multiple tools and scripts together can create friction and decrease your deployment velocity.

As a result, minor issues in staging can slip into production, leading to potential downtime and a poor user experience. Developers need a way to quickly set up alerts for targeted pre-production exceptions without incurring steep costs or heavy overhead.

Introducing Dynatrace Simple Workflows for early alerting

Dynatrace Simple Workflows helps your team overcome the challenges of manual processes by offering easy, no-cost automation for single-step tasks.

Go to Workflows and start creating a new workflow. By default, the Simple Workflow will be selected to give you the most cost-effective experience.

You can select any trigger that’s available for standard workflows, including schedules, problem triggers, customer event triggers, or on-demand triggers.

After you’ve selected + Workflow, you can name your workflow and select a specific trigger.

Here, you can select a specific event or a timed trigger like a cronjob. Additionally, with an on-demand trigger, you can trigger the workflow through REST API from your CI/CD, scripts, or any other automation flows.

After specifying your trigger, you can select a specific action. This action will be executed each time an event occurs, a timer kicks in, or an on-demand request triggers the action. You can easily send Slack/email messages to your teams, create JIRA issues, or trigger PagerDuty alerts. 

Safeguard your pre-production environments.

Imagine you’re running a busy e-commerce platform composed of multiple containerized microservices running on Kubernetes. Each microservice is responsible for a specific and critical business function—managing user sessions, processing payment and authentication, and tracking inventory.

To orchestrate the different logging services, you use Fluent Bit to forward these logs to your centralized logging system, like Dynatrace.

Recently, in production, your team encountered an out-of-memory error exception that killed all the processes for Fluent Bit. This caused you to lose complete visibility of your containers’ logs, performance, and error data, and you could not tell if the system was down or not.

kubernetes.event.message": "Memory cgroup out of memory: Killed process 1146778 (fluent-bit) total-vm:952432kB, anon-rss:303872kB, file-rss:1108kB, shmem-rss:0kB, UID:1000 pgtables:1424kB oom_score_adj:-997",
kubernetes.event.reason": "OOMKilling",

The error message indicates that the Linux cgroup assigned to your Fluent Bit container has reached its memory limit. Because Kubernetes enforces memory constraints for each pod, the kernel OOM (OutOfMemory) killer forcibly terminates the Fluent Bit process once memory usage exceeds the threshold set in the container/pod specification.

You have fixed the error, and as a follow-up to your RCA documentation, you want to ensure you’re alerted of these errors early on staging, so you want to set up a quick alert on staging for your feature branch.

For this example, we go to Simple Workflows and select Trigger > Davis® event trigger to find these out-of-memory errors. You can learn more about event triggers in Dynatrace Documentation.

When Davis AI event triggers find these, we will set a simple Slack or Email notification for our team so the team can ensure the fix is working and other future changes are caught early.
If you ever need more advanced capabilities—like custom code, multi-step logic, or conditional tasks—you can seamlessly upgrade to standard workflows with just a few clicks.

This ensures you’re never locked into an overly complex or expensive solution before you’re ready. It’s as simple as that!

Deep dive into Simple Workflows

How Simple Workflows work:

• Simple Workflows provide core Dynatrace automation capabilities
• You can use any trigger type, problem-event triggers, schedule-based triggers, or external webhooks.
• They trigger a single, out-of-the-box action: for example, Sending a Slack Notification, Creating a Jira Issue, Sending an Email, or Executing an HTTP Request.
• Unlike multi-step workflows, these don’t consume workflow hours.

Limitations:

• You can’t add more than one task.
• JavaScript actions, task conditions, and options are not allowed.

What’s next for Workflows?

Looking ahead, Dynatrace aims to expand the usability of Simple Workflows with more out-of-the-box actions and deeper integrations.

Both simple and standard workflows will benefit from future improvements like:

• A growing catalog of out-of-the-box actions
• Introduction of draft workflows, enabling you to work on your workflows without impact until you’re satisfied with your changes
• Watch Workflows: Get notifications whenever a workflow fails, is updated, or throttled.

For now, Simple Workflows delivers a fast way for developers and SREs to receive automated alerts—exactly when needed—at no extra cost.

Ready to try Simple Workflows?

Check out the Simple Workflows documentation to explore more use cases. For advanced functionality—like multi-step actions or running JavaScript—it’s free to upgrade to a standard workflow and take advantage of the full Dynatrace AutomationEngine capabilities.

With Simple Workflows, you can quickly eliminate manual monitoring steps, reduce time-to-detection, and keep staging errors where they belong—out of production.

The post Create simple workflows to automate alerts during development appeared first on Dynatrace news.

Navigate digital infrastructure complexity

In today’s rapidly evolving digital environment, organizations face increasing pressure from customers and competitors to deliver faster, more secure innovations. The complexity of IT infrastructure management continues to grow, and each new application deployment and every change in infrastructure potentially impact user experience.

In some cases, managing your digital infrastructure is like playing Jenga, where the placement of every piece is critical, and the smallest change can destabilize the entire structure. On one hand, the complexity of systems demands precise control; on the other, staying competitive requires frequent updates and rapid service enhancements. This dynamic creates a challenge: keeping the tower stable while continuously adding and removing pieces.

To keep up with current demands, DevOps and platform engineering teams need a solution that can fully embrace and understand complexity, delivering precise answers that enable the creation of trustworthy automation. The effectiveness of this automation relies on the quality of the underlying data. Observability, therefore, has become crucial in DevOps, offering insights into IT infrastructure stability, performance, and user experience. The DevOps Automation Pulse 2023 report notes that 78% of organizations employing observability-driven automation experience faster incident response and resolution times.

Synthetic monitoring enhances observability by enabling proactive testing and monitoring systems to identify potential issues before they quickly impact users. Returning to the Jenga metaphor, synthetic monitoring observes the tower from a distance, from the end user’s perspective, and triggers instability warnings immediately. Incorporating synthetic monitoring and observability-driven automation can greatly streamline the workflow for DevOps teams, allowing for continuous improvements in system reliability and efficiency.

Automation + Synthetic = Perfect match

This is why we integrated Dynatrace Synthetic Monitoring into Workflows. With this enhancement, Dynatrace can respond to any event and execute synthetic monitors within your workflows to assess the impact of events on user experience. Depending on the outcome, workflows can notify your teams by creating a Jira ticket, sending a Slack message, or initiating a remediation process. This enhancement improves reaction time to incidents impacting user experience and simplifies incident management. It reduces the time spent verifying the impact of changes and engages DevOps or platform engineers only when necessary, allowing them to focus on more strategic initiatives.

With the intuitive and easy-to-use web UI of Workflows, you can quickly build automation based on synthetic monitors with just a few steps, thanks to a dedicated Synthetic for Workflows task.

In the first step, you choose which monitors to execute. You have three options:

• Select a fixed list of monitors.
• Provide a list of tags; all monitors with the tags will be selected.
• Provide a list of applications; all monitors associated with those applications will be selected.

You can also provide a list of monitors, tags, or applications in the incoming event and extract the list using an expression, which allows you to build a generic workflow.

Use case: Automated release validation

Integrating Dynatrace Synthetic Monitoring into delivery pipelines is a strategic move that ensures all critical user journeys are validated as early as possible, whether a release involves minor bug fixes, updates to a microservice, or an entire monolithic application.

This integration is particularly valuable in progressive delivery models like canary deployments. In such scenarios, a new version of an application is initially rolled out to a small portion of users. By adding a synthetic user as “user zero,” the risk of negatively impacting real users is significantly reduced.

Combining Synthetic for Workflows with Site Reliability Guardian (which evaluates adherence to availability, performance, or security objectives based on synthetic results) automatically determines whether the given application version should proceed in the delivery pipeline. This integration facilitates the easy construction and implementation of solutions that enhance automatic release validation.

Use case: Digital infrastructure change

The problem is not always in the application. As illustrated by the Jenga game metaphor, even minor changes in complex infrastructure can topple the entire tower. In 2021, Facebook and its other platforms, Instagram and WhatsApp, experienced a major outage. This global outage lasted for several hours and affected billions of users worldwide.

Interestingly, the applications were functioning perfectly; the root cause was a configuration change in the backbone routers.

Incorporating synthetic monitoring directly into deployment pipelines ensures that every infrastructure modification—regardless of scale—is automatically validated against performance and functional criteria.

This integration is particularly valuable within GitOps frameworks, where infrastructure changes are managed through code and deployed automatically. Synthetic monitors can simulate API clients and mimic user behavior, enabling IT teams to evaluate the impact of changes. Such preemptive testing significantly reduces potential downtime and mitigates performance degradation.

Synthetic for Workflows and Site Reliability Guardian can—continuously or on-demand—verify that all infrastructure modifications meet SLOs from the perspectives of users and API clients.

Use case: Custom synthetic scheduling

The traditional approach to scheduling synthetic monitors involves fixed intervals, which don’t always align with the varying operational demands of modern digital environments.

Workflows in Dynatrace provide a powerful mechanism for triggering tasks in response to various platform events. By leveraging Synthetic for Workflow combined with this event-driven architecture, you can achieve a level of customization in scheduling synthetic monitors that goes beyond the capabilities of the built-in synthetic scheduler.

This flexibility is particularly useful in scenarios where operational conditions can change rapidly. For instance, if a peak load period triggers a test failure, Synthetic for Workflow can initiate retries until the load stabilizes. This retry logic can help triage the problem and ensure that monitoring results are accurate and reliable.

Conclusion

By integrating Dynatrace Synthetic Monitoring with Workflows, Dynatrace offers a comprehensive solution for evaluating the impact of infrastructure changes, enhancing release validation, and improving custom scheduling. However, utilizing Synthetic for Workflows extends beyond these use cases, enabling the easy construction of any automation based on synthetic monitors, where assessing the impact on user experience is crucial.

The post Automate digital excellence with Dynatrace Synthetic Monitoring and Workflows appeared first on Dynatrace news.

To address this challenge, the Dynatrace App Toolkit allows you to create custom workflow actions, providing seamless integration with third-party systems. This extends the capabilities of AutomationEngine for all users in your environment.

Prerequisites

Before you begin, please make sure you have the following:

• Access to a Dynatrace environment (for example https://environment-id.apps.dynatrace.com) where you can install and access Dynatrace® Apps and run Workflows.
• Most recent LTS version of js (version 20, as of May 2024).
• IDE or text editor of your choice (VS Code recommended)

Now, consider your use case! What do you wish to automate? Is there an API available for this purpose? Write down details regarding authentication and the necessary API Calls; you can also experiment with these API calls using tools like Postman, Insomnia, Apigee, or similar.

In this example, we will query electricity prices for the next 24 hours from https://www.awattar.de/services/api (for example, at the time of writing, no authentication is required for GET https://api.awattar.de/v1/marketdata), and figure out what time of day electricity will be the cheapest tomorrow.

Let’s create an app

We’re ready to create an app and implement our first custom Workflow action.

1. Use the Dynatrace App Toolkit to create a new app:

   npx dt-app@latest create --environment-url https://<environment-id>.apps.dynatrace.com

   When asked for the name, choose something short and catchy, and adhere to app naming guidelines. In our example, we will call the app awattar-actions.

2. Once the command above has finished, you should see a new folder (based on the provided app name), which you can open directly in your IDE.
3. Within your IDE, open a new terminal window (you can, of course, also perform the same commands in a terminal window of your choice outside of the IDE) and execute the following command to verify that everything is working:
   npm run start
4. You should now be greeted with your app’s welcome page, which states, “Welcome To Your Dynatrace App.”
   
5. Stop this command in the terminal by pressing CTRL C.
6. Create a new custom workflow action using
   npx dt-app generate action fetch-data
7. Within your IDE (or File Explorer), you should be able to see newly created files in the actions folder (feel free to take a closer look at fetch-data.action.ts and fetch-data.widget.ts).
   
8. Again, run npm run start to verify the setup’s success. Feel free to take a closer look at the output of this command, as it should provide some details about the workflow action.
   
9. We now need to include our changes in the newly created action, so let’s open fetch-data.action.ts and change its implementation to the following.

   interface FetchDataInput {
     start: string | undefined;
     end: string | undefined;
   }
   export default async (payload: FetchDataInput) => {
     // Best Practice: Separate Base URL of the Service and the targetURL of the API
     const baseUrl = 'https://api.awattar.de';
     const apiEndpoint = '/v1/marketdata';
     // re-construct the URL using the built-in URL
     const targetUrl = new URL(apiEndpoint, baseUrl);
     // add query parameters for start and end (if they are defined)
     if (payload.start) {
       targetUrl.searchParams.append('start', payload.start);
     }
     if (payload.end) {
       targetUrl.searchParams.append('end', payload.end);
     }
     // use fetch to call the API
     return await fetch(targetUrl.toString())
       .then((response) => response.json());
   };

10. We also need to adapt our widget fetch-data.widget.tsx to properly reflect the two input parameters in the action: start and end. Change the implementation to what’s shown below.

    import { AutomationTextInput } from '@dynatrace/automation-action-components';
    import { FormField, Hint, Label } from '@dynatrace/strato-components-preview';
    import { ActionWidget } from '@dynatrace-sdk/automation-action-utils';
    import React from 'react';
    interface FetchDataInput {
      start: string | undefined;
      end: string | undefined;
    }
    const FetchDataWidget: ActionWidget<FetchDataInput> = (props) => {
      const { value, onValueChanged } = props;
      const updateValue = (newValue: Partial<FetchDataInput>) => {
        onValueChanged({ ...value, ...newValue });
      };
      return (
        <>
          <FormField>
            <Label>Start</Label>
            <AutomationTextInput value={value.start} onChange={(start) => updateValue({ start })} />
            <Hint>Optional - see https://www.awattar.de/services/api for details</Hint>
          </FormField>
          <FormField>
            <Label>End</Label>
            <AutomationTextInput value={value.end} onChange={(end) => updateValue({ end })} />
            <Hint>Optional - see https://www.awattar.de/services/api for details</Hint>
          </FormField>
        </>
      );
    };
    export default FetchDataWidget;

11. Now, as a final step, deploy your app to your environment using npm run deploy.

Try it out

1. To try it out, go to your environment and navigate to Workflows.
2. Create a new workflow, choose On demand trigger, and add a new task.
3. Search for the action you created, for instance, awattar, and select it.
   
4. You can omit any configuration for now. Let’s try it out; select Save and Run on the top.
5. You’ll likely get the following error message about a blocked request. Please follow the guide provided to allow outbound API access to the host.

   Error: Blocked request to 'api.awattar.de' (host not in allowlist). To find out about how to manage allowed domains.
   please visit https://dt-url.net/allow-outbound-connections

6. Once done, you can edit and rerun the workflow, and you should see a result.
   

Conclusion

We created a simple workflow action within a few minutes to fetch data from a third party API. In an additional step, we can process the received data points and figure out the minimum. This can be done using a run javascript action and the following code snippet:

import { execution } from '@dynatrace-sdk/automation-utils';
export default async function ({ execution_id }) {
  const ex = await execution(execution_id);
  const result = (await ex.result('fetchdata_1')).data;
  // Initialize with first item
  var minItem = result[0];
  // Loop over array and find the actual minimum
  for (var item of result) {
    if (item.marketprice < minItem.marketprice) {
      minItem = item;
    }
  }
  return {
    // convert start and end into something human readable
    start_timestamp: (new Date(minItem.start_timestamp)).toLocaleString(),
    end_timestamp: (new Date(minItem.end_timestamp)).toLocaleString(),
    // keep marketprice and unit
    marketprice: minItem.marketprice,
    unit: minItem.unit
  };
}

When executing this workflow, you should eventually see a result like this:

The post Build custom workflow actions using the Dynatrace App Toolkit appeared first on Dynatrace news.

Knowledge is power. Knowing who is responsible for specific areas and services is a vital skill that saves time and hassle. This can quickly become overwhelming and hard to manage in today’s complex microservice environments. Combining services and responsible teams increases transparency, simplifies the identification of the right people, enables automation, and saves time and money.

As always, work must be done before you can reap the rewards. This includes preparing the environment with the required metadata to provide the ownership information transparently and when needed. These efforts include:

• Extending entities with ownership metadata.
• Enriching ownership information with desired metadata.
• Ensuring ownership coverage in the environment.
• Automating incident triage via targeted remediation or notification tasks.

What to consider when adding ownership information

Introducing ownership information requires several simple steps and typically depends on the desired application. In any case, three building blocks are required to connect the right information to software artifacts.

• Linking team ownership with the specific service or component they own.
• Enriching team ownership information with desired metadata.
• Providing easy access to team ownership information.

The first challenge is the assignment of team ownerships to the appropriate entities. At first glance, this sounds easy: Team A takes care of component B. However, as software changes continuously due to new deployments and releases, so can responsibilities. Revisiting an environment to ensure the assignments are still correct is almost impossible when done manually. The automated extraction of ownership information, for example, from Kubernetes annotations, is therefore essential.

Secondly, knowing who is responsible is essential but not sufficient, especially if you want to automate your triage process. Essential metadata about team ownership helps in addressing the right team with the appriopriate responsibilities via the desired channels. As teams and their structure and metadata are often maintained in a dedicated database, such as Microsoft Entra ID (formerly Azure Active Directory) or ServiceNow. This information needs to be reusable, avoiding the need to maintain the same information twice and ensuring data is not out of sync.

Keeping ownership teams and their properties up to date is essential, as is having the right contact information available when needed.

Finally, the best information is still useless if users can’t retrieve it quickly when needed and use it accordingly. Be it a visual representation or an automated task in a workflow, easily resolving ownership information for services is where the full value becomes noticeable.

How to efficiently introduce team ownerships

Dynatrace provides different ways of associating team ownership with entities and adding desired team metadata, such as contact details, to your environments.

Import teams

It is necessary to get ownership team information into the system and keep it updated. Dynatrace offers several ways to ingest ownership team information. Besides supporting UI and API input for ownership teams, a dedicated workflow action for importing, storing, and updating ownership teams is available.

The import_teams workflow action can be used in either an on-demand or a regularly triggered workflow to get ownership team information and store it accordingly. As data sources can vary, the import_teams action supports different options. Besides the generic import option of accepting JSON objects in the ownership schema (find details on how to use the import teams workflow actions here), two often-used databases for storing and maintaining team information are available to you:

• Microsoft Entra (formerly Azure Active Directory)
• ServiceNow (sys_user_groups)

The ownership team information can be queried regularly from the databases, maintaining them as the source of truth for the team and its properties. A dedicated workflow allows automatic synchronization of the database information with Dynatrace and keeps the ownership team information up-to-date.

Dynatrace ownership functionality supports configuration-as-code via its proprietary Monaco (Monitoring as code) CLI or Terraform. A workflow template for importing teams can be found in this public GitHub repository.

Assign teams to services

This correlation between people and software is crucial; if a problem occurs or a new security vulnerability is detected, it’s critical to quickly and easily know who is responsible and owns each specific software artifact.

As the ownership of certain components is highly dependent on the organization structure and possibly quite dynamic or heterogeneous, the linking is realized via key-value pairs added to the entities. This ensures flexibility in adding the ownership information when needed.

Since adding key-value pairs to entities is highly flexible, there are several ways of adding ownership metadata to an environment. However, to increase reliability, it’s recommended to add the ownership information depending on the environment characteristics. In cloud-native environments Kubernetes annotations or labels are recommended; these can be further used to propagate certain information within the software topology. Dedicated environment variables or custom properties are additional options to add ownership related metadata. More details on the supported ways of enriching your environment are described in Best practices for ownership information documentation.

When adding ownership teams, the key must start with one of five customizable indicators to be recognized as ownership metadata. By default, Dynatrace supports dt.owner or owner as the key prefix, while the value needs to reflect the unique team identifier. This guarantees an unambiguous identification of the right owners per component. The team identifier can be set only when creating the ownership team for the first time, either through the previously explained ownership importer or manually via the web UI or API.

If certain naming conventions or key-value pairs are already used to tag services or apps with responsible teams, it’s possible to reuse the same keys easily. For example, if team:myTeamName is already used to mark selected components, then team can be added as a supported key. Dynatrace will automatically recognize the existing tags as ownership metadata. Otherwise, it’s recommended that you reuse the default key prefix to stay consistent throughout the environment.

To keep an overview of which areas in an environment are already covered and where there might be potential blind spots, ownership information (or the lack thereof) can be easily visualized in notebooks and dashboards. The dashboard shown in figure 4 below can be found in the publicly accessible Github Repository:
https://github.com/dynatrace-perfclinics/platform-engineering-demo/blob/main/dynatraceassets/dashboards/team-ownership-dashboard.json

The dashboard serves as an example and likely needs to be adapted to your specific user needs.

Ownership  

If it comes to the worst case, and a security vulnerability is introduced or a severe incident happens, it’s important to automatically inform the responsible teams and start remediation actions.

With Dynatrace Workflows, triaging time can be minimized and eliminated. Based on the detected issue or event, a workflow can automatically retrieve the associated ownership teams during the execution. The dedicated get_owner workflow action, queries all teams of affected entities. Further the ingested ownership metadata, such as contact details can be used to prepare Jira tickets or notify the right teams with all the crucial information they need to immediately start issue resolution.

Some examples of how ownership information enables automated notification and remediation workflows are listed below.

• Automated Release Validation and notification of the responsible teams using the Site Reliability Guardian
  • https://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.automations/workflows/b9305953-0512-33be-9504-75039b00f5d5

• Problem Remediation Automation utilizing Red Hat Ansible Automation Platform

• Kubernetes Workload Optimization

• Security Vulnerability Processing

Another example of utilizing targeted notifications via Dynatrace Workflows can be found in this public GitHub repository. These examples can be extended to cover similar use cases as above.

What’s next

Dynatrace already provides the possibility to reference ownership team information from entities, ingest crucial ownership team metadata such as contact information in an efficient way, and offers automated access to this information either via the user interface or Dynatrace Workflows. This enables a wide possibility of use cases where automated and targeted notifications play a significant role in reducing MTTR and increasing efficiency and reliability.

To further improve the user handling of the ownership functionality, we are currently working on making this information easily accessible via DQL (Dynatrace Query Language). Making use of all relations and dependencies will further simplify the retrieval of necessary information reliably when needed.

Follow these steps to increase an environment’s transparency and prepare environments to automatically inform and notify the right people at the right time:

• Ingest ownership metadata, such as contact details, into a Dynatrace environment.
  • Using the teams-importer to keep ownership data automatically in sync.
  • An example of how to set up an ownership import workflow is given in our workflow-samples repository.
• Extend components and services with ownership information.
  • Use object-specific and reliable association methods such as Kubernetes annotations, environment variables, host metadata, etc. More details can be found in Dynatrace Documentation.
• Detect blind spots and ownership gaps in environments.
  • Use dashboards and notebooks to keep an overview of the current state and proactively detect blind spots.
    • An example can be seen in our platform engineering demo (If you want to try out the platform engineering demo, please look at this repository, where you can run the demo within GitHub codespaces.
  • Set up automation workflows, tailoring and targeting ticketing and remediation activities based on the affected entities and occurring events, problems, or security vulnerabilities. Checkout the already available examples within our publicly available Configuration as Code GitHub repository or our Dynatrace Discover tenant.

The post The right person at the right time makes all the difference: Best practices for ownership information appeared first on Dynatrace news.

“Great! I have ingested important custom data into Dynatrace, critical to running my applications and making accurate business decisions… but can I trust the accuracy and reliability?”

Welcome to the world of data observability.

The Dynatrace open platform is well-positioned to take advantage of the exponential increase in data generation. However, coupled with the increase of external data sources that can now be ingested, there are new challenges in data management that need to be addressed.

 “Every year, poor data quality costs organizations an average $12.9 million”
– Gartner

Data observability is a practice that helps organizations understand the full lifecycle of data, from ingestion to storage and usage, to ensure data health and reliability. Data observability involves monitoring and managing the internal state of data systems to gain insight into the data pipeline, understand how data evolves, and identify any issues that could compromise data integrity or reliability. At its core, data observability is about ensuring the availability, reliability, and quality of data.

Data observability is crucial to analytics and automation, as business decisions and actions depend on data quality. In the age of AI, data observability has become foundational and complementary to AI observability, data quality being essential for training and testing AI models.

Dynatrace now addresses many of the issues customers experience around the health, quality, freshness, and general usefulness of data that is externally sourced into Dynatrace Grail™, allowing them to make better-informed decisions and optimize their efforts for digital transformation and data-driven operations.

The rise of data observability in DevOps

Data forms the foundation of decision-making processes in companies across the globe. Data is the foundation upon which strategies are built, directions are chosen, and innovations are pursued. Consequently, the importance of continuously observing data quality, and ensuring its reliability, is paramount. Surveys from our recent Automation Pulse Report underscore this sentiment: 57% of C-level executives say the absence of data observability and data flow analysis makes it difficult to drive automation in a compliant way. This not only underscores the universal significance of data, it also hints at its pivotal role within DevOps. For DevOps teams that inform deployment strategies, optimize processes, and drive continuous improvement, the integrity and timeliness of data are of significant importance.

As organizations scale and accelerate their digital transformation journeys, a major hurdle to proper DevOps adoption is the trustworthiness of the massive volume of data coming from various sources, much of which goes into data silos such as log management tools, SIEM solutions, and others.

The rise of data observability needs is where Dynatrace capabilities around Grail, analytics, and Davis® AI are in an outstanding and unmatched position to deliver the currently missing value to the market: a leading and single solution for all data observability analytics needs. This reduces the demand for further data flow analysis tools and clears any hurdles to making data useable for DevOps automation use cases.

Davis AI, Grail, and data observability

By grouping common data observability issues into industry-standard pillars, we can provide tangible examples and showcase current capabilities. The five pillars we focus on are freshness, volume, distribution, schema, and lineage.

Freshness: Timeliness of data

In an ideal ecosystem, actionable data should be as recent as possible, supported by learnings from accurate, historical data. Observing the freshness of data helps to ensure that decisions are based on the most recent and relevant information.

Scenario: Due to an undetected configuration issue, a flight status system from a popular airline had been buffering data for the last two hours before sending it on in one batch. Downstream dashboards and system automations were using outdated data, leading to incorrect statuses of flights in reports.

Solution: After setting up data ingestion into Grail, Dynatrace Query Language (DQL) is used to add a freshness field (Figure 1) which is calculated from the delta between when the signal was written and when it was ingested. This freshness measurement can then be used by out-of-the-box Dynatrace anomaly detection to actively alert on abnormal changes within the data ingest latency to ensure the expected freshness of all the data records. Furthermore, the new Alert on missing data feature in the Anomaly Detector panel can be used to trigger notifications when data is not coming in as expected after being baselined.

Value: The possibility of alerting on data freshness issues, based on a learned baseline through Davis AI, allows for faster time-to-detect where there are seemingly no infrastructure issues. Normally this would have left an issue undetected for much longer, providing a false sense of security, eventually leading to a much bigger customer and monetary impact for the organization.

Volume: Quantity of data generated or processed within a given timeframe

Unexpected increases or drops in the volume of data are often a good indication of an undetected issue.

Scenario: For many B2B SaaS companies, the number of reported customers is an important metric. It heavily influences downstream reports, and dashboards, shaping decisions from daily operations to strategic monthly reviews. In this scenario, a manually triggered run of a production pipeline had the unintended consequence of duplicating the reported customer metric. If left unchecked, this misrepresentation of a single KPI could lead to misguided decision-making processes through multiple layers of the organization.

Solution: Like the freshness example, Dynatrace can monitor the record count over time. Once a DQL query has been set up, it can be used in an automation workflow (Figure 2) where scheduling, prediction, comparison to actual value, and, finally, alerting are all taken care of to enable a fully flexible way to detect anomalies in data volume.

Value: KPIs and metrics such as the number of reported customers are central to an organization’s business and strategic processes. Any issues here will result in a loss of trust in the data, and, if left undetected, they will eventually lead to monetary impact, including loss of reputation for an organization.

Distribution: The statistical spread or ranges of data

The distribution of data is essential in identifying patterns, outliers, or anomalies in the data. Deviation from the expected distribution can signal an issue in data collection or processing.

Scenario: A financial institution processes millions of transactions daily, ranging from credit card purchases and mortgage payments to interbank transfers and ATM withdrawals. An erroneous change in the database system leads to a subset of the data being categorized incorrectly. After several days, the fraud detection system starts triggering on a frequent basis, and liquidity management dashboards begin showing questionable values.

Solution: Baselining and raising alerts on anomalies are core capabilities of Davis AI. After setting up ingestion for the data that you want to monitor, it’s simple to use Dynatrace full AI capabilities to observe and alert on any anomalies in the data. In the example above, ingesting the number of transactions as business events, anomaly detection could be based on this to proactively alert and trigger mitigation activities.

Value: While variations are expected in financial trends, anomalies should be auto-detected, and manual detection should not be relied on. Earlier detection of these issues will keep the fallout as low as possible.

Schema: Structure and relationships of data between entities

Observing the schema can help identify and flag unanticipated changes, such as the addition of new fields or deletion of existing fields.

Scenario: An externally connected database system made an update that inadvertently dropped the account_id column in the customers table. The automated data pipeline propagated these changes, leading to downstream reports, dashboards, and applications breaking as the previous field reference is now missing.

Solution: Using the DQL FieldsSummary command, we can keep track of the number of distinct field keys within a given family of data records. Once confirmed in a notebook, the number of field keys can be used in an automated workflow to continuously monitor the count and write it back to a new metric (Figure 3). Once the new metric is established, out-of-the-box Dynatrace anomaly detection can be used to alert on either a static threshold or a learned baseline.

Value: Observing incoming data Schemas, and thus placing expectations on what the external data should look like and must contain, allows for pro-active alerting and mitigation of issues long before they can lead to widespread business impact such as broken reports, dashboards, or further analytics on top of the data.

Lineage: Journey of data through a system

Data lineage provides insights into where the data came from (upstream) and what is impacted (downstream). It plays a crucial role in root cause analysis as well as informing impacted systems about an issue as quickly as possible.

Scenario: The hourly_consumption table was deprecated and removed by an overzealous database administrator as there were no known downstream consumers of this data, breaking a monthly integration check used for consumption reporting for shareholders.

Solution: In the future, Dynatrace Smartscape® could be used, which already builds a dependency graph, to enable a data lineage view. This would enable faster root cause analysis of any data-related problems, as well as allow for easy notification of downstream consumers who would be impacted.

Value: A proper understanding of the source of the data, as well as where it is used, helps drive down time-to-alert and time-to-repair. Time-to-alert is achieved by quickly and automatically alerting those who are impacted by a data issue by quickly understanding downstream consumers of the data, while time-to-repair informs on the source of where the data originated from, to quickly drill down into those systems.

Data availability: A prerequisite

You could implement the most contemporary, accurate, and useful data observability solution possible, but what good will it be if all the data simply does not arrive as expected? Broken pipelines or missing data sources would mean that there is simply no data to observe and that data may never arrive, forever lost.

A truly valuable data observability solution should be able to alert on data issues as early in the process as possible. This requires monitoring of the upstream infrastructure, applications, or platform supporting those data streams. This is where the power of Dynatrace end-to-end observability comes into play. Dynatrace can leverage existing Infrastructure Monitoring and Application Observability solutions to surface problems that can affect later data observability workstreams—long before a traditional data observability solution would pick up the issue.

Leverage the power of Dynatrace and Davis AI—now and into the Future

Anomaly Detection

Anomaly detection is grounded in the idea of baselining typical patterns of ingested data, designed to alert where a change or deviation from the norm is observed. These patterns typically go beyond simple flat or trend lines, often exhibiting complex seasonal behaviors, such as business hours or weekly patterns related to the industry. Dynatrace is particularly strong in this area: Davis predictive AI has been enriched over the years with a set of advanced machine learning (ML) algorithms optimized for time-series observability datasets to cope with these challenges. Davis AI anomaly detection, leveraging these ML algorithms, can already be used on the results of DQL queries. (Embedding ML algorithms into DQL as functions is on the Dynatrace platform roadmap.)

Considering the examples and solutions provided above, anomaly detection plays a pivotal role in numerous data observability use cases and can be harnessed to effectively address these challenges.

Triage and resolution of a data incident

Triaging requires an ability to identify the root cause of a data incident, which is particularly challenging as an organization scales up the volume and speed of data ingest typical of an enterprise environment. It’s easy to see how Davis causal AI problem detection could be extended in the future to identify root-cause data observability issues.

Depending on the incident, there might be different paths to resolution. One acceptable path could be full auto-remediation, whereby Dynatrace AutomationEngine could be triggered, scripts executed, permissions granted, security checked, and data corrected. A second path might require Jira tickets to be created and human intervention through an approval process. A data problem alert could be used as the event allowing for multiple methods to notify the correct data owners, stewards, governors, or data teams.

An incident requires not only resolution but also understanding and alerting upstream data providers and downstream data subscribers to the potential impact. Dynatrace is strong on the observability of data pipelines ingesting data into Grail and consumers of Grail data, although this is an area that will be enhanced and improved in the future product roadmap.

Prevention of future incidents

Not all data quality incidents can be prevented, especially because ELT/ETL data pipelines typically tend to grow over time and span many different heterogeneous collectors that have different ownerships. There are, however, mitigation techniques you can use, for example:

• Health tracking of key datasets or streams over time—alerting on anomalies
• Monitoring standard query results and changes over time
• Well-designed, data-focused dashboards for monitoring
• Auto remediation where appropriate with built-in audit logging
• Forensic abilities for ad-hoc data analysis

Summary

Dynatrace is uniquely positioned to provide even more value by extending our world-class observability platform into the data observability realm. To achieve this, we leverage Infrastructure Monitoring and Application Observability for early warnings on data pipeline issues and use DQL, Workflows, and Grail for data observability—all enabled by our best-in-class Davis AI engine.

Ensuring the quality and reliability of underlying data is more crucial than ever now that many organizations are deploying Generative AI models. Data observability is becoming a mandatory part of business analytics, automation, and AI. Davis AI and data observability together uniquely ensure the quality and reliability of data at the level of hypermodal AI—predictive, causal, and generative.

You can now monitor sources and incoming data pipelines for freshness, volume, distribution, lineage, and availability issues early on without added noise and in a central location, the Dynatrace platform. This gives your teams additional confidence over data quality, saves time, prevents inaccurate analyses and automation outcomes, leads to more trustworthy AI models, and supports efforts to consolidate or reduce the number of IT tools they rely on.

Ready to get started with Dynatrace data observability? For complete details, best practices, and detailed use cases, see Dynatrace data observability documentation.

The post Introducing Dynatrace built-in data observability on Davis AI and Grail appeared first on Dynatrace news.

Bridge the gaps while staying secure

While more and more applications are moving to the cloud, the need for secure environments isn’t going away. EdgeConnect provides a secure bridge for SaaS-heavy companies like Dynatrace, which hosts numerous systems and data behind VPNs. EdgeConnect facilitates seamless interaction, ensuring data security and operational efficiency.

With the increasing adoption of SaaS platforms and escalating security concerns caused by the proliferation of cyber threats, companies are becoming increasingly aware of the importance of safeguarding their systems and data. In this hybrid world, IT and business processes often span across a blend of on-premises and SaaS systems, making standardization and automation necessary for efficiency. Enterprises seek solutions that enable these processes to interact across their entire system landscape without compromising security.

Due to a variety of challenges, organizations face significant difficulties in achieving seamless interaction between on-premises and SaaS systems. Security concerns make it risky to bridge these systems, leaving enough flexibility for users to achieve their goals and keeping IT in control of infrastructure and deployments.

A look behind the curtain: EdgeConnect connects to the outside world

An EdgeConnect instance establishes a WebSocket secure connection (WSS/443) to the Dynatrace platform that doesn’t require opening ports or inbound connections. EdgeConnect acts as a bridge between Dynatrace and the network where it’s deployed

Setting up an EdgeConnect is simple. You can manage your EdgeConnect configuration in the Settings app under General > External requests. Once the initial setup is complete, Dynatrace routes all HTTP(s) service requests that match your configured rules via EdgeConnect.

Efficiency and control

EdgeConnect boasts a range of features designed for efficiency and control. It’s built for container-based deployments, operating either as a standalone container or within your Kubernetes or Openshift clusters. EdgeConnect is designed to forward HTTP(s) requests exclusively, ensuring secure data transmission. It supports multi-instance high availability and load balancing, providing robust performance and reliability.

EdgeConnect empowers operators with an optional allow list for domains during deployment. This feature enables IT to maintain control without the need for any configuration in Dynatrace, further simplifying the process.

The versatility of EdgeConnect enables many use cases:

• Automated creation and update of tickets in your local Jira instance using Dynatrace Workflows whenever Davis® AI identifies problems.
• Integration of data from your internal data warehouse to Dynatrace dashboards, blended with data from Grail.
• Flexible and adaptable connections to internet-based services like ServiceNow via a dedicated host and IP that you control.

What’s next

EdgeConnect is available for all Dynatrace SaaS environments, version 1.275+. Have a look at our documentation to learn more about how to configure and deploy EdgeConnect. Once deployed, you can manage your EdgeConnect configuration in the Settings app under General > External requests.

We’d love to hear your feedback about EdgeConnect. Let us know your thoughts or share your ideas about how we can improve EdgeConnect in the future.

The post Dynatrace EdgeConnect securely connects your local systems to Dynatrace SaaS appeared first on Dynatrace news.

Dynatrace Configuration as Code enables complete automation of the Dynatrace platform’s configuration, ensuring that software is secure and reliable. With Configuration as Code, developers can manage their observability and security tasks with config files that can be developed alongside source code conveniently and at scale. Dynatrace offers Configuration as Code for the entire platform, covering all aspects, including app settings built for the AppEngine.

As software development grows more complex, managing components using an automated onboarding process becomes increasingly important. This is especially crucial in microservice architectures, where the number of components can be overwhelming. Furthermore, increasing the frequency of releases requires additional product lifecycle automation and setup.

While infrastructure has historically been treated as a bottleneck where proper scaling and compute power are applied to improve performance, these aspects are now typically addressed by hyperscalers that offer cloud-based infrastructure and infrastructure as a service. However, scaling up software development requires more tools along the software product lifecycle, which must be configured promptly and efficiently.

To handle this challenge, enterprises need to automate and streamline the onboarding and lifecycle of tool configurations in the software development processes, including aspects of observability, security, alerting, and remediation.

Efficient environment configuration at scale

One of software engineers’ most significant challenges is managing the numerous tools and technologies required for the software product lifecycle. Development teams must set up tailored configurations for each tool and component they’re responsible for. Developers can’t focus on software development while also managing the details of each tool’s configuration options, and tool administrators can’t scale to keep up with multiple development teams, numerous configurations, and the many environments they need to support. This is why it’s so essential to offer sharable configuration templates that can be easily reused and customized for specific teams, components, and environments.

Configuration as Code in Git repos, automatically applied by Dynatrace

Analogous to infrastructure as code, Configuration as Code, or “everything as code” is now essential for tackling software development challenges. With Dynatrace Configuration as Code, teams can meet these challenges while working with the IDEs, Git repos, and tools that they’re already familiar with. Configuration files allow for the automatic creation, update, and management of configurations for dashboards, synthetic monitors, alerts, SLOs, and security settings across multiple environments.

While developers edit files, a simple CLI command applies configurations to Dynatrace and, for example, automates the setup of a quality gate, including workflows and Site Reliability Guardians.

This can all be done safely and consistently in a repeatable manner. Configuration files can be reused, versioned, and shared across teams. Configuration as Code supports all the mechanisms and best practices of Git-based workflows, including pull requests, commit merging, and reviewer approval.

GitOps is a best-practice methodology for handling operation-relevant configurations that can be applied across the entire Dynatrace platform. With Dynatrace configuration files in Git repos, you gain:

• Persistent configuration state available in files
  • Easy copy and paste
  • Easy templating
• Established Git software development workflows
  • Repo branching
  • Change management
  • Reviews and approvals

From the developer perspective, you only need to fill out a form (YAML file) with key-value pairs (for example, to provide the thresholds for the API endpoint of a service) and an email address for escalation to get the benefit of the Dynatrace platform when you deliver a new service.

Enable self-service configuration at scale

You can easily set up observability, security, and automation by filling out a few key-value pairs in YAML format. Reduce the complexity of configuration down to only the parameters that vary within your organization, such as for different services and environments. Teams can quickly set the parameters they need to configure the Dynatrace platform to their specific requirements. We provide a CLI that is natively built for Dynatrace APIs and platform configurations, as well as apps built for our AppEngine that can be run and customized without third-party dependencies and with a lightweight setup. Additionally, we offer a Terraform provider if you already have a working Terraform-built environment in your organization.

Achieve automated observability and Site Reliability Engineering

To minimize any negative impact caused by problems such as security vulnerabilities, it’s crucial to respond and remediate the issues as quickly as possible. Dynatrace provides automation for detecting problems, and you can opt to automatically run a change-impact analysis report to proactively validate important objectives. This same mechanism can also be leveraged to validate the impact of new software releases on resources, logs, performance, reliability, or business measures. The screenshot below displays a workflow that listens for a deployment event of the easytrade service in the production stage.

The validation process is automated based on events that occur, while the objectives’ configuration, which is validated by the Site Reliability Guardian, is stored in a separate file. The screenshot below displays such a configuration. In summary, Configuration as Code enables the automatic execution of validations for a full set of configuration objectives and workflow definitions. Development teams can easily adopt this by providing key-value pairs in a YAML config file, such as the name of a service and the stage to listen for deployment events.

Service-Level Objectives (SLOs) can provide additional support for the insights and outlook of services and components. Whether tracking internal, workload-centric indicators such as errors, duration, or saturation or focusing on the golden signals and other user-centric views such as availability, latency, traffic, or engagement, SLOs-as-code enables coherent and consistent monitoring throughout the environment at scale.

Proper notifications or escalations are automated based on ownership information. The Dynatrace platform supports ingesting ownership team information, as shown in the screenshots below. Dedicated configuration files are used to create teams and maintain relevant information, such as responsibilities and contact details, in a scalable and automated way. While this can be achieved through UIs and APIs, providing contact details and links to any supporting material when things go wrong in any environment is most conveniently done by a developer, as code, side-by-side with the software source code.

Furthermore, by utilizing workflows, it’s easy to set up dynamically-queried ownership-team information from affected entities if a problem or a security vulnerability is detected. Leverage the full power of the Dynatrace AutomationEngine to automatically inform the right people and automatically resolve problems, all managed and configured easily via configuration as Code.

What’s next

Get started with Dynatrace Configuration as Code, natively built for the Dynatrace platform and third-party independent. You can read all about it in our Configuration as Code documentation. If Terraform is your tool of choice, please have a look at our Terraform provider documentation.

Stay tuned for more examples and easy-to-adopt automations provided in our public Github project.

The post Automated observability, security, and reliability at scale appeared first on Dynatrace news.

Our recent blog post, Stay ahead of the game: Forecast IT capacity with Dynatrace Grail and Davis AI, showed how Dynatrace Notebooks are used to predict the future behavior of time series data stored in Grail™. This follow-up post introduces Davis^® AI for Workflows, showing you how to fully automate prediction and remediation of your future capacity demands. The anticipation of future capacity demands makes it possible to completely avoid critical outages by notifying you days in advance, well before incidents arise.

Predictive capacity management starts within a Dynatrace Notebook, where the operations team explores important capacity indicators, such as the percentage of free disks, as shown below.

After exploring and selecting the most important capacity indicators for your environment, a workflow triggers forecast reporting at regular intervals. The example workflow below is triggered every Monday at 8:00 AM to provide a capacity report for all the disks that will likely run out of space within the next week.

Define the forecast

The workflow uses the Davis for Workflows action to automatically trigger a forecast for a selected set of disks. The forecast operation is selected within the Davis action, and a DQL query is used to specify the set of disks and the capacity indicator metric that should be predicted. Note that you can use any time series data you can fetch from Grail using DQL within the forecast action.

While this example uses the metric dt.host.disk.free, you can choose any kind of capacity metric, such as host CPU, memory, or network load—you can even extract a metric value from a given log line.

The forecast is trained on a relative timeframe (for example, the last seven days) which is specified in the configured DQL query. The DQL query example below trains forecasting on a relative timeframe of the last seven days:

timeseries avg(dt.host.disk.free), by:{dt.entity.host, dt.entity.disk}, bins: 120, from:now()-7d, to:now()

The configuration below shows that a forecast horizon of 100 data points is requested, which means that 100 additional predicted points will expand the initially fetched 120 data bins of the source DQL query. This predicts one week into the future.

The prediction action returns all its forecasted time series lines, which can include hundreds or even thousands of individual disk predictions.

Evaluate the forecast results

Within the following TypeScript action, each disk prediction is tested against a threshold to determine if the disk will run out of space in the next week. The TypeScript code snippet below is responsible for checking for threshold violations and for preparing all the violations in a result object for subsequent actions to follow up on:

The TypeScript action returns a custom object that uses a Boolean flag (violation) to tell the follow-up actions about violations and an array of all the violation details (violations).

const predictionSummary = { violation: false, violations: new Array<Record<string, string>>() };

Tip: Download the TypeScript template from our documentation.

Trigger remediation actions

A collection of remediation actions can be used to follow up on predicted capacity shortages. In this example, two parallel actions are defined. One action sends out an email notification; the other raises a Davis problem for each violating disk. All remediation actions use the Boolean violation flag of the previous workflow action to avoid invocations when there are no violations.

Here you can see the invocation condition used in the follow-up actions that control the invocation.

Raise events in case of disk capacity shortage!

A TypeScript remediation action is used to iterate through all the predicted disk shortages and to raise individual alarm events. Each alarm event has custom event properties that can be used to deliver further details about the situation and to further identify the disk or host.

Tip: Download the TypeScript template from our documentation.

Review all Davis-predicted capacity problems

Navigating to the Davis problems feed, the operations team can review all the predicted disk capacity shortages. Remember, raising events and problems is an optional remediation step that can be skipped entirely by directly sending emails or Slack messages to the responsible teams.

The creation of alerting events within this workflow example highlights the flexibility and power of the Dynatrace AutomationEngine combined with the analytical capabilities of Davis AI and Grail.

Summary

The combination of Davis AI forecasts with Dynatrace AutomationEngine and Grail opens the door for many valuable use cases—anticipative management of capacity being the most prominent of these. Predicting future capacity shortages for thousands of disks or hosts allows operations teams to anticipate critical situations weeks before incidents occur. The flexibility and power of the Dynatrace AutomationEngine allow operations teams to react to detected shortages flexibly and to customize and implement their remediation flows.

You can install Davis® for Workflows via the Dynatrace Hub. As a starting point for implementing your own anticipative capacity management workflow, you can download all the TypeScript code used in this example from our documentation:

• Evaluate the forecast results
• Raise a Davis problem

For full details, see Davis AI analysis in workflows documentation.

Predictive capacity management in action (14-second video)

The post Automate predictive capacity management with Davis AI for Workflows appeared first on Dynatrace news.