Organizations are evolving from human-driven operations to supervised autonomous operations, where AI investigates, recommends, and remediates, and humans stay in control of what matters most. A big part of delivering on that vision is working with the agents that customers already run in their cloud environments.

Harness the power of hyperscale agents

Each hyperscaler has AI agents that automatically investigate and help resolve production incidents using native cloud telemetry and tools. They act like embedded site reliability engineers, analyzing issues and recommending or executing remediation steps without waiting for a human to start the process.

AWS DevOps Agent provides investigation and remediation in AWS using native tooling. An Azure SRE Agent specializes in investigating and remediating Azure issues. And Google Gemini Cloud Assist is for incident analysis across Google Cloud Platform (GCP).

Over the past year, we’ve published how Dynatrace supercharges each of these cloud agents individually. When an issue occurs, Dynatrace Intelligence combines causal, predictive, and agentic AI using the Smartscape dependency graph to automatically link related symptoms and root causes across the environment into one unified problem card.

When Dynatrace integrates with the AWS DevOps Agent, dependency-aware root cause analysis combines with AWS frontier-agent capabilities, and joint customers report up to 70% reductions in mean time to resolution. When Azure SRE Agent connects with Dynatrace, deterministic, causation-based AI flows directly into Azure-native remediation workflows, cutting the back-and-forth between teams. And with Google Gemini Cloud Assist, Dynatrace delivers the same production context layer to GCP-hosted incidents: precise root cause, full topology, real business impact.

From integrations to intelligent orchestration

Many enterprises run workloads across AWS, Azure, and Google Cloud simultaneously, and managing three separate integrations with separate routing logic and separate cost controls is its own operational tax. Cloud SRE Agents provides a single orchestration layer that routes problems to specific hyperscaler agents based on configurable profiles to see everything happening across all three cloud agents.

The Cloud SRE Agents app writes findings back to Dynatrace, and provides your team with measurable visibility into autonomous actions.

How Cloud SRE Agents works

When Dynatrace Intelligence detects a problem and identifies the root cause, Cloud SRE Agents calls dedicated cloud-native agents from AWS, Azure, and Google Cloud to retrieve deeper insights from the sources that only they can reach: CloudTrail history, Azure subscription policy, GCP project IAM, recent deployments, and native runbooks. These agents run in parallel, gathering evidence as soon as the problem is detected. Their findings, and, where applicable, the recommended remediation path, are displayed in the same Dynatrace problem view that the on-call SRE is already using in their day-to-day workflow.

One view. No tab-switching. The work starts without you.

Three workflows do the orchestration in the background:

• Investigate evaluates your Interaction Profiles and dispatches matching problems to the right agents in parallel.
• Periodic Tasks polls each cloud provider for completion, detects stalled or timed-out investigations, and writes findings back as problem annotations.
• Event Handlers normalize the cloud-provider event stream so every action correlates back to its originating problem, end to end.

Cloud SRE Agents has the insights and intelligence to decide which agent gets which problem, tracks each run to completion, and brings the answers back together in a single view. The Overview tab provides a real-time, interactive network graph of problems, agents, and activities. The replay view allows the user to step back in time and get an overview of what has happened when, as well as the status of each investigation.

Intelligent routing with Interaction Profiles

In agentic operations, routing rules make the difference between turning autonomous systems loose on every alert and pointing them precisely where they earn their keep. Interaction Profiles are how you express routing judgment in Cloud SRE Agents. Each profile pairs a set of conditions with the agent or agents that should handle the problems flagged by the profile, and evaluates the conditions whenever Dynatrace Intelligence detects a problem.

The conditions you can write are deliberately broad. You can route by the cloud account, subscription, or project an incident touches; by problem category (availability, error, slowdown, resource contention); by affected entity type (a Kubernetes cluster, a database, a Lambda function); by tag, label, or any custom attribute carried in the problem record. Conditions combine with AND/OR logic and nest as deeply as you need, keeping real production routing policy inside the app rather than spilling into custom workflows or scripts.

Three ways teams put it to work

Route problems to the right cloud, automatically

A spike in Lambda error rates belongs to AWS DevOps Agent. An Azure App Service degradation calls for Azure SRE Agent. A Pub/Sub latency issue lands with Gemini Cloud Assist. In a multicloud estate, none of those decisions should fall to a human at 2:00 AM. A profile filtered by AWS Account ID, Azure Subscription ID, or GCP Project ID, then narrowed by resource type or tag, settles the routing question once. Every matching problem is automatically routed to the right specialist with the right cloud-native context.

Optimize spend with budget-aware routing

Cloud AI agents do work, and that work has a cost. Cloud SRE Agents lets you set a Monthly Duration Budget per agent and gate dispatch on it via a Has Available Budget filter: once the budget is exhausted, new investigations either stop (in strict enforcement mode) or proceed with a logged warning. The duration figure itself is a proxy, derived from Dynatrace event timestamps rather than the cloud provider’s clock, which makes it useful as a circuit breaker and directional signal, not a substitute for AWS, Azure, or GCP usage reports. The governance value is what matters: you decide how much autonomous investigation you’re willing to underwrite each month, and the system holds the line.

Tier autonomous investigation by problem type and entity

Not every Dynatrace problem warrants an autonomous investigation. Problem Category filters let you dispatch agents only to the problem categories that warrant it, for example, availability or error problems that require immediate action, rather than slowdowns or custom alerts where human triage might still be the right call. Layer on Entity Type filters, and you can further focus on specific infrastructure tiers (hosts, services, process groups, Kubernetes clusters). The result is a tiered model: high-severity issues receive immediate autonomous investigation, lower-severity signals queue for human review, and your team controls the threshold.

Governance that makes autonomous work measurable

Agentic operations earn trust when teams can see what the agents did, why, and whether it worked. Cloud SRE Agents treats that as a first-class concern, with two views built for the two audiences who care about it.

The Activity tab is the audit trail. Every investigation and mitigation appears as a card on a unified timeline; expand any card to see the agent’s full findings, the evidence it pulled, and the action it took or recommended. Each response can be rated Good, OK, or Bad, building a quality signal grounded in what your team actually saw rather than what the system predicted. When a single problem triggers work across multiple agents, those activities roll up to a single status (in progress, done, or stalled), so you always know where things stand without having to reconstruct the run from individual records.

The Statistics tab is where autonomous operations become a number you can show to a leadership team: problems handled, mitigations executed, average investigation time, MTTR and MTTI trends, success rates, and satisfaction scores broken down by agent. The same view doubles as a directional cost lens, since agent working time is the dominant driver on the cloud side of the bill. Treat the number as a trend signal and a circuit-breaker input, not a billing record (reconcile against AWS, Azure, and GCP usage reports for exact spend), and it makes the case for expanding agentic coverage with evidence rather than anecdote.

Why production context multiplies the value

What changes Cloud SRE Agents from a smart dispatcher into something more is what Dynatrace Intelligence contributes before an agent ever begins its analysis. Dynatrace delivers deterministic, causation-based root cause analysis grounded in Dynatrace’s Smartscape real-time dependency mapping, alongside business impact assessment and correlated telemetry. That context shapes the entire direction of the investigation. A cloud agent arriving with that foundation starts from “this specific service on this specific host is the root cause, and here’s the customer impact” rather than “something is wrong somewhere in this account.”

The numbers reflect it. According to AWS, organizations using the AWS DevOps Agent with Dynatrace see up to a 75% reduction in mean time to resolution.

Western Governors University, which runs a fully online learning environment for 200,000 students, uses AWS DevOps Agent with Dynatrace to automate cross-system correlation that previously required manual effort across multiple tools. At a larger scale, United Airlines transports more than 500,000 passengers daily across a hybrid environment that includes more than 500 AWS accounts, 20,000 Lambda functions, and 38,000 OneAgent deployments.

The team’s description of the before and after status is direct: previously, multiple tools with overlapping functions created gaps and black boxes during troubleshooting. With AWS DevOps Agent and Dynatrace, Dynatrace identifies the responsible layer, the agent investigates and provides resolution steps, and everything surfaces in a single Dynatrace view. No 3:00 AM tool-switching required.

Get started

For a closer look at the individual integrations, read the posts on AWS DevOps Agent and Dynatrace and Azure SRE Agent and Dynatrace, or see how Dynatrace Intelligence powers autonomous operations. To put your cloud agents to work today, install Cloud SRE Agents from the Dynatrace Hub. Cloud SRE Agents is currently available as a community-supported app.

The post Orchestrate multicloud AI agents for autonomous incident resolution appeared first on Dynatrace news.

“How do I structure my observability team?” is one of the most common questions folks leading software teams ask me. My advice: Don’t create a centralized “observability team” that’s responsible for all the observability within an organization.

Observability shouldn’t exist as a silo. It touches many parts of an organization, from development to production, and should be treated as a team sport.

As we know, our systems can only be considered observable if they emit telemetry. No data means that we can’t understand what is happening in our systems. Fortunately, the OpenTelemetry® (OTel) ecosystem from the Cloud Native Computing Foundation (CNCF) has become the de-facto standard for instrumenting, generating, collecting, and exporting telemetry data.

What does this mean for observability adoption in an organization? Let’s dig in.

Observability is everyone’s responsibility

Reliability can’t happen without observability. Observability must be looked at holistically. It is not the sole responsibility of any one team or individual. Everyone has an important part to play, and to a certain extent, the parts weave into each other.

Instrumenting code

There are two types of OpenTelemetry instrumentation:

• Code-based (manual) instrumentation
• Zero-code (automatic) instrumentation

Code-based instrumentation should be done by application developers, and not by an “observability team.” Developers know their applications best. Asking someone else to instrument your application is like asking someone else to write your code comments. Please never do that.

Zero-code instrumentation usually involves a shim or bytecode instrumentation wrapper around your code. If you’re a developer writing code in a language that supports OpenTelematry auto-instrumentation, you should understand how to implement both zero-code and code-based instrumentation. In doing so, you can use the instrumentation to troubleshoot your own code.

In some environments, zero-code instrumentation may be managed by the OTel Operator. If this is the case, the responsibility often falls to SRE or platform engineering teams. Event in those cases, developers should understand; at least at a high level, how zero-code instrumentation is configured with the OTel Operator.

Managing observability infrastructure

Observability infrastructure still needs to be managed, whether you’re using a SaaS vendor (e.g. Dynatrace) or an open source stack. If you’re using OpenTelemetry, chances are you’re managing at least one OTel Collector, and perhaps many. If you’re running your applications on Kubernetes, you’ll likely deploy and manage Collectors within the cluster as well. In most organizations, this responsibility falls under platform engineering or SRE teams, and these teams are essential to robust, reliable software delivery in large, complex environments.

That said, developers should still understand how the OpenTelemetry Collector is configured. It’s true that you don’t need to go through a Collector to send OTel data to an observability backend for non-production. However, the Collector still offers some nice things that direct-from-application doesn’t (e.g. batching data, masking data, and automatic retries), and I still highly recommend using it, even in development.

Making CI/CD pipelines observable

DevOps engineers can’t escape observability either, because guess what? We can make CI/CD pipelines observable too. While CI/CD pipelines may not be a production environment that external users interact with, they most certainly are a production environment that internal users interact with (i.e. software engineers, platform engineers, and SREs).

CI/CD pipelines are defined by code, and like it or not, that code can still fail. Making our application code observable helps us make sense of things when they fail in production. So, it stands to reason that having pipeline observability can help us understand what’s going on when CI/CD pipelines fail.

There’s been some great buzz around the observability of CI/CD pipelines, especially now that there’s an official OTel CI/CD Special Interest Group (SIG). This will give our favorite CI/CD tools a shared language for the observability of CI/CD pipelines, creating a foundation for them to support OpenTelemetry tools in this context.

We’re not there yet, which means that right now we must stitch a few tools together to achieve CI/CD observability. Fortunately, things are moving nicely in this space, and if you haven’t considered CI/CD pipeline observability in your organization before, now’s the time to start thinking about it. To learn more about what’s happening with OTel CI/CD observability, check out the #otel-cicd channel on CNCF Slack.

Troubleshooting

The beauty of observability is that once you instrument your code, you put the ability to troubleshoot in the hands of many. Consider the ripple effect when developers instrument their code:

• Developers: Instrumentation allows developers to debug their code as they’re writing it.
• QA testers: Instrumentation allows testers to troubleshoot failed tests, allowing them to file more detailed bug reports. If QAs can’t track down the issue, then it means that there is missing instrumentation that developers need to add to their code. This turns observability into a quality gate.
• SREs: Instrumentation allows SREs to troubleshoot production issues, gain insight into system performance, and ensure overall system reliability.

Ensuring adherence to observability practices

Remember how I advised against creating an “observability team” responsible for all observability within an organization? I still stand by that. That said, I do believe that organizations should have an observability team responsible for enterprise-wide observability oversight and advocacy. A team that defines and disseminates observability standards and practices within that organization. This team would need to stay up to date in the latest observability practices, vendor offerings, and the OpenTelemetry  ecosystem— not just as an observer, but also as a project contributor, while also encouraging developers, platform engineers, and SREs to contribute.

This “observability practices team,” can’t, however, exist on an island. First off, it needs to be aligned with leadership to ensure that everyone is on the same page when it comes to observability. The team also needs support from individual practitioners. As a result, the team also needs to work with developers, SREs, platform engineers, QAs, and DevOps engineers to ensure that the practices and standards that it comes up with make sense.

If observability is to be a team sport, it needs coordination and guidance. There should be guardrails in place, to ensure that you have standard tooling, practices, and enforcement of said practices. Practices and standards include things like standard Collector configurations, and standard attributes emitted to your chosen observability backend(s).

Standardizing tooling is important because I’ve seen far too many “tool jungles” in organizations, where each team or department has their own tooling and practices, and it ends up being a recipe for disaster. Too much redundancy and overlap.

In addition, the observability practices team should not be responsible for instrumenting developers’ code, nor should it be managing infrastructure. It’s there to work with these other groups and to make sure that things are done right.

Final thoughts

Observability weaves its way into various aspects of an organization. It’s not just a developer concern. It’s not just an SRE concern. It’s not just a QA concern. It’s certainly not the concern of a single “observability team.” Doing so downplays its importance, takes away our collective responsibility towards observability, and dilutes the promise of observability. The only way to make this work is by ensuring that the teams participating in this team sport that we call observability don’t operate in silos.

The post Observability is a team sport appeared first on Dynatrace news.

In practice, most organizations attempt to solve this with one of two compromises: lock everything down centrally and become a bottleneck for every pipeline change, or copy rules across pipelines and hope nothing drifts. Neither approach scales. And as data requirements shift faster than ever to address new services, new regulatory mandates, and new object types to observe, the cost of such compromise keeps growing.

Pipeline Groups change that dynamic. Generally available now in Dynatrace SaaS version 1.332, Pipeline Groups let platform engineering teams, the teams with admin permissions over the observability platform’s configuration, mandate and standardize ingest behavior across many pipelines, while safely delegating day-to-day configuration of individual pipelines to the teams best suited to own them.

The problem: Pipeline governance at scale

In large enterprises, dozens of teams may each operate their own custom pipelines. Common requirements, security context enrichment, cost allocation tagging, and storage bucket assignment must be applied consistently across the board. However, without a structured mechanism to enforce such standards, platform teams face a choice that gets harder with every new team, every new data source, and every new regulation:

• Lock everything down and become the gatekeeper for every pipeline change. Every adjustment, no matter how small, becomes a ticket, a review, a delay. Innovation stalls. Teams wait days for changes that should take minutes.
• Copy rules everywhere and accept the risk of configuration drift. What starts as a manageable set of shared rules gradually fragments—one team forgets to apply the latest cost allocation tag, another skips the security enrichment step, a third routes data to the wrong bucket. The inconsistencies compound silently until they surface as compliance gaps or billing surprises.

Customers consistently tell us that this approach doesn’t scale. What they need is a way to separate what must always happen from what teams should be free to decide, and to encode that separation directly into the platform, not just into process documents that team members forget to follow.

What are Pipeline Groups?

A Pipeline Group is a first-class configuration object that separates a global pipeline list into distinct sets; the pipelines in these sets are then members of the group. A Pipeline Group orchestrates a special type of reusable pipeline to define what happens before or after the stages of the member pipelines.

Pipeline Groups allow you to:

• Organize pipelines into groups with clearly defined membership.
• Define execution order so processing happens in a predictable, layered sequence.
• Control stage execution for the member pipelines so that teams can turn on or turn off specific pipeline stages.
• Enforce mandatory global processing that no team can bypass or override.
• Allow team-level customization within boundaries defined by the platform team.

Pipeline Groups determine how data flows through your pipelines. This ensures that governance isn’t an afterthought bolted onto a pipeline configuration, but rather it’s built into the execution model itself.

How Pipeline Group ownership works

The design behind Pipeline Groups draws a deliberate line between group-level configuration and member-pipeline configuration. How organizations map this division to team ownership is up to them. Pipeline Groups provide the mechanism, not a prescriptive policy. The pattern we see most often is straightforward: platform or SRE teams own the pipeline groups, and application teams own their member pipelines within those groups.

To make this more concrete, consider a platform team that’s responsible for observability across multiple business units. They create a Pipeline Group that adds a business segment field to every record for organizational attribution, applies cost allocation tags for accurate chargeback, sets security context so sensitive data is handled consistently, and assigns data to the correct storage bucket based on retention and compliance needs. These are the rules that must always apply, and because they operate at the group level, no member pipeline can bypass or override them.

Member pipeline ownership can be more granular than a simple admin-vs-team split. While a group is always admin-owned, the individual pipelines that make up the Pipeline Group can each have different owners. For example, a pipeline that enriches every record with organizational metadata or classifies data sensitivity might be globally relevant and owned by the platform team, or by a specialist team like the security team. A pipeline that handles domain-specific compliance logic, such as PCI field masking for a financial services division, might be owned by that compliance team because they have the expertise that the platform team lacks. This arrangement reflects how responsibility within that organization is distributed.

Within that group, application teams each get their own member pipeline. One team configures parsing for a specific log format. Another sets up filtering to reduce noise. A third extracts metrics tailored to their services. Each team works independently within its own pipeline without touching the platform-level rules above and without needing to coordinate with other teams or the platform team for routine changes.

The value of this separation becomes clear when things change, as they do frequently in large enterprises. Say that one of those business units onboards a new microservice that generates a completely new log format. Under a centralized-only model, the team would file a request, wait for the platform team to update the pipeline, validate, and iterate. With Pipeline Groups, the team simply adds their parsing and filtering rules to their own member pipeline. The mandatory enrichment, cost tagging, and routing are already guaranteed by the group. The new service is observable in hours, not weeks, and the platform team doesn’t need to be involved at all.

Or picture the reverse direction: a new regulatory requirement lands, say, a mandate that all log data from EU-based services must be assigned to region-specific storage. The platform team updates the group configuration once. Every member pipeline inherits the change immediately.

Why the flexibility of Pipeline Groups matters

The pace at which enterprise data requirements evolve has fundamentally changed. New services are spun up in days, not months. Regulatory landscapes shift across jurisdictions. The types of software entities that organizations need to observe, from traditional infrastructure to AI model outputs, edge devices, and third-party SaaS telemetry, keep expanding. In this environment, a rigid, centralized-only pipeline configuration becomes a constraint rather than a benefit.

Pipeline Groups are built for this reality. They give enterprises a mechanism that:

1. Creates room for innovation. Application teams can iterate on their pipeline configurations independently, experimenting with new parsing rules, extracting new metrics, and adapting to new data shapes without waiting for central approval on every change.
2. Eliminates bottlenecks. Platform teams define the rules once and let the system enforce them. Teams are freed from serving as gatekeepers for routine changes and can focus on architecture, standards, and strategy.
3. Ensures that ever-changing guardrails are applied. As compliance requirements, security policies, or cost structures evolve, platform teams can update them all in one place. The changes propagate automatically to every pipeline in the Pipeline Group.

Pipeline Groups give platform teams the governance controls and flexibility they need

Pipeline Groups give enterprise platform teams the governance controls they’ve been asking for, mandatory processing, centralized ownership of sensitive stages, and structured delegation, all without taking away the flexibility that makes Dynatrace OpenPipeline® valuable to individual teams in the first place.

Ready to get started with OpenPipeline and Pipeline Groups?

If your organization manages observability across multiple teams and struggles with consistency, Pipeline Groups are now generally available.

Learn how to define global guardrails once and empower your application teams to build their own pipelines confidently within standardized boundaries.

The post Pipeline Groups in Dynatrace OpenPipeline: Enterprise-grade governance explained appeared first on Dynatrace news.

Enterprise automation rarely fails because teams lack ideas. It fails because scaling automation safely is challenging, and conventional workflows often remain static after they ship, while systems and conditions continue to evolve.

The new update to Dynatrace Workflows helps teams accelerate their innovation loop and run automation like production software. Five improvements make it easier to iterate and operate at scale: workflow drafts, sub-workflows, approval requests, persistent execution data, and real-time notifications. Together, they support safer change, reusable standards, built-in governance, and data-driven refinement.

The innovation loop in enterprise automation

The innovation loop is a repeatable cycle for quickly and safely improving enterprise automation that treats workflows as living systems that evolve with your environment, self-improving as systems, teams, and requirements change, rather than remaining static:

• Teams start by designing or updating a workflow and validating it safely before it goes live.
• Next, they standardize the parts that work into reusable building blocks so future workflows are faster to create and more consistent.
• Then they add governance where it matters, such as approval checkpoints for high-impact actions, so speed does not bypass policy.
• After execution, they observe what actually happened using execution records and signals, including errors, latency, and outcomes, to understand reliability and impact.
• Finally, they refine the workflow based on those findings by tuning logic, tightening controls, improving reuse, and adjusting notifications.

The result is a compounding impact: faster delivery over time, fewer incidents at scale, automation that integrates changing circumstances, and greater organizational trust, as transparency, auditability, and tight feedback loops back improvements.

How Dynatrace Workflows accelerates the innovation loop

Dynatrace Workflows helps you accelerate your innovation loop with powerful new features designed for adaptability, collaboration, transparency, and control. Each feature is designed to empower teams to safely iterate on automation in dynamic enterprise environments.

Faster iteration with safer testing and experimentation with workflow drafts

The new workflow drafts feature allows teams to create, edit, and refine workflows in a draft state, allowing them to validate logic and configuration without triggering live automation. Workflows that remain in draft mode don’t consume workflow hours.

Workflow drafts support the first step of the innovation loop: iterate safely. Teams can make changes, review them with peers, and confirm the workflow is ready before anything runs in production.

Key benefits

• Reduce risk by testing configuration changes without production impact.
• Allow faster iteration and cleaner change management.
• Improve collaboration during the design and review stages.
• Control costs with included workflow-hour consumption for draft-only workflows.

Standardize reusable blocks for reliability and speed with sub-workflows

Sub-workflows allow a workflow to be executed as a task inside another workflow, so teams can package repeated logic into reusable sub-workflows and compose larger automations from consistent building blocks.

This supports the “standardize” step of the innovation loop. When teams standardize and reuse proven components, automation becomes more consistent, easier to maintain, and quicker to expand across teams and environments.

Key benefits

• Enhance reliability and consistency by reusing proven workflow components.
• Speed up delivery by composing workflows from standard blocks.
• Simplify maintenance by updating shared logic in a single location.
• Keep complex automations understandable by breaking them down into smaller, manageable units.

Example use cases

• Notification workflows: Automatically send notifications based on incident severity or timing.
• Data processing workflows: Run filter/transform/store stages as distinct sub-workflows, processing large datasets in stages.
• Incident response workflows: Automate repetitive tasks as reusable steps, such as creating tickets, logging incidents, notifying stakeholders, and triggering remediation steps.

Supervised human-in-the-loop control where it matters with approval requests

With Dynatrace Workflows, you can now insert an approval request into a workflow and pause execution until a human reviews and approves the next step.

This feature supports the “governance” step in the innovation loop. Approvals turn your workflows into supervised autonomous operations. This allows the automation of more processes while ensuring strict compliance with enterprise requirements for governance, risk management, and accountability, particularly for high-impact actions.

Key benefits

• Add explicit human control at high-risk decision points.
• Reduce errors in sensitive operations (changes, remediations, escalations).
• Ensure compliance with organizational standards.
• Increase trust in automation across security, ops, and platform teams.
• Minimize operational risks.

End-to-end visibility with an audit trail and persistent execution data

Dynatrace Workflows now persists workflow execution data in DQL. All workflow activities are recorded as system events in the dt.system.events table, creating a queryable execution record for tasks, states, and outcomes as a comprehensive audit trail.

Persisting execution data supports the “observe” step of the innovation loop. When workflow execution is captured as data, teams can troubleshoot failures, provide transparency, prove what happened, and improve automation based on evidence rather than anecdotes.

Key benefits

• Provide an audit trail for enterprise traceability and review.
• Monitor workflow health and reliability through execution outcomes.
• Speed up troubleshooting by pinpointing where failures occur.
• Optimize automation using DQL-driven actionable insights (error rates, trends, hotspots, and more)

Example: Workflow health overview

This example demonstrates how to use execution data to support users in identifying those workflows that need improvement.

The following DQL query…

fetch dt.system.events
| filter event.kind == "WORKFLOW_EVENT" and event.provider == "AUTOMATION_ENGINE" and event.type =="TASK_EXECUTION"
| filter dt.automation_engine.state == "ERROR"
| summarize count = count() , by:{dt.automation_engine.workflow.title}

…counts task execution errors per workflow. Using a honeycomb visualization, it displays error distribution by workflow and provides visibility into those workflows that fail too often.

Instant alerts for failures and workflow changes with real-time notifications

Easily set up and customize real-time notifications by email for all workflow errors or changes made by others, ensuring you’re always informed.

These real-time notifications support the “respond and refine” step of the innovation loop. Proactive alerting allows you to manage workflows effectively, minimize disruptions, and maintain seamless operations.

Key benefits

• Get immediate visibility into failures to address potential issues before they escalate.
• Detect workflow changes early to reduce drift and surprises.
• Improve operational coordination across teams that share workflows
• Ensure uninterrupted operations with timely, actionable alerts.

Empower your team: Build smarter, safer, and scalable workflows today

Scaling automation today is hard, changes are risky to test, automation becomes outdated and buggy as soon as one part of the organization makes changes, and when something inevitably breaks, you’re left hunting for answers across scattered logs and screenshots. But it doesn’t need to be like this.

Are you ready to make things easier, run your workflows like production software, and accelerate your innovation loop? Here’s how you get started:

• Explore our documentation to learn more about these features.
• Join the Dynatrace Community to connect with other users and share your insights in the Workflow and AutomationEngine feedback channel.

The post Reliable enterprise automation at scale: Accelerate the innovation loop with Dynatrace Workflows appeared first on Dynatrace news.

The integration of Dynatrace with Microsoft Azure SRE Agent establishes a new benchmark for cloud operations by leveraging AI-based root cause analysis and real-time production insights, alongside a comprehensive understanding of complex, large-scale IT environments. You can leverage the combined strengths of Dynatrace and Microsoft, enabling teams to resolve complex problems in large-scale IT environments more quickly and efficiently, and automate incident remediation, moving one step closer to driving autonomous operations across their complex environments.

In today’s cloud-first world, reliability isn’t just a goal; it’s a competitive advantage. As more services move online and LLM-powered assistants evolve into autonomous agents, maintaining the reliability, scalability, and cost-efficiency of critical systems becomes essential.

That’s why Dynatrace and Microsoft teamed up to integrate Dynatrace® AI-powered observability with the Azure SRE Agent. This collaboration allows site reliability engineers (SREs) to ensure seamless operations while proactively planning for future scalability and reliability requirements.

Transform your incident management through the combined capabilities of Azure SRE Agent and Dynatrace AI

Azure SRE Agent, introduced earlier this year, provides SREs and developers with the tools they need to increase the speed and efficiency of incident responses, diagnostics, and collaboration, allowing them to resolve problems quickly.

Seamlessly integrated with incident management tools such as ServiceNow, as well as the developer ecosystem, represented by GitHub Copilot or Azure DevOps, the agent runs in the background 24/7, learning and monitoring the health and performance of your cloud environment.

As a reliability assistant, Azure SRE Agent supports teams by efficiently diagnosing and resolving production issues. You can ask the agent questions in natural language, easily access clear and concise problem summaries, and coordinate incident workflows with integrated human-in-the-loop approvals.

Dynatrace enhances Azure SRE Agent’s troubleshooting and automation capabilities with advanced observability insights. By mapping topology, data, and business context, Dynatrace gains a comprehensive understanding and delivers production-accurate visibility across your entire IT system. This visibility feeds Dynatrace deterministic AI, allowing precise root-cause identification and impact analysis. All these insights are now seamlessly supplied to the Azure SRE Agent, equipping it with real-time production context and reliable root cause analysis.

This allows your teams to move beyond simply receiving alerts; teams are now provided with AI that acts, guides safe mitigations, and accelerates resolution within Azure-native workflows.

Gain efficiency across every stage of the incident lifecycle

Using the Model Context Protocol (MCP), the Azure SRE Agent is securely connected with Dynatrace. Whether a team member uses the agent to ask questions in plain natural language, or the agent interacts with Dynatrace directly—sharing insights, asking for real-time observability data, or root cause analysis, together with remediation steps—the close collaboration supports use cases across every stage of incident management, allowing you to:

• Cut MTTR by automating routine runbooks and diagnostics, with safe, approved mitigation actions based on full context.
• Reduce security risk by triaging vulnerabilities faster with production evidence, triggering guided fixes, and validating outcomes.
• Accelerate delivery with contextual GitHub issues and PRs that include root cause, blast radius, and tests, minimizing issue reproduction time and rework.
• Improve fix accuracy by correlating Azure and Dynatrace telemetry for precise root-cause and impact analysis.
• Prevent incidents before they happen using real-time signals and historical trends to stop regressions and reduce toil.

Illustrating the value: Proactively detect and remediate security vulnerabilities

Let’s take a look at a concrete example, which we presented at Microsoft Ignite. Imagine you run a Java-based payroll app on Azure, and a new security warning (CVE) appears. Every second matters now, and there’s no room for error: you need the issue fixed quickly, without lots of back-and-forth between teams.

• Once the vulnerability is detected, Dynatrace automatically identifies the library that caused the vulnerability, opens a GitHub issue containing all relevant information, such as which parts of your app are affected, and informs Azure SRE Agent.
• The SRE agent reviews the GitHub issue and requests additional information from Dynatrace via the MCP server, such as the number of users affected, how often it happens, which endpoints are involved, or which customers might be affected, to assess the scope and impact of the vulnerability.

• After gathering all necessary details, the SRE Agent synthesizes the information and creates a new GitHub issue, assigning it to GitHub Copilot for remediation.
• GitHub Copilot then takes action by updating the configuration and code in the GitHub repository to resolve the vulnerability automatically.
• The pull request not only includes the necessary version changes but also includes documentation, highlighting all findings as well as how the issue was remediated, along with unit tests, to prevent the issue from recurring.

Try the power of Agentic AI for incident resolution

Dynatrace delivers deep, causation-based insights into your live systems, now seamlessly integrated with Azure SRE Agent to elevate your incident management. With this integration, you can unlock:

• Smarter detection and remediation: Deep contextual observability from Dynatrace, correlated with Azure telemetry, enhances issue identification and resolution across complex environments.
• Automated operations: Routine runbook actions and diagnostic workflows can be automated, reducing mean time to repair and freeing teams to focus on innovation.
• Proactive reliability: Continuous analysis of real-time and historical data identifies leading indicators of failure, allowing teams to prevent incidents before they impact customers.

Azure customers can now access Azure SRE Agent directly in the Azure portal. To connect Dynatrace with the agent and learn how to set up Dynatrace MCP Server, see Dynatrace Documentation.

The post Boost cloud reliability: Dynatrace and Azure SRE Agent unite for autonomous operations appeared first on Dynatrace news.

Developers deploy on production systems at lightning speed every day. Site Reliability Engineers (SREs) strive to maintain production stability yet face an impossible trade-off: hamper releases with rigid security policies or risk shipping software riddled with vulnerabilities. Dynatrace Runtime Vulnerability Analytics and Security Posture Management flips this dynamic by giving SREs real-time visibility into exposed vulnerabilities and misconfigurations in production, allowing them to prioritize and remediate threats more quickly. This blog explores how an observability-driven approach puts SREs in control of security, compliance, and uptime—without sacrificing delivery velocity.

Software teams outpace security

Modern software teams have embraced agile development practices, DevOps, and continuous integration/continuous deployment (CI/CD) pipelines. While this speeds up innovation and value delivery to the end customer, it also floods production environments with frequent code changes and new infrastructure—often outpacing the ability to fully verify security compliance before deployment.

Challenges for the modern SRE

Oftentimes, organizations try to address this challenge by rolling out security tools throughout the entire software development lifecycle (SDLC), making sure no unscanned code makes it into production.

However, a couple of challenges arise with this setup, which leave SREs in a position of balancing a difficult tradeoff:

• Mission-critical role. SREs are accountable for system stability, ensuring users and businesses aren’t affected by downtime or compromised data.
• Conflicting pressures. SREs want to enforce a “zero vulnerabilities in production” policy. However, developers need to ship product features quickly to remain competitive – a difficult decision when SDLC tools find thousands of vulnerabilities.
• Lack of visibility. For SREs to gain insight into their vulnerability risks, they have to painstakingly pull together multiple data sources from their development teams. However, those only represent a point in time when the scan happened. In other words, they are blind to real-time risk and don’t have visibility into what is actually exposed in production.
• Exposure to zero-day vulnerabilities. Every week, approximately 700 new CVEs get published – a nightmare for any SRE who is responsible for the security of production systems. With the frequent turnover of deployments and complex microservice architectures, it’s impossible to maintain an overview of which deployed assets are vulnerable and need attention.
• Mounting anxiety. To enable fast-value creation with digital products, SREs are forced to allow vulnerabilities to make it into production. Without real-time insights into which they are exposed and where they reside, pinpointing the truly critical issues can feel like guesswork.

This friction leads to watered-down policies, sleepless nights, and a constant sense that there is a gap in the defense—right where it matters most: in production.

Closing the gap

Dynatrace Runtime Vulnerability Analytics and Security Posture Management bridges exactly this gap between continuous delivery speed and SREs’ need for operational security. Rather than relying solely on static scans or best-effort CI/CD or exposure checks, it brings risks caused by vulnerabilities and misconfigurations into clear focus while applications and infrastructure are actually running.

Key attributes of a robust runtime vulnerability analytics solution: 

• Production-centric insights. Identifies which vulnerabilities are actively exposed and which are connected to critical data assets, enabling SREs to prioritize real threats and reduce noise. 
• Observability at scale. Works alongside an observability platform that tracks infrastructure, application performance, and security events in real time. 
• Automated detection. Identifies exposed vulnerabilities or configuration drifts the moment they appear in production. (Learn more about Dynatrace Workflow Automations.) 
• Integrated with DevOps workflows. Serves as a seamless layer in the existing CI/CD pipeline, ensuring it doesn’t obstruct the pace of innovation. (Learn more about security integrations.) 

Securing production environments with Dynatrace

How can customers secure their production environments with Dynatrace security products?

In many proof-of-concept (POC) scenarios, development teams are overwhelmed by long, unprioritized lists of vulnerabilities and alerts. Manually triaging, assigning, and resolving these issues slows down delivery and often leads to weakened security admission policies—just to keep value flowing to customers.

Dynatrace helps cut through the noise by empowering SREs to focus on what truly matters. Here’s how:

• Vulnerabilities App (Runtime Vulnerability Analytics)
  Eliminate endless lists of vulnerabilities. Instead, leverage intelligent prioritization that highlights critical exposures based on real-time observability context—so SREs and developers can focus on what’s actually exploitable in production.
• Security Posture Management
  Continuously monitor your live production environments for misconfigurations and policy violations. This enables your systems to stay compliant and secure without manual checks or guesswork.
• Contextual Insights & Remediation Guidance (Code-Level Vulnerabilities)
  Go beyond surface-level alerts. Dynatrace pinpoints the exact library or code snippet responsible for a vulnerability, enabling developers to fix issues quickly and accurately—no more hunting through codebases.
• Continuous Feedback Loop (Automations, Workflows, and Integrations)
  Share real-time vulnerability data across development and product teams. This capability fosters a culture of secure coding and supports security becoming a shared responsibility— not a bottleneck.

Looking ahead

Dynatrace continues to evolve its security capabilities to meet the growing demands of modern production environments. Our roadmap includes deeper integrations with incident management platforms, expanded support for SDLC security tools, and enhanced AI-driven threat intelligence.

We’re also investing in more out-of-the-box workflow automations to streamline vulnerability detection and remediation—reducing manual effort and accelerating response times.

Our mission remains clear: to empower Site Reliability Engineers with end-to-end visibility and control, from code creation to live production. With Dynatrace, SREs can focus on strengthening system resilience, confident that security risks won’t silently undermine operational stability.

By tapping into real-time, production-focused insights, Site Reliability Engineers can align reliability and agility under one cohesive framework—and truly rest easier knowing that vulnerabilities won’t stay hidden for long.

Continue reading to learn more about how Dynatrace is empowering SREs to improve availability, performance, and user experience and solve problems proactively with full-stack visibility and real-time insights.

The post Empowering SREs with runtime vulnerability analytics and security posture management appeared first on Dynatrace news.

Streamlining site reliability at scale can be daunting, particularly with large-scale AWS environments and architecture that rely on hundreds—or even thousands—of Amazon EC2 instances. However, you can simplify the process by automating guardians in the Site Reliability Guardian (SRG) to trigger whenever there are AWS tag changes, helping teams improve compliance and effectively manage system performance.

This step-by-step guide will show you how to configure your architecture to trigger guardians whenever EC2 tags are updated. Note that EC2 is an example; this guide can be made to work generically for tag changes on any AWS resource. By the end of this guide, you’ll be ready to automate guardians at scale and optimize Amazon EC2 management with ease.

Why automate guardians for AWS tag changes?

Before diving into the technical setup, here’s why automating guardians whenever EC2 tags change is beneficial for your organization:

• Greater efficiency: Automatically triggering guardians removes the need for manual intervention, saving time for DevOps or site reliability engineering (SRE) teams and allowing for more efficient resource management at scale.
• Better compliance: Automating guardians ensures critical policies and checks are consistently applied after changes across your architecture, improving security and compliance efforts.
• Cost optimization: Immediate responses to tag changes lead to informed decisions about scaling, shutting down unused instances, or fine-tuning resource efficiency.
• Proactive site reliability: Automated guardians can monitor the four golden signals, enabling proactive reliability measures.

Now, let’s get started with the setup!

Step 1: Create an API token

First, create an API token to integrate AWS services with Dynatrace for guardian automation.

1. Log into your Dynatrace tenant

Log in to your Dynatrace tenant and note the first part of the URL (for instance, “abc12345”), which is your tenant ID.

2. Access token settings

Press Ctrl + K or CMD + K and search for “Access Tokens” within Dynatrace.

3. Generate a new token

Create a new access token and assign it “bizevents.ingest” permissions.

4. Save the token

Copy and securely store the token, which looks like “dt0c01.*****.*****”. You’ll use this later during configuration.

Step 2: Create the EventBridge connection

Amazon EventBridge acts as the bridge between AWS and Dynatrace. Here’s how to set it up:

1. Navigate to Amazon EventBridge

Log in to your AWS Management Console and go to EventBridge > Connections.

2. Recreate the cURL command

You can use this cURL command as a reference to establish your connection:

curl -X POST \
'https://abc12345.live.dynatrace.com/api/v2/bizevents/ingest' \
-H 'Authorization: Api-Token dt0c01.*****.*****' \
-H 'Content-Type: application/cloudevent+json' \
-d '{…}'

3. Set the Authorization Method

Create a new EventBridge connection with the Authorization Method set to “API Key” and use the API token from Step 1 as the value (i.e., “Api-Token dt0c01.*****.****”).

Step 3: Define and configure EventBridge rules

EventBridge rules define the exact conditions for triggering guardians:

1. Specify the input template:

Create an input template to modify your event data:

{
  "specversion": "1.0",
  "id": "<id>",
  "source": "aws.<source>",
  "type": "ec2.tag.change",
  "time": "<time>",
  "aws.region": "<region>",
  "aws.eventbridge.rule.arn": "<aws.events.rule-arn>",
  "aws.resources": <resources>,
  "data": <detail>
}

2. Set the event pattern

Create a rule in EventBridge with the following event pattern:

{
  "source": ["aws.tag"],
  "detail-type": ["Tag Change on Resource"],
  "detail": {
    "service": ["ec2"],
    "resource-type": ["instance"]
  }
}

3. Apply targets and permissions

Assign targets and permissions to ensure successful data ingestion into Dynatrace. Use an IAM role to permit EventBridge to call your API destination.

The input transformer should be set as follows:

{"detail":"$.detail","id":"$.id","region":"$.region","resources":"$.resources","source":"$.source","time":"$.time"}

Step 4: Test tag changes on Amazon EC2 instances

To validate your configuration, perform the following:

1. Change a tag

Modify a tag by going to your Amazon EC2 instances in the AWS Management Console. For instance, update the “Environment” tag with a new value.

2. Verify event logging

Check the EventBridge console to ensure your tag change triggered the appropriate event.

3. Confirm data in Dynatrace

Within Dynatrace, press CMD/Ctrl + K and search for “Notebooks.” Create a new notebook and run the following query:

fetch bizevents | filter event.type == "ec2.tag.change"

If the query returns results, your configuration is working correctly.

Step 5: Set up the guardian

1. Create a new guardian

In Dynatrace, search for “Site Reliability Guardian” (`CMD/Ctrl + K`) and create a new guardian. For best practices, use the “Four Golden Signals” template.

2. Automate the workflow

Either on the overview page showing all guardians or on the analysis page of a selected guardian, click the Automate button. This will generate a workflow that triggers the guardian based on incoming bizevents (Business events). Configure the event type as `bizevent` and set the filter query to:

event.type == "ec2.tag.change"

3. Add a pause

To allow your systems to stabilize before triggering the guardian, add a “wait before” step. For example, set a delay of 600 seconds (10 minutes).

Example timeline:

• 06:59: Tag changed on EC2 instance.
• 07:00: EventBridge triggers the workflow.
• 07:10: Guardian is executed after 10-minute pause.

4. Save the workflow

Save your final workflow to activate the automation.

Step 6: Validate and monitor the setup

Perform end-to-end validation by changing an EC2 tag again. Confirm the following:

• The tag change event reaches Dynatrace.
• The workflow triggers the guardian.
• The guardian results appear in Dynatrace (e.g., heatmaps or relevant logs).

Run the following query in Dynatrace for additional monitoring:

fetch bizevents | filter event.type == "ec2.tag.change"

You should see log entries confirming the successful execution of your guardian process.

Achieve more with Site Reliability Guardian

In this blog, we highlighted the significant benefits of automating Site Reliability Guardian  triggers for Amazon EC2 changes. With automation, SRG helps engineering teams achieve efficiency, improved compliance, and cost optimization.

The post Enhance efficiency and compliance with automated AWS tag change triggers: A step-by-step guide appeared first on Dynatrace news.

In this blog post, we guide you through configuring a project to visualize real-time CI/CD build and release data for your Azure DevOps pipelines. The resulting visibility enhances collaboration with transparent data, increases productivity by automating monitoring tasks, and enables teams to detect issues proactively. With insights into your Azure DevOps pipelines, you can continuously improve processes, ensuring smoother and more reliable deployments over time.

A notebook version of the blog is also available on the Dynatrace GitHub repository. Download the AzureDevOps - Dynatrace Integration.json file and upload it to your Notebooks app in Dynatrace.

Prerequisites

Before we begin, ensure you have the following:

• Access to your Dynatrace Tenant and permission to create tokens.
• Your tenant ID, which can be found in your environment URL: https://<YOUR_TENANT_ID>.live.dynatrace.com/.
• A token with Ingest Logs v2 scope.
• A token with Write Settings scope.

Create webhooks in Azure DevOps

First, we need to create two service hooks subscriptions in Azure DevOps: one for Builds Completed and one for Release Deployment Completed.

1. Navigate to https://{orgName}/{project_name}/_settings/serviceHooks.
2. During the configuration, do not apply any filters.
3. In the settings page of the subscription, fill in the following fields:
   • URL: https://<YOUR_TENANT_ID>.live.dynatrace.com/api/v2/logs/ingest
   • HTTP Headers: Authorization: Api-token <YOUR_LOG_INGEST_TOKEN>
   • Ensure the text above is copied exactly, replacing only the token.
   • Change “Messages to send” and “Detailed Messages to send” to Text.

Add a new Grail logs bucket

Follow these steps to create a new logs bucket in Dynatrace. Note: This step is optional. You’re welcome to use an existing Grail logs bucket.

1. Open the Storage Management app in your tenant: Select CTRL/CMD + K and enter Storage.
2. Create a new bucket by selecting + in the top right corner.
3. Name the bucket azure_devops_logs.
4. Set the retention time as desired.
5. Set the bucket type to logs.

Configure OpenPipeline with log processing rules

Using OpenPipeline™, you can easily define a rule that routes all relevant log lines you created in the previous step into the bucket. This process also performs the necessary processing steps, such as renaming fields or transforming log data into dedicated events.

1. Open the OpenPipeline app and select Logs in the left pane.
2. Select the Pipelines tab and create a new one by selecting + Pipeline.
3. Name the new pipeline AzureDevOps.
4. Go to Dynamic Routing and create the following new rule:

matchesPhrase(eventType,"ms.vss-release.deployment-completed-event") OR matchesPhrase(eventType,"build.complete")

5. From the Pipeline dropdown, select “AzureDevOps”.
6. Return to your pipelines and open “AzureDevOps”.
7. Under the Storage section, add a new processor > Bucket assignment, set a name and select the azure_devops_logs bucket from the dropdown.

8. Next, open the Processing tab and define a couple of rules.
   1. First, create a new Rename Fields rule and call it “Rename Build Fields” where the left value is the new field name and the right value is the existing field name.

1. resource.buildNumber: buildNumber
2. resource.result: result

1. 2. Second, create another Rename Fields rule called “Rename Release Fields” where the left value is the new field name and the right value is the existing field name.

1. resource.stageName: stageName  
2. resource.project.name: projectName 
3. resource.deployment.release.name: releaseName 
4. releaseStatus:resource.environment.status: releaseStatus

Use the following sample data to verify your rules are working as expected.

Release event sample data:

{ 
       "timestamp": "2024-11-11T15:14:51.104000000-05:00", 
       "loglevel": "NONE", 
       "status": "NONE", 
       "createdDate": "2024-11-11T20:14:50.6300269Z", 
       "detailedMessage.text": "Deployment of release Release-946 on stage Staging succeeded. Time to deploy: 00:14:14.", 
       "dt.auth.origin": "dt0c01.YFMJ6LUO43SFFDW2SF7EW5YZ", 
       "eventType": "ms.vss-release.deployment-completed-event", 
       "id": "1862ab11-c0d4-451a-8b9b-0dfe7f517297", 
       "message.text": "Deployment of release Release-946 on stage Staging succeeded.", 
       "resource.environment.status": "succeeded", 
       "resource.project.name": "devlove-alpha", 
       "resource.stageName": "Staging", 
       "resource.deployment.release.name": "Release-946" 
    }

Create an Azure DevOps dashboard and visualize log data

Now that we have ingested the logs coming from AzureDevOps, let’s visualize the data to get better and quicker insights into our CICD pipelines.

1. Go to the AzureDevOps Git Repository and download the AzureDevOps Dashboard (on Logs).json file.
2. Within Dynatrace, open the Dashboards app and select Upload at the top left corner.
3. Upload the JSON file to start visualizing your Azure DevOps data.

Extract “release” and “build” events

Taking this a step further, we can convert the ingested log data to SDLC (Software Development Lifecycle) events in case we detect a new release or build and discard the related log line afterward.

This helps reduce the number of stored log data and supports further platform engineering use cases, such as calculating DORA metrics, automating development processes, or observing the health of your engineering pipeline.

Disclaimer: The following instructions extract Business events from log data. Once supported by OpenPipeline we propose to extract Software Development Lifecycle Events (SDLC events), which are the preferred way of storing the extracted information.

Note: You may need to request a Log Content Length (MaxContentLength_Bytes) increase depending on how many steps your Release Events have. The integration generates ingest costs for logs and metrics (according to your rate card), depending on how many build/release events you ingest.

1. In Dynatrace, open OpenPipeline:
   • Go to OpenPipeline > Logs > Pipelines > AzureDevOps.
   • Navigate to the Data Extraction section.
2. Next, create a Business Event Processor rule, for all build events, using the following parameters:
   • Name: Build Result
   • Matching condition: matchesPhrase(eventType,"build.complete")
   • Event type: field name eventType
   • Event provider: Change to Static String: AzureDevOps
   • Fields to extract: result, buildNumber, resource.reason

1. Finally, we use another Business Event Processor rule, to extract release events:
   • Name: Release Result
   • Matching condition: matchesPhrase(eventType,"ms.vss-release.deployment-completed-event")
   • Event type: field name eventType
   • Event provider: Change to Static String: AzureDevOps
   • Fields to extract: stageName, projectName, releaseName, releaseStatus, resource.deplyoment.startedOn, resource.deployment.completedOn

Extract Davis® AI events (optional)

Besides transforming the log line into SDLC events, you can also extract Davis events in case something goes wrong. These events can be used to create an alert or trigger a (remediation) workflow.

1. First, add an event if a build has failed. Add a new “Davis event” processor using the following data:
   • Name: Build Complete Failed
   • Matching condition: matchesPhrase(eventType,"build.complete”) AND result== “build”
   • Event description: Unable to generate build {buildNumber}

Note: You can change the event.type in case you want to increase the severity level.

2. Next, extract a Davis event if the deployment is rejected:
   • Name: Release deployment rejected
   • Matching condition: matchesPhrase(eventType,"ms.vss-release.deployment-completed-event”) AND releaseStatus== “rejected”
   • Event description: Unable to deploy release {releaseName}

Discard logs (optional)

Now that we have successfully converted the log event into a SDLC (Business) and a Davis event, you can disable the storage assignment rule. This will reduce the amount of data stored in Grail and help you saving some money (and consequently speeding up your log queries). Go to OpenPipeline > Logs > Pipelines > AzureDevOps.

1. Open the OpenPipeline app and select the pipeline we created before.
2. Select the tab Storage and change the matching condition to false.

Analyzing data

After deleting the logs events we need to adapt our dashboard to consume business instead of log data. To speed up things you can upload a ready-to-use dashboard into your environment:

1. Go to the AzureDevOps Git Repository and download the AzureDevOps Dashboard (on BizEvents).json file.
2. Within Dynatrace, open the Dashboards, select Upload at the top left corner, and select the JSON file.

Additionally, we recommend that you create a segment to filter all of your monitored entities across different apps.

1. Go to the Segments app and create a new segment by selecting + in the top right corner.
2. Rename the segment to AzureDevOps.
3. Select + Business events and add the following filter: event.provider = AzureDevOps
4. Select + Logs and add the following filter: dt.system.bucket = azure_devops_logs
5. Select Preview to validate the filters and Save once you’re done.

What’s next

By following these steps, you’ll be able to seamlessly integrate Azure DevOps with Dynatrace, enabling efficient log management and insightful data visualization.

Happy monitoring! 🚀

The post Insights into your Azure DevOps pipelines appeared first on Dynatrace news.

Traditional monitoring approaches often require manual scripting and integration to get alerted about production-threatening issues in pre-production environments.
Dynatrace Simple Workflows make this process automatic and frictionless—there is no additional cost for workflows. By offering a single trigger and a single task, you can rapidly set up an action (like Slack or email notifications) to detect specific exceptions in your services. This means fewer surprises when deploying to production and more time spent delivering valuable features.

Why manual alerting falls short

As your product and deployments scale horizontally and vertically, the sheer volume of data makes it impossible for teams to catch every error quickly using manual processes. Your teams want to iterate rapidly but face multiple hurdles:

• Increased complexity: Microservices and container-based apps generate massive logs and metrics.
• Unstructured overview: Manually scanning logs or waiting for someone to notice an error in staging is time-consuming.
• Siloed tooling: Stitching multiple tools and scripts together can create friction and decrease your deployment velocity.

As a result, minor issues in staging can slip into production, leading to potential downtime and a poor user experience. Developers need a way to quickly set up alerts for targeted pre-production exceptions without incurring steep costs or heavy overhead.

Introducing Dynatrace Simple Workflows for early alerting

Dynatrace Simple Workflows helps your team overcome the challenges of manual processes by offering easy, no-cost automation for single-step tasks.

Go to Workflows and start creating a new workflow. By default, the Simple Workflow will be selected to give you the most cost-effective experience.

You can select any trigger that’s available for standard workflows, including schedules, problem triggers, customer event triggers, or on-demand triggers.

After you’ve selected + Workflow, you can name your workflow and select a specific trigger.

Here, you can select a specific event or a timed trigger like a cronjob. Additionally, with an on-demand trigger, you can trigger the workflow through REST API from your CI/CD, scripts, or any other automation flows.

After specifying your trigger, you can select a specific action. This action will be executed each time an event occurs, a timer kicks in, or an on-demand request triggers the action. You can easily send Slack/email messages to your teams, create JIRA issues, or trigger PagerDuty alerts. 

Safeguard your pre-production environments.

Imagine you’re running a busy e-commerce platform composed of multiple containerized microservices running on Kubernetes. Each microservice is responsible for a specific and critical business function—managing user sessions, processing payment and authentication, and tracking inventory.

To orchestrate the different logging services, you use Fluent Bit to forward these logs to your centralized logging system, like Dynatrace.

Recently, in production, your team encountered an out-of-memory error exception that killed all the processes for Fluent Bit. This caused you to lose complete visibility of your containers’ logs, performance, and error data, and you could not tell if the system was down or not.

kubernetes.event.message": "Memory cgroup out of memory: Killed process 1146778 (fluent-bit) total-vm:952432kB, anon-rss:303872kB, file-rss:1108kB, shmem-rss:0kB, UID:1000 pgtables:1424kB oom_score_adj:-997",
kubernetes.event.reason": "OOMKilling",

The error message indicates that the Linux cgroup assigned to your Fluent Bit container has reached its memory limit. Because Kubernetes enforces memory constraints for each pod, the kernel OOM (OutOfMemory) killer forcibly terminates the Fluent Bit process once memory usage exceeds the threshold set in the container/pod specification.

You have fixed the error, and as a follow-up to your RCA documentation, you want to ensure you’re alerted of these errors early on staging, so you want to set up a quick alert on staging for your feature branch.

For this example, we go to Simple Workflows and select Trigger > Davis® event trigger to find these out-of-memory errors. You can learn more about event triggers in Dynatrace Documentation.

When Davis AI event triggers find these, we will set a simple Slack or Email notification for our team so the team can ensure the fix is working and other future changes are caught early.
If you ever need more advanced capabilities—like custom code, multi-step logic, or conditional tasks—you can seamlessly upgrade to standard workflows with just a few clicks.

This ensures you’re never locked into an overly complex or expensive solution before you’re ready. It’s as simple as that!

Deep dive into Simple Workflows

How Simple Workflows work:

• Simple Workflows provide core Dynatrace automation capabilities
• You can use any trigger type, problem-event triggers, schedule-based triggers, or external webhooks.
• They trigger a single, out-of-the-box action: for example, Sending a Slack Notification, Creating a Jira Issue, Sending an Email, or Executing an HTTP Request.
• Unlike multi-step workflows, these don’t consume workflow hours.

Limitations:

• You can’t add more than one task.
• JavaScript actions, task conditions, and options are not allowed.

What’s next for Workflows?

Looking ahead, Dynatrace aims to expand the usability of Simple Workflows with more out-of-the-box actions and deeper integrations.

Both simple and standard workflows will benefit from future improvements like:

• A growing catalog of out-of-the-box actions
• Introduction of draft workflows, enabling you to work on your workflows without impact until you’re satisfied with your changes
• Watch Workflows: Get notifications whenever a workflow fails, is updated, or throttled.

For now, Simple Workflows delivers a fast way for developers and SREs to receive automated alerts—exactly when needed—at no extra cost.

Ready to try Simple Workflows?

Check out the Simple Workflows documentation to explore more use cases. For advanced functionality—like multi-step actions or running JavaScript—it’s free to upgrade to a standard workflow and take advantage of the full Dynatrace AutomationEngine capabilities.

With Simple Workflows, you can quickly eliminate manual monitoring steps, reduce time-to-detection, and keep staging errors where they belong—out of production.

The post Create simple workflows to automate alerts during development appeared first on Dynatrace news.

Establishing SLOs for Kubernetes clusters can help organizations optimize resource utilization. A good Kubernetes SLO strategy helps teams manage and make containerized workloads more efficient.

Kubernetes is a widely used open source system for container orchestration. It allows for seamless running of containerized workloads across different environments.

Effective resource provisioning and management is a critical aspect of a Kubernetes cluster. It involves a coordinated effort among stakeholders, including cluster owners, application teams, and business owners. The primary goal is to allocate sufficient resources to keep the applications running smoothly without overprovisioning and incurring unnecessary costs. This delicate balance requires a collaborative approach and transparent communication among all parties. Service-level objectives (SLOs) can play a vital role in ensuring that all stakeholders have visibility into the resources being used and the performance of their applications.

While the Kubernetes application from Dynatrace already provides detailed insights into key parameters such as memory or CPU slack—allowing you to immediately drill down to the workloads blocking and wasting resources—SLOs add a further layer on top of these insights by evaluating resource spending levels.

Service-level objectives are typically used to monitor business-critical services and applications. However, due to the fact that they boil down selected indicators to single values and track error budget levels, they also offer a suitable way to monitor optimization processes while aligning on single values to meet overall goals. Essentially, an SLO tracks a selected service-level indicator (SLI) and continuously evaluates its behavior over a given timeframe against a fixed (commonly agreed upon) threshold. This feature is valuable for platform owners who want to monitor and optimize their Kubernetes environment. Users can continuously evaluate a system’s performance against predefined quality criteria, making SLOs for Kubernetes clusters a good option for monitoring and improving a system’s overall performance.

Efficient coordination of resource usage, requests, and allocation is critical. All users must work together seamlessly to optimize resource utilization in a Kubernetes cluster. A Kubernetes SLO can be a transparent and trackable collaboration tool for different teams and collaborators.

Measure your Kubernetes resource efficiency and agree on an aligned approach between all stakeholders

If your team is responsible for setting up Kubernetes clusters, you might want to monitor and optimize the workload performance when setting up SLOs. However, if you’re part of the application team, the usage of reserved resources might differ significantly from the blocked resources

Monitoring your Kubernetes cluster allows you to proactively identify and resolve resource constraints, failures, and crashes before they impact end-user experience and your business.

Proper Kubernetes monitoring includes utilizing observability information to optimize your environment. By gaining insights into how your Kubernetes workloads utilize computing and memory resources, you can make informed decisions about how to size and plan your infrastructure, leading to reduced costs. A Kubernetes SLO that continuously evaluates CPU, memory usage, and capacity and compares these available resources to the requested and utilized memory of Kubernetes workloads makes potential resource waste visible, revealing opportunities for countermeasures.

Regarding resource utilization in a Kubernetes environment, there are two main perspectives to consider based on the primary stakeholders involved. One perspective focuses on the potential for optimization at the interface between the team responsible for managing the Kubernetes cluster and the application teams that are responsible for developing and deploying applications.

If your team is responsible for setting up Kubernetes clusters, you might want to monitor and optimize the workload performance when setting up SLOs. However, if you’re part of the application team, the usage of reserved resources might differ significantly from the blocked resources.

Requests represent the number of resources reserved or blocked for a container. Tracking the ratio between request and usage can provide valuable insights into optimization potential. As every container has defined requests for CPU and memory, these indicators are well-suited for efficiency monitoring.

Optimize memory and CPU utilization of your Kubernetes namespaces

One option is continuously tracking the memory and CPU utilization efficiency of existing Kubernetes objects, such as namespaces or workloads. Since teams typically run multiple workloads on namespaces, using this level of abstraction is a suitable option for a Kubernetes SLO. However, if you require more granular information, you can adjust the levels for resource utilization monitoring accordingly.

To calculate the service-level indicator for the Kubernetes namespace memory efficiency SLO, simply query the memory working set and request the memory metrics that are provided out of the box.

timeseries {
      memWorkSet=sum(dt.kubernetes.container.memory_working_set, default:0, rollup:avg)
      , memRequest=sum(dt.kubernetes.container.requests_memory, rollup:avg)}
      , nonempty:true, by:{dt.entity.cloud_application_namespace} 
      , filter: IN(dt.entity.cloud_application_namespace, "YOUR-NAMESPACE-ENTITY-ID")
| fieldsAdd sli = memWorkSet[]/memRequest[]*100
| fieldsRemove memWorkSet, memRequest

This DQL query can be inserted into the new SLO app on the Dynatrace platform, thereby supporting native Dynatrace Grail™ access.

The same procedure is applicable for measuring the computing resources of a single namespace.

timeseries {
      cpuUsage = sum(dt.kubernetes.container.cpu_usage, default:0, rollup:avg)
      , cpuRequest = sum(dt.kubernetes.container.requests_cpu, rollup:avg)}
      , nonempty:true, by:{dt.entity.cloud_application_namespace}
      , filter: IN(dt.entity.cloud_application_namespace, "YOUR-NAMESPACE-ENTITY-ID")
| fieldsAdd sli = cpuUsage[]/cpuRequest[]*100
| fieldsRemove cpuUsage, cpuRequest

These Kubernetes SLOs measure the ratio of used memory and computing resources relative to the requested memory and resources for an entire namespace. They provide insights into how efficiently the blocked resources are utilized. Because requested resources can’t be used elsewhere, the objective is to keep the difference between requested and used resources as small as possible. Utilizing such SLOs for your Kubernetes environment makes it possible to track efficiency transparently over time. The teams responsible for the cluster and the application teams that run their containers on the cluster can agree on the intended ratio between used and requested memory and CPU.

Optimize your Kubernetes cluster’s resource allocation

Measuring and optimizing the efficient usage of resources relative to resource requests is just one aspect of the full picture. As cluster owners must allocate cloud resources to meet their application team’s resource requests, it’s recommended that they measure the efficient provisioning of allocated resources over the entire Kubernetes cluster.

For cluster owners, monitoring resource usage at the node level provides better insights and information for taking sound actions. By tracking resource usage at each node, teams can gain insights into how many resources the entire cluster uses and whether the nodes work correctly and efficiently.

Teams should implement suitable SLOs to continuously monitor resource usage for the entire Kubernetes cluster. These SLOs should cover metrics like node memory utilization, which involves monitoring the ratio of requested versus allocated resources or the ratio of desired versus running pods per node.

For instance, if a node’s memory is high, it can lead to the undesired deletion of pods. This, in turn, can disrupt an application or service and incur additional costs for potential cluster upscaling.

A possible Kubernetes SLO for monitoring and evaluating the ratio of requested versus allocated memory can be expressed as follows:

timeseries {
      requests_memory = sum(dt.kubernetes.container.requests_memory, rollup:avg),
      memory_allocatable = sum(dt.kubernetes.node.memory_allocatable,rollup:avg)}
      , by:{dt.entity.kubernetes_cluster}
      , filter: IN (dt.entity.kubernetes_cluster, “YOUR-CLUSTER-ENTITY-ID”)
| fieldsAdd sli = (requests_memory[] / memory_allocatable[]) * 100
| fieldsRemove requests_memory, memory_allocatable

This SLO tracks the CPU utilization of a Kubernetes cluster:

timeseries {
      cpuUsage = sum(dt.kubernetes.container.cpu_usage, default:0, rollup:avg)
      , cpuRequest = sum(dt.kubernetes.container.requests_cpu, rollup:avg)}
      , nonempty:true, by:{dt.entity.cloud_application_namespace}
      , filter: IN(dt.entity.cloud_application_namespace, “YOUR-CLUSTER-ENTITY-ID”)
| fieldsAdd sli = cpuUsage[]/cpuRequest[]*100
| fieldsRemove cpuUsage, cpuRequest

In an ideal environment, the requested resources match the allocated resources, and hence, cloud resources will be used optimally, reducing the costs of invoiced but unused resources.

Ensure overall Kubernetes resource utilization efficiency

The four SLOs mentioned in this article provide valuable insights into the overall utilization efficiency of resources in a Kubernetes cluster and are a great starting point for optimizing them. Although both views (usage vs. request and request vs. allocation) must be analyzed together to obtain a holistic view, the split is extremely helpful in ensuring accountability. While the application teams have complete control over the usage/request SLO, the cluster owner can influence the request vs. allocation SLO. The SLOs serve as a simple way to recognize if the utilization over a certain period is on track or if resources are utilized inefficiently.

Though setting up proper SLOs to make these ratios visible and transparent does not reduce the need for collaboration and alignment, it provides a solid foundation for optimizing resource utilization efficiency.

The outlined SLOs for Kubernetes clusters guide you in implementing SRE best practices in monitoring your Kubernetes environment. By recognizing the insights provided, you can optimize processes and improve overall efficiency. This makes this an excellent solution for collaboration between contributors while holding the respective parties accountable.

What’s next with SLOs for Kubernetes clusters?

As getting started with efficiency and optimization efforts within a large environment is always tricky, you might ask, “Where should I start?” Dynatrace can help you here with all the insights the Dynatrace Kubernetes app provides out of the box. For more information about identifying and optimizing resources, go to the Optimize workload resource usage with Kubernetes app and Notebooks use case in Dynatrace Documentation. Also, visit the Kubernetes Dynatrace playground, where you can play around with the Kubernetes app on the Dynatrace platform. Finally, to try out the SLO examples described in this article, visit the Dynatrace Service-Level Objectives playground.

After identifying SLOs, stakeholders can begin measuring the agreed-upon resource utilization between clusters and namespaces. The ability to create and monitor SLOs will soon be natively integrated into the Kubernetes app, with dedicated templates, to quickly establish common objectives for resource utilization in a few clicks without needing to write DQL queries. These out-of-the-box templates and their visualization within the Kubernetes app will be available early next year.

Further best practices include adding dedicated ownership information to all Kubernetes objects so that responsible teams can be notified when anomalies or degradation are detected.

If you want to read and learn more about how cloud-native observability and automation can be achieved with Dynatrace, take a look at Michael Winkler’s recently published blog post, Cloud-native observability made seamless with OpenPipeline.

The post Kubernetes service-level objectives: Optimize resource utilization of Kubernetes clusters with SLOs appeared first on Dynatrace news.

An at-a-glance view of your system’s health and performance

Dynatrace guides you in quickly getting the most valuable service level objectives (SLOs) set up in just a few clicks. To provide maximum freedom in selecting the service-level indicators that matter most to your business, Dynatrace combines SLOs with the power of Dynatrace Grail™ data lakehouse, the central data platform with heterogeneous and contextually linked data. Service-level objectives now utilize highly flexible Dynatrace Query Language (DQL) for all Grail data sources. This lets you build your SLOs around the indicators that matter to you and your customers—critical metrics related to availability, failure rates, request response times, or select logs and business events.

Heterogeneous services require heterogeneous indicators

Metrics, logs, and traces are the core ingredients for making your environment observable. Depending on the environment, the different information types provide indicators that reveal potential problems for your customers.

Having insights into the context and dependencies of such heterogeneous data is essential for seeing the forest, not just the trees. This is where Grail, the Dynatrace central data platform, excels. Grail provides access to contextually linked information with just a few query lines.

It doesn’t matter if you need typically used failure-rate or response-time metrics to ensure your system’s availability and performance or if you need to rely on abnormal log drops to gain insights into raising problems—SLOs leveraged with Grail provide all the information you need.

Are you defining SLOs based on the four golden signals of a critical service? No problem—out-of-the-box templates simplify the setup of time series data based on service requests, and templates can quickly be adapted for more fine-grain needs, for example, to consider only single endpoints.

Do you need to see if selected logs or loglines are missing over time? It’s easy—just use log data, extract the desired information, and create an SLO based on the log format.

Are you experiencing an increase or degradation in certain events that indicate a rising problem? It doesn’t matter if the events are ingested business-related events like deal closures or simple login numbers automatically captured by Dynatrace OneAgent®. Fetch your events, target critical components in your environment, and get notified proactively if your required objectives are at risk.

If you can query something using DQL, you can use it as an indicator for your SLO. The only prerequisite is that the result of the SLI needs to be a time series named sli. The make time series command handles how time series are created from other data types.

Grail, the Dynatrace central data platform, provides the needed information and adds context. With the recently released new SLO capability, you can use Grail and all its information to specify selected SLOs to ensure your critical components meet your expectations now and in the future.

Having service-level objectives for leading and lagging indicators provides a more holistic picture of the status of emerging issues. Setting the SLO on the data types that matter provides a more tailored view of the performance stats and helps limit the number of SLOs to critical items and the most impactful indicators.

SLOs on events, metrics, and logs: How to bring SLOs to the next level

Leveraging Grail, SLOs offer an immersive opportunity to focus your SLOs on user expectations, business goals, and core user journeys. When getting started, this can seem overwhelming, and it’s easy to get lost in the sea of opportunities.

With the latest release, Dynatrace introduces convenient ways to give you a head start with SLOs on all data types. If you have already worked with classic Dynatrace SLOs, most steps will be familiar and easy to follow.

The latest Dynatrace SLO implementation relies on the existing SLO creation wizard templates and provides guided workflows for setting up SLOs with the most commonly used SLIs: service availability, service performance, and CPU utilization.

In the coming weeks and months, we will add to the current collection of templates for synthetic monitoring, digital experience management measures, Kubernetes resource optimization, and infrastructure monitoring. However, all of these can be created today using DQL queries. Simply extract and manipulate your desired time series from Grail within a notebook and copy the query into the SLO definition to get your customized SLO.

Service-level objectives come with a dedicated management overview page, where existing SLOs can be reviewed and modified as needed.

The SLO wizard can be called directly from the management overview or the dashboard using a dedicated shortcut.

An added tagging option simplifies how you interact with your SLOs. Adding ownership information or relevant application boundaries as additional metadata to your SLOs makes it easier for all stakeholders to react or reach out to the appropriate teams in case performance degradations or dependencies are identified.

Dedicated management makes it easy to maintain and run your SLOs, while highly customizable dashboard tiles allow you to integrate SLOs with your health and performance overview stats.

SLOs provide immediate information about critical services’ long-term performance and experiences. Hence, having a dedicated dashboard tile visualizing the key parameters of each SLO simplifies the process of evaluating them.

Automate your SLOs

To manage and evaluate your SLOs at scale, Dynatrace allows you to create, manage, and evaluate your SLOs either via the Dynatrace web UI or API. To implement SLOs in your software delivery cycle and consistently add observability measures from the beginning, Dynatrace configuration as code (Monaco and Dynatrace Terraform) will soon support the new API.

The next step will be improved error-budget burn rate alerting, which Dynatrace Davis® AI root cause analysis can recognize, detecting potential threats to your SLOs before they affect your customers. Additionally, the Dynatrace Automation Engine will leverage SLO alerts to create event-triggered workflows to inform relevant stakeholders, provide reports, or automatically kick off remediation activities.

What’s next

Dynatrace SLOs leverage the power of Grail, providing dedicated SLO management, a user-friendly creation wizard with templates, improved dashboarding options, and an API to interact with the new generation of SLOs. While the SLO management web UI and API are already available, the dashboard tile will be released within the next weeks.

Subsequent improvements will include improved error-budget burn rate alerting and a notification option to proactively inform you when an SLO is at risk. Furthermore, SLOs will find their place within all areas in Dynatrace, making it even easier to create SLOs on demand. At the same time, dedicated configuration-as-code support in Monaco and Terraform will provide a scalable, automated solution.

Get started

Measure what matters most to your business and end-users. Set up your first SLOs and gain insights into relevant and important processes.

We’re eager to hear your experience, feedback, and ideas for improvement. A dedicated feedback channel is set up in the Dynatrace Community. We look forward to hearing from you. Feedback channel for the new service-level objectives app – Dynatrace Community

The new SLOs are available within all Dynatrace SaaS environments and ready for you to try out, beginning with Dynatrace SaaS version 1.303. The new dashboard tiles will be available with version 1.304.

Start setting up your SLOs on indicators that matter to you and your customers

We created a few examples on our Dynatrace Playground tenant so that you can experience the potential and flexibility of SLOs leveraging DQL. You’ll find SLO examples for different data types, such as metrics, events, or logs. You’ll also find examples for entity types like services, synthetic monitors, Kubernetes clusters, and namespaces, and you’ll learn how to use DQL to filter for a selected group of entities or remove specific time periods from SLO-status calculations. Additionally, we set up a sample dashboard that presents an overview of selected critical services and their combined health and performance status aggregated over time.

The post Reliability indicators that matter to your business: SLOs for all data types appeared first on Dynatrace news.

Dynatrace introduced numerous powerful features to its Infrastructure & Operations app, addressing the emerging requirement for enhanced end-to-end infrastructure observability. The newly introduced Network devices and Details view within Hosts provide comprehensive health status information, relevant networking signals, and machine metrics—all analyzed and provided by the industry-leading combination of Dynatrace Grail™ data lakehouse and Davis^® AI.

These enhancements are designed to empower IT operations and SRE teams with more comprehensive visibility and increased efficiency at any time.

Let’s explore these exciting new features and see how they elevate infrastructure management.

^{Figure 1. Infrastructure & Operations app showing manual root cause and impact analysis}

Advanced filter components for powerful enterprise-grade search and visualization

One of the standout enhancements of the latest update is the improved filter components, designed to facilitate more complex and advanced searches across cloud, on-premises, or hybrid infrastructure hosting environments, including support for custom tags.

Whether you’re looking to zero in on specific hosts, operating system and application processes, or devices, the improved filters enable high-precision searches without sacrificing the operational performance of the Infrastructure & Operations app.

Specifically, background filter queries and visualizations have been optimized to ensure fast response times for complex and large enterprise datasets. A critical update that further enhances this experience is the increase in visual limits—now, up to 100,000 host elements can be displayed within the resulting filtered view.
You can interactively sort, filter, and reorder these host elements with no query delay. This means quicker access to insights, allowing IT teams to act faster during timely and critical incidents.
Whether you’re looking to swiftly review performance trends across thousands of cloud-native resources or your goal is to investigate a single component within your environment, Infrastructure & Operations provides the details you’re looking for. Speaking of single elements and resources brings us to the next enhancement:

Host detail view: Get the complete picture and deep dive into host health

The recently introduced host detail view has elevated manual health reviews and in-depth analysis to the next level. You can now explore related insights and AI-powered analytics within a single host detail view. This view offers DevOps and engineers metrics related to health, security status, service events, and service level objectives (SLOs), paired with quick access to drill deep into logs or process-level insights.

As shown in Figure 3, the host details view is accessible directly from the list of host resources in Infrastructure & Operations and from within the overview pane.

To increase efficiency and shorten operations cycles, the current filtered host detail view can be shared effortlessly by copying the URL in the browser’s address bar. Teams can share the current view via Teams, Slack, or any other messenger and include the current view and findings as part of a support ticket.

In cases where rich details of process and service level consumption are required, the host details provide you with a Processes page focused on utilization metrics and troubleshooting insights.

Technology-type tags can be added at the process and process group levels, splitting the technology tagging for hosts into main and sub-technologies. This simplifies filtering and automation efforts.

When deep monitoring is activated, drilling into the process detail view shows versioning and release information. Custom tags on the application level allow you to filter more easily while simultaneously enhancing automation capabilities.

A new feature makes it easier for IT teams managing large-scale environments to stay on top of every detail across any infrastructure or region. Now, with more granular insights into process performance and technology versions, teams can quickly review and detect anomalies, prioritize problems reported by Davis AI, and maintain SLOs and compliance—all in one place: Infrastructure & Operations.

Host networking metrics and visualization

Network performance bottlenecks are one of the most common troubleshooting challenges in large infrastructures. The new Host Detail view also introduces improved networking insights to streamline diagnostic efforts, including a summarized view of key network metrics, TCP packet transmission statistics, retransmit errors, and packet loss.

These metrics are critical for understanding network health on the application and host level, isolating issues related to network connectivity or connection outages.

This improvement directly helps DevOps and InfraOps teams by reducing the time spent identifying network-related issues in cross-functional efforts.

Network devices view: Extend end-to-end infrastructure observability

In the daily life of DevOps, InfraOps, or SRE, one of the most common routines and troubleshooting challenges in large infrastructures is staying ahead of network performance bottlenecks.

With SD-WAN appliances, firewalls, load balancers, routers, switches, and other network devices in place, this task is becoming quite challenging yet crucial.

Infrastructure & Operations features a newly introduced Networking Devices section, which provides networking visualizations to streamline these efforts within one place.

With extensions available for Cisco, Palo Alto, NetScaler, Juniper, F5, Fortinet, and HAProxy available within Dynatrace Hub, the Network devices section instantly provides value to teams. Existing, configured extensions provide metrics and Davis AI-powered analytics without additional configuration efforts. Networking devices and appliances not yet onboarded can swiftly be integrated via SNMP integration or technology-specific extensions, as highlighted above.

This new capability dramatically simplifies monitoring signals like ethernet interface availability status and network utilization metrics alongside hardware metrics and consumption. Dynatrace Davis AI plays an integral role in this newly introduced section by automatically identifying and raising problems that might otherwise go unnoticed. It enables IT teams to proactively manage network performance bottlenecks or outages quickly, potentially before these issues impact services and users.

Dynatrace: Powering the next level of end-to-end infrastructure observation

The continued integration of Grail™ data lakehouse and Davis AI into Infrastructure & Operations, alongside new features focusing on networking devices, show the Dynatrace commitment to providing IT teams with unmatched and holistic end-to-end infrastructure observability empowered by AI-driven insights.

Together, these technologies enhance the ability of DevOps and InfraOps teams to manage the growing complexity of modern IT environments within a single platform and with a single app—Dynatrace Infrastructure and Operations.

With AI-driven root cause analysis, IT teams no longer need to manually correlate logs, metrics, and traces. Instead, the platform autonomously identifies the root causes of issues, minimizing downtime and freeing up valuable time for more strategic initiatives.

Networking specialists or architects who desire to surface the corresponding log lines are provided with pre-filtered query views, offering service-specific log information and surrounding logs of related components.

Engineers, developers, and power users are provided with the all-new Logs app, which we introduced recently in our blog post Simplicity meets power: Introducing the all-new Dynatrace Logs app.

Beyond the two apps mentioned, Davis CoPilot integration supports novice Dynatrace users by translating questions from natural language into Dynatrace Query Language (DQL) when requests become more sophisticated

Kickstart the new era of AI-powered end-to-end infrastructure observation with Dynatrace today!

Discover how the new and enhanced Infrastructure & Operations app can transform your daily routines and troubleshoot workflows to drive better business outcomes.

Existing Dynatrace customers with licenses that include Infrastructure Observability have automatic access to the new app and all other features available with Dynatrace SaaS; you don’t need to take any action other than to start using Infrastructure & Operations and its features.

Explore the latest updates on the Dynatrace platform by starting a free trial or requesting a live demonstration and taking a custom-tailored tour.

The post Supercharge your end-to-end infrastructure and operations observability experience appeared first on Dynatrace news.

The previous blog post in this series discussed the benefits of implementing early observability and orchestration of the CI/CD pipeline using Dynatrace. This approach enhances key DORA metrics and enables early detection of failures in the release process, allowing SREs more time for innovation. This blog post explores the Reliability metric, which measures modern operational practices.

Why reliability?

DORA introduced the Reliability metric as “the fifth DORA metric” to address the fact that many organizations improved release times and deployment frequency without adequately considering the reliability of new releases. These releases often assumed ideal conditions such as zero latency, infinite bandwidth, and no network loss, as highlighted in Peter Deutsch’s eight fallacies of distributed systems. To enhance reliability, testing the software under these conditions is crucial to prepare for potential issues by leveraging chaos engineering or similar tools. Chaos engineering is a practice that extends beyond traditional failure testing by identifying unpredictable issues. While it is powerful, it presents several challenges that affect its adoption. In this blog post, we delve into these challenges and explore how Dynatrace can address them to enhance the reliability of released software.

Challenges

Limited awareness of cross-service interactions

In chaos engineering, “hypothesis” denotes the expected behavior of an application under defined conditions or stressors, whether within their owned service or across services. It forms the cornerstone of chaos engineering experiments. In distributed or microservices environments, application teams often lack visibility into how their service will perform under diverse conditions across other services or the entire system. This limitation affects the quality of hypotheses, leading to frequent revisions or inaccurate assumptions.

“It’s not about chaos – it’s about reliability” – Andrus Kolton, CTO & founder of Gremlin

Dynatrace addresses this challenge by offering comprehensive observability across the entire environment. With Dynatrace, teams can seamlessly monitor the entire system, including network switches, database storage, and third-party dependencies. This visibility can be cross-checked in real time using features like Smartscape topology or Service Flow. By leveraging these resources, SREs can formulate more informed hypotheses based on the behavior of their service or application.

Insights: Unclear starting system state

Along with hypotheses, it is imperative to have a clear starting state to validate the experiments’ success. These are referred to as baseline metrics for the starting state, and without appropriate insights into the application ecosystem, it is difficult to form hypotheses. Such baselines constitute a few metrics like:

1. What are the top five problems in your application – CPU spikes, slow response, database connections bottleneck, etc.
2. The problems that take maximum time to resolve – lowest MTTR.
3. Impact of fewer resources, for example, CPU and disk, available to different services and applications.
4. More critical services that are likely to take other services down.

For instance, the above dashboard offers visibility into service and distributed systems through different lenses, such as the slowest MTTR, top problems, or services frequently impacting others. This data simplifies establishing the starting state for chaos experiments or determining the priority services requiring reliability tests and the specific reliability standards to test against.

Blast radius and risk of prolonged outages

While tests in pre-production environments provide valuable insights, proper validation occurs when chaos experiments are conducted in the production environment. However, SREs are hesitant to use chaos engineering during testing or validation stages due to the risk of incorrect hypotheses, potentially impacting multiple services in the production system and leading to prolonged outages. Therefore, their primary concern is to avoid such outages and unnecessary damage to the production system, leading to low adoption of chaos engineering and hindering efforts to improve reliability.

These outages can be mitigated by limiting the scope of chaos engineering tests, known as the “blast radius,” and gradually expanding it as impacted entities are correctly identified and fixed. Dynatrace automatically discovers all components and dependencies within complex technology stacks end-to-end, identifying billions of causal dependencies across websites, applications, services, processes, hosts, networks, and infrastructure within minutes. With this comprehensive monitoring, SREs can utilize holistic monitoring, where situational awareness and Davis® AI-based alerting transition from correlation to causation-based analysis. This enables different teams to understand the semantics of cascading problems, or the domino effect, and pinpoint the root cause for actionable fixes.

In the screenshot below, a chaos engineering scenario introduced latency and resource stress on the “easytrade” demo application. This scenario introduces a 7-second delay for random requests and causes CPU stress on the container. This helps the app team observe the application’s behavior under these conditions and identify any bottlenecks.

In the video below, Davis AI identifies the interdependencies and pinpoints that database slowness impacted multiple services, identifying it as the root cause. Additionally, the details of the new deployment and chaos experiment were included in the root cause analysis. By leveraging service tools and visual resolution, developers can work towards building robust and reliable services. Similarly, by gradually increasing the blast radius, developers can enhance the reliability of the entire application.

Lastly, the SRE team can leverage Dynatrace workflows to automate outages, ensuring virtually no downtime for services or applications.

Conclusion

Ensuring builds are created accurately the first time is crucial, even if it means testing them by injecting failures or remote scenarios. This enhances build reliability and equips SREs and operations teams to swiftly resolve potential issues that could lead to outages. As John Allspaw said, “Incidents are unplanned investments; their costs have already been incurred. Your org’s challenge is to get ROI on those events.”

With Dynatrace advanced monitoring capabilities, SRE teams can confidently overcome challenges and embrace Chaos Engineering. This ensures optimal performance and reliability, enabling organizations to confidently navigate today’s dynamic IT landscapes.

What’s next

In this blog series’ final installment, we’ll explore how Davis AI root cause analysis can be used to autonomously manage your application’s day-to-day operations with the help of Dynatrace workflows.

The post Build systems more reliably with Dynatrace: Chaos Engineering appeared first on Dynatrace news.

In the previous blog post of this series, we discussed the crucial role of Dynatrace as an orchestrator that steps in to stop the testing phase in case of any errors. Additionally, Dynatrace equips SREs and application teams with valuable insights powered by Davis^® AI. In this blog post of the series, we will explore the use of Site Reliability Guardian (SRG) in more detail.

SRG is a potent tool that automates the analysis of release impacts, ensuring validation of service availability, performance, and capacity objectives throughout the application ecosystem by examining the effect of advanced test suites executed earlier in the testing phase.

Validation stage overview

The validation stage is a crucial step in the CI/CD (Continuous Integration/Continuous Deployment) process. It involves carefully examining the test results from the previous testing phase. The main goal of this stage is to identify and address any issues or problems that were detected. Doing so reduces the risk of production disruptions and instills confidence in both SREs (Site Reliability Engineers) and end-users. Depending on the outcome of the examination, the build is either approved for deployment to the production environment or rejected.

Challenges of the validation stage

In the Validation phase, SREs face specific challenges that significantly slow down the CI/CD pipeline. Foremost among these is the complexity associated with data gathering and analysis. The burgeoning reliance on cloud technology stacks amplifies this challenge, creating hurdles due to budgetary constraints, time limitations, and the potential risk of human errors. Additionally, another pivotal challenge arises from the time spent on issue identification. Both SREs and application teams invest substantial time and effort in locating and rectifying software glitches within their local environments. These prolonged processes not only strain resources but also introduce delays within the CI/CD pipeline, hampering the timely release of new features to end-users.

Mitigate challenges with Dynatrace

With the support of Dynatrace Grail™, AutomationEngine, and the Site Reliability Guardian, SREs and application teams are assisted in making informed release decisions by utilizing telemetry observability and other insights. Additionally, the Visual Resolution Path within generated problem reports helps in reproducing issues in their environments. The Visual Resolution Path offers a chronological overview of events detected by Dynatrace across all components linked to the underlying issue. It incorporates the automatic discovery of newly generated compute resources and any static resources that are in play. This view seamlessly correlates crucial events across all affected components, eliminating the manual effort of sifting through various monitoring tools for infrastructure, process, or service metrics. As a result, businesses and SREs can redirect their manual diagnostic efforts toward fostering innovation.

1. Configure an action for the Site Reliability Guardian in the workflow. The action should focus on validating the guardian’s adherence to the application ecosystem’s specific objectives (SLOs). Additionally, align the action’s validation window with the timeframe derived from the recently completed test events.
   
2. As the action begins, the Site Reliability Guardian (SRG) evaluates the set objective by analyzing the telemetry data produced during advanced test runs. At the same time, SRG uses DAVIS_EVENTS to identify any potential problems which could result in one of two outcomes.
   
   Outcome #1: Build promotion
   Once the newly developed code is in line with the objectives outlined in the Guardian—and assuming that Davis AI doesn’t generate any new events—the SRG  action activates the successful path in the workflow. This path includes a JavaScript action called promote_jenkins_build, which triggers an API call to approve the build being considered, leading to the promotion of the build deployment to production.
   
   Outcome #2: Build rejection
   If Davis AI generates any issue events related to the wider application ecosystem or if any of the objectives configured from the defined guardian are not met, the build rejection workflow is automatically initiated. This triggers the disapprove_jenkins_build  JavaScript action, which leads to the rejection of the build.
   
   Moreover, by utilizing helpful service analysis tools such as Response Time Hotspots and Outliers, SREs can easily identify the root cause of any issues and save considerable time that would otherwise be spent on debugging or taking necessary actions.  SREs can also make use of the Visual Resolution Path to recreate the issues on their setup or identify the events for different components that led to the issue. In both scenarios, a Slack message is sent to the SREs and the impacted app team, capturing the build promotion or rejection.The telemetry data’s automated analytics, powered by SRG and Davis AI, simplify the process of promoting builds. This approach effectively tackles the challenges that come with complex application ecosystems. Additionally, the integration of service tools and Visual Resolution Path helps to identify and fix issues more quickly, resulting in an improved mean time to repair (MTTR).

Validation in the platform engineering context

Dynatrace—essential within the realm of platform engineering—streamlines the validation process, providing critical insights into performance metrics and automating the identification of build failures. By leveraging SRG and Visual Resolution Path, along with Davis AI causal analysis, development teams can quickly pinpoint issues, and further rectify them ensuring a fail-smart approach. The integration of service analysis tools further enhances the validation phase by automating code-level inspections and facilitating timely resolutions. Through these orchestrated efforts, platform engineering promotes a collaborative environment, enabling more efficient validation cycles and fostering continuous enhancement in software quality and delivery.

In conclusion, the integration of Dynatrace observability provides several advantages for SREs and DevOps, enabling them to enhance the key DORA metrics:

• Deployment Frequency: Improved deployment rate through faster and more informed decision-making. SREs gain visibility into each stage, allowing them to build faster and promptly address issues using the Dynatrace feature set.
• Change Lead Time: Enhanced efficiency across stages with Dynatrace observability and security tools, leading to quicker postmortems and fewer interruption calls for SREs.
• Change Failure Rate: Reduction in incidents and rollbacks achieved by utilizing “Configuration Change” events or deployment and annotation events in Dynatrace. This enables SREs to allocate their time more effectively to proactively address actual issues instead of debugging underlying problems.
• Time to restore service: While these proactive approaches can help improve Deployment Frequency and Change Lead Time, telemetry observability data with Dynatrace AI causation engine Davis AI can aid in improving Time to restore service.

In addition, Dynatrace can leverage the events and telemetry data that it receives during the Continuous Integration/Continuous Deployment (CI/CD) pipeline to construct dashboards. By using JavaScript and DQL, these dashboards can help generate reports on the current DORA metrics. This method can be expanded to gain a better understanding of the SRG executions, enabling us to pinpoint the responsible guardians and the SLOs managed by various teams and identify any instances of failure. Addressing such failures can lead to improvements and further enhance the DORA metrics. Below is a sample dashboard that provides insights into DORA and SRG execution.

In the next blog post, we’ll discuss the integration of security modules into the DevOps process with the aim of achieving DevSecOps. Additionally, we’ ll explore the incorporation of Chaos Engineering during the testing stage to enhance the overall reliability of the DevSecOps cycle. We’ll ensure that these efforts don’t affect the Time to Restore Service turnaround build time and examine how we can improve the fifth key DORA metric, Reliability.

What’s next?

Curious to see how it all works? Contact us to schedule a demo and we’ll walk you through the various workflows, JavaScript tasks, and the dashboards discussed in this blog series.

If you’re an existing Dynatrace Managed customer looking to upgrade to Dynatrace SaaS, see How to start your journey to Dynatrace SaaS.

The post Automate CI/CD pipelines with Dynatrace: Part 4, Validation stage appeared first on Dynatrace news.

In the last blog post of this series, we delved into how Dynatrace, functioning as a deploy-stage orchestrator, solves the challenges confronted by Site Reliability Engineers (SREs) during the early of automating CI/CD processes. Having laid the foundation during the deployment stage, we’ll now explore the benefits of Dynatrace visibility and orchestration during the testing phase.

The testing stage plays a crucial role in ensuring the quality of newly built code through the execution of automated test cases. Testing includes integration tests, which assess whether the code functions as intended when interacting with other services and application functionalities. It can also include performance testing to determine if the application can effectively handle the demands of the production environment.

Mitigate challenges during the testing stage

Like the build stage, the testing stage is time-sensitive and can consume a significant amount of time for execution, in turn creating a substantial waiting period for SREs to determine the success or failure of tests and the need for retesting. This slow feedback and time spent rerunning tests can hinder the overall software deployment process.

To mitigate these challenges, building upon the groundwork established during the deploy phase, Dynatrace can effectively pinpoint any issues encountered during testing. Moreover, by configuring alert notifications through native features such as ownership and alerting profiles, teams can receive prompt alerts in the event of failures. This proactive strategy significantly minimizes wait times and empowers SREs to redirect their focus toward innovative endeavors.

The steps outlined below show you how to achieve such a proactive solution.

1. Pass annotation events to Dynatrace, leveraging the Ingest events API, at the beginning and end of each test. These events provide additional context to the Davis^® AI causation engine in case of issues and function as logic operators for the execution of advanced testing, such as soak, integration, or chaos engineering.
2. Integrate performance test tools with Dynatrace by adding headers to HTTP requests. Further, harness request attributes and telemetry data gathered from requests to gain insights into the requests.
3. (Optional) Set web request naming rules using the earlier configured request attributes to facilitate easy identification of requests across different releases or test suites.
   
4. During the execution of tests, the telemetry data generated by the newly built code is transmitted to Grail through OneAgent.
   The data is examined and can fall into one of the below two categories:

   If there are errors in the telemetry data

   If there are errors in the telemetry data, the task immediately marks the stage as problematic and triggers the process to stop the ongoing job. The stop_the_build task utilizes the CI/CD API (in this case, Jenkins) to halt the build. This task also retrieves additional job metadata to include in the generated message.

   In tandem with stopping the job, another task in the workflow, send_load_test_failure, is triggered: a Slack message is dispatched to the team responsible for initiating the job. The immediate alerting mechanism ensures that SREs are promptly informed about test errors, thereby circumventing the high wait times.

   If there are no errors and the outcome is successful

   If there are no errors reported in the data, the pipeline job will proceed to conduct integration and reliability tests, which will be validated in the final validation stage by the Site Reliability Guardian (SRG). During this phase, SRG will validate the test coverage to assess the impact of the newly introduced code and determine its suitability for promotion to the production environment.

Optional best practices

Optionally, the following best practices are recommended to achieve the most effective outcomes.

• During the testing stage, generate on-demand synthetic monitors to monitor the performance of the application as experienced by your end users in different geolocations.
• Generate SLOs, leveraging the Dynatrace API, that will be validated in the next stage to determine if the build should be rejected or promoted to production.

Test stage orchestration in a platform engineering context

Dynatrace, integral to platform engineering, offers invaluable visibility into test results and accelerates the test cycle by swiftly identifying and resolving underlying issues. Utilizing service tools like response hotspots and distributed traces, along with Davis causal AI, dev teams can achieve fast turnarounds in the event of test failures. Dynatrace root cause analysis enhances collaboration, providing a comprehensive solution for optimizing testing phases so that they allow for a more agile and productive development cycle, fostering a culture of continuous improvement and accelerating overall software delivery.

Also, to optimize the workflow for both SREs and application teams, leverage Dynatrace dashboard capabilities to expand your data visibility. This allows for easy comparisons of telemetry data across different releases and enables the utilization of service tools for in-depth analysis, ultimately contributing to service optimization.

Conclusion

In conclusion, this blog post examined how Dynatrace capabilities and telemetry data can enhance the success of the testing stage. In the next blog post in this series, we delve deeper into how Dynatrace can further streamline the promotion of new code to production.

The post Automate CI/CD pipelines with Dynatrace: Part 3, Testing stage appeared first on Dynatrace news.

In the previous installment of this blog series, we explored how to set up Dynatrace as a build-stage orchestrator to effectively address the challenges faced by Site Reliability Engineers (SREs). In this blog post, we’ll transition to the next pipeline stage, the Deploy stage, and examine the visibility advantages that Dynatrace provides during this critical phase.

Deploy stage

In the deployment stage, the application code is typically deployed in an environment that mirrors the production environment. This step is crucial as this environment is used for the final validation and testing phase before the code is released into production. This stage ensures the code meets the required quality standards before it goes live.

Configuration drift

Configuration drift is a challenge in the deployment stage. Configuration drift occurs when a staging environment’s configuration deviates from the production environment. Such deviations often result from undocumented or un-versioned environmental changes, potentially causing unexpected behavior and production outages upon deployment.

Even when the staging environment closely mirrors the production environment, achieving a complete replication of all potential scenarios, such as simulating extremely high traffic volumes to assess software performance, remains challenging. This can lead to a lack of insight into how the code will behave when exposed to heavy traffic.

Creating a staging environment that faithfully replicates the production setup is crucial to effectively addressing such challenges. Furthermore, augmenting test coverage to mirror the scenarios encountered in production is imperative. These strategies can play a vital role in the early detection of issues, helping you identify potential performance bottlenecks and application issues during deployment for staging. With this approach, you gain increased confidence that a release will succeed when promoted to the production environment.

Leverage OneAgent functionality

Ingesting configuration changes into Dynatrace through the events API call and utilizing OneAgent^® to detect configuration changes for supported technologies help maintain close alignment between your staging and production environments. This approach effectively combats configuration drift. Furthermore, the Dynatrace Davis^® AI engine can accurately identify whether such changes have resulted in any issues and allows you to roll back configurations if necessary.

By harnessing OneAgent native features, the generated telemetry data can be effectively combined with Davis AI prediction and forecasting capabilities. This enables the prediction of trends that might not have been thoroughly tested due to variations in different environments. This proactive approach proves instrumental in anticipating and mitigating potential performance issues before they result in adverse impacts.

Combat configuration drift

Dynatrace actively tracks modifications to the standard configuration files of supported technologies. Whenever a change is detected, Dynatrace automatically generates a Deployment change event for the corresponding process and the host on which the process runs. The predefined set of files monitored for configuration alterations is maintained within ruxitagentproc.conf. In the event of any incidents stemming from these configuration adjustments, Davis AI utilizes the Deployment change data to pinpoint potential root causes.

As an illustration, in the screenshot below, when a modification was made to the nginx.conf file, Dynatrace promptly detected the configuration change and reported it as a Deployment change event. In this specific instance, the misconfiguration of nginx led to a critical issue where nginx crashed and could not restart, prompting an alert from Dynatrace. Davis AI efficiently identified the deployment change as the potential root cause for the malfunctioning of nginx.

By harnessing these deployment events, SREs can effectively track all configuration changes that have occurred over a specified time. This capability empowers them to replicate the staging environment to match the production setup, proactively combating configuration drift.

Although Dynatrace can detect configuration changes, the best practice is to systematically ingest a configuration change or deployment change event each time the team modifies a configuration file. This approach ensures proper record-keeping and assists in achieving consistency between the staging and production environments, effectively mitigating configuration drift.

Predictive traffic analysis

Deploying OneAgent within the staging environment facilitates the availability of telemetry data for analysis by Davis AI. Davis AI can leverage this data to enable predictive analysis. This enables a scenario where—even if test cases can’t replicate identical traffic to production due to various limitations—predictive analysis can offer insights into the capacity of the new build to withstand production-level loads.

To illustrate this concept, consider the scenario below. During testing with limited generated traffic, Davis AI predictive analysis offered valuable insights into how Elastic Book Storage (EBS) might perform and how the rate of Input/Output Operations Per Second (IOPs) might change under heavier traffic load.

Other proven strategies

We recommend following best practices to achieve the most effective observability outcomes.

• At the onset of the deployment stage, send a deployment event to Dynatrace notifying it about the new deployment. These deployment events are important as they provide additional context to the Davis AI causation engine, aiding in evaluating deployment quality.
• To further aid with postmortem analysis, use the DT_TAGS metatag to clearly distinguish between different build and service requests.
• Additionally, consider using the DT_CUSTOM_PROP environment variable to include extra metadata about the build, providing valuable information for monitoring and analysis.

The framework outlined above provides a comprehensive view of the deployment process and facilitates comparisons across different releases. For instance, in the animation below, we utilized deployment events and DT_TAGS to assess the quality of releases and capture golden signals across various builds. This approach can be a valuable resource for developers in understanding underlying issues and identifying opportunities for enhancements.

Deploy stage orchestration in a platform engineering context

Deployment events automatically identified by OneAgent or manually triggered by app owners play a pivotal role in ensuring consistency across all environments. When issues are detected, causal AI determines if the problem is linked to deployment changes. This contributes significantly to increased productivity for development teams, as they spend less time troubleshooting and identifying discrepancies across environments. Additionally, the integration of predictive analysis, fueled by Davis AI, instills confidence in developers that their application’s behavior will withstand the anticipated load, including potentially high traffic.

Dynatrace as a deploy-stage orchestrator

You’ll find all the Jenkins pipeline and other scripts you need to set up Dynatrace as a deploy-stage orchestrator, including the referenced dashboard, in thisGit repository.

What’s next?

This blog post delved into how Dynatrace capabilities and telemetry data from a staging environment can effectively align staging with the production setup. In the next blog post in this series, we’ll further explore how Dynatrace can aid in streamlining the testing phase within an SRE pipeline.

The post Automate CI/CD pipelines with Dynatrace: Part 2, Deploy stage appeared first on Dynatrace news.

In the first blog post of this series, we explored how the Dynatrace^® observability and security platform boosts the reliability of Site Reliability Engineers (SRE) CI/CD pipelines and enhances their ability to focus on innovation. This blog post guides you through configuring Dynatrace to automate CI/CD processes to achieve these objectives.

Dynatrace observability architecture can be classified into three layers:

1. Orchestration (Dynatrace)
2. CI/CD toolset (Jenkins / Chef / Puppet / Bamboo, etc.)
3. Infrastructure layer (Kubernetes Cluster, GCP, Standalone server, etc.)

A conventional pipeline has the following stages:

1. Build
2. Deploy
3. Test
4. Validation

Now, let’s explore the challenges that arise in the different stages of the pipeline and examine how the role of Dynatrace as the orchestrator can contribute to overcoming these challenges. In this blog post, we’ll focus on the first stage of the pipeline, the Build stage.

Build stage

The build stage involves pulling the latest code, compiling it, and ensuring its proper functionality by running feature tests.

Mitigate challenges

While the application is building, you might see compilation problems that necessitate debugging, and there might be considerable delay before the root causes of such issues are identified. The build process must be restarted once a solution is identified and deployed. A high rate of build failures indicates that the team is investing significant time into diagnosing and rectifying issues. This waiting and problem-solving can hinder your overall software development process.

Enhancing visibility during the build stage can help identify compilation and integration errors earlier in the development cycle. This proactive approach reduces wait times and allows SREs to redirect their efforts toward innovation.

Leverage Dynatrace features

The instrumentation flow currently operates as follows:

1. After code commits, a GitHub action is employed to trigger the workflow, utilizing the workflow API. Within this workflow, a designated task, powered by JavaScript, initiates the build stage of the pipeline job, commencing a new build. The below screenshot shows the code initiating the Jenkins job.
   
2. During the job pipeline execution, logs are extracted and pushed to Grail. Log reading can be facilitated by installing OneAgent or employing a log ingestion system to ingest the logs into Dynatrace. Subsequently, a workflow DQL task named look_for_build_errors will scrutinize the build stage for errors.
3. Based on the outcome of the log analysis, one of two paths is pursued:
   1. If build errors are detected
      Should the job logs reveal errors, the task promptly flags the build as erroneous and initiates the process to halt the ongoing job. The task stop_the_jenkins_job leverages the CI/CD API (in this case, Jenkins) to stop the build, as indicated in the output from step 2. This task also extracts additional job metadata to incorporate in the generated alert.
      
      In tandem with stopping the job, another task in the workflow, send_alert_to_team, activates, sending a Slack message to the team responsible for initiating the job. The immediate alerting mechanism ensures that SREs are promptly informed about build errors, thereby circumventing high wait times.
      
   2. If no errors are detected
      If no errors are identified, the workflow seamlessly advances the job to the Deploy stage. As a good practice, we recommend that you incorporate a workflow with a JavaScript task that periodically retrieves job details and ingests them as business events to enhance visibility into the pipeline stages and provide insights into current trends.
      Above is a sample JavaScript workflow that extracts data from the Jenkins pipeline via API calls and the dashboard capturing current trends.The Jenkins Build Stage Insights dashboard provides visualizations of the build stages represented as business events.

Build stage orchestration in a platform engineering context

In the realm of platform engineering, the focus is on relieving the cognitive burden on development teams and boosting their efficiency by offering predefined pathways, essentially treating the development platform as a product. 

With build-log monitoring and the initiation of diverse workflows within Dynatrace based on log outcomes, automation is now a reality for many development teams. Furthermore, the insights gained from the build stage provide SREs and app teams with valuable information for optimizing existing pipelines and areas for improvement. 

What’s next

Now, it should be clear how Dynatrace orchestration can be introduced early in the pipeline process and how it can significantly decrease wait times during the build stage by notifying the teams of any errors during the application build process.

In the next blog post in this series, we’ll delve into how this orchestration further aids SRE teams in addressing the challenges encountered during the Deploy stage.

All the resources you need to set up Dynatrace as a build-stage orchestrator

In the meantime, you’ll find all the resources you need to set up Dynatrace as a build-stage orchestrator (workflows, JavaScript tasks, and the referenced dashboard) in this Git repository.

The post Automate CI/CD pipelines with Dynatrace: Part 1, Build stage appeared first on Dynatrace news.

Golden Paths for rapid product development

Modern software development aims to streamline development and delivery processes to ensure fast releases to the market without violating quality and security standards. DevOps practices have been established in the last decade to accomplish this goal and deal with the dynamics of modern, cloud-native software architectures. To bring these practices to life within an organization at scale, the discipline of platform engineering has gained popularity. From a high-level point of view, platform engineering aims to:

• Reduce the cognitive load on development teams.
• Improve reliability and resiliency of products that rely on platform capabilities.
• Accelerate product development and delivery by reusing and sharing platform tools and knowledge.
• Reduce risk of security, regulatory, and functional issues in products and services.
• Enable cost-effective and productive use of services.

While it takes multiple capabilities to achieve these goals, implementing “Golden Path” templates is a fundamental ingredient. The Cloud Native Computing Foundation defines a Golden Path as a “templated composition of well-integrated code and capabilities for rapid project development.” Simply put, a Golden Path is a self-service template for common tasks that allows for autonomy while providing guardrails that safeguard production environments.

Imagine that instead of development teams fending for themselves amidst a sea of tools and infrastructure, well-defined and enterprise-wide templates are provided for the development of all new product services. Such a template should contain a get-started tutorial, sample source-code framework, policy guardrails, CI/CD pipeline, infrastructure-as-code templates, and reference documentation. This approach helps you quickly integrate best practices within your organization and provides cloneable artifacts for rapid product development.

No developer is left in the dark to fend for themselves, Golden Paths light the way.

Ensure governance across your organization

While Golden Paths are key to bringing DevOps best practices to development teams, distributing them is challenging. This challenge is partially addressed by internal development platforms (IDP), which have been adopted by many, but not all, organizations.

Consequently, Dynatrace provides templates in the context they are needed—with or without the internal use of an IDP. This ensures governance across your organization with the proper templates in the right place. At the same time, this does not restrict you from managing them in your IDP, as explained below.

Leverage opinionated, self-service, and optional templates

The latest enhancements of the Site Reliability Guardian have introduced opinionated Golden Path templates. For now, they concentrate on Kubernetes, host, and security objectives but they will grow to cover even more reliability, performance, and security aspects. These templates enable development teams to instantiate a guardian for automating release validation in a self-service manner. Along the journey, monitored entities can be selected to provide the context to fetch the right data from Dynatrace Grail™.

After completing this two-step process, a ready-to-use guardian is created. It’s possible to add new objectives if needed or to tailor the objectives and their thresholds.

Allow for flexibility

Custom query variables are available to fine-tune guardian objectives and maintain flexibility in fetching data from Grail. An example of such a variable is the version number of a service. In many cases, you want to retrieve the logs, metrics, or traces of a particular version, which is unknown upfront. Consequently, the custom variable version can be defined in the DQL query, which retrieves its value before executing.

Custom query variables can be defined by starting with the $ sign followed by the variable name in the DQL query editor of a guardian objective. An overlay component then helps you add this variable and define the default value in the context of the guardian.

While a variable will fall back to its default if no value is provided, there are three ways of ingesting the value:

1. When triggering a guardian validation from within the UI by selecting the Validate button, set a value for the configured variables using the Set variables option.
2. The Site Reliability Guardian workflow action allows you to set a variable value. In this case, it is possible to consume data from the triggering event or a previous workflow action using Jinja expressions.
3. If a workflow is used to automate the validation process, the workflow trigger event can provide automatically mapped properties to variables. Therefore, the properties must be within the execution_context property, as shown below. Given this example, the version number 0.1.0 is set where the variable $version is defined.

Since variables provide important context for validation, you can view validation-result values over time. The following screenshots depict the version number associated with the last 10 validation results.

Integrate with existing internal developer platforms

If you have an internal developer platform that serves as a central product engineering hub, you need Golden Path templates incorporated there so that you can scale out the platform for use by different teams. With Dynatrace, this can be achieved by extracting an instantiated template using Dynatrace Configuration as Code or cloning it from our public repository. This declarative format of a template, consisting of a guardian and workflow configuration, as depicted by the sample below, can then be added to the template repository managed and distributed by an IDP.

What’s next?

Site Reliability Guardian is built to ensure and maintain reliability and resiliency of products by leveraging automated release validations. Therefore, Golden Path templates in a self-service manner are offered for platform engineers to enable development teams to get started easily. The next enhancements of the Site Reliability Guardian will bring Davis® AI even closer to the Site Reliability Guardian by recommending relevant objectives and baselines for comparison.

• The new functionality has already been released and is available for your use. If you’re using the Site Reliability Guardian already, update to the latest version. Otherwise, navigate to Dynatrace Hub and install it from there.
• If the Site Reliability Guardian is new to you and you’re curious about it, go to Dynatrace Discovery and explore its capabilities in a playground environment.

We’d love to hear your feedback about Site Reliability Guardian. Let us know your thoughts or share your ideas about further improvements.

The post Ensure safe and secure releases at scale by providing Golden Paths appeared first on Dynatrace news.

Establishing SLOs for Kubernetes clusters can help organizations optimize resource utilization. A good Kubernetes SLO strategy helps teams manage and make containerized workloads more efficient.

Kubernetes is a widely used open source system for container orchestration. It allows for seamless running of containerized workloads across different environments. However, managing Kubernetes optimally can be a daunting task due to its complex architecture. Properly monitoring a Kubernetes cluster or any related environment can be difficult.

Effective resource provisioning and management is a critical aspect of a Kubernetes cluster. It involves a coordinated effort among various stakeholders, including cluster owners, application teams, and business owners. The primary goal is to allocate sufficient resources to keep the applications running smoothly without overprovisioning and incurring unnecessary costs. This delicate balance requires a collaborative approach and transparent communication among all parties involved. Service-level objectives (SLOs) play a vital role in ensuring that all stakeholders have visibility into the resources being used and the performance of the applications.

SLOs are often used to monitor business-critical services and applications for customers. However, they can also be used to monitor optimization processes effectively. Essentially, SLOs track a selected service-level indicator (SLI) and continuously evaluate its behavior over a given timeframe against a fixed threshold. This feature is valuable for platform owners who want to monitor and optimize their Kubernetes environment. By considering historical behavior, SLOs provide an excellent way to track and evaluate optimization tasks. Users can continuously evaluate the system’s performance against predefined quality criteria, making SLOs for Kubernetes clusters a good option for monitoring and improving the system’s overall performance.

Efficient coordination among resource usage, requests, and allocation is critical. To optimize resource utilization in a Kubernetes cluster, all users must work together seamlessly. A Kubernetes SLO can serve as a transparent and trackable collaboration tool for different teams and collaborators.

Optimize memory utilization of your Kubernetes Namespaces

Monitoring your Kubernetes cluster allows for proactively identifying and resolving resource constraints, failures, and crashes before they impact the end-user experience and your business.

Proper Kubernetes monitoring includes utilizing observability information to optimize your environment. By gaining insights into how your Kubernetes workloads utilize computing and memory resources, you can make informed decisions about how to size and plan your infrastructure, leading to reduced costs. A Kubernetes SLO that continuously evaluates CPU, memory usage, and capacity and compares these available resources to the requested and utilized memory by Kubernetes workloads makes potential waste of resources visible, revealing opportunities for countermeasures.

When it comes to resource utilization in a Kubernetes environment, there are two main perspectives to consider based on the primary stakeholders involved. One perspective focuses on the potential for optimization at the interface between the team responsible for managing the Kubernetes cluster and the teams responsible for developing and deploying applications.

When setting up SLOs for Kubernetes clusters, it is important to choose the right metrics to track based on your objective. If your team is responsible for setting up the Kubernetes cluster, you might want to monitor and optimize the workload performance. However, if you’re part of the application team, the usage of reserved resources may differ significantly from the blocked resources.

Requests represent the number of resources being reserved or blocked for a container. Tracking the ratio between request and usage can provide valuable insights into optimization potential. As every container has defined requests for CPU and memory, these indicators are well-suited for efficiency monitoring.

One option is to continuously track the memory utilization efficiency of existing Kubernetes objects, such as namespaces or workloads. Since teams typically run multiple workloads on namespaces, using this level of abstraction is a suitable option for a Kubernetes SLO. However, if you require more granular information, you can adjust the levels for resource utilization monitoring accordingly.

This Kubernetes SLO measures the ratio between the requested memory and the memory used for an entire namespace. It provides insights into how efficiently the blocked resources are being utilized. Since resources that have been requested cannot be used elsewhere, the objective is to keep the difference between requested and used resources as small as possible. Utilizing such SLOs for Kubernetes clusters makes it possible to track efficiency transparently over time. The teams responsible for the cluster and the application teams that run their containers on the cluster can agree on the intended ratio between used and requested memory.

Optimize Kubernetes cluster’s resource allocation

One aspect of managing cloud resources is tracking and adjusting the ratio between the requested and used memory resources. On the other hand, the cluster owners must allocate cloud resources to meet the application team’s resource requests.

For cluster owners, monitoring resource usage at the node level provides better insights and information for taking sound actions. By tracking resource usage at each node, teams can gain insights into how many resources the entire cluster uses and whether the nodes are working correctly and efficiently.

Teams should implement suitable SLOs to continuously monitor resource usage for Kubernetes cluster nodes at the node level. These SLOs should cover metrics like node memory utilization, which involves monitoring the ratio of requested versus allocated resources or the ratio of desired versus running pods per node.

For instance, if a node’s memory or CPU utilization is high, it can lead to the undesired deletion of pods. This, in turn, can disrupt an application or service and incur additional costs for potential cluster upscaling.

A possible Kubernetes SLO for monitoring and evaluating the ratio of requested versus allocated memory can be expressed as follows:

In an ideal environment, the requested resources would almost match the allocated ones, and hence, cloud resources would be optimally used, reducing costs of invoiced but unused resources.

Ensure overall Kubernetes resource utilization efficiency

The two SLOs mentioned above provide valuable insights into the overall utilization of resources in a Kubernetes cluster and are a good starting point for improving efficiency. Although both interfaces (usage vs. request and request vs. allocation) must be analyzed together to obtain a holistic view, the split is extremely helpful in ensuring accountability. While the application teams have complete control over the usage/request SLO, the cluster owner can influence the request vs. allocation SLO.

Setting up proper SLOs to make these ratios visible and transparent does not reduce the need for collaboration and alignment, but it provides a solid foundation for optimizing resource utilization efficiency.

The outlined SLOs for Kubernetes clusters guide you in implementing SRE best practices in monitoring your Kubernetes environment. By recognizing the insights provided, you can optimize processes and improve overall efficiency. This makes it an excellent tool for collaboration between different contributors while also holding the respective parties accountable.

What’s next with SLOs for Kubernetes clusters?

Increase your resource utilization efficiency with Dynatrace using the SLOs mentioned in the blog. If you want to learn how to set up SLOs with Dynatrace, have a look here:

• Service level objective examples: 5 SLO examples for faster, more reliable apps
• Lessons learned from enterprise service-level objective management
• SLOs done right: how DevOps teams can build better service-level objectives

Or check the following Kubernetes blogs if you’d like to learn more about Dynatrace capabilities to increase efficiency and reliability in your Kubernetes environment:

• Dynatrace memory analysis helps Product Architects identify unknown unknowns
• How to maximize Kubernetes monitoring with AI-driven automation
• Kubernetes in the wild report 2023

The post SLOs for Kubernetes clusters: Optimize resource utilization of Kubernetes clusters with service level objectives appeared first on Dynatrace news.

In the realm of SRE, time and effort allocation planning are crucial factors that involve a delicate balance between operational management and project improvements. This intricate allocation strategy can be categorized into two main domains. In this blog post, we’ll delve deeper into these categories to gain a comprehensive understanding of their significance and the challenges they present.

Planned effort

Site Reliability Engineering (SRE) effort and time allocation planning typically fall into two domains:

• Operations Management (50%)
  Operations Management includes on-call responsibilities, post-mortem assessments, addressing other interruptions, and buffer time. These tasks collectively ensure uninterrupted production service.
• Process Improvements (50%)
  The allocation for process improvements is devoted to automation and continuous improvement SREs help to ensure that systems are scalable, reliable, and efficient. This improves the current project and paves the way for future innovation.

Reality

In practice, while both these categories have equal attention, project improvements hold paramount importance for business outcomes. SREs invest significant effort in enhancing software reliability, scalability, and dependability. Regrettably, recent reports indicate that SREs spend a substantial portion of their time addressing build issues and managing production incidents. This challenge escalates with the growing complexity of cloud systems and organizational aspirations for digital transformation, often leaving minimal time for substantial project improvements. Consequently, organizations grapple with various issues that impact software reliability.

Process Improvements

Organizations are strategically integrating observability into the initial stages of their release processes to tackle these challenges. As they embark on this initiative, SREs are tasked with identifying an observability solution that aligns seamlessly with their application teams and seamlessly integrates with their existing toolset.

Outcome

Dynatrace plays a pivotal role in this endeavor by empowering application teams through its seamless integration. The Dynatrace integration leverages native features and events that pass through the pipeline. Events serve as logic operators that can trigger or stop subsequent tasks within the pipeline. Additionally, the Site Reliability Guardian serves as the governing entity, making decisions on whether to proceed or halt a specific build based on observable telemetry data supplied by OneAgent during the CI/CD pipeline process.  This proactive strategy significantly enhances the chances of success for SREs, providing them with more time to focus on substantial project improvements (50%) and broaden the buffer zone (30%). This empowers them to spearhead innovations that ensure the business is prepared for future expansions.

Integrate DevSecOps with Dynatrace

Software delivery is structured around CI/CD pipelines, which play a critical role in the SRE process and represent the first step toward effective automation. Automated CI/CD pipelines greatly reduce the occurrence of manual errors. They provide continuous feedback to developers and enable rapid product iterations. Elevating the efficiency of release pipelines is one key to producing high-quality software; it mitigates the need for post-incident analyses and on-call duties for the SRE team. It also returns valuable time back to the SRE team.

A few avenues for elevating CI/CD pipelines are:

• Enhancing the extent of automated test coverage during the testing phase.
• Embracing the tenets of DevOps and DevSecOps methodologies anchored in engineering principles.
• Identifying and automating the validation of business/application Service Level Objectives (SLOs) during release cycles.
• Streamlining the CI/CD process to ensure optimal efficiency.

To realize these goals, SRE teams can seamlessly follow a three-step process within their Dynatrace environment:

1. Commencement (optional): If desired, initiate a Dynatrace workflow task to establish a connection with the DevOps tool using HTTP. This step is only necessary if you intend to control the build creation process through Dynatrace.
2. Push events: Configure your DevOps tool to dispatch deployment events at the inception of the deployment process. Additionally, introduce annotation events to notify Dynatrace of the progress within your testing phase. These events serve as logical operators that dictate the course of the release process.
3. Automated validation and progression: Depending on the configuration of your tasks, Site Reliability Guardian (SRG) validation can be automatically activated to promote or disapprove the advancement of a build towards production.

This integration leads to complete automation with end-to-end pipeline visibility, thereby reducing the heavy lifting of release management for SRE teams and empowering them to focus on innovation, which catalyzes organizational growth.

Designing systems for reliability

Engineering teams must extend their focus beyond functional and load tests to instill assurance in the software release process via automated cycles. The rationale is that, during actual production, variables can induce outcomes that are different from those predicted in standardized tests. Thus, more comprehensive testing becomes essential to embrace unpredictability and mirror real-world conditions. These practices are commonly known as “chaos engineering.”

By embracing chaos engineering practices, development teams cultivate a higher degree of confidence in the robustness of their applications within specific production scenarios. However, the very nature of chaos engineering introduces a challenge: identifying the responsible service when a failure occurs. This is where Dynatrace Davis AI comes into play, leveraging telemetry data from your services. Davis AI automatically establishes a baseline of each service’s behavior, learned during the development and functional testing phases, and subsequently identifies the underlying causes of failures, thereby eliminating the uncertainty of the root cause and “war room” scenarios.

To illustrate, a CI/CD pipeline setup has a functional test phase followed by a chaos-engineering test. The chaos engineering test was structured to randomly slow down a Docker container and, thereby, a critical service for the application. During this phase, Davis AI automatically identifies that a key business request has breached its automated baseline due to a significant Mongo database slowdown. Moreover, Davis AI identifies the reason behind the slowdown and pinpoints the exact query that caused the problem.

With Davis AI’s contextual capabilities, embracing chaos engineering and making application code robust and better prepared for production deployments is easy.

Reduce operations overhead

Ideally, no bugs are reported once software is deployed to production. However, this is highly unlikely. Therefore, it’s important to have a process in place to minimize downtime in the event of a failure.

Davis AI assists with automated root cause analysis, providing details of the underlying services, traces, logs, and user sessions that caused the failure. This can save SRE teams from the time and effort of debugging or a war room scenario. By identifying the exact code or trace that causes a failure, Davis AI helps teams fix problems quickly and significantly reduces MTTR. In addition, if you configure an automated remediation workflow, Dynatrace invokes it and restores applications with virtually no downtime.

In the above screenshot, one of the requests in the application reports errors. Davis AI identifies that this anomaly was reported in the newer releases. Davis AI captures and relays the details of previous and current builds to the remediation workflow. The remediation workflow uses this information to roll back the build, leaving Davis AI to automatically validate and close detected problems.

Dynatrace Observability empowers the SRE team, propelling them towards LowOps/NoOps. This allows them to prioritize innovation, strategic goals, and minimize downtime while optimizing the delivery of perfect software. In our next blog post in this series, we’ll delve into the step-by-step process of automating the release pipeline using Dynatrace best practices.

The post Accelerate and empower Site Reliability Engineering with Dynatrace observability appeared first on Dynatrace news.

SLOs cover a wide range of monitoring options for different applications. This article explores SLOs for service performance. If you’re new to SLOs and want to learn more about them, how they’re used, and best practices, see the additional resources listed at the end of this article.

According to the Google Site Reliability Engineering (SRE) handbook, monitoring the four golden signals is crucial in delivering high-performing software solutions. These signals (latency, traffic, errors, and saturation) provide a solid means of proactively monitoring operative systems via SLOs and tracking business success.

How to measure performance

SLOs are based on service-level indicators (SLIs), which are used to track performance against pre-defined service-level agreements (SLAs).

While this connection might sound simple, finding the right metrics to measure the needed SLIs takes time and effort. Moreover, after selecting an SLI, complex metric expressions might be required to extract and interpret the result and come to the right conclusions and decisions.

SLOs, as a measure of service quality, can track the related availability, reliability, and performance. Hence, service performance is one key indicator for defining and monitoring service-side objectives. Performance typically addresses response times or latency aspects and contributes to the four golden signals.

Generally, response times measure the total duration of receiving, processing, and completing a request. Among other measures, response times significantly contribute to evaluating frictionless and satisfying user experiences and hence are included in calculating the Application Performance Index (Apdex). Proactively monitoring these indicators is vital to achieving the expected and desired performance.

Service-performance template

Latency is often described as the time a request takes to be served. This is what Dynatrace captures as response time. An excellent way to establish an SLO based on latency is to have a certain percentage of all service requests returned within a selected time frame of, for example, 300 ms. Once the SLI metric is defined, an SLO can be created based on the ratio of high-performing partitions (time intervals) compared to all partitions.

To simplify the process, Dynatrace provides dedicated templates for SLO creation via the SLO Wizard. One template explicitly targets service performance monitoring.

The service performance SLO depicts the ratio of the count of partitions (time intervals) rated as “fast” compared to the total count of all partitions. The respective metric expression looks like this:

((builtin:service.response.time:avg:partition("latency",value("good",lt(10000))):splitBy():count:default(0))/(builtin:service.response.time:avg:splitBy():count)*(100)):default(100,always)

How to use and interpret the template

Looking back to the provided metric expression, different metric transformations, such as avg, partition, splitBy, lt, count, and default, are used to calculate the numerator and denominator to compute the SLO’s service performance status.

Partition Operator

An important operator is represented via the so-called partition transformation.

The partition transformation separates the time series into equally sized time intervals over the timeframe. The longer the timeframe, the larger the timeslots. Therefore, the SLO evaluation’s precision depends on the evaluation length (=query) period.

Functionality-wise, it adds another dimension based on the selected criteria, such as “good” or “bad,” by considering the input parameter, such as lt (lower than) 10000us. After calculating the ratio between good and overall timeslots the desired percentage value for the performance SLO is returned.

Note: The partition operation adds an additional dimension to the timeseries. This requires the usage of splitBy or merge commands to align the numerator and denominator accordingly.

A further pre-requisite to using the partition operator effectively is shown in the aggregation requirement. An aggregation operator processes the datapoints of the single partitions, influencing whether they’re counted as good or bad. Typical and recommended aggregations are the avg (average) or percentile(N^th) function. While avg applies the arithmetic means, the percentile(N^th) aggregation can help by removing large outliers, due to calculating the N^th percentile of the given data points. These two aggregation transformations can be exchanged based on the desired outcome. Using the percentile(90) aggregation in the described example results in only counting partitions where 90% of all datapoints depict a faster response time than 10000 us (0,01 seconds).

Further details about the partition syntax can be found in the documentation.

The partition operator requires a threshold for dividing the timeslots into “good” and “bad”. The latency command provides a means of estimating service-performance values. lt() represents “lower-than,” followed by an integer value indicating the microseconds used for the threshold.

Default Operator

While the splitBy and count operator, as standard operators for metrics, remove the different dimensions via aggregation and return the amount of datapoints, the default() operator helps in case of data gaps within the given time series.

The default operator replaces null values in the payload with the specified value. In the example above, the default operator replaces null values with 0, in order to ensure that only values matching the latency criterion are considered for the SLO status calculation. However, in cases where there is no traffic on the selected service, this does not mean that performance is bad. This possibility is covered by adding default(100, always) at the end, which ensures that “no traffic” situations are not counted as bad performance.

Increasing the precision of service-performance SLOs

The service-performance SLOs described are based on built-in metrics, which are captured, calculated, and provided out-of-the-box by Dynatrace. For higher precision (for example, smaller time slots of good and bad samples), custom-calculated metrics can create specific time series for the numerator.

NOTE: When combining calculated service metrics with built-in metrics, the muted request option for excluding muted requests in the calculated service metric is advised, as the built-in metric applies it by default.

Set up your first service-performance SLO

The SLO templates are available to all users and can be applied via the SLO wizard. To create an SLO, select the Add SLO button on the relevant service page or via the SLO application.

As soon as the SLO is created, it can be monitored and visualized via a dashboard. By applying alerting capabilities, a notification can be sent as soon as the service level objective is at risk of being violated.

To get more information about how SLOs can be created and support you in reaching your business goals, please have a look at the resources below:

• Service level objective examples: 5 SLO examples for faster, more reliable apps
• Lessons learned from enterprise service-level objective management
• SLOs done right: how DevOps teams can build better service-level objectives

The post Maximize user experience with out-of-the-box service-performance SLOs appeared first on Dynatrace news.

Dynatrace Configuration as Code enables complete automation of the Dynatrace platform’s configuration, ensuring that software is secure and reliable. With Configuration as Code, developers can manage their observability and security tasks with config files that can be developed alongside source code conveniently and at scale. Dynatrace offers Configuration as Code for the entire platform, covering all aspects, including app settings built for the AppEngine.

As software development grows more complex, managing components using an automated onboarding process becomes increasingly important. This is especially crucial in microservice architectures, where the number of components can be overwhelming. Furthermore, increasing the frequency of releases requires additional product lifecycle automation and setup.

While infrastructure has historically been treated as a bottleneck where proper scaling and compute power are applied to improve performance, these aspects are now typically addressed by hyperscalers that offer cloud-based infrastructure and infrastructure as a service. However, scaling up software development requires more tools along the software product lifecycle, which must be configured promptly and efficiently.

To handle this challenge, enterprises need to automate and streamline the onboarding and lifecycle of tool configurations in the software development processes, including aspects of observability, security, alerting, and remediation.

Efficient environment configuration at scale

One of software engineers’ most significant challenges is managing the numerous tools and technologies required for the software product lifecycle. Development teams must set up tailored configurations for each tool and component they’re responsible for. Developers can’t focus on software development while also managing the details of each tool’s configuration options, and tool administrators can’t scale to keep up with multiple development teams, numerous configurations, and the many environments they need to support. This is why it’s so essential to offer sharable configuration templates that can be easily reused and customized for specific teams, components, and environments.

Configuration as Code in Git repos, automatically applied by Dynatrace

Analogous to infrastructure as code, Configuration as Code, or “everything as code” is now essential for tackling software development challenges. With Dynatrace Configuration as Code, teams can meet these challenges while working with the IDEs, Git repos, and tools that they’re already familiar with. Configuration files allow for the automatic creation, update, and management of configurations for dashboards, synthetic monitors, alerts, SLOs, and security settings across multiple environments.

While developers edit files, a simple CLI command applies configurations to Dynatrace and, for example, automates the setup of a quality gate, including workflows and Site Reliability Guardians.

This can all be done safely and consistently in a repeatable manner. Configuration files can be reused, versioned, and shared across teams. Configuration as Code supports all the mechanisms and best practices of Git-based workflows, including pull requests, commit merging, and reviewer approval.

GitOps is a best-practice methodology for handling operation-relevant configurations that can be applied across the entire Dynatrace platform. With Dynatrace configuration files in Git repos, you gain:

• Persistent configuration state available in files
  • Easy copy and paste
  • Easy templating
• Established Git software development workflows
  • Repo branching
  • Change management
  • Reviews and approvals

From the developer perspective, you only need to fill out a form (YAML file) with key-value pairs (for example, to provide the thresholds for the API endpoint of a service) and an email address for escalation to get the benefit of the Dynatrace platform when you deliver a new service.

Enable self-service configuration at scale

You can easily set up observability, security, and automation by filling out a few key-value pairs in YAML format. Reduce the complexity of configuration down to only the parameters that vary within your organization, such as for different services and environments. Teams can quickly set the parameters they need to configure the Dynatrace platform to their specific requirements. We provide a CLI that is natively built for Dynatrace APIs and platform configurations, as well as apps built for our AppEngine that can be run and customized without third-party dependencies and with a lightweight setup. Additionally, we offer a Terraform provider if you already have a working Terraform-built environment in your organization.

Achieve automated observability and Site Reliability Engineering

To minimize any negative impact caused by problems such as security vulnerabilities, it’s crucial to respond and remediate the issues as quickly as possible. Dynatrace provides automation for detecting problems, and you can opt to automatically run a change-impact analysis report to proactively validate important objectives. This same mechanism can also be leveraged to validate the impact of new software releases on resources, logs, performance, reliability, or business measures. The screenshot below displays a workflow that listens for a deployment event of the easytrade service in the production stage.

The validation process is automated based on events that occur, while the objectives’ configuration, which is validated by the Site Reliability Guardian, is stored in a separate file. The screenshot below displays such a configuration. In summary, Configuration as Code enables the automatic execution of validations for a full set of configuration objectives and workflow definitions. Development teams can easily adopt this by providing key-value pairs in a YAML config file, such as the name of a service and the stage to listen for deployment events.

Service-Level Objectives (SLOs) can provide additional support for the insights and outlook of services and components. Whether tracking internal, workload-centric indicators such as errors, duration, or saturation or focusing on the golden signals and other user-centric views such as availability, latency, traffic, or engagement, SLOs-as-code enables coherent and consistent monitoring throughout the environment at scale.

Proper notifications or escalations are automated based on ownership information. The Dynatrace platform supports ingesting ownership team information, as shown in the screenshots below. Dedicated configuration files are used to create teams and maintain relevant information, such as responsibilities and contact details, in a scalable and automated way. While this can be achieved through UIs and APIs, providing contact details and links to any supporting material when things go wrong in any environment is most conveniently done by a developer, as code, side-by-side with the software source code.

Furthermore, by utilizing workflows, it’s easy to set up dynamically-queried ownership-team information from affected entities if a problem or a security vulnerability is detected. Leverage the full power of the Dynatrace AutomationEngine to automatically inform the right people and automatically resolve problems, all managed and configured easily via configuration as Code.

What’s next

Get started with Dynatrace Configuration as Code, natively built for the Dynatrace platform and third-party independent. You can read all about it in our Configuration as Code documentation. If Terraform is your tool of choice, please have a look at our Terraform provider documentation.

Stay tuned for more examples and easy-to-adopt automations provided in our public Github project.

The post Automated observability, security, and reliability at scale appeared first on Dynatrace news.

To ensure high standards, it’s essential that your organization establish automated validations in an early phase of the software development process—ideally when code is written. The Dynatrace Site Reliability Guardian is designed for this practice; it allows development teams to define quality objectives in their code, which is validated throughout the delivery process before the code reaches production.

While quality standards and validations should ideally be implemented throughout the software development lifecycle, we see teams adopting the Site Reliability Guardian at various stages, from development to production environments. Our discussions with teams who use Site Reliability Guardian in production reveal that starting with post-deployment validation provides quick insights into the status quo and helps identify areas that need improvement via pre-production validations. This approach, known as “shift-left,” is preferred by organizations that derive insights from their production environments and plan improvements from there.

How Dynatrace uses Site Reliability Guardian

In each of these Dynatrace examples, insight is made in a production-like environment. Validation tasks are then extended left to cover performance testing and release validation in a pre-production environment. These examples can help you define your starting point for establishing DevOps and SRE best practices in your organization.

Ensure expected production behavior

One Dynatrace team is responsible for the demo applications we use to demonstrate Dynatrace capabilities. We use monitored demo applications to deliver constant load and a defined set of business transactions. Without demo applications, it would not be possible for our sales teams in the field to show off the full power of Dynatrace. To ensure these objectives, the team leverages the Site Reliability Guardian to check for a potential decrease in business transactions whenever a new application release is deployed.

Two guardians, which are managed by the team’s Site Reliability Guardian and contain different sets of objectives, are used in this setup. While the first guardian validates the traffic, the second guardian checks the business transactions generated during the observation period. The functionality is implemented via an automated workflow.

The Workflows screenshot below shows that a task is triggered by a change event related to the application, execution of the guardians, and final aggregation of the results. Ultimately, the result is shared via a Slack notification to report on the current business behavior.

Resilient applications with chaos testing in pre-production

Another Dynatrace team uses a guardian as a safeguard during chaos testing. More precisely, this team uses AWS Fault Injection Simulator (FIS) to run fault injection to improve the application’s performance and resiliency. These experiments are accompanied by well-defined objectives formulated as Dynatrace service-level objectives (SLOs). The guardian is used to bring the experiment of the fault injection simulator and the SLOs together. Hence, it validates the status of the SLOs for the experiment period to derive potential regressions that might have been introduced in a newer release.

While this team uses Fault Injection Simulator (FIS) to run their chaos testing experiments, the Site Reliability Guardian is not limited to this testing practice. It fits well into the entire performance testing domain where reliability, performance, and security indicators must be continuously validated.

Release validation on the four golden signals in pre-production

Another Dynatrace team adopted Site Reliability Guardian with an early version of our Traces in Grail capability for a classic approach to release validation within its delivery pipeline. In other words, they use it as a quality gate that detects bad releases before they reach production. They first explore relevant objectives in their production environment. Based on those insights, they implemented automated validation tasks, and shifted left in their software delivery pipeline.

In this case, the four golden signals (latency, traffic, errors, and saturation) are derived from span attributes and DQL metric queries via Dynatrace Grail™. The queries are depicted below (sensitive data has been removed).

Latency – Response time (below 50 ms) derived from spans

fetch spans
| filter application.name == "APPLICATION-NAME"
| filter application.environment == "hardening"
| filter dt.entity.service_method.name == "getEffectivePolicies"
| filter span.kind == "SERVER"
| summarize toDouble(countIf(duration <= 50ms)) / count()

Traffic – Number of requests derived from log lines

fetch logs
| filter log.source == "projects/APPLICATION-NAME/logs/requests"
| parse content,"JSON:request"
| fieldsAdd httpRequest = request[httpRequest]
| summarize count = count()

Error rate – Number of HTTP 5xx errors derived from spans

fetch spans
| filter application.name == "APPLICATION-NAME"
| filter application.environment == "hardening"
| filter dt.entity.service_method.name == "getEffectivePolicies"
| filter span.kind == "SERVER"
| summarize response_500_error_rate = toDouble(countIf(http.status_code >= 500 AND http.status.code < 600))/count()

Saturation – Memory requested by a Kubernetes workload

timeseries avg(dt.kubernetes.workload.requests_memory),
filter: in(dt.kubernetes.workload.name, "WORKLOAD-NAME")

Please note that “Traces in Grail” are currently in private preview, and the DQL syntax could be subject to change. 

What’s next?

Dynatrace strives to enable DevOps and SREs to ship software that works perfectly. Accordingly, we provide the Site Reliability Guardian as one way to establish reliability, security, and quality practices early in your software development lifecycle.

Stay tuned for an upcoming blog post that shows how you can shift your DevOps and SRE practices even further left by declaring your standards in code—leveraging the Dynatrace configuration-as-code approach—and validating them via Site Reliability Guardian as part of your delivery process.

Check out the Site Reliability Guardian for yourself by installing it using the Dynatrace Hub, then share your feedback with us and let us know how you intend to use it. Head over to the Dynatrace Community to see our plans for additional features. We’d love to hear your suggestions and ideas.

The post How Dynatrace boosts its production resilience with Site Reliability Guardian appeared first on Dynatrace news.

Service-level objectives help IT teams define technical success and align with top-line business objectives.

But not all service-level objectives (SLOs) are created equal. So how do development and operations (DevOps) teams and site reliability engineers (SREs) distinguish among good, great, and suboptimal SLOs? In the 2023 Perform session “SLOs done right: A practitioners guide,” Michael Cabrera, SRE lead at Vivint, and Andreas Grabner, DevSecOps activist at Dynatrace, break down the state of SLOs and discuss how teams can adopt successful SLOs, avoid less-than-ideal objectives, and ultimately build better SLOs.

The state of service-level objectives

While SLOs play a critical role in helping DevOps and SRE teams align technical objectives with business goals, they’re not always easy to define. In many cases, issues are tied to operational overload. Enterprises now have access to myriad metrics they can track and measure, but an abundance of choice doesn’t equal actionable insight.

Indeed, 54% of SREs say they handle too many metrics, making it increasingly difficult to find the most relevant ones for a particular service, according to the Dynatrace State of SRE Report. 22% of SREs said they don’t know what makes a good SLO, while 18% said they don’t know which metrics to track. The same number said they’re not sure how or where to get started.

The result? From issues with an overwhelming set of metrics to problems with implementation, 99% of SREs said they encounter challenges when defining and creating SLOs.

The four components of a good SLO

To help SREs and DevOps teams create and deploy effective SLOs, Cabrera and Grabner examine the following four factors that make a good SLO:

1. Monitors signals

The first attribute of a good SLO is the ability to monitor the four “golden signals“: latency, traffic, error rates, and resource saturation. While not every SLO must report on all these signals simultaneously, good service-level objectives help teams uncover and understand how at least one of the four signals can affect current operations.

2. Includes the end user

While SLOs for back-end silos and processes help eliminate internal issues, back-end-based objectives tell only half the story. As a result, effective SLOs must account for end users. “You might have an internal dashboard where everything is green and it makes you feel good,” Grabner noted. “But on the outside, people are suffering. That means you can’t define SLOs purely on back-end metrics.”

3. Bridges the business gap

Good SLOs should also help bridge the gap between chief executive officers’ concerns and those of chief technology officers. By providing data relevant to both parties — such as user adoption metrics for CEOs and application crash data for CTOs — organizations can find common ground.

4. Prioritizes incident response

SLOs should target problems that represent potential business impact. This enables teams to prioritize incident response and ensure immediate issues are resolved before working to address more long-term concerns.

SLOs that teams should avoid

Any SLO that accurately measures and reports data can benefit organizations. In practice, however, SLOs’ value varies significantly based on how teams design, deploy, and manage them.

Grabner and Cabrera offer the example of an iOS app experiencing crash issues after a team deploys a new version. Dynatrace OneAgent provided information about failure rates, latency, and throughput, along with iOS data for users, crashes, and error rates. Creating objectives tied to these metrics alone, however, didn’t benefit end users. Put simply, these metrics are the symptom rather than the root cause; knowing how many crashes have taken place and how often provides evidence that applications or other IT systems aren’t working as intended, but it doesn’t offer insight to help fix the problem at scale.

As a result, SLOs can’t simply alert on metrics. Instead, they must alert on both front- and back-end processes to help teams understand where problems occur and how they’re related, setting the stage for meaningful correction.

How to build a better SLO

So, how do teams build a better SLO?

Building better SLOs starts by understanding the dual nature of these service-level objectives. To be effective, SLOs must deliver business and technical results. Cabrera and Grabner cited the example of a tandem approach in monitoring application performance. On the technical side, SLOs account for metrics such as availability, response time, and error rate. On the consumer side, SLOs focus on application adoption, overall application rating, and the total number of mobile crashes.

By combining these data sets, SRE and DevOps teams can make decisions that improve back-end performance and end-user experience. When it comes to creating a better SLO, “Make it visible and tangible,” Grabner advised.

It’s also worth considering implementing automation to help scale SLOs and encourage shift-left operations that allow teams to quickly and easily build their own SLO frameworks.

The post SLOs done right: how DevOps teams can build better service-level objectives appeared first on Dynatrace news.