For developers, the ultimate win is creating impactful features, shipping them quickly, and confidently implementing fixes when needed. Distributed tracing plays a critical role in achieving these goals. As a core component of observability, it provides the insights developers need to write better code, collaborate efficiently, and increase application reliability.

Developers know how challenging it can be to follow traces from beginning to end across multiple entities and data providers. Harder still is following those traces in context with metrics, logging data, security details, and real user experience data. Certain tools may provide visibility into small segments or isolated details. But developers often have to piece these clues together manually, which makes it difficult to extract meaningful and reliable results.

Dynatrace delivers distributed tracing with capabilities designed to streamline workflows. From understanding the impact of code changes to quickly resolving issues flagged in production, here’s how distributed tracing can help you increase developer velocity and deliver meaningful innovation throughout the software development lifecycle (SDLC).

Distributed tracing best practices key takeaways:

1. Visualize and understand your system. Distributed tracing plays a key role in uncovering the topology, interactions, and dependencies of your services.
2. Analyze and fix problems fast. Combine distributed tracing with AI analysis, context-driven querying, and collaboration tools.
3. Build better software with telemetry data in context. Unify telemetry data
4. Automate to ship better code faster. Use distributed tracing to automate and streamline workflows and set up intelligent alerts.

Visualize and understand your system

Before implementing changes or designing new features, it’s essential to have a clear understanding of your system’s current state. Without a clear picture of your current system, any change risks overlooking critical dependencies and misalignment with your architecture, whether in the cloud or on-premises. Distributed tracing plays a key role in uncovering the topology, interactions, and dependencies of your services, giving developers the visibility they need to make informed decisions.

Dynatrace Smartscape® technology automatically maps horizontal service relationships, offering real-time visualization of service connections and dependencies. This end-to-end view helps developers quickly understand how components interact within the overall system.

For a more granular understanding, inspecting individual traces allows you to focus on specific workflows or problematic behaviors. This depth of insight can help you plan better and identify challenges early, reducing surprises during later stages of development.

Trace example: Verifying service for “platinum” loyalty users

Say you’re a developer on an app called “AuthenticationService.” You can select a trace that lets you visualize its path through your system and examine each span’s properties to ensure “Platinum” loyalty users are being served effectively. While understanding the state of your system is important at each stage of the SDLC, it’s often one of the first steps in planning, requirements gathering, and designing changes.

In Dynatrace, you would open the Distributed Tracing app and enter a filter for the request attribute that indicates loyalty status. This filter returns a list of traces you can inspect individually to understand any bottlenecks or issues that could be impacting this specific subset of customers.

Analyze and fix problems fast using distributed tracing for developers

Identifying root causes in complex microservice-based architectures can be tough, but Dynatrace Davis® AI analyzes dependencies in real time, helping you quickly and confidently resolve issues.

But what if you want to visualize data outside of an individual problem occurrence? Some issues occur repeatedly in specific areas of a system, while others arise from application changes. Problems often span multiple teams, requiring collaboration to resolve effectively.

For broader insights, the Dynatrace Distributed Tracing app provides flexible access to raw trace data. You can filter by namespace, release version, or database calls, then Dynatrace automatically generates a DQL (Dynatrace Query Language) query so you can view results in notebooks or pin visuals to dashboards, which enable you to collaborate seamlessly with other teams. This approach is particularly useful during testing, deployment, and maintenance, where visibility into system behavior is critical.

This example shows a plot generated by a DQL query that indicates the number of database calls an application makes and how they perform over time.

Let’s assume you’re also interested in how your database queries are performing. To get an instant performance summary, you can easily add a table to your dashboard that summarizes the count, average, and percentiles of the durations of your database spans.

Build better software with telemetry data in context

Developers juggle many responsibilities, from testing and deploying code to maintaining stability. However, fragmented workflows and the endless search for the right information can often slow progress and create frustration, making it harder to meet deadlines and maintain a focus on innovation.

As a developer, how can you get the information you need without wasting time chasing down data and trying to understand all its upstream and downstream effects? The answer: distributed traces and telemetry data in context.

Flexible access to unified telemetry data is a game-changer for developers. With Dynatrace, you can seamlessly query multiple telemetry data sets and visualize the results in ways that align with your specific needs. This flexibility empowers you to connect the dots across complex workflows, analyze incidents within their broader context, and make data-driven decisions that support automation and enhance software quality.

The example below brings together spans and logs. Dynatrace automatically enriches every log with trace and span IDs, making it simple to track correlations among related data. With Dynatrace Query Language (DQL), you can dynamically combine spans and related log content on-the-fly.

To understand the impact of an error, you can chart which services and endpoints are emitting this log error.

As a developer, you can unlock deeper insights, linking operations to specific errors and enabling you to assess the scope and impact of incidents accurately. This gives you and your teams the ability to move beyond reactive problem-solving into proactive, intelligent development.

Use distributed tracing for developers to automate and ship better code faster

Automation is key to accelerating development while maintaining quality. By streamlining workflows, automating repetitive tasks, and setting up intelligent alerts, teams can proactively address issues and minimize manual effort.

For example, as a developer, you can use trace data to establish quality gates or set alerts for specific exceptions, ensuring you can quickly flag and resolve any recurring problems. This approach boosts efficiency and ensures that teams can consistently deliver high-performing, reliable code to production.

Assume you’ve been working on a bug that appeared as an exception in your traces. You want to check if the bug has been fixed and set up an alert to get a reminder if this particular bug comes back.

Let’s start by creating a metric based on this particular exception in the Dynatrace stream-processing data ingestion technology, OpenPipeline™. By creating a metric at ingestion using OpenPipeline’s centralized data handling and advanced data processing capabilities, you can continually monitor for evidence of the exception recurring.

You can use this extracted metric to visualize its frequency over time in a dashboard.

Finally, once you’ve set up a metric, you can use it to create an alert and send a notification to your system of choice (such as Slack, Teams, ServiceNow, and so on).

Increase reliability and automate more with distributed tracing best practices

As development responsibilities continue to expand in today’s complex environments, distributed tracing has emerged as a critical tool for developers throughout the entire software development lifecycle.

Applyin these distributed tracing best practices helps developers visualize systems, resolve issues quickly, and streamline workflows. By providing deep insights and context of upstream and downstream events, traces and spans give teams the data they need to automate tasks and deliver more reliable, higher-quality software faster and with greater confidence. The result allows you to focus on what matters most: creating impactful features.

Start integrating distributed tracing into one part of your workflow today and see how quickly it becomes indispensable to your development practice.

The post Distributed tracing best practices for the software development lifecycle appeared first on Dynatrace news.

Establishing SLOs for Kubernetes clusters can help organizations optimize resource utilization. A good Kubernetes SLO strategy helps teams manage and make containerized workloads more efficient.

Kubernetes is a widely used open source system for container orchestration. It allows for seamless running of containerized workloads across different environments.

Effective resource provisioning and management is a critical aspect of a Kubernetes cluster. It involves a coordinated effort among stakeholders, including cluster owners, application teams, and business owners. The primary goal is to allocate sufficient resources to keep the applications running smoothly without overprovisioning and incurring unnecessary costs. This delicate balance requires a collaborative approach and transparent communication among all parties. Service-level objectives (SLOs) can play a vital role in ensuring that all stakeholders have visibility into the resources being used and the performance of their applications.

While the Kubernetes application from Dynatrace already provides detailed insights into key parameters such as memory or CPU slack—allowing you to immediately drill down to the workloads blocking and wasting resources—SLOs add a further layer on top of these insights by evaluating resource spending levels.

Service-level objectives are typically used to monitor business-critical services and applications. However, due to the fact that they boil down selected indicators to single values and track error budget levels, they also offer a suitable way to monitor optimization processes while aligning on single values to meet overall goals. Essentially, an SLO tracks a selected service-level indicator (SLI) and continuously evaluates its behavior over a given timeframe against a fixed (commonly agreed upon) threshold. This feature is valuable for platform owners who want to monitor and optimize their Kubernetes environment. Users can continuously evaluate a system’s performance against predefined quality criteria, making SLOs for Kubernetes clusters a good option for monitoring and improving a system’s overall performance.

Efficient coordination of resource usage, requests, and allocation is critical. All users must work together seamlessly to optimize resource utilization in a Kubernetes cluster. A Kubernetes SLO can be a transparent and trackable collaboration tool for different teams and collaborators.

Measure your Kubernetes resource efficiency and agree on an aligned approach between all stakeholders

If your team is responsible for setting up Kubernetes clusters, you might want to monitor and optimize the workload performance when setting up SLOs. However, if you’re part of the application team, the usage of reserved resources might differ significantly from the blocked resources

Monitoring your Kubernetes cluster allows you to proactively identify and resolve resource constraints, failures, and crashes before they impact end-user experience and your business.

Proper Kubernetes monitoring includes utilizing observability information to optimize your environment. By gaining insights into how your Kubernetes workloads utilize computing and memory resources, you can make informed decisions about how to size and plan your infrastructure, leading to reduced costs. A Kubernetes SLO that continuously evaluates CPU, memory usage, and capacity and compares these available resources to the requested and utilized memory of Kubernetes workloads makes potential resource waste visible, revealing opportunities for countermeasures.

Regarding resource utilization in a Kubernetes environment, there are two main perspectives to consider based on the primary stakeholders involved. One perspective focuses on the potential for optimization at the interface between the team responsible for managing the Kubernetes cluster and the application teams that are responsible for developing and deploying applications.

If your team is responsible for setting up Kubernetes clusters, you might want to monitor and optimize the workload performance when setting up SLOs. However, if you’re part of the application team, the usage of reserved resources might differ significantly from the blocked resources.

Requests represent the number of resources reserved or blocked for a container. Tracking the ratio between request and usage can provide valuable insights into optimization potential. As every container has defined requests for CPU and memory, these indicators are well-suited for efficiency monitoring.

Optimize memory and CPU utilization of your Kubernetes namespaces

One option is continuously tracking the memory and CPU utilization efficiency of existing Kubernetes objects, such as namespaces or workloads. Since teams typically run multiple workloads on namespaces, using this level of abstraction is a suitable option for a Kubernetes SLO. However, if you require more granular information, you can adjust the levels for resource utilization monitoring accordingly.

To calculate the service-level indicator for the Kubernetes namespace memory efficiency SLO, simply query the memory working set and request the memory metrics that are provided out of the box.

timeseries {
      memWorkSet=sum(dt.kubernetes.container.memory_working_set, default:0, rollup:avg)
      , memRequest=sum(dt.kubernetes.container.requests_memory, rollup:avg)}
      , nonempty:true, by:{dt.entity.cloud_application_namespace} 
      , filter: IN(dt.entity.cloud_application_namespace, "YOUR-NAMESPACE-ENTITY-ID")
| fieldsAdd sli = memWorkSet[]/memRequest[]*100
| fieldsRemove memWorkSet, memRequest

This DQL query can be inserted into the new SLO app on the Dynatrace platform, thereby supporting native Dynatrace Grail™ access.

The same procedure is applicable for measuring the computing resources of a single namespace.

timeseries {
      cpuUsage = sum(dt.kubernetes.container.cpu_usage, default:0, rollup:avg)
      , cpuRequest = sum(dt.kubernetes.container.requests_cpu, rollup:avg)}
      , nonempty:true, by:{dt.entity.cloud_application_namespace}
      , filter: IN(dt.entity.cloud_application_namespace, "YOUR-NAMESPACE-ENTITY-ID")
| fieldsAdd sli = cpuUsage[]/cpuRequest[]*100
| fieldsRemove cpuUsage, cpuRequest

These Kubernetes SLOs measure the ratio of used memory and computing resources relative to the requested memory and resources for an entire namespace. They provide insights into how efficiently the blocked resources are utilized. Because requested resources can’t be used elsewhere, the objective is to keep the difference between requested and used resources as small as possible. Utilizing such SLOs for your Kubernetes environment makes it possible to track efficiency transparently over time. The teams responsible for the cluster and the application teams that run their containers on the cluster can agree on the intended ratio between used and requested memory and CPU.

Optimize your Kubernetes cluster’s resource allocation

Measuring and optimizing the efficient usage of resources relative to resource requests is just one aspect of the full picture. As cluster owners must allocate cloud resources to meet their application team’s resource requests, it’s recommended that they measure the efficient provisioning of allocated resources over the entire Kubernetes cluster.

For cluster owners, monitoring resource usage at the node level provides better insights and information for taking sound actions. By tracking resource usage at each node, teams can gain insights into how many resources the entire cluster uses and whether the nodes work correctly and efficiently.

Teams should implement suitable SLOs to continuously monitor resource usage for the entire Kubernetes cluster. These SLOs should cover metrics like node memory utilization, which involves monitoring the ratio of requested versus allocated resources or the ratio of desired versus running pods per node.

For instance, if a node’s memory is high, it can lead to the undesired deletion of pods. This, in turn, can disrupt an application or service and incur additional costs for potential cluster upscaling.

A possible Kubernetes SLO for monitoring and evaluating the ratio of requested versus allocated memory can be expressed as follows:

timeseries {
      requests_memory = sum(dt.kubernetes.container.requests_memory, rollup:avg),
      memory_allocatable = sum(dt.kubernetes.node.memory_allocatable,rollup:avg)}
      , by:{dt.entity.kubernetes_cluster}
      , filter: IN (dt.entity.kubernetes_cluster, “YOUR-CLUSTER-ENTITY-ID”)
| fieldsAdd sli = (requests_memory[] / memory_allocatable[]) * 100
| fieldsRemove requests_memory, memory_allocatable

This SLO tracks the CPU utilization of a Kubernetes cluster:

timeseries {
      cpuUsage = sum(dt.kubernetes.container.cpu_usage, default:0, rollup:avg)
      , cpuRequest = sum(dt.kubernetes.container.requests_cpu, rollup:avg)}
      , nonempty:true, by:{dt.entity.cloud_application_namespace}
      , filter: IN(dt.entity.cloud_application_namespace, “YOUR-CLUSTER-ENTITY-ID”)
| fieldsAdd sli = cpuUsage[]/cpuRequest[]*100
| fieldsRemove cpuUsage, cpuRequest

In an ideal environment, the requested resources match the allocated resources, and hence, cloud resources will be used optimally, reducing the costs of invoiced but unused resources.

Ensure overall Kubernetes resource utilization efficiency

The four SLOs mentioned in this article provide valuable insights into the overall utilization efficiency of resources in a Kubernetes cluster and are a great starting point for optimizing them. Although both views (usage vs. request and request vs. allocation) must be analyzed together to obtain a holistic view, the split is extremely helpful in ensuring accountability. While the application teams have complete control over the usage/request SLO, the cluster owner can influence the request vs. allocation SLO. The SLOs serve as a simple way to recognize if the utilization over a certain period is on track or if resources are utilized inefficiently.

Though setting up proper SLOs to make these ratios visible and transparent does not reduce the need for collaboration and alignment, it provides a solid foundation for optimizing resource utilization efficiency.

The outlined SLOs for Kubernetes clusters guide you in implementing SRE best practices in monitoring your Kubernetes environment. By recognizing the insights provided, you can optimize processes and improve overall efficiency. This makes this an excellent solution for collaboration between contributors while holding the respective parties accountable.

What’s next with SLOs for Kubernetes clusters?

As getting started with efficiency and optimization efforts within a large environment is always tricky, you might ask, “Where should I start?” Dynatrace can help you here with all the insights the Dynatrace Kubernetes app provides out of the box. For more information about identifying and optimizing resources, go to the Optimize workload resource usage with Kubernetes app and Notebooks use case in Dynatrace Documentation. Also, visit the Kubernetes Dynatrace playground, where you can play around with the Kubernetes app on the Dynatrace platform. Finally, to try out the SLO examples described in this article, visit the Dynatrace Service-Level Objectives playground.

After identifying SLOs, stakeholders can begin measuring the agreed-upon resource utilization between clusters and namespaces. The ability to create and monitor SLOs will soon be natively integrated into the Kubernetes app, with dedicated templates, to quickly establish common objectives for resource utilization in a few clicks without needing to write DQL queries. These out-of-the-box templates and their visualization within the Kubernetes app will be available early next year.

Further best practices include adding dedicated ownership information to all Kubernetes objects so that responsible teams can be notified when anomalies or degradation are detected.

If you want to read and learn more about how cloud-native observability and automation can be achieved with Dynatrace, take a look at Michael Winkler’s recently published blog post, Cloud-native observability made seamless with OpenPipeline.

The post Kubernetes service-level objectives: Optimize resource utilization of Kubernetes clusters with SLOs appeared first on Dynatrace news.

You’re in the middle of threat-hunting activities using Dynatrace and discover that some of your assets are trying to resolve a suspicious DNS name to extract data via covert channels like DNS tunneling. You now want to detect such events automatically by creating a custom Dynatrace security event. This blog post explains how to achieve this goal.

Following the Route53 log use case in Dynatrace threat-hunting documentation, a DNS query record is structured in the following way:

{
    "version": "1.100000",
    "account_id": "684399999995",
    "region": "us-east-1",
    "vpc_id": "vpc-026e4d09999995",
    "query_timestamp": "2024-08-17T07:24:57Z",
    "query_name": "250607.56755a335668636d52634969497343694167496e4a6c59584e766269493649.434a4762334a696157526b5a5734694c416f6749434a6b5a5852686157787a.tiitha-maliciousdomain.com.",
    "query_type": "A",
    "query_class": "IN",
    "rcode": "NOERROR",
    "answers": [
        {
            "Rdata": "34.667.67.208",
            "Type": "A",
            "Class": "IN"
        }
    ],
    "srcaddr": "172.31.1.6",
    "srcport": "42241",
    "transport": "UDP",
    "srcids": {
        "instance": "i-0b56150a99995e6d",
        "resolver_endpoint": "rslvr-out-c8e7c49999999fb5a"
    }
}

Let’s say that the meaningful fields you need for the security event are:

• timestamp when the DNS query was executed (query_timestamp);
• source IP where the query was performed (srcaddr);
• DNS query requested (query_name).

The JSON matcher, with its ability to parse out selected members, allows you to use the following simple DQL expression to extract only the fields you need:

json{
  timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp,
  string:query_name,
  ipaddr:srcaddr
}(flat=true)

As a result, you now have three additional fields with their respective types and values.

To add more accuracy to the query you’re about to automate, you need to extract the domain portion from the query_name field. This can be achieved with the following DQL expression:

LD* '.'? ( (LD '.' LD):domain '.' EOS)

That DQL pattern translates to something like this:

“Please extract two line-data (LD) portions from the content separated by a dot LD '.' LD) that is immediately followed by a single dot ('.') and then reaches the end of the string (EOS). A single dot and some line data might precede it.”

So, the pattern will match the highlighted DNS names below:

Adding a filter for the correct domain and limiting the query to only the last 5 minutes gives you the following DQL query to automate:

fetch logs, from: -5m
| filter aws.log_group == "/aws/route53/unguard-secla-demo/resolver-logs"
| parse content, "json{timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp, string:query_name, ipaddr:srcaddr}(flat=true)"
| parse query_name, "ld* '.'? ( (ld '.' ld):domain '.' eos) "
| filter domain == "tiitha-maliciousdomain.com"
| fields query_timestamp, srcaddr, query_name

Automate the query

Dynatrace AutomationEngine provides the ability to automate such periodic query executions. When you’re satisfied with your DQL query and are ready to automate it, select the Open with button in the lower right-hand corner of the query window.

As a result, a modal dialog will display, providing you with different possibilities about where to open the DQL query. In the current use case, you should open the query in Workflows.

Modify the default workflow

To execute the workflow automatically every 5 minutes, change the default on-demand trigger to either a cron or time-interval trigger. You can easily achieve this by selecting the trigger box in Workflows and following the instructions in the right-hand pane. You can read more about workflow triggers in Workflow schedule trigger documentation

Ingest query results as security events

The simplest way to do this is to use Dynatrace OpenPipeline. OpenPipeline allows you to create custom endpoints for data ingestion and process the events in the pipeline (for example, adding custom pipe-dependent fields to simplify data analysis in a later phase).

Set up a custom pipeline

The best way to set up a security event ingestion to Dynatrace is via Dynatrace OpenPipeline. You can create your own ingest endpoint that will then perform custom processing activities, create additional fields, store these events in separate buckets, and much more. For the current use case, we can create a custom security event pipeline to which we can send all the automated security events we have created from the Security Investigator, add respective fields for future analysis, and store them in the default security events bucket.

Now, create a general ingest endpoint for this use case and any automated detections we want to initiate from the Security Investigator in the future.

1. Go to OpenPipeline.
2. Choose Events > Security Events.
3. On the Ingest Sources tab, select + Source.
4. Give your pipeline a name, for example, Automated discovery from Security Investigator.
5. Choose a path to the new ingest source, for example, Signals.
6. Since we currently don’t want to create any dynamic rules to our pipeline, leave Routing set to static and choose the Custom events pipeline as the route.
7. Go to the Pre-processing tab and add a new processor with the type Add Fields.
8. Give the processor a name, for example, My custom processor.
9. Add the fields that you want to be automatically added when an event passes the pipeline. In our case, we will add the following fields:
   – kind == “SECURITY_EVENT”
   – provider == “Automated Detection Script”
   – category == “AUTOMATED_DETECTION”
   – provider_product == “Security Investigator App”
10. To figure out which fields to add for custom security events, refer to the semantic dictionary.
    Following the overall convention here will simplify analyzing this data later, together with other security events created by other sources.
11. You can test the processor with sample data to see what kind of event will be created based on the processor, and if you’re happy with the result, select Save.

For the current demo, this is enough, however, there are lots of things that you could configure additionally. Take a look at the Ingest custom security events via API documentation to learn more about sending security events to OpenPipeline.

Create and secure the token for event ingest

To create the token needed to ingest events, follow the steps in How to ingest data (events) documentation. When you have created the token, we recommend that you protect it in the Credential Vault instead of just using it in plain text in the automation engine.

1. Open Credential Vault.
2. Select Add new credential.
3. Select token as the credential type and give it a meaningful name.
4. Select App Engine as the credential scope.
5. Paste your ingest token into the token field and select Save.

After saving the credentials, an ID is presented to you in the form of CREDENTIALS_VAULT-12345. Write this down for later usage.

Ingest records through the OpenPipeline custom endpoint

Let’s return to our workflow and add a follow-up task for the DQL query. Choose the JavaScript task type and start writing the code to ingest security events.

Consider the following code for your function:

import { execution } from '@dynatrace-sdk/automation-utils';
import { credentialVaultClient } from '@dynatrace-sdk/client-classic-environment-v2';

export default async function ({ execution_id }) {

  const ex = await execution(execution_id);

  // load the token from the credential vault by its' ID
  const token = await credentialVaultClient.getCredentialsDetails({
    id: "CREDENTIALS_VAULT-F12345",
  }).then((credentials)=> credentials.token);

  // loop through the DQL query results and ingest them one by one
  var result = await ex.result('execute_dql_query_1');
  result.records.forEach (record => {
    ingest_event(token, record);
  });
}

async function ingest_event(token, payload) {

  payload["event.description"] = "Suspicious DNS query from AWS DNS Logs";

  fetch('https://<dynatrace_env>.dynatrace.com/platform/ingest/custom/events.security/signals', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json; charset=utf-8",
      "Authorization": `Api-Token ${token}`
    },   
    body: JSON.stringify(payload)
  }).then(function(response) {
    console.log(response);
  })
}

The code loads the token from the Credential Vault that will be used to ingest the events to OpenPipeline. Then, we iterate through the result set we got from the execute_dql_query_1 task and pass each event along with the ingest token to the ingest function, which will POST the event to the OpenPipeline ingest endpoint that we created earlier. The response to the request is shown in the console for debugging purposes.

We’re also adding a custom field called event.description to the event. You can add custom fields to your event from the OpenPipeline processing as well, but since we want to keep the ingest source as general as possible, it makes more sense to add the event-specific fields near the event, not the pipeline.

Now, give it a distinct name and test if it runs perfectly. Ingesting events to Grail via OpenPipeline should result in a status: 202 that is visible in the run_javascript_1 log console.

Fetch the security events from Grail

To view the newly created security events, follow the same principles as with fetching any other security data from Grail. Using the following DQL query, you can see all the events you have started to ingest:

fetch events
| filter event.kind == "SECURITY_EVENT"
| filter event.category == "AUTOMATED_DETECTION"

To feel the power of the semantic dictionary and to illustrate what you gain from using it, consider the following query. If you want to visualize all the security events ingested to Grail over the last 12 hours by event.category, it is quite simple with the following query thanks to the unified field names:

fetch events, from: -12h
| filter event.kind == "SECURITY_EVENT"
| makeTimeseries count(), by: { event.category }, interval:1h

The result could be a line chart like this:

Of course, you can follow your own naming convention, but this would make it much harder to view all the security events (both custom and Dynatrace-created events) in conjunction.

Operationalize the results

You can create additional workflows for your security events in Grail. For example, you can create a workflow that notifies you in Slack when a new event is created or integrate your workflows with Jira to create a ticket every time such events arise.

You can create a custom integration to your own SIEM system (similarly as we did for the OpenPipeline) for events that can enrich your security data in other environments. Consider your use case and work processes to determine the best way to approach security events available in Grail.

The post Generate security events from Dynatrace Security Investigator via OpenPipeline appeared first on Dynatrace news.

In today’s data-driven world, businesses across various industry verticals increasingly leverage the Internet of Things (IoT) to drive efficiency and innovation. IoT is transforming how industries operate and make decisions, from agriculture to mining, energy utilities, and traffic management.

Agricultural businesses use IoT sensors to automate irrigation systems, while mining and water supply organizations traditionally rely on SCADA to optimize and monitor water distribution, quality, and consumption.

Mining and public transportation organizations commonly rely on IoT to monitor vehicle status and performance and ensure fuel efficiency and operational safety.

As businesses increasingly embrace these technologies, integrating IoT metrics with advanced observability solutions like Dynatrace becomes essential to gaining additional business value through end-to-end observability.

Dynatrace offers a feature-rich agent, Dynatrace OneAgent^®, and an agentless open source approach perfectly tailored for edge-IoT use cases, leveraging OpenTelemetry. Both methods allow you to ingest and process raw data and metrics.

In the following sections, we’ll explore how aircraft monitoring fits into this broader IoT landscape and how to implement OpenTelemetry and Dynatrace to enhance the monitoring and management capabilities of IoT data in aviation.

Get started with IoT: Aircraft monitoring

Have you ever wondered how services like flightradar24 or FlightAware collect aircraft location details and report them live?

Each aircraft equipped with an Automatic Dependent Surveillance-Broadcast (ADS-B) transponder broadcasts ADS-B data packets at a frequency of 1090 MHz that include GPS-derived position, altitude, velocity, and other information.

These broadcasts occur periodically and are received by ground stations and other aircraft equipped with ADS-B receivers. They enable real-time tracking and enhanced situational awareness for air traffic control and collision avoidance systems.

The ADS-B protocol differs significantly from web technologies. While web technologies use the internet and HTTP for data transfer, enabling communication between web clients and servers, ADS-B relies on radio frequency broadcasts for direct, real-time communication. ADS-B transmits structured data packets containing specific aviation-related information such as position and velocity. Unlike web technologies, which support a wide range of applications from webpage serving to API interactions, ADS-B is designed explicitly for real-time physical tracking and monitoring in aviation—just like any other IoT monitoring solution in the earlier mentioned verticals.

The hardware

We’ll delve into collecting ADS-B data with a Raspberry Pi, equipped with a software-defined radio receiver (SDR) acting as our IoT device, which is a RTL2832/R820T2 based dongle, running an ADS-B decoder software (dump1090).

To transport our ADS-B application’s JSON log files into our Dynatrace tenant, we’ll leverage an agentless approach using open source software (OpenTelemetry).

The data format

The aircraft.json file in /run/dump1090-mutability/ contains a timestamp (now), the processed Mode S messages (messages), and an array of JSON objects for all known aircraft metrics, such as identifiers, position, altitude, speed, and signal power.

Critical data includes the aircraft’s ICAO identifier, squawk code, flight callsign, position coordinates, altitude, speed, and the time since the last message was received.
This information is essential for later advanced analytics and aircraft tracking.

Locations of critical data

• Aircraft data: /run/dump1090-mutability/aircraft.json
• History data: /run/dump1090-mutability/history.json

View the raw data

The data in dump1090, once received, can be viewed through the application user interface, accessible at: http://<your-raspberry-pi-IP-address>/dump1090/gmap.html

Why observability and data ingestion?

In our example, the ADS-B application provides an excellent visual representation for short-term live monitoring purposes. IoT devices are not meant to store data for longer periods. This is why the log files’ default location is set to /run within the device—a temporary location.

By additionally sending this data to Dynatrace, you can store data for longer periods, from 1 day up to 10 years, while gaining enhanced insights as these metrics can be correlated with other business processes.

If your IoT devices hold business-relevant metrics that can provide additional value when combined with other business processes, you should store them in Dynatrace Grail™, as demonstrated in this blog post.

Ingest decoded ADS-B data into Dynatrace

The specific log files created by the ADS-B software contain aircraft data (aircraft.json) and historical metrics (history.json). They provide detailed information that, when sent to Dynatrace, enables data analytics and improved decision-making capabilities.

The data in aircraft.json can be ingested into Dynatrace using the Dynatrace OpenTelemetry Collector, our officially supported OpenTelemetry Collector.

After extracting the collector, we simply update its configuration, the otel-collector-config.yaml file, to send the aircraft.json live feed to your tenant’s Dynatrace OpenTelemetry endpoint.

receivers:
  filelog:
    include: [/run/dump1090-mutability/aircraft.json]
    start_at: beginning
    include_file_path: true
    multiline:
      line_start_pattern: '^\{'

processors:
  attributes:
    actions:
      - key: log.source
        value: opentelemetry-iot-dump1090-collector
        action: upsert
      - key: iot-device
        value: rpi24-adsb-receiver
        action: upsert
      - key: loglevel
        value: INFO
        action: upsert
      - key: device.address
        value: IWJHUD768H
        action: upsert

exporters:
  otlphttp/dynatrace:
    endpoint: "https://<your-tenant-id>.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlphttp/dynatrace]
      processors: [attributes]
  telemetry:
    logs:
      level: "info"
    metrics:
      address: "0.0.0.0:8998"

Note the OpenTelemetry processor log attributes log.source and opentelemetry-iot-dump1090-collector. We’ll use these values later to filter the log data and focus on the log events from this specific IoT device and application log.

View, query, and analyze ingested IoT data

The ingested data in Dynatrace Notebooks shows that most critical information like flight callsign, location (latitude/longitude), altitude, vertical rate, speed, and aircraft category are stored in JSON format. Sample JSON data is shown below:

Dynatrace Notebooks and Dashboards allow you to analyze and visualize the ingested data. We’ll begin querying the JSON data and start extracting critical information.

Parsing JSON data is an amazingly simple and fast process, as DQL (Dynatrace Query Language) commands like expand and fieldsFlatten place all the above data into the appropriate columns and fields:

With the data now available, we’ll again use DQL to analyze and summarize various metrics further, such as the total number of unique aircraft monitored and the highest altitude recorded.

Some outputs can be displayed as single values, categorical charts, line charts, or other dashboard elements, as shown below.

Some example queries that help populate this data are shown below:

DQL query to find the aircraft traveling at the highest altitude at a given time

fetch logs, from:now() - 7d //fetches data from the past 7 days
| filter matchesValue(log.source, "opentelemetry-iot-dump1090-collector") // Filter ADS-B logs ingested by the opentelemetry collector on our IoT device
| parse content, "JSON:json_content"
| fieldsAdd aircraft=json_content[aircraft]
| expand aircraft
| fieldsFlatten aircraft
| filter isNotNull(aircraft.flight)
| filter isNotNull(aircraft.altitude)
| fields aircraft.flight, aircraft.altitude
| sort aircraft.altitude desc
| limit 1
| fields aircraft.altitude // or | fields aircraft.flight

DQL query to find out the aircraft type based on the aircraft category codes that are transmitted via ADS-B

fetch logs // fetching logs without any limitation
| filter matchesValue(log.source, "opentelemetry-iot-dump1090-collector") // Filter ADS-B logs ingested by the opentelemetry collector on our IoT device
| parse content, "JSON:json_content"
| fieldsAdd aircraft=json_content[aircraft]
| expand aircraft
| fieldsFlatten aircraft
| filter isNotNull(aircraft.flight)
| filter isNotNull(aircraft.category)
| fields aircraft.flight, aircraft.category
| sort aircraft.category desc
| fields aircraft.category

 | fieldsAdd aircraft.type = if(matchesValue(aircraft.category,"A1"), "light",
 else:if(matchesValue(aircraft.category,"A2"), "medium",
 else:if(matchesValue(aircraft.category,"A3"), "medium",
 else:if(matchesValue(aircraft.category,"A0"), "heavy",
 else:if(matchesValue(aircraft.category,"A5"), "heavy",
 else:if(matchesValue(aircraft.category,"B4"), "small",
 else:if(matchesValue(aircraft.category,"A7"), "rotorcraft")))))))
 | summarize count(), by:{aircraft.type}

DQL Query to count the number of flights tracked per day, each hour over two months

fetch logs, from:now() - 60d // fetching the past two month of logs
| filter matchesValue(log.source, "opentelemetry-iot-dump1090-collector") // Filter ADS-B logs ingested by the opentelemetry collector on our IoT device
| parse content, "JSON:json_content"
| fieldsAdd aircraft=json_content[aircraft]
| expand aircraft
| fieldsFlatten aircraft
| filter isNotNull(aircraft.flight) AND  isNotNull(aircraft.altitude)
| filterOut aircraft.altitude == "ground"
| fields aircraft.flight, timestamp
| summarize no_of_flights = countDistinct(aircraft.flight), by:{bin(timestamp, 1h), alias:timestamp}
| fields timestamp,`Flight Count` = no_of_flights

Like the above queries, we can extract critical aircraft statistics and data to report them on a dashboard like the one seen below. Some of the statistics that were captured and useful for various other verticals are:

• Total unique flights tracked over the selected time frame
• Time since the aircraft was last seen
• Callsign of the aircraft with the lowest altitude
• Flight tracked per day and hour

(Interested in using this dashboard in your Dynatrace environment? Download the JSON data)

Often, we need to examine data related to a specific aircraft in depth. Adding the aircraft callsign as a variable can achieve this.

In the dashboard below, we dynamically use a variable resulting from a DQL query. This variable is then used as a filter in other dashboard tiles, restricting the view to the specific aircraft the user selects.

(Interested in using this dashboard in your Dynatrace environment? Download the JSON data)

Advanced mathematics and data science in Dynatrace

Looking at the raw data we have just seen, how are we able to tell the distance to an aircraft from a given point? Additionally, how can we determine the distance of the aircraft to the destination airport?

This information is not directly available from the raw ingested logs and metrics but is calculated using the aircraft’s latitude and longitude within Dynatrace. In the previous dashboard, you might have noticed a tile titled “Distance.” That tile’s value value was calculated using DQL.

To provide such insights, we need to apply mathematical formulas to calculate the distance between an aircraft and an airport. We use the Haversine formula to calculate the distance as we already have the aircraft’s latitude and longitude through the ingested logs.

The Haversine formula is an equation used in navigation to calculate the shortest distance between two points on the surface of a sphere, given their longitudes and latitudes. It accounts for the Earth’s curvature and is helpful in determining great-circle distances between two locations.

Distance calculations using trigonometric functions

Dynatrace supports advanced analytics and mathematical functions such as trigonometric, allowing us to accurately compute the distance from the aircraft to the airport on demand.

Applying this formula in DQL provides us with the distance from the Aircraft to the airport. Here is an example of the formula written in DQL:

The trigonometric calculations used above in DQL are shown here:

// Haversine formula calculates the distance between two points on the Earth's surface given their latitude and longitude.
| fieldsAdd a = sin(toDouble(dLat) / 2) * sin(toDouble(dLat) / 2) + cos(toDouble(myLatRad)) * cos(toDouble(aircraft.lat2rad)) * sin(toDouble(dLong)/2) * sin(toDouble(dLong) /2)
| fieldsAdd C = 2 * atan2(sqrt(a), sqrt(1-a))
| fieldsAdd distance = (earthRadius * C)/1000 // Divide by 1000 to convert from mts to kms

The DQL queries above demonstrate the powerful capabilities of Dynatrace in solving complex business problems encountered in the field.

Similar mathematical calculations can be utilized in other industries, such as:

• Warehouse location planning: Combine sales and shipping metrics to determine optimal locations for new warehouses based on distance to the customer.
• Emergency services: Calculate the nearest hospitals or medical facilities for ambulances and emergency responders to ensure quick response times.
• Agriculture field management: Calculate distances between different plots of land to optimize the use of machinery and labor.
• Mining: Leverage real-time distance monitoring of mining trucks and heavy machinery for operational safety purposes.

Monitor IoT devices and environmental data

IoT devices are often deployed outdoors, making monitoring their network signal strength and quality crucial, considering typical IoT networks such as LoRa, BLE, cell signal, and health metrics such as temperature or other environmental data, rather than just CPU and memory utilization.

To monitor such advanced metrics, you can leverage Dynatrace custom metrics functionality. A script can periodically collect additional data and send it to the Dynatrace metric ingestion endpoint.

Conclusion

IoT devices are now common in industrial, agricultural, space, utilities, hospitals, mining, and several other industries. Extracting critical business data from these devices in real time is now possible using OpenTelemetry and the powerful analytical capabilities of Dynatrace.

By integrating ADS-B airplane data and using trigonometric calculations, we’ve provided advanced analytics for actionable insights while visualizing them within dynamic dashboards.

This blog post provides real-world use cases and demonstrates how Dynatrace can extend existing real-time IoT monitoring systems by leveraging OpenTelemetry. It also highlights the value of IoT monitoring across various industries and provides a hands-on guide to using DQL for advanced analytics, empowering you to extract deeper insights from your data.

Start leveraging Dynatrace for your IoT- and edge-computing needs today.

The post Advanced analytics: Leverage edge IoT data with OpenTelemetry and Dynatrace appeared first on Dynatrace news.

The Digital Operational Resilience Act (DORA) adopted by the EU addresses the digital and cybersecurity resilience of financial institutions. Although DORA helps banking and financial institutions prevent, respond to, and recover from critical cybersecurity incidents, DORA compliance could present many hurdles for financial institutions. Applying the DORA framework is especially challenging as organizations move more workloads to the cloud and manage sprawling hybrid and multi-cloud environments.

By proactively improving application security and reliability, organizations can reduce exposures and prevent the type of incidents that DORA is meant to address. An observability and security platform approach can help by automating checks for DORA compliance requirements and improving your overall application security and reliability.

Challenges with DORA compliance

DORA, while aiming to bolster cybersecurity and reliability in the financial sector, presents several challenges for organizations to navigate. Here are some of the key hurdles that institutions need to overcome:

• Complex digital ecosystems. Delivering financial services requires a complex landscape of applications, hybrid cloud infrastructure, and third-party vendors. This complexity can increase cybersecurity risk, introduce many points of failure, and increase the effort required for DORA governance and compliance.
• Third-party risk management. While DORA emphasizes managing risks associated with some third-party ICT service providers, financial institutions have limited control over security practices of these vendors. Effectively assessing and mitigating these external risks requires robust vendor due diligence and continuous monitoring of their cybersecurity posture.  For example, look for vendors that use a secure development lifecycle process to develop software and have achieved certain security standards.
• Resource constraints. Implementing and maintaining DORA compliance can be resource-intensive, requiring skilled personnel, advanced technologies, and ongoing investment. This can divert attention and resources from delivering better customer experience and innovation.
• Integration with existing processes. DORA compliance needs to be seamlessly integrated with existing risk management, incident response, and business continuity processes within the organization. This can require process re-engineering to fill gaps and ensuring clear communication and collaboration across security, operations, and development teams.

Classifying requirements in the DORA framework

DORA defines a wide range of requirements for organizations to achieve compliance. To help determine how customers can comply with DORA requirements, DORA’s articles can be classified in the following three categories:

1. Definition: Describes terms and the scope of the act.
2. Governance: Addresses organizational policies and procedures related to information and communication technology (ICT) risks.
3. Technical: Specifies technical requirements for ICT systems within an organization.

Achieving DORA compliance using an observability and security platform approach

Achieving DORA compliance can involve passing annual audits conducted by independent third parties. Automatically and continuously checking systems to see if they meet the latest security standards not only helps organizations pass annual compliance audits but also reduces the risks of cyber security incidents. Continuous checking helps you update your systems with the latest security standards to meet DORA requirements and reduce cyber risks.

A comprehensive observability approach combined with advanced AI and automation supports organizations toward DORA.

Using continuous Security Posture Management to keep systems compliant

Using the Dynatrace observability and security platform, Dynatrace Security Posture Management (SPM) can digitally and automatically verify the technical requirements specified in the DORA regulations. Specifically, Dynatrace SPM can verify the technical requirements of DORA Chapter II (ICT risk management) and Chapter III (ICT-related incident management, classification, and reporting) and the relevant Regulatory Technical Standards (RTSs) applicable.

While DORA provides high-level definitions, other regulatory frameworks (such as CIS or DISA-STIG) offer technical specifications used as a basis for technical best practices. By combining technical best practices with DORA technical specifications, Dynatrace creates technical checks to monitor your organization’s security posture.

By employing LLMs capable of handling large context windows, the Dynatrace Security Research team maps technical requirements from frameworks (such as CIS) to DORA paragraphs and establishes a knowledge base of technical best practices.

The Dynatrace process involves a unique collaboration between AI and human experts. After obtaining individual mappings between technical recommendations and DORA requirements, Dynatrace compliance experts perform a thorough review. This collaboration ensures accuracy, identifies redundancies, and verifies that technical articles align with relevant checks.

Effective ICT risk management

Dynatrace Runtime Vulnerability Analytics offers AI-powered risk assessment and intelligent automation for continuous real-time exposure management throughout your entire application stack. Dynatrace promptly alerts your security teams about identified exposures and leverages the Dynatrace platform’s topology map to visualize any affected dependencies. Moreover, the Davis AI engine assists in prioritizing what needs to be fixed first.

The Dynatrace platform also delivers runtime application protection for common attack types. Leveraging code-level insights and transaction analysis, teams can detect and thwart malicious activity. With the capability to identify and halt attacks in real time and without configuration requirements, Dynatrace aids in safeguarding your organization against zero-day vulnerabilities. Dynatrace Security Analytics can also improve the effectiveness and efficiency of threat hunts.

Faster mean time to respond (MTTR) for ICT incidents

Dynatrace automatically collects and analyzes large volumes of data to make sense of sprawling multi-cloud application environments. The Dynatrace causal AI engine, Davis, analyzes observability and security data in the context of topology information to group anomalies, pinpoint root causes, and automatically prioritize based on business impact. This single source of truth tears down information silos that traditionally separate teams that perform different functions on many application components. This eliminates the need for manual diagnostics for remediation, improving and reducing the likelihood of material downtime incidents.

Automated business impact analysis

Organizations can use the Dynatrace platform to analyze and automate business analysis, like the business impact analysis required for DORA incident reporting. With Dynatrace Digital Experience Monitoring, organizations can detect and diagnose disruptions to financial services caused by an outage, broken functionality, or performance issues.

Additionally, Dynatrace can capture business data from a variety of sources, like OneAgent, from real user monitoring sessions, log files, and external tools and data sources. Using this information all stored in the Grail data lakehouse, business impact due to an incident can be measured and compared with historical data.

Early warning indicators

Dynatrace provides metrics including service-level objectives (SLOs) and service-level indicators (SLIs) that allow teams to predict problems before they occur and especially before they impact customers. By holding DevOps teams accountable for SLOs, they can take proactive action to increase resilience and reliability and avoid actual downtime. This trains your teams to be proactive in maintaining software quality and saves you money by avoiding downtime.

Better understanding of third-party ICT risk

While DORA requires third-party ICT vendors to disclose critical vulnerabilities to financial services customers, the complexity of modern applications means that some may slip through unnoticed. Dynatrace   can detect and prioritize exposures in any monitored application, including third-party software like COTS and open source. This eliminates the dependency on analyzing a Software Bill of Materials (SBOM) provided by the vendor or completely relying on the vendor’s internal security processes. Additionally, financial services customers can hold their vendors accountable to fix truly critical vulnerabilities in a timely manner.

Continuous digital operational resilience testing

Before any deployment or software release, Dynatrace can automate change impact analysis required for DORA’s digital operational resilience testing requirement with Site Reliability Guardian. For each service, a guardian can automatically validate service-level objectives, “golden signals,” or security vulnerabilities before and after any deployment or configuration changes. It detects regressions and deviations from previously observed behavior, including latency, traffic, error rates, saturation, security coverage, vulnerability risk levels, and memory consumption. Thus, Site Reliability Guardian supports DevOps and SREs in continuously testing application releases to improve release quality.

What’s next

Learn more about how Dynatrace assists with addressing DORA requirements and how Dynatrace complies with DORA in our DORA blog series, and contact your Dynatrace account representative to find the best solution for your organization.

The post Taming DORA compliance with AI, observability, and security appeared first on Dynatrace news.

The previous blog post in this series discussed the benefits of implementing early observability and orchestration of the CI/CD pipeline using Dynatrace. This approach enhances key DORA metrics and enables early detection of failures in the release process, allowing SREs more time for innovation. This blog post explores the Reliability metric, which measures modern operational practices.

Why reliability?

DORA introduced the Reliability metric as “the fifth DORA metric” to address the fact that many organizations improved release times and deployment frequency without adequately considering the reliability of new releases. These releases often assumed ideal conditions such as zero latency, infinite bandwidth, and no network loss, as highlighted in Peter Deutsch’s eight fallacies of distributed systems. To enhance reliability, testing the software under these conditions is crucial to prepare for potential issues by leveraging chaos engineering or similar tools. Chaos engineering is a practice that extends beyond traditional failure testing by identifying unpredictable issues. While it is powerful, it presents several challenges that affect its adoption. In this blog post, we delve into these challenges and explore how Dynatrace can address them to enhance the reliability of released software.

Challenges

Limited awareness of cross-service interactions

In chaos engineering, “hypothesis” denotes the expected behavior of an application under defined conditions or stressors, whether within their owned service or across services. It forms the cornerstone of chaos engineering experiments. In distributed or microservices environments, application teams often lack visibility into how their service will perform under diverse conditions across other services or the entire system. This limitation affects the quality of hypotheses, leading to frequent revisions or inaccurate assumptions.

“It’s not about chaos – it’s about reliability” – Andrus Kolton, CTO & founder of Gremlin

Dynatrace addresses this challenge by offering comprehensive observability across the entire environment. With Dynatrace, teams can seamlessly monitor the entire system, including network switches, database storage, and third-party dependencies. This visibility can be cross-checked in real time using features like Smartscape topology or Service Flow. By leveraging these resources, SREs can formulate more informed hypotheses based on the behavior of their service or application.

Insights: Unclear starting system state

Along with hypotheses, it is imperative to have a clear starting state to validate the experiments’ success. These are referred to as baseline metrics for the starting state, and without appropriate insights into the application ecosystem, it is difficult to form hypotheses. Such baselines constitute a few metrics like:

1. What are the top five problems in your application – CPU spikes, slow response, database connections bottleneck, etc.
2. The problems that take maximum time to resolve – lowest MTTR.
3. Impact of fewer resources, for example, CPU and disk, available to different services and applications.
4. More critical services that are likely to take other services down.

For instance, the above dashboard offers visibility into service and distributed systems through different lenses, such as the slowest MTTR, top problems, or services frequently impacting others. This data simplifies establishing the starting state for chaos experiments or determining the priority services requiring reliability tests and the specific reliability standards to test against.

Blast radius and risk of prolonged outages

While tests in pre-production environments provide valuable insights, proper validation occurs when chaos experiments are conducted in the production environment. However, SREs are hesitant to use chaos engineering during testing or validation stages due to the risk of incorrect hypotheses, potentially impacting multiple services in the production system and leading to prolonged outages. Therefore, their primary concern is to avoid such outages and unnecessary damage to the production system, leading to low adoption of chaos engineering and hindering efforts to improve reliability.

These outages can be mitigated by limiting the scope of chaos engineering tests, known as the “blast radius,” and gradually expanding it as impacted entities are correctly identified and fixed. Dynatrace automatically discovers all components and dependencies within complex technology stacks end-to-end, identifying billions of causal dependencies across websites, applications, services, processes, hosts, networks, and infrastructure within minutes. With this comprehensive monitoring, SREs can utilize holistic monitoring, where situational awareness and Davis® AI-based alerting transition from correlation to causation-based analysis. This enables different teams to understand the semantics of cascading problems, or the domino effect, and pinpoint the root cause for actionable fixes.

In the screenshot below, a chaos engineering scenario introduced latency and resource stress on the “easytrade” demo application. This scenario introduces a 7-second delay for random requests and causes CPU stress on the container. This helps the app team observe the application’s behavior under these conditions and identify any bottlenecks.

In the video below, Davis AI identifies the interdependencies and pinpoints that database slowness impacted multiple services, identifying it as the root cause. Additionally, the details of the new deployment and chaos experiment were included in the root cause analysis. By leveraging service tools and visual resolution, developers can work towards building robust and reliable services. Similarly, by gradually increasing the blast radius, developers can enhance the reliability of the entire application.

Lastly, the SRE team can leverage Dynatrace workflows to automate outages, ensuring virtually no downtime for services or applications.

Conclusion

Ensuring builds are created accurately the first time is crucial, even if it means testing them by injecting failures or remote scenarios. This enhances build reliability and equips SREs and operations teams to swiftly resolve potential issues that could lead to outages. As John Allspaw said, “Incidents are unplanned investments; their costs have already been incurred. Your org’s challenge is to get ROI on those events.”

With Dynatrace advanced monitoring capabilities, SRE teams can confidently overcome challenges and embrace Chaos Engineering. This ensures optimal performance and reliability, enabling organizations to confidently navigate today’s dynamic IT landscapes.

What’s next

In this blog series’ final installment, we’ll explore how Davis AI root cause analysis can be used to autonomously manage your application’s day-to-day operations with the help of Dynatrace workflows.

The post Build systems more reliably with Dynatrace: Chaos Engineering appeared first on Dynatrace news.

The Dynatrace third-party vulnerabilities feature provides insights into vulnerable software packages by matching detected package identifiers and version numbers with known vulnerable software packages and related common vulnerabilities and exposures (CVEs).

The risk of impact from an existing known vulnerability also depends on whether certain processes are using the vulnerable parts of a software component. To give users additional insights, Dynatrace provides vulnerable function usage information for certain vulnerable software packages. This information specifies which function in the source code relates to a vulnerability. The insights provided by this feature help teams to prioritize CVEs based on their reachability through monitoring the vulnerable functions.

Vulnerable function monitoring

Tracking vulnerable open source software components efficiently is one of the most important pillars of managing attack surfaces. The Dynatrace third-party vulnerabilities solution provides key capabilities for detailed and continuous insights into vulnerable software components present in an IT system.

In addition to providing information about which software components are vulnerable, the ability of Dynatrace to gather application insights also enables teams to track which software function relates to a specific vulnerability.

This ability gives security analysts and professionals the advantage of being able to evaluate the risk of a vulnerable software component in detail, depending on how vulnerable functions are used.

Let’s assume the Java library shown in figure 1 is affected by vulnerability CVE-2024-XYZ. The vulnerable function in the software package is highlighted in red. Process group 1 doesn’t use the function that contains the vulnerability. The risk of CVE-2024-XYZ is low in this scenario, since the process isn’t accessing the vulnerable part of the package.

In contrast to figure 1, figure 2 shows a scenario where the process group uses the vulnerable function of the Java package.

In the scenario shown in figure 2, Process Group 2 uses the vulnerable part of the Java package by accessing the vulnerable function as highlighted. In this case, there could be an increased risk of exploitation, for example from a user-provided input sent through Process Group 2 to the vulnerable function.

The example in figure 3 demonstrates the function tracking functionality explained above on a real-world example using the architecture of the Spring Framework.

The Spring Framework has many components that offer different functionalities. In our example, we monitor a simple web application that uses the URIPathHelper utility class to clean user input with the function removeSemicolonContent, which removes semicolons(;) from a given URI.

In the Spring Framework version we use for this example, the spring-web component contains vulnerability CVE-2020-5421, a high-severity CVE that exposes the application to a reflected file download (RFD) attack. This vulnerability affects the functions such as the removeSemicolonContent function we use in our application, as shown in figure 4.

To demonstrate the vulnerable function monitoring, we replace the removeSemicolonContent function with a custom one as shown below.

Now, figure 5 shows that no vulnerable function is now being used.

As we show in this example, vulnerable function monitoring provides users with additional insights and possibilities to prioritize CVEs. In this scenario, we only replace a vulnerable function with a custom one for demonstration purposes. In practice, however, we recommend upgrading third-party dependencies to fixed versions whenever possible.

Leveraging vulnerable function information to focus on the real risks

Software packages often provide a long list of different functionalities that can be used independently of each other. A vulnerability might only affect a very specific part of a software package and may never be called in a specific execution scenario.

Vulnerable functions provide this exact insight and show users of the Dynatrace third-party vulnerabilities feature which process groups are using the specific vulnerable function(s) of a CVE and which are not.

Prioritize CVEs

Figure 6 shows the vulnerable functions that are in use for CVE-2019-0232. , an RCE vulnerability. Although the CVE is high severity, none of the current process groups are using the vulnerable function of the org.apache.catalina package.

Meanwhile, figure 7 shows the vulnerable functions view for CVE-2022-25857, a DoS vulnerability. In contrast to the CVE shown in figure 6, three process groups are currently using the vulnerable functions of the org.yaml.snakeyaml package.

Although both CVEs have a high-risk score of 7.5 as shown in the National Vulnerability Database (NVD), security teams should focus on fixing CVE-2022-25857 shown in Figure 7 since it poses a greater risk to the currently monitored environment.

Monitor for changes in vulnerable functions usage

While the vulnerable functions feature helps to prioritize remediating vulnerabilities, you can also use it to monitor changes in the behavior of process groups. A sudden spike in the usage of vulnerable functions that aren’t typically used by the monitored process group in a software package could be an indicator of an ongoing exploit. In such a case, teams should check the specific process group.

Extending coverage of vulnerable functions

Currently, the Dynatrace third-party vulnerabilities feature covers 784 vulnerable functions from 254 different software packages. The top 10 software packages with the most vulnerable functions are shown in Table 1.

Table 1: Top 10 software packages with the most vulnerable functions in our vulnerability feeds

The Dynatrace Threat Intelligence teams and the Snyk vulnerability feed are constantly extending the coverage of vulnerable functions in software components to provide the best insight for our customers. Figure 8 shows the growth of vulnerable functions coverage in 2024, and we are planning to further extend the number of CVEs where we provide vulnerable functions insights.

On the Dynatrace webpage, you can learn more about our Runtime Vulnerability Analytics offering. See Dynatrace documentation for how to efficiently manage third-party vulnerabilities.

The post Third-party vulnerabilities: Prioritize CVEs to boost digital resilience appeared first on Dynatrace news.

Something not working as expected? Not sure what an error message means? At Dynatrace, we’re committed to helping you get the most out of observability and security to keep your software working perfectly. The perfect place to start is the Dynatrace troubleshooting community forum.

What is the Dynatrace troubleshooting community?

The Dynatrace troubleshooting community is a website that hosts articles written by Dynatrace experts with quick answers to common issues.

If you’re unsure about something, it’s easy to engage Dynatrace support directly. Either open a chat right on the platform or open a support ticket. But you can also find self-service guidance and troubleshooting tips and tricks in the Troubleshooting Community. The forum hosts technical experts throughout Dynatrace.

Offering 24/7 self-service, the Dynatrace Troubleshooting community forum is the perfect place to start your Dynatrace support journey. You’ll find fantastic articles that can help you resolve issues and connect to like-minded practitioners to assist along the way.

Dynatrace engineering teams now contributing directly

For the first time, the Dynatrace Troubleshooting community forum is officially enabling our Engineering (R&D) teams to create customer-facing content in collaboration with our support team and community. With this addition, we’re bringing the best possible knowledge directly to you.

Experts from our R&D teams, Customer Success group, and Support organization share their expertise and experience through these troubleshooting articles to provide you with help and guidance if something doesn’t work out as planned.

For example, the Kubernetes and SSO R&D teams recently collaborated with the support team to provide Dynatrace tips and troubleshooting articles for a professional deep dive into specific issues and resolutions such as the following:

• Tip: Introducing built-in Persistent Volume Claim Monitoring for Kubernetes
• Troubleshooting: Troubleshooting common SAML federation issues with Dynatrace SSO
• Tip: How to use SAML Federation to set up Single Sign On (SSO) in Dynatrace
• Troubleshooting: Troubleshooting common SAML federation issues with Dynatrace SSO

Dynatrace troubleshooting and expertise 24×7

Questions are welcome—the article owner, contributor, or one of our Community super-users, whom we affectionately call DynaMights, will answer them.

We want to offer our customers the best possible self-service options for handling issues. We want searchers to find the abundance of know-how and expertise that welcomes all in to learn and grow with us.

While we have only highlighted a few articles here, we’re always releasing more, and more fantastic information is available to help you perfect your software. We’ll continue to innovate and give out the best knowledge possible.

If you can’t locate the answer to your issue, leave us your feedback so we can improve.

The post The Dynatrace troubleshooting community: Experts tips and tricks to get the most out of Dynatrace appeared first on Dynatrace news.

As an app developer, you have many recurring tasks, such as starting the development server, creating app functions, querying data stored in Grail, managing app configurations, and building and deploying apps. Sound familiar? The Visual Studio Code extension Dynatrace Apps is here to streamline your development process and simplify app building.

App configuration and build management

Let’s start with the basics: Use the App Configuration form (select Configure app in the VS Code Project tree) to set up all necessary options from the app manifest, including name, icon, version, and environment URL, as well as app scopes and content security policies. The extension wraps all important functionality from the Dynatrace App Toolkit to build and deploy your app, manage app dependencies, and test the new version of your app without redeployment.

Working with data

Dynatrace Apps are often used to “bring logic to data,” addressing new use cases on top of data stored in Dynatrace. Now you can easily query live data directly within VS Code using the Dynatrace Query Language (DQL).

First, create a new file, such as logQuery.dql, and enter your new query, utilizing autocompletion similar to Notebooks. The Run query button allows you to select a timeframe, executes the query directly from within the IDE, and shows the result in a new editor window. Then, you can explore the result’s data structure and access the JSON object’s properties.

(Re)Using queries within your app

Once you are happy with the result of your query, you can easily use it in your app code. Adding the annotation @name(“logsByLevel”) and saving the file auto-generates the corresponding TypeScript function runQueryLogsByLevel, ready for import and use within your app.

Dynamic queries

In real-world scenarios, most queries in the context of an app need some dynamic parameterization. This is achieved by adding another annotation. In the following example, we introduce a new parameter, logLevel, and define a default value ERROR by adding @param(“loglevel”, “ERROR”).

When saving the file, the generated function is automatically updated, and the new parameter can be used within the code.

Best practices when working with DQL

We recommend organizing multiple queries within a single file or across different DQL files to enhance the workspace structure. To seamlessly integrate query functions into React state management, utilize our react-hooks Dynatrace SDK package. This aids in effectively handling execution, loading, and error states.

The previously described process of generating a function for your query additionally produces a function named getQueryLogsByLevel, which returns the query along with the specified parameters. This feature proves particularly useful if you plan to implement an IntentButton. Such a button allows users to seamlessly open the query in another application, like Dynatrace Notebooks, for in-depth analysis and further exploration.

Working with app functions

App functions represent the backend, including all the business logic, of a Dynatrace app. They’re used for access to third-party APIs, heavy data processing and manipulation, or encapsulating functionality that needs elevated access rights.

You can create a new app function by selecting the Generate app function and entering a name. Once the function is generated, you can test the functionality by clicking the Play button next to the function name and reviewing the output in the console.

Now it’s your turn

Are you interested in learning more? Watch the latest video from Inside Dynatrace Apps, and see a live demo of the Dynatrace Apps VS Code extension.

If you want to start your own developer journey, head to the Microsoft Visual Studio Marketplace, download the extension, and benefit from increased developer efficiency thanks to simplified app configuration, effortless integration of DQL, and enhanced functionality through Dynatrace functions.

Are you new to the topic of developing apps for the Dynatrace platform? Head to our Developer Portal and learn how to get started by watching our new tutorial.

Happy coding!

The post Turbocharge Dynatrace app development with the new Visual Studio Code extension `Dynatrace Apps` appeared first on Dynatrace news.

To address this challenge, the Dynatrace App Toolkit allows you to create custom workflow actions, providing seamless integration with third-party systems. This extends the capabilities of AutomationEngine for all users in your environment.

Prerequisites

Before you begin, please make sure you have the following:

• Access to a Dynatrace environment (for example https://environment-id.apps.dynatrace.com) where you can install and access Dynatrace® Apps and run Workflows.
• Most recent LTS version of js (version 20, as of May 2024).
• IDE or text editor of your choice (VS Code recommended)

Now, consider your use case! What do you wish to automate? Is there an API available for this purpose? Write down details regarding authentication and the necessary API Calls; you can also experiment with these API calls using tools like Postman, Insomnia, Apigee, or similar.

In this example, we will query electricity prices for the next 24 hours from https://www.awattar.de/services/api (for example, at the time of writing, no authentication is required for GET https://api.awattar.de/v1/marketdata), and figure out what time of day electricity will be the cheapest tomorrow.

Let’s create an app

We’re ready to create an app and implement our first custom Workflow action.

1. Use the Dynatrace App Toolkit to create a new app:

   npx dt-app@latest create --environment-url https://<environment-id>.apps.dynatrace.com

   When asked for the name, choose something short and catchy, and adhere to app naming guidelines. In our example, we will call the app awattar-actions.

2. Once the command above has finished, you should see a new folder (based on the provided app name), which you can open directly in your IDE.
3. Within your IDE, open a new terminal window (you can, of course, also perform the same commands in a terminal window of your choice outside of the IDE) and execute the following command to verify that everything is working:
   npm run start
4. You should now be greeted with your app’s welcome page, which states, “Welcome To Your Dynatrace App.”
   
5. Stop this command in the terminal by pressing CTRL C.
6. Create a new custom workflow action using
   npx dt-app generate action fetch-data
7. Within your IDE (or File Explorer), you should be able to see newly created files in the actions folder (feel free to take a closer look at fetch-data.action.ts and fetch-data.widget.ts).
   
8. Again, run npm run start to verify the setup’s success. Feel free to take a closer look at the output of this command, as it should provide some details about the workflow action.
   
9. We now need to include our changes in the newly created action, so let’s open fetch-data.action.ts and change its implementation to the following.

   interface FetchDataInput {
     start: string | undefined;
     end: string | undefined;
   }
   export default async (payload: FetchDataInput) => {
     // Best Practice: Separate Base URL of the Service and the targetURL of the API
     const baseUrl = 'https://api.awattar.de';
     const apiEndpoint = '/v1/marketdata';
     // re-construct the URL using the built-in URL
     const targetUrl = new URL(apiEndpoint, baseUrl);
     // add query parameters for start and end (if they are defined)
     if (payload.start) {
       targetUrl.searchParams.append('start', payload.start);
     }
     if (payload.end) {
       targetUrl.searchParams.append('end', payload.end);
     }
     // use fetch to call the API
     return await fetch(targetUrl.toString())
       .then((response) => response.json());
   };

10. We also need to adapt our widget fetch-data.widget.tsx to properly reflect the two input parameters in the action: start and end. Change the implementation to what’s shown below.

    import { AutomationTextInput } from '@dynatrace/automation-action-components';
    import { FormField, Hint, Label } from '@dynatrace/strato-components-preview';
    import { ActionWidget } from '@dynatrace-sdk/automation-action-utils';
    import React from 'react';
    interface FetchDataInput {
      start: string | undefined;
      end: string | undefined;
    }
    const FetchDataWidget: ActionWidget<FetchDataInput> = (props) => {
      const { value, onValueChanged } = props;
      const updateValue = (newValue: Partial<FetchDataInput>) => {
        onValueChanged({ ...value, ...newValue });
      };
      return (
        <>
          <FormField>
            <Label>Start</Label>
            <AutomationTextInput value={value.start} onChange={(start) => updateValue({ start })} />
            <Hint>Optional - see https://www.awattar.de/services/api for details</Hint>
          </FormField>
          <FormField>
            <Label>End</Label>
            <AutomationTextInput value={value.end} onChange={(end) => updateValue({ end })} />
            <Hint>Optional - see https://www.awattar.de/services/api for details</Hint>
          </FormField>
        </>
      );
    };
    export default FetchDataWidget;

11. Now, as a final step, deploy your app to your environment using npm run deploy.

Try it out

1. To try it out, go to your environment and navigate to Workflows.
2. Create a new workflow, choose On demand trigger, and add a new task.
3. Search for the action you created, for instance, awattar, and select it.
   
4. You can omit any configuration for now. Let’s try it out; select Save and Run on the top.
5. You’ll likely get the following error message about a blocked request. Please follow the guide provided to allow outbound API access to the host.

   Error: Blocked request to 'api.awattar.de' (host not in allowlist). To find out about how to manage allowed domains.
   please visit https://dt-url.net/allow-outbound-connections

6. Once done, you can edit and rerun the workflow, and you should see a result.
   

Conclusion

We created a simple workflow action within a few minutes to fetch data from a third party API. In an additional step, we can process the received data points and figure out the minimum. This can be done using a run javascript action and the following code snippet:

import { execution } from '@dynatrace-sdk/automation-utils';
export default async function ({ execution_id }) {
  const ex = await execution(execution_id);
  const result = (await ex.result('fetchdata_1')).data;
  // Initialize with first item
  var minItem = result[0];
  // Loop over array and find the actual minimum
  for (var item of result) {
    if (item.marketprice < minItem.marketprice) {
      minItem = item;
    }
  }
  return {
    // convert start and end into something human readable
    start_timestamp: (new Date(minItem.start_timestamp)).toLocaleString(),
    end_timestamp: (new Date(minItem.end_timestamp)).toLocaleString(),
    // keep marketprice and unit
    marketprice: minItem.marketprice,
    unit: minItem.unit
  };
}

When executing this workflow, you should eventually see a result like this:

The post Build custom workflow actions using the Dynatrace App Toolkit appeared first on Dynatrace news.

Logs provide answers, but monitoring is a challenge

Manual tagging is error-prone

Making sure your required logs are monitored is a task distributed between the data owner and the monitoring administrator. Often, it comes down to provisioning YAML configuration files and listing the files or log sources required for monitoring. This manual, error-prone approach can lead to monitoring gaps, which become critical when a host or service has an outage or incident.

Finding the right logs is cumbersome

Even if your logs are monitored, you need to make sense of the vast data volume. As the scale and complexity of your tech stack grows, you might need to navigate the maze of hosts or Kubernetes clusters, apps, and microservices and understand the relevance and risks associated with logs originating from these entities. Challenges compound: Manual tagging of log sources has long been difficult regarding monitoring coverage. And you can’t assume the tagging is 100% correct to pinpoint the correct logs.

In the past, more work was needed to understand the context of log data. What about correlated trace data, host metrics, real-time vulnerability scanning results, or log messages captured just before an incident occurs? This context is vital to understanding issues.

Dynatrace automatically puts logs into context

Dynatrace Log Management and Analytics directly addresses these challenges. First, OneAgent takes care of log autodiscovery. Once logs are selected for monitoring, OneAgent enriches log data with the topological context you need. For example, OneAgent helps you monitor the logs from a Kubernetes environment with automatic enrichment that identifies the right cluster, namespace, container, and pod ID.

Once logs are stored in Dynatrace Grail™, our purpose-built data lakehouse for observability data, the logs are automatically shown in the right context. Finding answers begins with opening the right app for your use case.

You can easily pivot between a hot Kubernetes cluster and the log file related to the issue in 2-3 clicks in these Dynatrace® Apps: Infrastructure & Observability (I&O), Databases, Clouds, and Kubernetes.

Open a host, cluster, cloud service, or database view in one of these apps, and you immediately see logs alongside other relevant metrics, processes, SLOs, events, vulnerabilities, and data offered by the app.

By eliminating slow and manual correlation, lack of context, and getting visibility into the surrounding data, you reduce the risk of prolonged outages, mean time to repair, and tool sprawl.

Get quicker answers

Let’s look at how logs in context can make your teams more effective.

Log histograms: Insight into log volumes and patterns

Open one of these Dynatrace Apps and select Logs for any listed entity (host, Kubernetes workload, cloud service, or database instance):

• Infrastructure & Operations
• Kubernetes
• Databases
• Clouds

You’ll see a histogram chart of log data with various severity levels (such as Error, Info, or Warning) relevant to the selected Dynatrace entity, giving you a clear understanding of log patterns and volumes over time. Is there a sudden spike in errors? A sudden drop in received log data? Depending on which app is in use, one glance at a histogram provides invaluable insight into managing clouds, databases, Kubernetes environments, and infrastructure.

Log analytics simplified: Deeper insights, no DQL required

Your team will immediately notice the streamlined log analysis capabilities below the histogram. Jump directly into log insights by selecting a recommended query, for example, to see the errors related to a problem detected by Davis® AI during the selected timeframe. Furthermore, your team can easily access all error logs within the specified timeframe displayed on the histogram or view all logs within that timeframe, all without writing any queries from scratch.

Surrounding logs display: Effortlessly navigate log context

You can see the result after opening a recommended query without leaving an app’s context. Upon expanding a single log entry, all relevant context provided by OneAgent during the ingestion process is displayed, making it easy to expand your analysis to the infrastructure or entity related to the error logs. For a single log record found, you can easily see the surrounding logs.

Look at this example of an online store payment service generating errors. The application owner found error logs related to unsupported credit cards. Select Surrounding logs to view the log messages for the whole transaction, based on the trace ID, that ended up with an error and a failed order.

In Infrastructure & Operations, surrounding logs can also be displayed based on other criteria, like the host file or log source from which logs are collected. This allows quick and easy troubleshooting without writing or editing queries.

Logs in context across Dynatrace Apps

• Infrastructure & Operations leverages advanced AI capabilities that automatically discover and map all components within your infrastructure, including hosts, virtual machines, containers, and cloud instances.
• Databases offers comprehensive database monitoring capabilities, providing organizations with real-time visibility into the performance and health of their database environments.
• Clouds is a central hub for monitoring and managing multicloud environments, providing organizations with a unified view of their cloud infrastructure and services.
• Kubernetes delivers comprehensive monitoring and management capabilities for Kubernetes environments, enabling organizations to ensure the performance, availability, and scalability of their containerized workloads.

Stay tuned for even wider support of log data embedded seamlessly into the context of Dynatrace Apps, and better ways to get answers from logs without writing queries.

See for yourself

• Watch a demo of logs in context within various Dynatrace Apps in this Dynatrace University course.
• Open your free Dynatrace trial now to get started.

The post Unlock log analytics: Seamless insights without writing queries appeared first on Dynatrace news.

An earlier blog post introduced how Akamas helps optimize Kubernetes clusters without “breaking the bank.” As one of the first Dynatrace partners, Akamas used its domain expertise and unique product capabilities to build a custom app on the Dynatrace platform. The app empowers platform engineering teams with insights into infrastructure health, FinOps, and security combined with alerting and automatic lifecycle management. In this way, the app reduces costs and improves the reliability and performance of all Kubernetes applications.

For example, the Akamas app optimized a Dynatrace-monitored Kubernetes environment, found 22 workloads with reliability issues and 53 workloads with sub-optimal performance, and identified potential savings of about $50,000 per month.

Insights into your Kubernetes environment

Leveraging Dynatrace observability data, Akamas analyzes all available Kubernetes workloads and identifies optimization opportunities such as cost reduction, reliability, and performance improvements.

As you can see in the screenshot below, you get a summary of all optimization opportunities identified in the environment. The table below lists all the individual Kubernetes workloads that can be optimized.

There is a cost reduction opportunity for the notification workload of $2,300 per month and other opportunities related to performance improvements and reliability issues due to misconfigured resource settings.

Optimize your workloads

Select Optimize next to the notification workload to open the Optimize workload page, which offers options for optimizing the workload. Akamas is a goal-driven optimization solution, which means you choose if you want to improve the application’s performance or lower the cost, which in Kubernetes translates to reducing the resource requirements of your containers for CPU, memory requests, and memory limits.

When you manually reduce your container’s resources to save costs, you risk impacting service performance and reliability. To avoid slowing down your apps or—even worse—harming SLOs, you can define constraints that need to be considered, such as response time or error rate. Once constraints are set, Akamas AI considers application-level performance signals so cost-reduction recommendations don’t impact your SLOs.

Before you can start optimizing, you need to define the scope. When you choose Container, Akamas tunes the Kubernetes CPU and memory limits. It also supports full stack optimization, which optimizes JVM parameters such as maximum heap size and garbage collection.

Stay in control: Monitor the optimization process

Returning to the overview, you can switch to the Optimizations tab, which summarizes all your running optimizations, including the optimization created in the previous step.

Select See details for any service to dive into more details about running optimization tasks. Besides showing a short summary, including optimization goals, constraints, and scope, you can dive into cost trends and the relevant SLOs.

In this example, despite the cost going down (from over $80 down to about $50), the application performance was not impacted and stayed well below the defined threshold of 340 milliseconds.

Apply configurations

Akamas can apply configurations automatically or suggest configurations that SRE teams can use for further manual improvements. In the example below, you can see the current CPU and memory limits and the new values suggested by Akamas: change the server.cpu_limit from 1,000 down to 982 millicores and reduce the server.memory_limit by 53 MB. Select the button in the top-right of the Pending Recommendation pane to reveal the kubectl command, which you can use to apply the suggested recommendations.

Summary

The Akamas app helps achieve three critical goals by optimizing Kubernetes application configurations, harnessing the power of artificial intelligence of the Akamas platform, and leveraging the capabilities of the Dynatrace platform:

• Cost reduction: Identify opportunities to trim unnecessary expenses related to Kubernetes workloads. Imagine saving thousands of dollars each month by fine-tuning your resource allocations.
• Reliability enhancement: Pinpoint reliability issues within your Kubernetes environment and ensure your applications run smoothly, minimizing downtime and improving overall system stability.
• Performance optimization: Fine-tune your software stack configurations to maximize performance parameters, ensuring applications meet or exceed their service level objectives (SLOs).

This is a perfect example of how to easily build custom apps on top of observability data stored within Dynatrace and leverage its enterprise-grade platform. Utilize the power of the Dynatrace platform, with its easy-to-use building blocks, to address specific use cases based on your business requirements.

Interested in learning more?

Watch the recording of one of this year’s Perform breakout sessions, where Stefano Doni, CTO and Co-founder of Akamas, and Alois Mayr, Principal Product Manager at Dynatrace, gave a quick intro to how Dynatrace and Akamas work better together, introducing the new Dynatrace Kubernetes monitoring app and how Akamas helps optimize your Kubernetes environment.

Start building your own Dynatrace app to address the specific needs of your users:

• Familiarize yourself with the possibilities for extending Dynatrace.
• Watch a recording and learn how to build your own Dynatrace app in less than 10 minutes.
• Head over to Dynatrace Developer and learn about the tools and technologies of Dynatrace AppEngine.

Are you interested in learning more about Akamas and why Dynatrace and Akamas are better together? Have a look at the Dynatrace Hub, watch a video tour, or contact Akamas directly to book a demo and get more insights into how Akamas can optimize your Kubernetes environment.

The post Dynatrace® Apps showcase: Akamas Kubernetes optimization appeared first on Dynatrace news.

In this blog post, we explore real-life scenarios shared by our customers, partners, and internal users at Dynatrace, offering inspiration to kickstart your own journey with custom solutions powered by AppEngine.

For many, the driving force behind custom solution development is today’s dynamic and rapidly evolving digital landscape, in which organizations find themselves in a race to automate repetitive tasks, drive data-backed decisions, and do more with less. Increasingly, organizations are exploring unified software platforms that eliminate data silos while offering both flexibility and extensibility to safeguard their investments and streamline their diverse tools.

Bring logic to data with easy-to-build apps

As a unified observability and security platform, Dynatrace is designed to be open and customizable from the ground up. By seamlessly integrating custom solutions, Dynatrace empowers data-driven decision-making, automation, and tool consolidation—all while maintaining the familiar out-of-the-box experience. The introduction of Dynatrace^® AppEngine empowers customers and partners to build custom, compliant, and intelligent data-driven applications, ushering in a new era of platform extensibility.

Custom solutions come in all shapes and sizes, providing value for customers, partners, and internal Dynatrace users. Let’s explore some real-life custom solution examples that leverage the power of the Dynatrace core platform technologies to go beyond observability.

Automate disk resizing operations with Davis AI predictive analytics

The Dynatrace Site Reliability Engineering (SRE) team was looking for a way to automatically adjust disk space for cloud volumes on a regular basis to avoid over- or under-provisioning them. They needed a way to automatically calculate the disk resizing point and the target disk size to trigger the resizing operation. Using AppEngine as the foundation, they fed disk utilization data into Davis^® AI to forecast the required resizing operation. With Automation Engine, they were able to call the cloud provider’s API to trigger the desired resizing action.

Developer Observability app provides a shared view of issues

Dynatrace production systems generate about 10 TB of log data each day, which contains valuable information about errors and other issues. Of course, development teams need to understand how their code behaves in production and whether any issues need to be fixed. With hundreds of teams, distributed systems, and terabytes of log data, this is a daunting task. The SRE team’s solution was to create a Developer Observability app that identifies potential problems in log data, grouping, triaging, and identifying the correct teams to work on fixes. The app scans all incoming logs for various patterns and uses DQL (Dynatrace Query Language) to collect context like hosts, other processes, and traces for each potential problem. The app uses Dynatrace EdgeConnect to automatically create a ticket in our on-premises Jira issue-tracking system.

The app provides a dashboard view that gives all development teams an overview of their services, reducing the required effort per issue from many hours to a few minutes.

Custom visualization of user flows

Another example comes from one of our partners, Spindox, who created a custom visualization of user flows to replace an incumbent point solution for one of their customers. Using Dynatrace AppEngine, Spindox was able to implement funnel analytics according to the customer’s specifications, providing them with all the information they needed within a single platform.

AWS cloud cost analysis

To optimize cloud spend, the FinOps team at Dynatrace built a solution that automatically ingests hourly Amazon Web Services (AWS) billing data on a daily basis as BizEvents into a Dynatrace Grail™ bucket. This allows the FinOps team to use DQL to perform analysis, such as identifying how many compute resources are running that could be covered with a savings plan. DQL also allows the team to identify spikes and anomalies in their daily cloud costs. The Notebooks app is used to drill down into the origins of the costs, while Workflows is used for automatic alerting via Slack. Pre-built custom dashboards enable the team to share the hourly billing data with development teams, giving them insights into how architecture and design decisions drive costs.

Improve cloud security posture by analyzing and auto-remediating security findings

Public cloud providers offer automated tools that detect security-related events as they occur. These events require triage, analysis, and remediation by the owners of the affected resources. However, this process can become tedious with hundreds of thousands of events flooding in daily. At Dynatrace, thousands of security tickets must be generated from these events, followed by manual remediation. Once the risk is mitigated, the resource owner must validate the resolution with the security team.

Hyperscaler security events are ingested into Grail as BizEvents. Subsequently, they undergo severity-based filtering, followed by analysis and grouping based on findings and resources. Finally, actionable Jira tickets are generated for the identified issues and assigned to the relevant owners for remediation.

The possibilities are boundless

As these examples demonstrate, the possibilities are boundless, and we’ve seen many more use cases shared by our customers and partners:

• File transfer monitoring: When migrating files from one server to another, know when a file arrives, leaves, is early, late, or missing. Use Dynatrace as a single source of truth to identify missing files immediately and take corrective measures.
• Industry software monitoring: Import telemetry data emitted by industry vertical enterprise apps (such as Temenos for banking) and build a custom app to analyze and enrich the data to trigger automated actions.
• Problem enrichment: Enhance Dynatrace-detected problems with external data. For instance, when Dynatrace identifies an issue, you can pinpoint the impacted customers using Salesforce data.
• GitHub actions profiler: Analyze data generated by GitHub action workflows and get insights into their performance and efficiency.
• Business process monitoring: Create custom dashboards for business process monitoring and trigger automated remediation actions.

Remember, as an open and extensible platform, Dynatrace doesn’t confine you to pre-defined use cases. We equip you with the necessary tools and support to extend Dynatrace by leveraging AppEngine, AutomationEngine, Davis AI, Dashboards, and more. Alternatively, you can collaborate with our AppEngine-certified partners to bring your custom use case to the Dynatrace platform.

The post Automate your work with unified observability and custom solutions appeared first on Dynatrace news.

Perform is our company’s event once a year in Las Vegas, where our customers and partners visit us to learn more about our product and industry.

It was not my first time in Las Vegas. Therefore, the slot machines next to the baggage carousels and the bright lights that beamed into the night sky did not surprise me when I landed in this desert land’s organized chaos. However, it was my first time at Perform, and although I knew I would learn a thing or two in the next week, I was unaware of how beneficial taking part in this event would be.

Getting my hands dirty: Hands-on training at Perform

I was invited to join Perform to help host one of the HOT classes two days before the main conference event. These classes allow our customers and partners to learn more about specific areas of the Dynatrace product that interest them and their company. This year, the workshop-styled classes attracted nearly 1,500 attendees eager to learn more about “Power dashboarding,” “Advanced Diagnostics,” “Intro to DQL,” and a dozen other topics.

Over the two days, I helped run four HOT classes with Dirk Wallerstorfer, Radu Stefan, and Vagiz Duseev. Our classes were focused on building custom apps with AppEngine.

HOT class: Building apps with Dynatrace AppEngine

As this class name implies, we guided  attendees through the journey of building a Dynatrace app with a code-along example app. EasyTrade Analytics is a hypothetical stockbroker app developed by two members of the Platform enablement team, Sinisa Zubic and Edu Campver. The app displays account data of the virtual stockbroker “EasyTrade” and visually indicates if there have been any fraudulent activities on the accounts.

Our goal was that by the end of each HOT class, our attendees would feel confident to start brainstorming how they can use Dynatrace Apps to extend their current Dynatrace experience and to know where to go for all the resources they need to get started building their own apps.

While building the app with the attendees, we got a real sense of who our attendees were and what they wanted to achieve. And for me, it was the first time I had the opportunity to engage directly face-to-face with our customers, allowing me to put myself in their shoes and see their needs and struggles. This face-to-face interaction helped stimulate great conversation with the attendees and allowed us to go deeper into relevant topics of app development and help them get the most from the class.

Developer personas

To my surprise, I discovered that our attendees were a scattered mix across industries, with customers from airlines, healthcare, consulting, and insurance to supply chain, distribution, financial services, and likely more. This showed me how vast and versatile the Dynatrace product is and how it can be used for many use cases. And yet, our customers’ needs and wants for custom app development are quite similar.

There was one part of our class where we showed the attendees how to add annotations to the timeseries chart that will indicate the fraudulent activities in the EasyTrade analytics app. In each HOT class, this seemed to be the eye-opening moment for our attendees, where the app went from just a nice UX experience to a purposeful use case with something each of our attendees related to.

As expected, we had a wide range of coding abilities in each class, ranging from the advanced developer who finished building the app when we reached the second exercise to those who had recently started coding. One of my favorites was an attendee who had already tried to develop their app before the HOT class, having self-studied at a TypeScript and React crash course.

This proved to me that our assumption is correct: customers that want to create apps come with all types of experience in programming, reminding me of the importance of clear and well-structured documentation to ensure all levels can find the appropriate information when developing apps.

My takeaways from the HOT class

After completing the exercise, we encouraged our attendees to share what they would like to create using Dynatrace Apps. The attendees didn’t hold back and seemed more than happy to share their ideas, which caused a waterfall effect with every idea becoming more adventurous than the last; below are some of the ideas I took note of during the classes:

• An app for fraud alerts and alerts for fraudulent links.
• An app for tracking the custom user behavior of their customers.
• An app to create alerts that could be sent to their engineers.
• An app for helping diagnose bot traffic.
• An app with a highly customizable dashboard for a call center.

The big show starts

After the two days, with a mix of jet lag, early mornings, and extended teaching sessions–it didn’t stop!

In the following days, I helped manage the booth for Exploratory Analytics and Custom Solutions; this allowed me to discuss at length with our customers and many of the HOT class attendees what they wished to achieve when creating custom apps.

Additionally, I was able to sit in on many of the main stage events, learning more about the vision of Dynatrace from our CEO, Rick McConnell, and how Dynatrace faces many of the industry challenges from our CTO, Bernd Greifeneder. Even an exceptional performance from a local group of acrobats took the stage midway through the day!

Also, I sat in on one of our breakout sessions, hosted by Dirk Wallerstorfer and Paul Schumacher, who presented in a very comical way, “Dynatrace Apps: build your own in 10 minutes or less.” You can watch the recording from Perform on-demand.

Finally, I managed to squeeze in some time to record an interview for our newest episode of our YouTube series, Inside Dynatrace Apps, where I interviewed Dirk Wallerstorfer, a product manager at Dynatrace about the why and when you would want to create a Dynatrace App. You can watch the episode now on YouTube, by visiting Dynatrace Apps: Why and When.

From the HOT classes to the main conference, I would be lying if I told you I was not tired after the event. Although it left me feeling revitalized after the experience. Within a few days I had learned so much. Perform was a great event to widen my perspective, see first-hand from the customers, and learn more about the other areas of the Dynatrace product. And, of course, it’s nice to put names to the faces of many of my colleagues!

What you should do next

Are you interested in learning more about the work of us “Developer advocates” or do you want to share your personal experiences from this year’s Perform? Head over to the Dynatrace Community and let us hear your feedback or join our monthly office hours for Dynatrace App development and learn more from myself or my colleagues, Sarah and Sini.

The post Experiencing Perform: The diary of a Developer Advocate appeared first on Dynatrace news.

The growing complexity of modern multicloud environments has created a pressing need to converge observability and security analytics.

Security analytics is a discipline within IT security that focuses on proactive threat prevention using data analysis. As attackers become more skilled, threats can become more detrimental to organizations if they go undetected. A proactive approach to application security is essential: by constantly collecting and analyzing data, security analytics enables teams to catch problems before they escalate.

In the past five years, delivering innovation more efficiently has remained at the forefront of customer demand. As environments began to scale with cloud-native and microservices architectures to meet this demand, security approaches didn’t evolve accordingly. The result: Environments are more vulnerable to threats and more difficult to secure. In fact, according to a recent survey, nearly 70% of chief information security officers agreed that vulnerability management has become more difficult as the complexity of their software supply chain and cloud ecosystem has increased.

The ripple effect of increased risk compounds the problem. With more alerts and greater difficulty identifying false positives, security teams experience frustration and burnout. A lack of common tooling, common language, and collaboration further inhibits productivity and prolongs remediation.

At the 2024 Dynatrace Perform conference in Las Vegas, Gerhard Byrne, Dynatrace principal product manager, and Susan St. Clair, principal security solutions engineer, discussed how organizations can take a more proactive approach to threat detection and incident resolution.

During their breakout session, Byrne and St. Clair demonstrated how the Dynatrace platform accelerates remediation by enriching security data with observability context and actionable insights to protect environments against exploitation or lateral movement.

Threat hunting expectations vs. reality

In a perfect world, threat hunting and incident resolution would be a linear, straightforward process. Ideally, after fetching data and filtering, an organization could enrich the findings with observability data to get better insights into the nature of the alert. With these insights, the team could identify the threat and understand the nature of the incident. This allows them to react accordingly and return the system to a secure state.

But with the complexity of modern cloud environments — including the associated software supply chains and siloed toolchains — and the increasing sophistication of today’s attackers, threat hunting is unpredictable. In reality, security teams aren’t aware of all the unknown unknowns in their environments. After fetching, filtering, and enriching information with observability data, security teams might change their hypothesis about what’s occurring. This revision of assumptions might happen multiple times, causing engineers to lose track of previous hypotheses, patterns, and evidence. Keeping threats documented is a challenge: Engineers typically open numerous tabs to maintain context, which is tedious and can create error. Remediating a vulnerability can thus take far longer than anticipated, which can be detrimental when the risk is high.

“As defenders, we need to embrace different paths and possibilities like our adversaries are doing today,” Byrne said. “Just going down a checklist will not help you find new threats.”

Streamline threat hunting and accelerate resolution with Dynatrace Security Investigator

During the conference, Dynatrace announced the new Security Investigator app on the platform. The app enables security teams to investigate threats faster, obtain accurate and observability-enriched results, and maintain context throughout the weaving paths on which security investigations might lead.

Security Investigator demo

St. Clair began her demonstration of the app with the following scenario: She receives a Slack alert that an anomaly was detected and there has been unauthorized access to a Kubernetes cluster monitored via OneAgent.

To begin, St. Clair filtered the alert for a time window of a couple of hours to focus her analysis. Using Dynatrace Query Language in Grail, St. Clair determined what log data was available to her. With each execution, data appears in a query tree. “As I’m building out this investigation, each of these nodes is being created for me automatically,” she said. “As part of that documentation, I can easily go back and forth to see what was executed.”

St. Clair then used the Dynatrace Pattern Language (DPL) to make the data more usable. She used the DPL Architect to create her own patterns in addition to the platform’s out-of-the-box patterns for parsing the data. After running an audit log pattern, she wanted to understand why she received the Slack alert and which object the attacker had accessed so she could then focus on the unauthorized responses. Running this query, suspicious IPs arose, and she saved them as evidence in a new folder she created within the app.

As she continued to execute queries, St. Clair’s hypothesis began to change. She saw substantial traffic in a specific port, which was not necessarily malicious, but it was abnormal enough to warrant additional investigation. Fortunately, the query tree automatically creates new “branches” to support changing hypotheses and help engineers keep track of evidence and patterns. The query tree thus frees security professionals from tedious manual documentation, allowing them to focus entirely on finding the unknown unknown.

Finally, St. Clair found running malware on the system, pivoting the investigation from the initial authentication alert. By the end of an investigation, she had a visual representation of the process from start to finish.

“I can keep track of where I went. [The data is] documented, shareable, collaborative, and available for further investigation,” St. Clair said.

The road ahead: enriched security analytics with Dynatrace

Organizations can benefit most from their security investigations using the abundant data in the Dynatrace platform. The platform’s broad and deep observability identifies where problems initiate as well as their dependencies using PurePath. Teams can use Real User Monitoring and Session Replay to track user-facing activity in real time. This helps them understand how an attacker accessed an application, how they interacted with it, how traces originated, and more. Teams can also create management reports and use Dynatrace Notebooks to easily share their security investigation data with their peers.

“Security is a team sport,” Byrne said. “The next time you’re in a war room, you can be the person who provides insights and conclusions based on the wealth of data that is available at your fingertips with Dynatrace platform.”

The post Advanced security analytics to resolve incidents quickly and streamline threat hunting appeared first on Dynatrace news.

Great news: OpenTelemetry endpoint detection, analyzing OpenTelemetry services, and visualizing Istio service mesh metrics just got easier. With the launch of unified services for OpenTelemetry, Dynatrace enables teams to analyze multiple aspects of OpenTelemetry services in a single view.

Here’s more about unified services and how the capability enhances OpenTelemetry observability.

Why Dynatrace is the best way to analyze OpenTelemetry services

OpenTelemetry is growing in popularity as a way for organizations to standardize observability instrumentation for all their services, languages, and frameworks. As a CNCF open source incubating project, OpenTelemetry provides a standardized set of APIs, libraries, agents, instrumentation, and specifications for logging, metrics, and tracing.

With more than 1,106 companies and 9,168 individuals contributing to OpenTelemetry since the project’s inception in 2019 and with adoption growing worldwide, OpenTelemetry is a good solution for bridging gaps with business-specific instrumentation. OpenTelemetry is also useful for getting a handle on all the third-party libraries and services an organization uses.

But while OpenTelemetry solves the problem of standardizing data instrumentation, the complexity and scale of cloud-native tools and environments mean that organizations still struggle with OpenTelemetry service observability.

Specifically, teams must jump from tool to tool and page to page to extract the benefits of all the signals OpenTelemetry collects. This context switching makes it hard for teams to effectively connect the dots when it comes to integrating the service-specific insights with surrounding architecture components such as service-mesh data. Moreover, the number of services in an enterprise’s portfolio makes it impractical to manually configure and adapt alerting for the tens of thousands of service endpoints teams need to monitor.

The new unified services type for OpenTelemetry solves these issues and more by auto-baselining and detecting all of a service’s endpoints, then bringing all logs, metrics, traces, and service-mesh data into a single unified, contextualized view.

A unified platform approach also makes OpenTelemetry data available to more teams across the organization for more diversified analysis.

Let’s highlight two major improvements provided by the Dynatrace Opentelemetry unified services view: Endpoint detection and in-context Istio service mesh metrics.

Unified services enable automatic OpenTelemetry endpoint detection

With unified services, Dynatrace detects the different endpoints of a service and automatically baselines and triggers Davis AI anomaly detection for the detected endpoints.

“This is important because different endpoints expose different functionality of a service,” says Arlindo Lima, product manager at Dynatrace. “[The endpoints] might have different SLAs (service level agreements) or completely different performance characteristics.”

For example, one endpoint might be more heavily used than others, and therefore more likely to experience failure. “Not monitoring endpoints separately can mask issues and make it more difficult to identify the root cause of problems,” Lima says. By automatically detecting these OpenTelemetry endpoints, Davis AI adds the endpoints to its service list for analysis and alerting with no additional setup or configuration required.

“The automated Davis AI baselining capabilities can now alert teams when their endpoints degrade, which accelerates resolution time,” Lima continues. “With one click, Davis analyzes thousands of signals, taking into consideration the domain and topology knowledge, and surfaces all the anomalies detected from the selected endpoint.”

Unified services include Istio service mesh metrics ingested from OpenTelemetry spans

With unified services, keeping track of service mesh metrics from Istio Envoy proxies just got easier, too.

Service meshes are a dedicated infrastructure layer added on top of an application that abstracts service-to-service communication using a proxy, often deployed as a sidecar, which runs alongside each service.

“When Dynatrace receives OpenTelemetry line protocol spans from Istio’s Envoy proxies, Dynatrace now automatically shows the performance of the proxy directly on the service screen,” Lima says. This detail from Istio service mesh is available using APIs now and will soon be available directly from OneAgent.

Having instant access to these granular performance metrics of Istio Envoy proxies provides teams with the following advantages:

• Immediately see if the proxy is causing any significant performance overhead
• Understand if requests are being blocked or not proxied to the actual microservice
• Identify any deviating failure rates that indicate problems
• Better understand how the service mesh layer is affecting overall service performance

Benefits of unified services and how to get them

With automatic OpenTelemetry endpoint detection and baselining, Davis causal AI can now operate on OpenTelemetry endpoints to give teams immediate, granular visibility into endpoint performance. The unified services view makes it easier for teams to track service calls, traces, logging, and events in context.

What’s more, the new service type simplifies the service and endpoint detection rules Dynatrace uses to detect new services and services endpoints, and makes them more transparent.

New Dynatrace customers already have these features enabled by default. Existing customers can activate the unified services view by following the link in the banner to Dynatrace settings.

Dynatrace OneAgent + OpenTelemetry: Bringing all your telemetry data under one roof

Once you start enjoying the benefits of using the Dynatrace unified service for OpenTelemetry analysis, it makes sense to get all your telemetry data under one roof.

Dynatrace OneAgent, the Dynatrace platform’s native telemetry data collector, can auto-instrument applications and infrastructure that are not already defined by OpenTelemetry. This means you can use OpenTelemetry to define traces for custom or specialty apps and let Dynatrace auto-instrument the rest of your cloud-native environment. OneAgent works out of the box with OpenTelemetry and automatically detects application traces and metrics, bringing data from the entire technology stack into a single analytics platform.

Put simply, Dynatrace extends OpenTelemetry observability, and OpenTelemetry extends Dynatrace observability.

________________________________

Many thanks to Arlindo Lima and Matthew Reider for contributing to this article.

The post OpenTelemetry services analysis and endpoint detection made easier with Dynatrace unified services appeared first on Dynatrace news.

We’ve all been there. An important fix is waiting to be delivered, and a manager asks you why it’s taking so long and says that the pipeline was much faster a few months ago. Although you disagree with the assessment, you can’t dispute it because you don’t have the data to show that the pipeline is not slower.

At Dynatrace, Jenkins is the most used Continuous Integration (CI) tool. It’s open source, allows for great scalability with the possibility to run workloads on the cloud and on-premises Kubernetes clusters, and is easily extensible via plugins.

To monitor these pipelines, OpenTelemetry can be used to trace every step of the build and send the traces to various backends. You’re probably already familiar with the view of a trace in Dynatrace, as shown below. If you’re also familiar with Jenkins, you’ll notice that the trace steps are the same as in your Jenkins builds—you get a one-to-one mapping.

In this trace, we see that the checkout took up a large portion of the build. The question is, what can be done with this data? How can you show things like the failure rate of a pipeline during the last two weeks with as little effort as possible?

What we measure at Dynatrace

It’s important to agree upon what should be measured; there is no right or wrong answer here. At Dynatrace, we measure the DORA metrics from Google.

These include:

• Deployment frequency: How often is code deployed?
• Lead time for changes: How long does it take to get committed code into production?
• Time to restore service: How long does it take to restore a service?
• Change failure rate: How often does the service fail following changes?

While tracking the time to restore a service and the change failure rate require an issue management system, the first two metrics can be acquired by looking at the pipelines.

Unified services: Easy, out-of-the-box service visualization

If you use GitHub or GitLab, you can use the Four Keys project (https://github.com/dora-team/fourkeys), which can be integrated into your project, or you can write your own service that takes the OpenTelemetry data and executes the necessary operations on it.

But if you want to get everything out of the box, you can use Dynatrace unified services. This service visualization experience for OpenTelemetry was introduced with  Dynatrace version 1.274. As soon as you have unified services enabled on your Dynatrace tenant, your OpenTelemetry API-ingested services will look like the image below.

There’s an endpoint for each branch that exists within Jenkins. You get the number of failures, the throughput, and the response time, which maps to the duration of the build.

Now, let’s jump to a real-world example. Let’s say that your developers believe that their pull requests sometimes take longer than usual, not a lot, but a noticeable amount. And sometimes, the pull requests aren’t triggered at all. In such a scenario, check the durations of all the pull requests. In this case, we can see a clustering of pull requests and some outliers. With this information, we can guess that the webhook to Jenkins isn’t correctly configured, and only when the pipeline is scanned on Jenkins are the PRs picked up.

Normally, you get one endpoint per pull request, but as you can see, we summarized all the pull requests into one chart. This can be achieved by adapting the query a bit.

builtin:service.request.response_time_chart:filter(and(eq("dt.entity.service",SERVICE-AE4EB9E449B856CA),eq("endpoint.name","EP/backstage/PR-156"))):splitBy("endpoint.name"):median

You can adapt the equals to a contains and remove the PR number from the endpoint name and you get data for all pull requests in one chart.

builtin:service.request.response_time_chart:filter(and(eq("dt.entity.service",SERVICE-AE4EB9E449B856CA),contains("endpoint.name","EP/backstage/PR-"))):splitBy("endpoint.name"):median

Now, let’s say we want to check the Backstage pipeline. The first DORA metric is rather easy; we simply look at the throughput of the main branch. You can view the throughput in the Dynatrace Data Explorer, which translates to the number of builds that ran. In the view below, you can see the number of builds in the last two hours.

To learn the amount of time it takes until a commit appears in production, you need the duration of pull requests and the duration of the main branch. The chart below shows you the duration of builds on the main branch.

The chart for the main branch works out of the box. For the pull requests, we must adapt the query a bit, because based on the built-in rules that come with the unified services, they create a new endpoint for every pull request, which allows for exact control, but this is not what we want in this case. So, we will leave the query as-is .

One last thing that you might be interested in is the failure rate of the main branch. As with the other metrics, simply search your main branch, view the data in the Dynatrace Data Explorer and be happy that your main branch only broke twice in the last seven days.

Sneak-peek time!

While Traces powered by Grail™ is still in development, we already use it for internal use cases and can give you a sneak peek. With a simple Dynatrace Query Language (DQL) query, we can filter for each and every step of a trace, or as is the case in this example, a Jenkins pipeline. As you can see, the allocation times look fine, but there is a huge outlier. This outlier points us to an overloaded cloud; too many workloads were running, and the cluster ran out of resources. Therefore, the pods had to wait a long time to be spawned. With this data in hand, we don’t need to rely on a gut feeling—you can now back up your report with data.

See the current state of your pipeline at a glance

With Dynatrace unified services, you can assess the state of your pipelines at a glance and can make confident statements based on data and not your gut feelings. You can now provide answers to tough questions like “Are the pipelines getting slower?” or “Are my builds breaking more often?” Another big advantage is the fact that once you have the services enabled, you have the data available all the time and no longer need to search when new questions arise. If you want a deeper look into this topic, watch Dynatrace Tips & Tricks #18 – Jenkins Pipeline Analytics with OpenTelemetry.

Try it yourself

Using Jenkins? The following links help you to export the traces from Jenkins and to ingest them into Dynatrace.

• Jenkins OpenTelemetry Plugin
• OpenTelemetry Collector
• Dynatrace Tips & Tricks #18 – Jenkins Pipeline Analytics with OpenTelemetry

The post Unified services: Easily keep track of your pipelines appeared first on Dynatrace news.

Establishing SLOs for Kubernetes clusters can help organizations optimize resource utilization. A good Kubernetes SLO strategy helps teams manage and make containerized workloads more efficient.

Kubernetes is a widely used open source system for container orchestration. It allows for seamless running of containerized workloads across different environments. However, managing Kubernetes optimally can be a daunting task due to its complex architecture. Properly monitoring a Kubernetes cluster or any related environment can be difficult.

Effective resource provisioning and management is a critical aspect of a Kubernetes cluster. It involves a coordinated effort among various stakeholders, including cluster owners, application teams, and business owners. The primary goal is to allocate sufficient resources to keep the applications running smoothly without overprovisioning and incurring unnecessary costs. This delicate balance requires a collaborative approach and transparent communication among all parties involved. Service-level objectives (SLOs) play a vital role in ensuring that all stakeholders have visibility into the resources being used and the performance of the applications.

SLOs are often used to monitor business-critical services and applications for customers. However, they can also be used to monitor optimization processes effectively. Essentially, SLOs track a selected service-level indicator (SLI) and continuously evaluate its behavior over a given timeframe against a fixed threshold. This feature is valuable for platform owners who want to monitor and optimize their Kubernetes environment. By considering historical behavior, SLOs provide an excellent way to track and evaluate optimization tasks. Users can continuously evaluate the system’s performance against predefined quality criteria, making SLOs for Kubernetes clusters a good option for monitoring and improving the system’s overall performance.

Efficient coordination among resource usage, requests, and allocation is critical. To optimize resource utilization in a Kubernetes cluster, all users must work together seamlessly. A Kubernetes SLO can serve as a transparent and trackable collaboration tool for different teams and collaborators.

Optimize memory utilization of your Kubernetes Namespaces

Monitoring your Kubernetes cluster allows for proactively identifying and resolving resource constraints, failures, and crashes before they impact the end-user experience and your business.

Proper Kubernetes monitoring includes utilizing observability information to optimize your environment. By gaining insights into how your Kubernetes workloads utilize computing and memory resources, you can make informed decisions about how to size and plan your infrastructure, leading to reduced costs. A Kubernetes SLO that continuously evaluates CPU, memory usage, and capacity and compares these available resources to the requested and utilized memory by Kubernetes workloads makes potential waste of resources visible, revealing opportunities for countermeasures.

When it comes to resource utilization in a Kubernetes environment, there are two main perspectives to consider based on the primary stakeholders involved. One perspective focuses on the potential for optimization at the interface between the team responsible for managing the Kubernetes cluster and the teams responsible for developing and deploying applications.

When setting up SLOs for Kubernetes clusters, it is important to choose the right metrics to track based on your objective. If your team is responsible for setting up the Kubernetes cluster, you might want to monitor and optimize the workload performance. However, if you’re part of the application team, the usage of reserved resources may differ significantly from the blocked resources.

Requests represent the number of resources being reserved or blocked for a container. Tracking the ratio between request and usage can provide valuable insights into optimization potential. As every container has defined requests for CPU and memory, these indicators are well-suited for efficiency monitoring.

One option is to continuously track the memory utilization efficiency of existing Kubernetes objects, such as namespaces or workloads. Since teams typically run multiple workloads on namespaces, using this level of abstraction is a suitable option for a Kubernetes SLO. However, if you require more granular information, you can adjust the levels for resource utilization monitoring accordingly.

This Kubernetes SLO measures the ratio between the requested memory and the memory used for an entire namespace. It provides insights into how efficiently the blocked resources are being utilized. Since resources that have been requested cannot be used elsewhere, the objective is to keep the difference between requested and used resources as small as possible. Utilizing such SLOs for Kubernetes clusters makes it possible to track efficiency transparently over time. The teams responsible for the cluster and the application teams that run their containers on the cluster can agree on the intended ratio between used and requested memory.

Optimize Kubernetes cluster’s resource allocation

One aspect of managing cloud resources is tracking and adjusting the ratio between the requested and used memory resources. On the other hand, the cluster owners must allocate cloud resources to meet the application team’s resource requests.

For cluster owners, monitoring resource usage at the node level provides better insights and information for taking sound actions. By tracking resource usage at each node, teams can gain insights into how many resources the entire cluster uses and whether the nodes are working correctly and efficiently.

Teams should implement suitable SLOs to continuously monitor resource usage for Kubernetes cluster nodes at the node level. These SLOs should cover metrics like node memory utilization, which involves monitoring the ratio of requested versus allocated resources or the ratio of desired versus running pods per node.

For instance, if a node’s memory or CPU utilization is high, it can lead to the undesired deletion of pods. This, in turn, can disrupt an application or service and incur additional costs for potential cluster upscaling.

A possible Kubernetes SLO for monitoring and evaluating the ratio of requested versus allocated memory can be expressed as follows:

In an ideal environment, the requested resources would almost match the allocated ones, and hence, cloud resources would be optimally used, reducing costs of invoiced but unused resources.

Ensure overall Kubernetes resource utilization efficiency

The two SLOs mentioned above provide valuable insights into the overall utilization of resources in a Kubernetes cluster and are a good starting point for improving efficiency. Although both interfaces (usage vs. request and request vs. allocation) must be analyzed together to obtain a holistic view, the split is extremely helpful in ensuring accountability. While the application teams have complete control over the usage/request SLO, the cluster owner can influence the request vs. allocation SLO.

Setting up proper SLOs to make these ratios visible and transparent does not reduce the need for collaboration and alignment, but it provides a solid foundation for optimizing resource utilization efficiency.

The outlined SLOs for Kubernetes clusters guide you in implementing SRE best practices in monitoring your Kubernetes environment. By recognizing the insights provided, you can optimize processes and improve overall efficiency. This makes it an excellent tool for collaboration between different contributors while also holding the respective parties accountable.

What’s next with SLOs for Kubernetes clusters?

Increase your resource utilization efficiency with Dynatrace using the SLOs mentioned in the blog. If you want to learn how to set up SLOs with Dynatrace, have a look here:

• Service level objective examples: 5 SLO examples for faster, more reliable apps
• Lessons learned from enterprise service-level objective management
• SLOs done right: how DevOps teams can build better service-level objectives

Or check the following Kubernetes blogs if you’d like to learn more about Dynatrace capabilities to increase efficiency and reliability in your Kubernetes environment:

• Dynatrace memory analysis helps Product Architects identify unknown unknowns
• How to maximize Kubernetes monitoring with AI-driven automation
• Kubernetes in the wild report 2023

The post SLOs for Kubernetes clusters: Optimize resource utilization of Kubernetes clusters with service level objectives appeared first on Dynatrace news.

We’re giving you endless possibilities. While you were bound to existing product offerings and features in the past, you now have the power to do anything you want with Dynatrace.

But we know that with endless possibilities, fear can manifest. Fear of the unknown, what to create, how to create it, and so on. But we can’t let this fear set us back! Fear is not necessary, and throughout this blog post, we will cover some key points to inspire confidence in creating Dynatrace Apps.

Developer skills are helpful but not required to get started

It’s natural to feel unsure when confronted with something new or foreign. Just opening a new user interface of a familiar tool can feel so daunting that you can find yourself asking, where did all the buttons go? I want to go back to how things were before.

Web development over the years has made some significant improvements, and the job of a developer has become more comfortable with the assisted aids, such as Chrome Developer Tools or Visual Studio code (including GitHub Copilot), that the industry now offers.

Developing Dynatrace Apps is no different, though the word “developing” suggests that this is something that only developers can do. This couldn’t be further from the truth because creating a Dynatrace app can be as simple as writing…

…into your terminal. This single line of code downloads everything you need, including the Dynatrace Apps Toolkit. Of course, there are a few other bits to consider, but even non-technical people can get started after watching this QuickStart video. I should know; I’m a developer newbie and made the QuickStart video!

By the end of the QuickStart tutorial, you’ll know how to develop, make changes, and deploy your first app without any prior developer experience. You just need a computer, a stable internet connection, and your Dynatrace environment URL.

Good developers copy; great developers paste

When joining the app development world, I came across this phrase, “Good developers copy, great developers paste,” with a critical comment underneath, “The best developers understand.”

It’s rare to find someone who wants to start something from the ground up, mainly because it is expensive and time-consuming. And that’s why it’s important to us at Dynatrace to ensure you have all the information you need to create your Dynatrace custom apps seamlessly.

Many building blocks go into creating your apps, such as the Strato design system that provides you with the fundamental and customizable UI components to make your app look the part, app functions that operate as the backend of your app and can be used within your apps for use cases like when you want to access third-party APIs within your app or need to process heavy data. You can see more use cases by visiting the app functions documentation.

To support using these building blocks, we provide you with our Dynatrace Developer documentation, which acts as a one-stop shop for all information about creating a Dynatrace custom app. Not only do we provide you with full code snippets, but we also provide extensive written and video documentation explaining each one of our tools–all to ensure that you can be one of the best Dynatrace custom app developers.

To top it off, this is the same documentation we use internally at Dynatrace—giving you the most up-to-date and precise documentation for Dynatrace custom app development. We even provide you with several sample apps that act as inspiration and a collection of best practices. These fully functional apps cover common tasks such as querying data, using the Dynatrace API, creating visualizations… and much more. You can download the source code from GitHub and kickstart your app development journey.

You’re not alone

We know there will always be moments of haze and confusion. Perhaps the documentation does not solve your problem, you have an error, or you can’t find the answer. This is why our Dynatrace Community provides you a developer space to connect with fellow Dynatrace custom app developers. Inclusive of the developer Q&A forum, where you can ask any questions related to the topics, and the developer blog section, which you’ve found as you’re reading this blog post.

So, given that you’re still reading this, you’re ready to develop your first Dynatrace app. Remember, you don’t need high-end developer skills to get started, and you have a vast amount of information to help you create features in your apps using the documentation. If all else fails, you can lean on our Dynatrace Community to help find the answers you need. As Albert Einstein said, “Everything should be made as simple as possible, but not simpler.”

What’s next?

That’s it! And, if after reading this blog post, you believe you’re ready for even more, we invite you to take a deep dive into learning the essentials of Dynatrace Apps development by visiting our Get started with the Dynatrace platform tutorial.

Relevant links

• For more information about developing apps, visit Dynatrace Developer documentation
• To create your first app, visit the Get started with the Dynatrace platform tutorial.
• For questions related to app development, visit the developer Q&A forum.
• To keep up with the latest developments and news, visit the developer blog.

The post Start strong: Words of wisdom for creating Dynatrace Apps appeared first on Dynatrace news.

To help you raise the quality of your investigation results, Dynatrace offers an easy way of structuring data using DPL Architect. This tool lets you quickly extract typed fields from unstructured text (such as log entries) using the Dynatrace Pattern Language (DPL), enabling you to extract timestamps, determine status codes, identify IP addresses, or work with real JSON objects. This allows you to answer even the most complex questions with ultimate precision. The best thing: the whole process is performed on read when the query is executed, which means you have full flexibility and don’t need to define a structure when ingesting data.

>> Scroll down to see Dynatrace DPL Architect in action (24-second video)

Investigating log data with the help of DQL

Let’s look at a practical example. The simplest log analysis use cases can be solved in seconds by applying a simple filter to the query results. If a CISO asks, “Have we seen the IP address 40.30.20.1 in our logs within the last year?” a simple DQL query seems to suffice:

fetch logs, from: -365d
| filter contains(content, “40.30.20.1”)

This query returns all the records that contain the string value 40.30.20.1.

In the next step, adding DQL aggregation functions enables you to answer more complex questions like “How many times in an hour has this IP address visited our website within the last two weeks?” Again, a simple DQL query helps you out:

fetch logs, from: -14d
| filter contains(content, “40.30.20.1”)
| summarize by: bin(timestamp, 1h), count()

However, this kind of simple filtering using a sub-string search is prone to errors and is not precise enough. The mentioned filters return all records that contain 40.30.20.1, but are not necessarily from the source IP portion of the log format. The string 40.30.20.1 might also appear elsewhere, for example, in query parameters, and a simple string filter will return all records containing this search term:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395
40.30.20.109 - - [28/Aug/2023:10:22:04 -0300] "GET / HTTP/1.1" 200 2216

The first log record is matched because of its query parameter—something we don’t care about in our case. The second log record came up because the source IP contains the IP address—this is what we’re really looking for. In a nutshell, if you’re querying terabytes of logs and wish to drill down to specific records, filtering strings from plaintext log content is not enough.

The issue is that questions from CISOs aren’t usually so trivial when it comes to security use cases. Or, the log format where the answers should be looked for is more complex (for example, AWS CloudTrail logs). As a consequence, we need to search structured data to avoid mistakes. To minimize false positives, increase the precision of queries, and get the maximum out of DQL, fields can be extracted from record content.

Extracting patterns using DPL

Dynatrace Pattern Language (DPL) is a pattern language that allows you to describe a schema using matchers, where a matcher is a mini-pattern that matches a certain type of data. These matchers can be used to extract new fields from the schemaless data in Grail to enable more precise log filtering. Consider the same Apache access log example from above:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395

To extract the client IP addresses from the beginning of the log, a simple DPL pattern can be used:

IPADDR:client_ip

The IPADDR matches both versions of IP addresses (IPv4 addresses in dot-decimal notation and IPv6 addresses in hextet notation), leaving all other IP-like strings unmatched. For example, if the log record contains a dot-decimal number that is NOT an IPv4 address (999.999.999.999), it won’t be matched.

DPL patterns can be applied in DQL using the parse command. The simplest way to get started with DPL is to use Dynatrace DPL Architect.

DPL Architect to the rescue

DPL Architect is a handy tool, accessible through the Notebooks app, which supports you in quickly extracting fields from records. It helps create patterns, provides instant feedback, and allows you to save and reuse DPL patterns, for faster access to data analytics use cases.

To open the DQL Architect, you have to execute a DQL query, select the content, and choose Extract fields.

Figure 1: Extract fields in Notebooks using DPL Architect (24-second video)

Starting with preset patterns

The simplest way to extract data is using one of the ready-to-use preset patterns available for the most popular technologies, such as AWS, Microsoft, or GCP. Start exploring AWS VPC Flow logs or analyzing Kubernetes audit logs by choosing the patterns from the panel on the left side. You can also customize the list by adding your own individual patterns.

Developing a new pattern using DPL Architect

DPL Architect helps you by creating your own patterns and provides instant feedback. Start typing a DPL expression in the pattern field, and review the matching data (highlighted below) in the preview editor.

For accessing the extracted fields in the resultset, it’s necessary to add a name. In the example below, we’re only interested in the client_ip and the related response_code. The pattern matches the whole record, but only two fields are being extracted since they have defined extract names. Any other data between those two fields won’t be visible in the results (however, it is very easy to add them later if necessary).

Remember, we started our journey with the DPL Architect by selecting query results in a notebook. The colored bar at the top of the DPL Architect shows how many records from your query result match the current DPL pattern. This enables you to modify the DPL pattern to ensure it matches all records in the resultset.

Records that don’t match the current pattern can easily be added to the Match preview panel by selecting Add to preview.

After changing the DPL pattern, any previously unmatched records will be highlighted and the progress bar on the top informs you that 100% of the records in the Notebooks resultset are matched. It’s now time to insert our pattern into the DQL query by selecting Insert Pattern.

Precise extraction of fields from complex data

DPL also provides matchers for more complex data structures, like key-value pairs, structures, and JSON objects, that enable you to parse individual sub-elements from the whole object. Consider the following log record:

1693230219 230.4.130.168 C "GET / HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" 186 {"username":"james","result":0 }

Imagine your CISO comes with a request: “Give me the list of all IP addresses where the user ‘james’ has successfully logged in from.” Extracting these three fields and building a DQL query is as easy as pie:

LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)

The DPL will extract three fields: an IP address from the record, a string from the JSON object element username, and an integer from the JSON element result.

Summarizing the results based on these fields can be done with the following DQL:

fetch logs
| parse content, "LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)"
| filter username == "james" and result == 0
| summarize by: src_ip, count()

Imagine that you don’t have DPL available and need to filter all log records based on only searching for string values: the result would contain a lot of false positives (which only add “noise” to time-critical investigations). This is especially true with more complex log records containing nested JSON objects. The above example was oversimplified intentionally, but considering complex log records like AWS CloudTrail log records, having precise access to data in a specific node of an object makes a huge difference.

Summary

When performing security investigations or threat-hunting activities, it’s important to have precision in place to get reliable results. Historical data needs to be available, and access to object details is required for precise answers. DPL Architect enables you to quickly create DPL patterns, speeding up the investigation flow and delivering faster results. With the possibility of using Dynatrace-provided patterns for selected technology stacks, investigators can deliver answers even faster!

Check out the following video, where Andreas Grabner and I teamed up for a new episode of Dynatrace Observability Clinic. In this video, we dig deeper into the topic of extracting data via DPL, including a live demonstration of DPL Architect.

Referral links

• Documentation: Dynatrace Pattern Language
• Documentation: Dynatrace Pattern Language Architect
• Blog post: Dynatrace Grail: The data lakehouse for observability, security analysis, and automation
• Blog post: Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail

The post Speed up your security investigations with DPL Architect appeared first on Dynatrace news.

In the realm of SRE, time and effort allocation planning are crucial factors that involve a delicate balance between operational management and project improvements. This intricate allocation strategy can be categorized into two main domains. In this blog post, we’ll delve deeper into these categories to gain a comprehensive understanding of their significance and the challenges they present.

Planned effort

Site Reliability Engineering (SRE) effort and time allocation planning typically fall into two domains:

• Operations Management (50%)
  Operations Management includes on-call responsibilities, post-mortem assessments, addressing other interruptions, and buffer time. These tasks collectively ensure uninterrupted production service.
• Process Improvements (50%)
  The allocation for process improvements is devoted to automation and continuous improvement SREs help to ensure that systems are scalable, reliable, and efficient. This improves the current project and paves the way for future innovation.

Reality

In practice, while both these categories have equal attention, project improvements hold paramount importance for business outcomes. SREs invest significant effort in enhancing software reliability, scalability, and dependability. Regrettably, recent reports indicate that SREs spend a substantial portion of their time addressing build issues and managing production incidents. This challenge escalates with the growing complexity of cloud systems and organizational aspirations for digital transformation, often leaving minimal time for substantial project improvements. Consequently, organizations grapple with various issues that impact software reliability.

Process Improvements

Organizations are strategically integrating observability into the initial stages of their release processes to tackle these challenges. As they embark on this initiative, SREs are tasked with identifying an observability solution that aligns seamlessly with their application teams and seamlessly integrates with their existing toolset.

Outcome

Dynatrace plays a pivotal role in this endeavor by empowering application teams through its seamless integration. The Dynatrace integration leverages native features and events that pass through the pipeline. Events serve as logic operators that can trigger or stop subsequent tasks within the pipeline. Additionally, the Site Reliability Guardian serves as the governing entity, making decisions on whether to proceed or halt a specific build based on observable telemetry data supplied by OneAgent during the CI/CD pipeline process.  This proactive strategy significantly enhances the chances of success for SREs, providing them with more time to focus on substantial project improvements (50%) and broaden the buffer zone (30%). This empowers them to spearhead innovations that ensure the business is prepared for future expansions.

Integrate DevSecOps with Dynatrace

Software delivery is structured around CI/CD pipelines, which play a critical role in the SRE process and represent the first step toward effective automation. Automated CI/CD pipelines greatly reduce the occurrence of manual errors. They provide continuous feedback to developers and enable rapid product iterations. Elevating the efficiency of release pipelines is one key to producing high-quality software; it mitigates the need for post-incident analyses and on-call duties for the SRE team. It also returns valuable time back to the SRE team.

A few avenues for elevating CI/CD pipelines are:

• Enhancing the extent of automated test coverage during the testing phase.
• Embracing the tenets of DevOps and DevSecOps methodologies anchored in engineering principles.
• Identifying and automating the validation of business/application Service Level Objectives (SLOs) during release cycles.
• Streamlining the CI/CD process to ensure optimal efficiency.

To realize these goals, SRE teams can seamlessly follow a three-step process within their Dynatrace environment:

1. Commencement (optional): If desired, initiate a Dynatrace workflow task to establish a connection with the DevOps tool using HTTP. This step is only necessary if you intend to control the build creation process through Dynatrace.
2. Push events: Configure your DevOps tool to dispatch deployment events at the inception of the deployment process. Additionally, introduce annotation events to notify Dynatrace of the progress within your testing phase. These events serve as logical operators that dictate the course of the release process.
3. Automated validation and progression: Depending on the configuration of your tasks, Site Reliability Guardian (SRG) validation can be automatically activated to promote or disapprove the advancement of a build towards production.

This integration leads to complete automation with end-to-end pipeline visibility, thereby reducing the heavy lifting of release management for SRE teams and empowering them to focus on innovation, which catalyzes organizational growth.

Designing systems for reliability

Engineering teams must extend their focus beyond functional and load tests to instill assurance in the software release process via automated cycles. The rationale is that, during actual production, variables can induce outcomes that are different from those predicted in standardized tests. Thus, more comprehensive testing becomes essential to embrace unpredictability and mirror real-world conditions. These practices are commonly known as “chaos engineering.”

By embracing chaos engineering practices, development teams cultivate a higher degree of confidence in the robustness of their applications within specific production scenarios. However, the very nature of chaos engineering introduces a challenge: identifying the responsible service when a failure occurs. This is where Dynatrace Davis AI comes into play, leveraging telemetry data from your services. Davis AI automatically establishes a baseline of each service’s behavior, learned during the development and functional testing phases, and subsequently identifies the underlying causes of failures, thereby eliminating the uncertainty of the root cause and “war room” scenarios.

To illustrate, a CI/CD pipeline setup has a functional test phase followed by a chaos-engineering test. The chaos engineering test was structured to randomly slow down a Docker container and, thereby, a critical service for the application. During this phase, Davis AI automatically identifies that a key business request has breached its automated baseline due to a significant Mongo database slowdown. Moreover, Davis AI identifies the reason behind the slowdown and pinpoints the exact query that caused the problem.

With Davis AI’s contextual capabilities, embracing chaos engineering and making application code robust and better prepared for production deployments is easy.

Reduce operations overhead

Ideally, no bugs are reported once software is deployed to production. However, this is highly unlikely. Therefore, it’s important to have a process in place to minimize downtime in the event of a failure.

Davis AI assists with automated root cause analysis, providing details of the underlying services, traces, logs, and user sessions that caused the failure. This can save SRE teams from the time and effort of debugging or a war room scenario. By identifying the exact code or trace that causes a failure, Davis AI helps teams fix problems quickly and significantly reduces MTTR. In addition, if you configure an automated remediation workflow, Dynatrace invokes it and restores applications with virtually no downtime.

In the above screenshot, one of the requests in the application reports errors. Davis AI identifies that this anomaly was reported in the newer releases. Davis AI captures and relays the details of previous and current builds to the remediation workflow. The remediation workflow uses this information to roll back the build, leaving Davis AI to automatically validate and close detected problems.

Dynatrace Observability empowers the SRE team, propelling them towards LowOps/NoOps. This allows them to prioritize innovation, strategic goals, and minimize downtime while optimizing the delivery of perfect software. In our next blog post in this series, we’ll delve into the step-by-step process of automating the release pipeline using Dynatrace best practices.

The post Accelerate and empower Site Reliability Engineering with Dynatrace observability appeared first on Dynatrace news.

Grail: Enterprise-ready data lakehouse

Grail, the Dynatrace causational data lakehouse, was explicitly designed for observability and security data, with artificial intelligence integrated into its foundation. We’ve further enhanced its capabilities to meet the high standards of large enterprises by incorporating record-level permission policies.

To fully utilize Grail features, it’s recommended that you incorporate its unique buckets and security policies at the beginning of your observability journey. Utilizing built-in mechanisms and customizing organization-specific policies maximizes the benefits of Grail capabilities.

Custom data buckets for faster queries, increased control, and custom retention periods

>> Scroll down to the bottom of this blog post to view a ~7-minute video demonstration of custom data buckets.

The first layer in the Grail data model consists of buckets and tables (and views for entities, which is outside this blog post’s scope).

Tables are a physical data model, essentially the type of observability data that you can store. Buckets are similar to folders, a physical storage location.

There is a default bucket for each table. Here is the list of tables and corresponding default buckets in Grail.

The default buckets let you ingest data immediately, but you can also create additional custom buckets to make the most of Grail.

Address specific use cases with custom buckets

It’s logical to segregate high-volume data into its own bucket. This allows the data to be frequently queried and used separately from other scenarios.

For example, a separate bucket could be used for detailed logs from Dynatrace Synthetic nodes. Debug-level logs, which also generate high volumes and have a shorter lifespan or value period than other logs, could similarly benefit from dedicated storage. Keeping these logs separate decreases the data volume for other troubleshooting logs. This improves query speeds and reduces related costs for all other teams and apps.

Address organizational structure with custom buckets

Depending on your organization’s structure, you may find it beneficial to keep logs used by specific business units or departments in separate buckets. This approach makes queries faster for individual units, as they only query relevant logs, and it ensures distinct access separation.

Suppose a single Grail environment is central storage for pre-production and production systems. In that case, the use of separate buckets makes it easier to distinguish different stages, keeping production data separate from development or staging data.

Adopting this level of data segmentation helps to maximize Grail’s performance potential. If your typical queries only target a specific use case, business unit, or production stage, ensuring they don’t include unrelated buckets helps maintain efficiency and relevance.

Custom buckets unlock different retention periods.

Segmenting your data into multiple buckets also puts you in control of the data retention period. The simplicity of storing data in Grail is reflected in its retention policies; you choose how long to store each portion of your data, and you never have to think about managing archives or retrieving archived data. Coupled with the transparent pricing of GiB/day, you can set up buckets to exactly match your business needs.

While the built-in default_logs bucket has a retention period of 35 days, you have more options to choose from. Use Grail’s public bucket management API to create new buckets for which you can select data retention periods of 1 day to 10 years , or use the new Storage Management app to that with just a few clicks.

In conjunction with the previous example of keeping high-volume and short-lived logs separate, you might also need to keep your application data longer. For example, transaction data and user-profile logs might need to be retained for 12 or 18 months.

Use buckets to query only the log data you need

Whether looking for a “needle in a haystack” or reporting on data stored in Grail, you can start an advanced query with DQL by fetching data from one of the tables. At this point, you should familiarize yourself with the blog post Tailored access management, Part 2: Onboard users to Grail and AppEngine, which covers access to Grail tables and buckets.

Now, let’s take a look at a query example that puts this all into use.

fetch logs

| filter loglevel=="ERROR"

In this example, we query a certain table (logs) and filter the results by a field (loglevel) with a certain value (ERROR). Note that with such a query, you fetch logs from all the buckets the end-user can access.

Although this initially only includes the default bucket, you might also include other buckets (if these are available to the user). As you bring in more data and users to Grail, relying just on the default buckets is not the optimal setup.

This is where filtering on custom buckets comes in. This allows you to query data from a specific bucket.

fetch logs

| filter dt.system.bucket=="prod_infra_logs" and loglevel=="ERROR"

This example now includes an additional filter that restricts data retrieval from a certain bucket (prod_infra_logs).

Using buckets to query only the data you need significantly speeds up queries and reduces query costs.

Buckets for data with high-security requirements

Buckets can also be used for managing high-level access control of data.

You can store access control logs or payment provider events in separate buckets to grant only limited SecOps or business administrators access to these logs.

Grail and AppEngine’s new policy-based access management provides a way to do this with buckets. For example, by adding a WHERE clause to the policy statement, you can define a specific bucket (=) or a range of buckets (STARTSWITH). In this example, the policy grants access to all buckets that have names starting with prod_infra_.

ALLOW storage:buckets:read WHERE storage:bucket-name STARTSWITH "prod_infra_";

However, creating access policies solely on the bucket and table level is not scalable in a enterprise landscape, as one Dynatrace tenant can have a limited number of custom buckets. Instead, access control based on specific attributes like host groups or team assignments can be achieved using different policies that are based on supported attributes or security context.

Record-level permissions and security context

As covered in the previously linked blog post about access management, Dynatrace Grail brings a new architecture to permissions management. The new approach that uses security policies provides you with new dynamic controls for user authorization.

As using custom buckets opened up a basic approach to access where users could get access to a whole bucket, record-level permissions allow you to take a fine-grained approach.

This means that whenever you run a DQL query to fetch data from Grail, your policy-based access rights are evaluated, and records without defined access are filtered out.

This means your teams’ permissions are not constrained by data management decisions on a bucket level. Let’s say it makes sense to consolidate all short-living app debug logs to a bucket that has a short retention period. With record-level permissions, you can now ensure multiple app owners can see only their data in that bucket.

Another example would be a business unit admin who needs to have access to departmental data across buckets.

Permissions based on DQL fields and security context

To implement this in the Log Management and Analytics context, you can create policies with additional clauses that provide access.

ALLOW storage:logs:read
WHERE storage:k8s.namespace.name="abc"

 AND storage:dt.host_group.id STARTSWITH "org1-";

In this example, the policy allows access to logs that have a certain Kubernetes namespace (abc) and originate from a host group whose name must start with a specific string (org1-).

The list of standard table fields used in security policies provides flexibility for defining individual policies.

When you look at the list of DQL fields, you’ll notice one reserved field, dt.security_context.

You can assign a value to dt.security_context during data ingest for use in a security policy, which is not covered by the list of previous DQL fields. You can explicitly set a value for dt.security_context for some logs, or take the value of an existing field.

In this example, logs from a particular source (dsfm) are enriched with a literal value for dt.security_context (sec-lvl-7) during log ingest. A policy with a specific clause can provide access to only logs with this security context. Security context rule management is available via settings In the Dynatrace web UI and API.

This approach to granular record-level permissions opens up the flexibility needed in enterprise environments. You can craft policies based on existing fields like Kubernetes cluster or namespace, a host or a host group, a service name, or a log source. Or you can use security context for any other use cases, like granting access based on an AWS account, a GCP project, an Azure subscription, a username, or a team name.

Take the first step now

Many organizations have found immediate value in working with logs in Grail. Starting from optimizing their business and opening revenue streams based on data in logs to unlocking real-time insights from observability data and eliminating hours of manual work per process, as the Bank of Montreal did recently. Utilizing the core aspects of Grail, like buckets and permissions, sets organizations on the path to success.

Next steps

• Start a Dynatrace free trial and explore Log Management and Analytics powered by Grail
• Read our documentation explaining buckets, permissions in Grail, security context for logs, and IAM
• Record-level permissions for Grail are generally available with Dynatrace version 1.277. Custom buckets, in addition to bucket and table permissions, are available with Dynatrace version 1.265.
• Use Storage Management app to create and manage custom Grail buckets and unlock custom retention times for your data since Dynatrace version 1.281.

Special thanks to Dominik Punz and Christian Kiesewetter for contributing to this blog post.

Video demo

Watch this 7-minute video to see how you can use custom data buckets to separate use cases, data retention, and access permissions in Dynatrace.

The post Enhance data management with Grail: Ultimate guide to custom buckets and security policies appeared first on Dynatrace news.

If you use AWS cloud services to build and run your applications, you may be familiar with the AWS Well-Architected Framework. This is a set of best practices and guidelines that help you design and operate reliable, secure, efficient, cost-effective, and sustainable systems in the cloud. The framework comprises six pillars: Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability.

But how can you ensure that your applications meet these pillars and deliver the best outcomes for your business? And how can you verify this performance consistently across a multicloud environment that also uses Microsoft Azure and Google Cloud Platform frameworks? Because Google offers its own Google Cloud Architecture Framework and Microsoft its Azure Well-Architected Framework, organizations that use a combination of these platforms triple the challenge of integrating their performance frameworks into a cohesive strategy.

This is where unified observability and Dynatrace Automations can help by leveraging causal AI and analytics to drive intelligent automation across your multicloud ecosystem. The Dynatrace platform approach to managing your cloud initiatives provides insights and answers to not just see what could go wrong but what could go right. For example, optimizing resource utilization for greater scale and lower cost and driving insights to increase adoption of cloud-native serverless services.

In this blog post, we’ll demonstrate how Dynatrace automation and the Dynatrace Site Reliability Guardian app can help you implement your applications according to all six AWS Well-Architected pillars by integrating them into your software development lifecycle (SDLC).

Dynatrace AutomationEngine workflows automate release validation using AWS Well-Architected pillars

With Dynatrace, you can create workflows that automate various tasks based on events, schedules or Davis problem triggers. Workflows are powered by a core platform technology of Dynatrace called the AutomationEngine. Using an interactive no/low code editor, you can create workflows or configure them as code. These workflows also utilize Davis®, the Dynatrace causal AI engine, and all your observability and security data across all platforms, in context, at scale, and in real-time.

One of the powerful workflows to leverage is continuous release validation. This process enables you to continuously evaluate software against predefined quality criteria and service level objectives (SLOs) in pre-production environments. You can also automate progressive delivery techniques such as canary releases, blue/green deployments, feature flags, and trigger rollbacks when necessary.

This workflow uses the Dynatrace Site Reliability Guardian application. The Site Reliability Guardian helps automate release validation based on SLOs and important signals that define the expected behavior of your applications in terms of availability, performance errors, throughput, latency, etc. The Site Reliability Guardian also helps keep your production environment safe and secure through automated change impact analysis.

But this workflow can also help you implement your applications according to each of the AWS Well-Architected pillars. Here’s an overview of how the Site Reliability Guardian can help you implement the six pillars of AWS Well-Architected.

AWS Well-Architected pillar #1: Performance efficiency

The performance efficiency pillar focuses on using computing resources efficiently to meet system requirements, maintaining efficiency as demand changes, and evolving technologies.

A study by Amazon found that increasing page load time by just 100 milliseconds costs 1% in sales. Storing frequently accessed data in faster storage, usually in-memory caching, improves data retrieval speed and overall system performance. Beyond efficiency, validating performance thresholds is also crucial for revenues.

Once configured, the continuous release validation workflow powered by the Site Reliability Guardian can automatically do the following:

• Validate if service response time, process CPU/memory usage, and so on, are satisfying SLOs
• Stop promoting the release into production if the error rate in the logs is too high
• Notify the SRE team using communication channels
• Create a Jira ticket or an issue on your preferred Git repository if the release is violating the set thresholds for the performance SLOs

SLO examples for performance efficiency

The following examples show how to define an SLO for performance efficiency in the Site Reliability Guardian using Dynatrace Query Language (DQL).

Validate if response time is increasing under high load utilizing OpenTelemetry spans

fetch spans 
| filter endpoint.name == "/api/getProducts" 
| filter k8s.namespace.name == "catalog" 
| filter k8s.container.name == "product-service" 
| filter http.status_code == 200 
| summarize avg(duration) // in milliseconds

Check if process CPU usage is in a valid range

timeseries val = avg(dt.process.cpu.usage) 
,filter in(dt.entity.process_group_instance, "PROCESS_GROUP_INSTANCE-ID") 
| fields avg = arrayAvg(val) // in percentage

AWS Well-Architected pillar #2: Security

The security pillar focuses on protecting information system assets while delivering business value through risk assessment and mitigation strategies.

The continuous release validation workflow powered by Site Reliability Guardian can automatically do the following:

• Check for vulnerabilities across all layers of your application stack in real-time, getting help from Dynatrace Davis Security Score as a validation metric
• Block releases if they do not meet the security criteria
• Notify the security team of the vulnerabilities in your application and create an issue/ticket to track the progress

SLO examples for security

The following examples show how to define an SLO for security in the Site Reliability Guardian using DQL.

Runtime Vulnerability Analysis for a Process Group Instance – Davis Security Assessment Score

fetch events 
| filter event.kind == "SECURITY_EVENT" 
| filter event.type == "VULNERABILITY_STATE_REPORT_EVENT" 
| filter event.level == "ENTITY" 
| filter in("PROCESSGROUP_INSTANCE_ID",affected_entity.affected_processes.ids) 
| sort timestamp, direction:"descending" 
| summarize  
{  
status=takeFirst(vulnerability.resolution.status), 
score=takeFirst(vulnerability.davis_assessment.score), 
affected_processes=takeFirst(affected_entity.affected_processes.ids) 
}, 
by: {vulnerability.id, affected_entity.id} 
| filter status == "OPEN"  
| summarize maxScore=takeMax(score)

AWS Well-Architected pillar #3: Cost optimization

The cost optimization pillar focuses on avoiding unnecessary costs and understanding managing tradeoffs between cost capacity performance.

The continuous release validation workflow powered by the Site Reliability Guardian can automatically do the following:

• Detect underutilized and/or overprovisioned resources in Kubernetes deployments considering the container limits and requests
• Determine the non-Kubernetes-based applications that underutilize CPU, memory, and disk
• Simultaneously validate if performance objectives are still in the acceptable range when you reduce the CPU, memory, and disk allocations

SLO examples for cost optimization

The following examples show how to define an SLO for cost optimization in the Site Reliability Guardian using DQL.

Reduce CPU size and cost by checking CPU usage

To reduce CPU size and cost, check if CPU usage is below the SLO threshold. If so, test against the response time objective under the same Site Reliability Guardian. If both objectives pass, you have achieved your cost reduction on CPU size.

Here are the DQL queries from the image you can copy:

timeseries cpu = avg(dt.containers.cpu.usage_percent), 
filter: in(dt.containers.name, "CONTAINER-NAME") 
| fields avg = arrayAvg(cpu) // in percentage

fetch logs 
| filter k8s.container.name == "CONTAINER-NAME" 
| filter k8s.namespace.name == "CONTAINER-NAMESPACE" 
| filter matchesPhrase(content, "/api/uri/path") 
| parse content, "DATA '/api/uri/path' DATA 'rt:' SPACE? FLOAT:responsetime "  
| filter isNotNull(responsetime) 
| summarize avg(responsetime) // in milliseconds

Reduce disk size and re-validate

If the SLO specified below is not met, you can try reducing the size of the disk and then validating the same objective under the performance efficiency validation pillar. If the objective under the performance efficiency pillar is achieved, it indicates successful cost reduction for the disk size.

timeseries disk_used = avg(dt.host.disk.used.percent), 
filter: in(dt.entity.host,"HOST_ID") 
| fields avg = arrayAvg(disk_used) // in percentage

AWS Well-Architected pillar #4: Reliability

The reliability pillar focuses on ensuring a system can recover from infrastructure or service disruptions and dynamically acquire computing resources to meet demand and mitigate disruptions such as misconfigurations or transient network issues.

The continuous release validation workflow powered by the Site Reliability Guardian can automatically do the following:

• Monitor the health of your applications across hybrid multicloud environments using Synthetic Monitoring and evaluate the results depending on your SLOs
• Proactively identify potential availability failures before they impact users on production
• Simulate failures in your AWS workloads using Fault Injection Simulator (FIS) and test how your applications handle scenarios such as instance termination, CPU stress, or network latency. SRG validates the status of the resiliency SLOs for the experiment period.

SLO examples for reliability

The following examples show how to define an SLO for reliability in the Site Reliability Guardian using DQL.

Success Rate – Availability Validation with Synthetic Monitoring

fetch logs 
| filter log.source == "logs/requests" 
| parse content,"JSON:request" 
| fieldsAdd httpRequest = request[httpRequest] 
| fieldsAdd httpStatus = httpRequest[status] 
| fieldsAdd success = toLong(httpStatus < 400) 
| summarize successRate = sum(success)/count() * 100 // in percentage

Number of Out of memory (OOM) kills of a container in the pod to be less than 5

timeseries oom_kills = avg(dt.kubernetes.container.oom_kills), 
filter: in(k8s.cluster.name,"CLUSTER-NAME") and in(k8s.namespace.name,"NAMESPACE-NAME") and in(k8s.workload.kind,"statefulset") and in (k8s.workload.name,"cassandra-workload-1") 
| fields sum = arraySum(oom_kills) // num of oom_kills

AWS Well-Architected pillar #5: Operational excellence

The operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improve supporting processes and procedures.

With continuous release validation workflow powered by the Site Reliability Guardian, you can:

• Automatically verify service or application changes against key business metrics such as customer satisfaction score, user experience score, and Apdex rating
  
• Enhance collaboration with targeted notifications of relevant teams using the Ownership feature
• Create an issue on your preferred Git repository to track and resolve the invalidated SLOs
• Trigger remediation workflows based on events such as service degradations, performance bottlenecks, security vulnerabilities
• Validate your CI/CD performance over time, considering the execution times, pipeline performance, failure rate, etc., which shows your operational efficiency in your software delivery pipeline.

SLO examples for operational excellence

The following examples show how to define an SLO for operational excellence in the Site Reliability Guardian.

Apdex rating validation of a web application

1. Navigate to “Service-level objectives” and click on “Add new SLO” button
2. Select “User experience” as a template. It will auto-generate the metric expression as the following:

   (100)*(builtin:apps.web.actionCount.category:filter(eq("Apdex category",SATISFIED)):splitBy())/(builtin:apps.web.actionCount.category:splitBy())

3. Replace your application name in the entityName attribute:

   type("APPLICATION"),entityName("APPLICATION-NAME")

4. Add a success criteria depending on your needs
   
5. Reference this SLO in your Site Reliability Guardian objective

AWS Well-Architected pillar #6: Sustainability

The sustainability pillar focuses on minimizing environmental impact and maximizing the social benefits of cloud computing.

The continuous release validation workflow powered by the Site Reliability Guardian can automatically do the following:

• Measure and evaluate carbon footprint emissions associated with cloud usage
• Leverage observability metrics to identify underutilized resources for reducing energy consumption and waste emissions

SLO examples for sustainability

The following examples show how to define an SLO for sustainability in the Site Reliability Guardian using DQL.

Carbon emission total of the host running the application for the last 2 hours

fetch bizevents, from: -2h 
| filter event.type == "carbon.report" 
| filter dt.entity.host == "HOST-ID" 
| summarize toDouble(sum(emissions)), alias:total // total CO2e in grams

Under-utilized memory resource validation

timeseries memory=avg(dt.containers.memory.usage_percent), by:dt.entity.host 
| filter dt.entity.host == "HOST-ID" 
| fields avg = arrayAvg(memory) // in percentage

Implementing AWS Well-Architected Framework with Dynatrace: A practical guide

To help you get started with implementing the AWS Well-Architected Framework using Dynatrace, we’ve provided a sample workflow and SRGs in our official repository. This resource offers a step-by-step guide to quickly set up your validation tools and integrate them into your software development lifecycle.

Quick Implementation: Follow the instructions in our Dynatrace Configuration as Code Samples repository to deploy the sample workflow and SRGs. This will enable you to immediately begin validating your applications against the AWS Well-Architected pillars.

For more about how Site Reliability Guardian helps organizations automate change impact analysis, performance, and service level objectives, join us for the on-demand Observability Clinic, Site Reliability Guardian with DevSecOps activist Andreas Grabner.

The post Implementing AWS Well-Architected pillars with automated workflows appeared first on Dynatrace news.

For most organizations, online service reliability that balances innovation and uptime is a primary goal. SLO monitoring and alerting on SLOs using error-budget burn rates are critical capabilities that can help organizations achieve that goal.

In the digital age, a company’s success depends on the reliability of its online services. To manage this crucial aspect, many companies adopt service level objectives (SLOs). SLOs are specifically processed metrics that help businesses balance breakthroughs with reliability. They help define how often and for how long services can afford to fail without severely impacting users. SLOs are pivotal for development, DevOps, and SRE teams because they provide a common language for discussing system reliability.

But just establishing SLOs isn’t enough. Without implementing robust SLO monitoring, anomaly detection, and alerting on SLOs, teams can miss issues that breach defined quality targets. Monitoring performance against these objectives using AI is where the true value lies, based on establishing error budgets and monitoring an error budget burn rate.

What is SLO monitoring? And what is an error budget burn rate?

SLO monitoring is an IT process that tracks microservices architecture health based on metrics that express specific performance targets.

An error budget is the acceptable number of errors or downtime a service can experience while still meeting its SLO, akin to a risk buffer.

An error budget burn rate is a measurement of how fast your service is using up its error budget.

This metrics-based approach to SLO monitoring helps organizations plan and adjust before problems escalate into outages.

Imagine your operations is a car, and the error budget is the tank of gas. An error budget burn rate is the rate at which your car consumes the gas. A high burn rate is a red flag indicating you may not meet your SLO of making it to the next exit, while a slow burn rate suggests you have room to take more risk and maybe go to the next town. An error budget encourages a healthy trade-off between innovation and reliability.

Why using error budget burn rates is important for SLO monitoring

Setting up alerts on SLOs using burn rates is an approach to anomaly detection that is valuable for several reasons:

• Sends alerts proactively: Alerting on SLOs using burn rate is based on how fast the service is consuming its error budget (the allowable threshold of errors within the SLO). When the error budget burn rate is too high, the system sends out alerts, helping teams to proactively address the issue before the SLO is violated (for example, you run out of gas).
• Provides granularity: Traditional threshold-based alerts on the SLO status might miss short but significant anomalies, whereas burn-rate metric alerts are more granular and can detect these.
• Reduces alert fatigue: By alerting only when the error budget burn rate is high, it reduces alert noise and helps prevent alert fatigue. Teams can focus on the most critical issues that might lead to an SLO breach.
• Improves decision making: By analyzing burn rates, teams can make better decisions about deploying new releases or initiating actions that could potentially affect the service reliability.

In the following example, the Dynatrace platform identifies seven problems with the service Availability BasketsResources in less than a week. It further identifies which problems had an impact on quality.

Using unified observability and causal AI for SLO monitoring and alerting on SLOs

The key to understanding precisely how services are delivering on their SLOs is having accurate, real-time intelligence and context about services and their dependencies. Most monitoring solutions and do-it-yourself setups simply correlate probabilities to estimate where things could be going wrong. Often, data comes from many individual point solutions, and teams must piece together evidence based on hunches and specialized knowledge. And because each solution measures the performance of various components using its own scale, teams often receive a flood of false positive alerts. As a result, teams waste time verifying which alerts matter instead of improving performance and maturing DevOps practices.

The Dynatrace approach brings all the data into a unified platform and converges SLOs with our causal AI engine, Davis®. Using real-time dependency mapping and fault-tree analysis, Davis AI determines the precise root cause of a problem in context, so it identifies problem areas before they become outages. This unified approach promptly alerts on the business-critical SLOs, which enables DevOps teams to manage service reliability without noisy false positive alerts. With fast alerts and the precision of Dynatrace Davis AI, teams gain immediate insight into the problem context and root causes affecting SLOs.

In the following example, Dynatrace identifies an SLO at risk, the three affected services, and the precise root cause of each issue.

Focus on key SLOs for each application to deliver great user experiences

Front-end services directly impact the user experience and indicate overall system health, including potential backend issues. With a few carefully selected SLOs on the frontend, you can maintain effective, focused visibility on your overall system health.

This strategic positioning of SLOs ensures that you can immediately identify any spike in error budget burn rate. That’s because the direct relationship between the front-end services and user experience minimizes the risk of false positives. Furthermore, this approach allows for rapid alerting on burn rate spikes. Fast alerts are critical for mitigating any potential impact on the frontend, thus assuring consistent and optimal user experiences. This approach aligns with business goals and supports a proactive, user-centric model of service delivery.

Setting up SLO monitoring using burn rates

Dynatrace provides a unified platform for monitoring and establishing SLOs, integrating crucial performance metrics seamlessly into a comprehensive, real-time observability environment.

If you are just getting started with SLOs in Dynatrace, we highly recommend this introductory video, which provides a comprehensive and user-friendly overview of this SLO functionality.

Set up SLOs on front-end services that focus on failure rates

1. Open the service you’re interested in and create your SLO by using the shortcut in the entity menu:
2. As mentioned earlier, focus on your front-end services. The direct correlation between the front-end services and user experience minimizes the risk of false positives and over-alerting.
3. For alerting, start with SLOs that focus on failure rate. Such SLOs monitor the percentage of successful calls for a specific service. To create such an SLO, choose the “service-level availability” template.

Once set up, you can seamlessly monitor your SLOs, their status, error budget, and burn rate. If a SLO is at risk of being violated, the Dynatrace AI engine automatically finds the root cause, so you can clearly see the related problems. Dynatrace then makes it easy to add issues to a dashboard or other reporting mechanism for follow-up.

Set up alerting on SLOs using error budget burn-rates

Once you configure the SLOs, you can set up alerting on them using your error budget burn rate. Automatic burn rate alerts let your team know when any SLO is at risk of being violated.

To create alerts, go to Service-level objectives, <your SLO>, and select More (…), Create alert (see Set up alerts for complete instructions). This generates a metric event in Dynatrace, which you can then configure as follows:

1. What burn rate is too high? Let Dynatrace help by making a “suggested threshold.” It automatically calculates a threshold based on the target value you defined for your SLO and historic data.
2. Since we’re monitoring the burn rate of crucial front-end services, we want to be alerted quickly when issues emerge. You can control how quickly you get alerted in the “advanced model properties.”
3. Understand how many alerts your configuration would have generated over the past week. If it generates too many problems, your SLO target may be too ambitious. Adjust these figures until you reach the right alerting threshold for your situation.

Alerting is just one side of the equation. You also need context if you’d like to proactively take action to keep your SLO healthy. Here Davis causal AI provides insights into possible root causes of SLO degradation.

1. Choose the right event type for the alert. They’re available via dropdown on the “event type” field. If you start with failure rate SLOs, choose the “Error” event type.
2. Allow Davis causal AI to merge the problem opened by the SLO with other problems.

Monitoring SLOs made easy with unified observability and Dynatrace causal AI

Alerting on SLOs using burn rates on your front-end services ensures that you can detect and resolve system anomalies swiftly. In the process, Davis causal AI provides automatic and comprehensive root cause analysis. This unified approach with observability and AI allows teams to focus on critical issues, reduce alert fatigue, and make informed decisions that drive efficiency and better business outcomes.

To see this use case in action, tune in to my conversation with Andi Grabner in the Observability Clinic, SLO Error Budget Burn Rate-Based Alerting.

For additional tips about scaling SLOs, see the blog post Service Level Objectives (SLOs) at Scale (Tips and Tricks).

The post SLO burn: SLO monitoring and alerting on SLOs using error-budget burn rates with unified observability and AI appeared first on Dynatrace news.