Observability data can tell you a lot, but often not in a language that your business understands. Traces fire, business events flow, and dashboards light up, yet P1 incidents still break out. The first question from the war room is typically, “Who’s impacted?” Too often, the honest answer is, “Let me check the database and get back to you.” This gap between raw telemetry and human-readable business context is exactly what lookup tables in Dynatrace are designed to close.

In our introductory blog post on Dynatrace lookup tables, we explained the fundamentals: what lookup tables are, how to ingest them, and initial use cases spanning security to business context enrichment.

In this blog, we’ll go deeper, with real-world examples of lookup tables, practical patterns, and how teams are already putting them to work.

The problem: Dashboards display cryptic identifiers, not insights

Daniel Adams, Observability Engineer at FreedomPay, put it plainly in a recent Dynatrace Tips & Tricks session:

“A lot of our apps are not coded to show customer names or plain text human-readable stuff. It’s mostly IDs being passed back and forth.”

The result is that dashboards are filled with cryptic identifiers that operations teams and business stakeholders can’t mentally decode. And splitting data by customer, country, or store requires trips back to the SQL database for further contextual configuration.

Many organizations that run distributed systems at scale face the same challenge: telemetry is rich, but the business context is missing.

The solution: Use lookup tables to make your reference data available in Grail

Lookup tables let you upload reference data (CSV, JSON, or XML files) directly into Dynatrace Grail®, where it lives alongside your logs, traces, metrics, and business events. Once uploaded, two DQL commands unlock the enrichment from your lookup tables:

• Load fetches the contents of a lookup table, working just like querying any other data type in Grail.
• lookup joins records from your observability data with matching rows in the lookup table, using a defined key field.

In FreedomPay’s case, the lookup table holds customer names, customer codes, store IDs, and countries. The join key is the store ID, the same cryptic identifier that was previously displayed on their dashboards. One DQL lookup command later, those IDs become readable names, and the data can be split by country, grouped by customer, and surfaced in a dashboard that anyone on the team can use.

Lookup tables reduce user cognitive load

The impact of lookup tables is not just cosmetic. Daniel described the before-and-after in P1 situations:

“Previously, everyone was asking: who’s impacted, which customer? Now it’s just very apparent and real-time. People can share screenshots directly from Dynatrace.”

The shift from “let me go check” to “it’s right here” compresses triage time and reduces the cognitive load on everyone in the room.

Beyond incident response, lookup tables also power dashboard variables. FreedomPay uses dashboard variables to populate dropdown filters (for example, clients and countries) directly from the lookup table itself, keeping the UI dynamic without manual maintenance. The load command behaves like any other fetch command but operates on reference data rather than telemetry, making it a natural fit for driving interactive dashboards.

And the enrichment isn’t limited to business events. The load and lookup commands work across all data types in Grail.

In practice: cost allocation with lookup tables

Cost allocation is a good example of how lookup tables bridge the gap between operational data and business reality. Many organizations already maintain mappings that associate users, teams, or services with products or cost centers, but that context rarely lives inside observability data. Lookup tables allow you to introduce such context.

Consider DPS (Dynatrace Platform Subscription) consumption driven by user activity such as running queries, triggering automation, or executing functions. On its own, such usage is hard to attribute to a specific part of the organization. By associating users with products or cost centers in a lookup table, you can attribute usage to organizational ownership rather than just technical activity, align cost views with existing financial or product structures, and keep that attribution logic centralized and reusable rather than embedding it in individual analyses.

What makes this pattern scale is that business mappings change over time, while usage data continues to flow. Lookup tables allow you to evolve the mapping without rewriting your analyses. The same mechanism used for cost allocation works equally well for mapping users to teams, services to portfolios, or environments to internal programs. Cost allocation simply highlights how powerful this becomes when operational data is interpreted through a business lens.

Importantly, this is just one way to approach cost allocation, not the only way. Some teams rely on host-based attribution, others on pipeline metadata or external financial systems. Lookup tables complement these models by making it easy to incorporate existing reference data wherever it adds clarity.

Keeping lookup data fresh

For lookup tables to deliver lasting value, their data must remain current. FreedomPay currently uses Postman to push updated CSVs via the upload API, but they’re building automation to pull this data from their SQL backend daily and sync changes using the overwrite: true parameter. The lookup table path stays the same; the data underneath gets refreshed. Downstream dashboards and queries are updated automatically.

More broadly, there are several approaches to keeping lookup data up to date:

• Dynatrace Workflows automate extraction and upload from SQL databases or APIs on a schedule.
• Periodic refresh updates your lookup data programmatically as part of CI/CD or data sync processes.
• Manual refresh uses Postman, cURL, or other API-based uploads with the overwrite flag when data changes infrequently.

Dynatrace lookup tables are now generally available

With the release of Dynatrace SaaS version 1.334, lookup tables are generally available for all customers running Dynatrace SaaS with an active Dynatrace Platform Subscription (DPS). GA brings production-readiness, improved query performance for the lookup command, and full integration with Grail’s scalability and access controls.

To get started

1. Identify a data enrichment use case: cost allocation, customer context, error code mapping, or security allow lists.
2. Prepare your reference data as a CSV, JSON, or JSONL (JSON Lines) file.
3. Upload the file to Grail using the Dynatrace API, Workflows, or a custom app.
4. Use the load and lookup commands in your DQL queries, notebooks, and dashboards.

For detailed instructions, visit our lookup data documentation. And to see how FreedomPay built its implementation from end to end, watch the full demo on YouTube.

The post How lookup tables turn observability data into business insight appeared first on Dynatrace news.

Transform how you derive actionable insights from data

Have you ever struggled to make sense of a massive data set or turn raw data into actionable business insights? Investigations handles all this for you—and more—at the scale modern organizations demand.

Whether you’re troubleshooting API call throttling, debugging AWS integration issues, or accelerating root cause analysis, the Investigations app provides a flexible toolkit for exploring your data, allowing you to:

• Analyze large DQL results in their original form at a detailed level
• Maintain your entire investigation flow and historical queries in context via the query tree
• Perform complex investigations on data stored in Dynatrace Grail®
• Build DQL queries quickly and efficiently based on your findings
• Save and reuse evidence to refine queries and uncover answers
• Pivot your queries based on the metainformation attached to log records
• Analyze the observability metrics connected to your log sources

With Investigations, you can easily navigate through investigation history to review queries and results—streamlining incident response, accelerating root cause analysis, and delivering deeper contextual insights from your data.

Get started with practical use cases

As strong advocates of “learning by doing,” we understand you may be curious about how to start using the Investigations app more broadly. You don’t need to start from scratch—there are plenty of documented scenarios you can customize for your own purposes. To help you get started, explore these informative tutorials:

• Investigate API call throttling using DQL and Investigations
• Debug AWS Integration issues
• Creating faster filters for DQL queries to speed up any investigation

If threat hunting is more your style, you can start with these tutorial-style investigation challenges:

• Christmas hands-on: Who stole Santa’s secret file
• Valentine’s Day hands-on: Heartbeat failed

There are, of course, many more use cases to explore—Investigations can be used with any data in Grail: logs, events, metrics, traces, and even performance data.

Drive insights across all your data

Since its launch, the recently renamed Investigations app has been key to unlocking insights in Grail, allowing teams to analyze metrics alongside logs and connect traces during incident response—this is a game-changer for both security and observability. Over time, the capabilities of the Investigations app were extended well beyond security, supporting a wide range of analytical scenarios.

Despite the name change, all the app’s existing features, including application IDs, which are used in intents and in the browser address bar, remain unchanged, so no updates to integrations or bookmarks are needed. And the roadmap will continue to prioritize incident response and accelerations of investigations.

For ease of use, Investigations will remain listed under “Security” on the Dynatrace platform and in Dynatrace Hub.

For complete details, please see Investigations documentation.

Ready to get started?

Investigations is a built-in app available in all SaaS environments beginning with Dynatrace SaaS version 1.330. Best of all, this app requires no additional subscriptions or privileges—so you can start using it immediately without waiting for access permissions and unlock deeper insights for your business.

The post From incident response to everyday analytics: Introducing Dynatrace Investigations appeared first on Dynatrace news.

Enriching observability data with additional context means improved data quality, which leads to better decision-making and faster troubleshooting. Instead of switching to an external data source and searching for a specific identifier across multiple documents, you now gain immediate insights at query time, effectively streamlining your work.

In this blog post, you’ll learn how to ingest lookup data and use it to effortlessly enrich your observability data. Practical use cases outline scenarios in which lookup data improves user workflows and makes root cause analysis and troubleshooting more efficient.

How to ingest lookup data

Lookup data files can be uploaded in formats such as CSV, JSON, or XML. You can upload data files using Workflows, via API, or by creating your own custom app. Once ingested, you can query lookup data just like any other Grail data, using Dynatrace Query Language (DQL) commands like lookup and join, or built-in Dynatrace® Apps like Dashboards, Notebooks, and Security Investigator for exploratory analytics.

In addition to an uploaded data file, you also need to provide a parse pattern written in Dynatrace Pattern Language (DPL) that defines the structure of the lookup data.

Once uploaded, you can access lookup tables via the load command. Be aware that files are organized in a directory-like structure in Grail. To make it easier to find stored files, we’ve introduced autocomplete functionality. Just start typing and jump directly to the respective file.

To learn more about supported file types, available attributes for data ingest, or the structure of parse patterns, please have a look at our documentation.

Practical use cases

Populating lookup data is a fantastic choice for enriching data with additional context in several scenarios:

• Mapping error codes in your logs to readable text for streamlined troubleshooting,
• Enriching IP addresses or IDs with respective account names to convert meaningless identifiers into meaningful qualifiers that speed up triage and root cause analysis.
• Accelerating security investigations with allow lists for security data.

Enrich your data with business context

Imagine that your system’s business-relevant events logged in Grail contain product IDs, and you’d like to enrich the IDs with the vendor’s name and some additional information from an external source.

This can easily be done with lookup tables by ingesting data containing the product and vendor information. In the example below, we use the product ID as the lookup field and enrich the business events with the mapped vendor values from the lookup table.

fetch bizevents
| lookup [ load "/lookups/vendorlist" ],
    sourceField: product.id,
    lookupField: product.id

Improved insights when working with security data

In another use case, imagine a security analyst is tasked with finding suspicious login attempts to your company’s network outside of business hours. Let’s assume corporate policy allows IT engineers to work from home any day, but that is not the case for accountants. The security analyst wants to understand which usernames belong to which role. Doing this manually would mean spending considerable time cross-referencing employees with their respective roles and manually creating filters based on usernames.

Creating a lookup table containing employees’ usernames and roles could significantly streamline this work, allowing the analyst to use external data to filter and summarize more accurate results.

Filtering for malicious IP addresses

Suppose your security analyst has obtained a list of fraudulent IP addresses from a threat intelligence feed that tracks malicious IP activity. These IP addresses are associated with spam, malware, botnets, or other malicious activities that expose your applications to potential threats.

The security analyst can now store this suspicious IP list as lookup data in Grail, update it whenever necessary, use it to detect and flag requests from any listed IP addresses, and leverage the data for further analysis in Security Investigator.

Flag TOR exit nodes

Going a step further, your security analyst can identify, flag, and track requests from TOR networks. TOR is an anonymizer that hides your tracks on the internet. By rerouting your internet activity via at least three other nodes before reaching your website, the TOR network obscures where requests originate, allowing bad actors to hide their identity and explore the internet with malicious intent.

The analyst creates a lookup table and populates it regularly with the latest list of TOR exit nodes. This data is used for further analysis, for example, in Security Investigator to detect login attempts that originate from TOR.

To utilize the Dynatrace® platform’s full power and set the TOR data in context, the analyst automates the fetching, writing, and uploading of the list of IP addresses using Workflows and visualizes the data with Dashboards.

What’s next?

Lookup tables provide a method to efficiently add context to any type of data stored in Grail. They can be used to integrate operational and transactional data, supporting your users in their day-to-day lives.

Stay tuned for further updates, such as improving our existing Snowflake Workflow Connector by adding capabilities to create and manage lookup tables, and using Security Investigator to create new lookup tables or view and filter existing tables.

Are you interested in trying out lookup tables in your own environment? This new capability is available as a public preview for all customers running the latest version of Dynatrace SaaS with an active Dynatrace Platform Subscription (DPS). It is super simple to activate; head over to our documentation to learn how.

The post Enrich your Dynatrace data with the newly introduced lookup tables appeared first on Dynatrace news.

When you’re working with complex, multi-layered data, pinpointing the root cause of an issue or identifying anomalies can feel like trying to find a flickering light in a city skyline. Effective troubleshooting demands a deep understanding of data structures, filters, and query logic. But what happens when you’re staring at a mountain of logs or metrics and don’t even know what to look for? For instance, imagine you’re investigating a sudden spike in error rates across your application. Without knowing which service triggered the cascade or how to filter the noise, you’re left guessing where to begin.

With the introduction of deep search functionality in Grail, we’ve introduced a new way to cut through the noise of complex data. You can now explore data effortlessly without needing to understand every single layer of complexity in your data structure.

Investigating data is a time-consuming process

Difficult data investigations can be taxing. Examining data takes too long, and probing for issues is tricky for those who don’t know the complexities of the data structure. In the query example below, taken before the introduction of DQL search, we attempt to filter within spans where any event sent from an Oracle database contains the phrase “error”.

Search within complex structures

Grail’s unique data warping technology allows for index-free, schema-on-read, high-performance queries, enabling you to flexibly analyze diverse datasets. While Grail has long been capable of dynamically interpreting complex data structures at query time, with the addition of the DQL Search command, you can now deep dive into your metrics, logs, traces, and other data stored in Grail in a way more agile and intuitive way.

Simplified string-based filtering across multiple fields

You can now search across vast and complex data sets without needing to understand the underlying structure. By leveraging string-based search, the system offers intuitive exploration of deeply nested data—no manual filters or query syntax are required. This approach eliminates the traditional barriers of schema knowledge and filter construction, making data investigation accessible to everyone. (See Figure 2 below.)

Surface the needle in the haystack faster

You can search across all fields, not just indexed or predefined ones, making it easy to surface any data without needing to pre-tag or preprocess it. DQL search is optimized to be highly performant, even with terabytes of data.

Search in nested structures

Finding relevant data within nested elements or arrays is cumbersome, requiring a lot of effort to ensure you don’t exclude necessary parts of the data structure. With the new DQL Search command, you can search across complex data structures, including nested elements and arrays, to find exactly what you need.

Imagine a DevOps engineer troubleshooting a failed deployment in a development environment. Suspecting a misconfiguration or runtime error, the engineer needs to find traces containing the term “exception”—buried deep within the span.events array of nested records. The system scans across all nested fields and surfaces relevant spans instantly—saving time, reducing friction, and accelerating root cause analysis.

Now, consider another example with Dynatrace Security Investigator. You’re investigating a security incident in your Amazon cloud environment. You need to track a specific event from your AWS CloudFront logs, and you have the CloudTrail event ID value. You have no idea where to find the event ID. It could be buried in the content, tucked away in header values, or stored in an obscure field. The DQL Search command empowers you to search across all fields instantly—no filters, no field-mapping, just fast, accurate results when you need them the most.

As simple as using a search bar, inside DQL

Experience the flexibility and power of exploratory analytics in Dynatrace without worrying about indexes or schemas. The DQL Search command allows for deep, detailed investigations while maintaining simplicity, making it accessible to a wider range of users.

The post Deep Search: Transform your data exploration and investigation with Dynatrace Grail appeared first on Dynatrace news.

For developers, the ultimate win is creating impactful features, shipping them quickly, and confidently implementing fixes when needed. Distributed tracing plays a critical role in achieving these goals. As a core component of observability, it provides the insights developers need to write better code, collaborate efficiently, and increase application reliability.

Developers know how challenging it can be to follow traces from beginning to end across multiple entities and data providers. Harder still is following those traces in context with metrics, logging data, security details, and real user experience data. Certain tools may provide visibility into small segments or isolated details. But developers often have to piece these clues together manually, which makes it difficult to extract meaningful and reliable results.

Dynatrace delivers distributed tracing with capabilities designed to streamline workflows. From understanding the impact of code changes to quickly resolving issues flagged in production, here’s how distributed tracing can help you increase developer velocity and deliver meaningful innovation throughout the software development lifecycle (SDLC).

Distributed tracing best practices key takeaways:

1. Visualize and understand your system. Distributed tracing plays a key role in uncovering the topology, interactions, and dependencies of your services.
2. Analyze and fix problems fast. Combine distributed tracing with AI analysis, context-driven querying, and collaboration tools.
3. Build better software with telemetry data in context. Unify telemetry data
4. Automate to ship better code faster. Use distributed tracing to automate and streamline workflows and set up intelligent alerts.

Visualize and understand your system

Before implementing changes or designing new features, it’s essential to have a clear understanding of your system’s current state. Without a clear picture of your current system, any change risks overlooking critical dependencies and misalignment with your architecture, whether in the cloud or on-premises. Distributed tracing plays a key role in uncovering the topology, interactions, and dependencies of your services, giving developers the visibility they need to make informed decisions.

Dynatrace Smartscape® technology automatically maps horizontal service relationships, offering real-time visualization of service connections and dependencies. This end-to-end view helps developers quickly understand how components interact within the overall system.

For a more granular understanding, inspecting individual traces allows you to focus on specific workflows or problematic behaviors. This depth of insight can help you plan better and identify challenges early, reducing surprises during later stages of development.

Trace example: Verifying service for “platinum” loyalty users

Say you’re a developer on an app called “AuthenticationService.” You can select a trace that lets you visualize its path through your system and examine each span’s properties to ensure “Platinum” loyalty users are being served effectively. While understanding the state of your system is important at each stage of the SDLC, it’s often one of the first steps in planning, requirements gathering, and designing changes.

In Dynatrace, you would open the Distributed Tracing app and enter a filter for the request attribute that indicates loyalty status. This filter returns a list of traces you can inspect individually to understand any bottlenecks or issues that could be impacting this specific subset of customers.

Analyze and fix problems fast using distributed tracing for developers

Identifying root causes in complex microservice-based architectures can be tough, but Dynatrace Davis® AI analyzes dependencies in real time, helping you quickly and confidently resolve issues.

But what if you want to visualize data outside of an individual problem occurrence? Some issues occur repeatedly in specific areas of a system, while others arise from application changes. Problems often span multiple teams, requiring collaboration to resolve effectively.

For broader insights, the Dynatrace Distributed Tracing app provides flexible access to raw trace data. You can filter by namespace, release version, or database calls, then Dynatrace automatically generates a DQL (Dynatrace Query Language) query so you can view results in notebooks or pin visuals to dashboards, which enable you to collaborate seamlessly with other teams. This approach is particularly useful during testing, deployment, and maintenance, where visibility into system behavior is critical.

This example shows a plot generated by a DQL query that indicates the number of database calls an application makes and how they perform over time.

Let’s assume you’re also interested in how your database queries are performing. To get an instant performance summary, you can easily add a table to your dashboard that summarizes the count, average, and percentiles of the durations of your database spans.

Build better software with telemetry data in context

Developers juggle many responsibilities, from testing and deploying code to maintaining stability. However, fragmented workflows and the endless search for the right information can often slow progress and create frustration, making it harder to meet deadlines and maintain a focus on innovation.

As a developer, how can you get the information you need without wasting time chasing down data and trying to understand all its upstream and downstream effects? The answer: distributed traces and telemetry data in context.

Flexible access to unified telemetry data is a game-changer for developers. With Dynatrace, you can seamlessly query multiple telemetry data sets and visualize the results in ways that align with your specific needs. This flexibility empowers you to connect the dots across complex workflows, analyze incidents within their broader context, and make data-driven decisions that support automation and enhance software quality.

The example below brings together spans and logs. Dynatrace automatically enriches every log with trace and span IDs, making it simple to track correlations among related data. With Dynatrace Query Language (DQL), you can dynamically combine spans and related log content on-the-fly.

To understand the impact of an error, you can chart which services and endpoints are emitting this log error.

As a developer, you can unlock deeper insights, linking operations to specific errors and enabling you to assess the scope and impact of incidents accurately. This gives you and your teams the ability to move beyond reactive problem-solving into proactive, intelligent development.

Use distributed tracing for developers to automate and ship better code faster

Automation is key to accelerating development while maintaining quality. By streamlining workflows, automating repetitive tasks, and setting up intelligent alerts, teams can proactively address issues and minimize manual effort.

For example, as a developer, you can use trace data to establish quality gates or set alerts for specific exceptions, ensuring you can quickly flag and resolve any recurring problems. This approach boosts efficiency and ensures that teams can consistently deliver high-performing, reliable code to production.

Assume you’ve been working on a bug that appeared as an exception in your traces. You want to check if the bug has been fixed and set up an alert to get a reminder if this particular bug comes back.

Let’s start by creating a metric based on this particular exception in the Dynatrace stream-processing data ingestion technology, OpenPipeline™. By creating a metric at ingestion using OpenPipeline’s centralized data handling and advanced data processing capabilities, you can continually monitor for evidence of the exception recurring.

You can use this extracted metric to visualize its frequency over time in a dashboard.

Finally, once you’ve set up a metric, you can use it to create an alert and send a notification to your system of choice (such as Slack, Teams, ServiceNow, and so on).

Increase reliability and automate more with distributed tracing best practices

As development responsibilities continue to expand in today’s complex environments, distributed tracing has emerged as a critical tool for developers throughout the entire software development lifecycle.

Applyin these distributed tracing best practices helps developers visualize systems, resolve issues quickly, and streamline workflows. By providing deep insights and context of upstream and downstream events, traces and spans give teams the data they need to automate tasks and deliver more reliable, higher-quality software faster and with greater confidence. The result allows you to focus on what matters most: creating impactful features.

Start integrating distributed tracing into one part of your workflow today and see how quickly it becomes indispensable to your development practice.

The post Distributed tracing best practices for the software development lifecycle appeared first on Dynatrace news.

Get ready to experience a whole new world of limitless tracing power. With our latest enhancements, we’re transforming the way you work with trace data. The Dynatrace® platform now enables comprehensive data exploration and interactive analytics across data sets (trace, logs, events, and metrics)—empowering you to solve complex use cases, handle any observability scenario, and gain unprecedented visibility into your systems. Whether you’re using OpenTelemetry or OneAgent, operating in the cloud or on-premises—we’ve got you covered.

Introducing a new era of distributed tracing with advanced analytics

In today’s complex systems landscape, understanding the root causes of issues can be daunting, especially as applications scale and OpenTelemetry adds complexity.

Davis® AI automatically pinpoints root causes, offering immediate answers. For deeper exploration, our Distributed Tracing app empowers you to analyze raw trace data and uncover insights, whether troubleshooting errors, optimizing performance, or discovering the “unknown unknowns.”

But why stop there? Building on this solid foundation, we’re thrilled to announce two powerful platform enhancements. Say hello to advanced trace analytics and new data storage and capture options. These game-changing features elevate your data interactions, opening up vast possibilities for advanced queries and efficient data management tailored to your needs.

Site reliability engineers, performance architects, and developers can now leverage dynamic analysis tools like dashboards and workflows to explore trends, automate processes, and maintain control at an unprecedented level. Additionally, these queries serve as excellent starting points for more complex data explorations with Notebooks.

Get ready to maximize the full potential of your trace data—unlock deeper insights and automate like never before, all within a single platform.

Level up your analytics game: Enhanced team collaboration and advanced data insights

With traces now stored in Dynatrace Grail™, our scalable data lakehouse, you can unlock powerful new analytics capabilities, handle massive volumes of data, and run complex queries seamlessly. Combining traces with logs, metrics, Kubernetes events, and telemetry attributes gives you a complete, contextual view of your environment for unmatched end-to-end observability.

Unlock deeper insights

Using Dynatrace Query Language (DQL), you can extract game-changing insights from raw span data with precision. Use these queries to start more complex data exploration with Notebooks. This enables you to uncover hidden patterns, discover unknown unknowns, and make confident, data-driven decisions. These powerful insights can easily be transformed into interactive dashboards.

Example: Exception analysis

Understanding patterns, especially regarding exceptions, is no easy feat. However, you can begin unlocking additional insights using the Distributed Tracing app. For example, you can filter to understand endpoint performance where exception messages contain the string, access denied.

Once filtered, you can easily open a notebook with a pre-populated DQL query. You can then add additional details or modify the query as needed. Combine multiple findings in a notebook or dashboard to share your analysis with your team, allowing them to see these focused updates live in real time.

Achieve superior analytics

Transform trace data into intelligent insights by combining trace data with logs, metrics, and events. This combination allows you to enrich all data and get details in context. Use this intelligent data to see what matters to you most in real time by creating interactive dashboards and driving better decision-making. This gives you the power to break down silos, spark collaboration, and extract actionable insights with ease. It democratizes access to critical data, ensuring all teams can leverage the same reliable insights to drive impactful outcomes.

Example: Combine trace data with logs

A common scenario is understanding which frontend API requests have log messages on the backend indicative of a specific problem. Let’s look at an example where the log message contains timeout, and we want to understand the response time of traces in the context of these messages.

By linking trace data with logs, you can query across spans and related log messages. You can also summarize with DQL to understand how often a specific pattern occurs. To visualize span duration, use p99 to see the slowest percentile of span response times and then navigate directly to the traces.

Automate with Dynatrace OpenPipeline

With Dynatrace, you can create custom metrics from trace data using Dynatrace OpenPipeline™, unlocking powerful new automation capabilities. It’s now possible to create metrics on OpenTelemetry and OneAgent spans with any available attribute, giving you the power to define operational, request, and method-level metrics.

OpenPipeline provides the flexibility to build metrics tailored to your specific needs, enabling you to integrate metrics seamlessly with advanced Dynatrace automation features, such as AutomationEngine and SRE Guardian. You can streamline workflows, intelligently automate repetitive tasks, proactively resolve issues, and spend more time innovating with automation, ensuring that only reliable, high-performing code reaches production.

We’ve only scratched the surface of scenarios where advanced analytics can make an impact—the possibilities are virtually endless. By combining advanced trace analysis, intuitive query capabilities, and seamless automation, your team can enable sharper analysis, streamline workflows, and foster innovation. This powerful approach ensures you can focus on delivering better outcomes with greater efficiency, empowering your organization to tackle complex issues with precision and agility, ultimately bringing unprecedented value.

Extended trace retention: Retain data longer when it matters

Need to analyze trends over the long term or adhere to compliance requirements? With extended trace retention, you can store trace data for up to 10 years. This feature ensures your organization is well-equipped for trend analysis and detailed post-mortem reviews—all while meeting regulatory requirements.

Retention policies are fully configurable in OpenPipeline. You can share data in buckets with varying retention times depending on the use case, allowing you to target specific applications or error-prone services for longer storage. This precision reduces storage costs while ensuring you retain the data that matters most.

Extended trace ingest

You can now customize trace ingestion rates to meet your specific needs. While most trace data is already ingested at high coverage rates, this option gives you more granular control over your trace volume. Ingest as much data as you want, above and beyond what is already included in your Dynatrace license.

Maximize the value of your OpenTelemetry data

At Dynatrace, we love OpenTelemetry. We champion open source innovation and recognize OpenTelemetry’s influence in setting observability standards (we’re also a top contributor). That’s why our advanced capabilities were designed from the ground up with OpenTelemetry at its core. We built our entire new tracing experience on OpenTelemetry semantic conventions and expanded from there. Now, OpenTelemetry users can troubleshoot and analyze while leveraging  OTel standards. This commitment empowers you to simplify complexity and innovate faster by extracting maximum value from your data, regardless of origin.

Experience the future of Distributed Tracing

At Dynatrace, we believe that observability should be effortless and completely on your terms. With our latest advancements, we’re helping you manage complexity, innovate faster, and push boundaries. Our solution adapts seamlessly to your ecosystem, whether you use OpenTelemetry or run cloud-native or on-premises workloads. Built to handle enterprise scale, the Dynatrace platform processes massive volumes of data in real time while unifying insights across all teams in your organization.

We’re rolling out this functionality to existing Dynatrace Platform Subscription (DPS) customers. Elevate your observability journey with these new possibilities—tailored to your needs, your way.

If you’re not a DPS customer, you can try out the new Distributed Tracing experience with prepopulated data on the Dynatrace Playground.

If you’re new to Dynatrace and want to try out the new Distributed Tracing experience with your own data, check out our free trial

Take the leap today and discover how Dynatrace can revolutionize your approach to observability.

The post Distributed tracing with Dynatrace just got even better appeared first on Dynatrace news.

Deal with data overload in the enterprise

In today’s rapidly evolving digital landscape, enterprises are inundated with vast amounts of data. Extracting meaningful insights from this data is crucial for staying competitive. However, traditional data analysis techniques can be time-consuming and demand specialized expertise, limiting how quickly and easily insights can be obtained.

Empower deep data analysis with natural language queries

Davis CoPilot enhances efficiency and productivity by seamlessly integrating generative AI throughout the Dynatrace platform. This feature allows you to effortlessly gain insights and generate queries without needing to learn new syntax or manage complex commands. Consequently, Dynatrace becomes accessible to a broader audience, including non-technical users and those who don’t work with Dynatrace on a daily basis, and empowers teams to make faster data-driven decisions.

Empower users with intuitive data access—without compromising security

At Dynatrace, we recognize the complexities associated with data environments. DQL, the query language employed to analyze data stored in Dynatrace Grail™ data lakehouse, offers remarkable versatility and power, serving as an essential tool for experienced users seeking to fully harness Grail’s capabilities. Davis CoPilot simplifies the data querying process for both professionals and beginners by enabling interactions through natural language. This democratizes data access, allowing all users to generate valuable insights swiftly and effortlessly. Consequently, the data analysis process is accelerated, empowering teams to make informed, data-driven decisions with increased speed and precision.

At Dynatrace, we prioritize the protection of your data. Our solutions are engineered to be secure, reliable, and entirely transparent. Davis CoPilot guarantees that your confidential information is never at risk of being leaked or disclosed across environments, as we ensure continuous protection of your prompts and data. Furthermore, there is no automatic model training or fine-tuning based on your usage, ensuring that your data is employed strictly for its intended purpose—to generate DQL and provide swift insights. This steadfast dedication to security and transparency enables you to use our tools confidently, trusting that your data is well-protected. Look at our documentation to get more insights into the privacy and security aspects of Davis CoPilot.

Get started with quick analysis in Notebooks and Dashboards

Davis CoPilot allows you to perform rapid data analysis in Notebooks and Dashboards by translating natural language prompts into Dynatrace Query Language (DQL). The results are automatically executed and returned, making complex data analysis more accessible than ever before.

Simply create a new notebook or dashboard, then select + Add > Davis CoPilot. Enter your prompt (or try one of our suggestions), and select Run. Davis CoPilot will generate and auto-execute the DQL so you can go from question to data insights in seconds. If you’d rather refine your query before executing it, open the dropdown list next to the run button and select Generate DQL only (this feature is currently only available in Notebooks).

Environment-aware queries unlock full data-context awareness

Davis CoPilot is much more than an AI tool that helps you create queries. Davis CoPilot knows the context of your data, which results in more precise answers using a feature called environment-aware queries.

Having environment-aware queries configured allows Davis CoPilot to identify unique data fields and custom metrics in your environment. You can now run more complex analyses and get better results by crafting more accurate queries that identify and reference relevant entities, events, spans, and metrics straight from your environment. And, of course, we do this without putting you or your data at risk. This functionality is opt-in, and you have full control over which data tables and buckets are accessible to Davis CoPilot. Let’s look at some examples:

If you’re an application owner tracking travel bookings for new trips on a travel website, you’ll likely need to track:

• profit made on each booking  (as a business event)
• applicable discounts (as a business event)
• length of time it takes customers to complete a booking (as a custom metric)

With this in mind, you might give Davis CoPilot the following command: “Show me the average revenue and price reduction for new trips over the last month.”

If you have environment-aware queries configured, the following DQL will be generated automatically, and you’ll get the relevant results you’re looking for.

fetch bizevents , from:now() – 30d 
| filter event.type == “new trip” 
| makeTimeseries interval:1h, {profit= avg(profit), discount= avg(discount)

With environment-aware queries configured, Davis CoPilot infers that “revenue” refers to the profit field and “price reduction” refers to the discount field, even though your prompt doesn’t use the correct field names. However, if you don’t have environment-aware queries configured, Davis CoPilot can’t identify all relevant fields. For example, the following incorrect DQL will be generated if the same conversational command is issued when environment-aware queries are not configured. In such cases, you won’t get any results since the fields mentioned in the command don’t exist in your environment.

fetch bizevents, from:now() – 30d 
| filter event.type ==  “new trip”
| makeTimeseries interval:1h, {avg_revenue = avg(revenue), 
  avg_price_reduction = avg(price_reduction)

Alternatively, you might ask Davis CoPilot the following: “On average, how long does it take customers to book new trips?” If you have environment-aware queries enabled, the following DQL will be generated, and you’ll get the relevant results you need.

timeseries avg(new_trip_booking_duration)

Conversely, if you don’t have environment-aware queries configured, you’ll likely receive an error message because Davis CoPilot can’t correctly map your question to your custom metric key. In this case, Davis CoPilot can’t generate a valid DQL query since it won’t be able to find a matching built-in metric.

User permissions are enforced both with and without environment-aware queries, ensuring that Davis CoPilot provides relevant responses that comply with individual data-access rights. Environment-aware queries truly unlock the power of Grail for everyone in your organization.

What’s next for Davis CoPilot

This is just the beginning of our new AI assistant journey. We’re committed to making Davis CoPilot even better, and we’ve got some fantastic features coming your way, from query explanations to problem insights, document generators, and more.

We value your feedback and are continuously working to enhance our product. Want to share your thoughts? You can share your learnings directly from the Davis CoPilot interface. Your feedback helps us refine the functionality and better meet your needs. You can also request to participate in ongoing or upcoming Preview programs. Get in touch with your Dynatrace account manager if you’re interested.

Get started today and embrace the future of data analytics

The launch of Davis CoPilot marks a significant advancement in data analysis capabilities. If you have a Dynatrace Platform Subscription, Davis CoPilot is available for you with the release of Dynatrace SaaS version 1.301. If you have a classic license, Davis CoPilot is available for you with the release of Dynatrace SaaS version 1.304.

Empower your team with the ability to effortlessly transform natural language prompts into actionable insights. Activate Davis CoPilot in your Dynatrace environment today and explore how it can transform your data analysis workflows.

For more information and to get started, please visit our documentation. Thank you for being part of this exciting journey with us. We look forward to your feedback and seeing how Davis CoPilot helps you achieve your goals.

The post Announcing General Availability of Davis CoPilot: Your new AI assistant appeared first on Dynatrace news.

The new Dynatrace Logs app, fully powered by Grail™ data lakehouse, significantly enhances the experience for novice and seasoned users. Logs delivers unparalleled simplicity with powerful, sharable views and insights to address these needs.

Enhanced log ingestion and seamless out-of-the-box integration

The Logs app is supported by comprehensive log ingestion capabilities provided by the Dynatrace platform and Dynatrace Grail while interwoven with other use-case-specific Dynatrace Apps:

While this is just an overview of the commonly used ingest routes and methods you can leverage, the Dynatrace Hub offers the complete set, currently with 500 supported technologies, 150 additional public extensions, and 65 Dynatrace Apps.

Harness log source sprawl

It’s common for large enterprises to provide hundreds of applications to employees and to host more than a dozen critical line-of-business applications. “Digital workers are now demanding IT support to be more proactive,” is a quote from last year’s Gartner Survey

Understandably, a higher number of log sources and exponentially more log lines would overwhelm any DevOps, SRE, or Software Developer working with traditional log monitoring solutions. The Dynatrace logs in context and surrounding logs features ensure that Dynatrace users are provided with a view that’s tailored to their needs and supports their daily tasks.

The Logs app enables Dynatrace users, whether novice or experienced power users, to rapidly filter and investigate results manually within the app or experience a seamless drill-down handover from use-case-specific apps.

Whether it’s cloud applications, infrastructure, or even security events, this capability accelerates time to value by surfacing logs that provide the crucial context of what occurred just before an error line was logged. For example, if one of your customers unexpectedly uploaded a 1 GB file instead of a 1 MB file, was there an error with the buffer overflowing, or was the network stack unable to handle the unexpected load? Even more importantly, how was the error handled, and did the process end successfully for the customer? With Dynatrace and Surrounding logs, answers to such questions are never more than a click away, with no need to write complex queries or patterns.

Simplicity for novice and power users

For users who seek quick access to relevant logs without the need to write complex queries, easy filtering capabilities are available from within individual log line details or by adding and selecting fields in the menu bar.

For those who aspire to become power users, the new in-app DQL editor (Dynatrace Query Language) translates manually selected filters into the DQL code executed in the backend.

Power users can edit and enhance DQL queries to update results, or they can start directly within the DQL editor view. This allows deep-dive analysis when manual investigation or complex queries are required.

As the screenshot above shows, you can transition back to the filter selection menu bar by selecting Back to previous filters or by sharing the query and results with other teams.

This ensures a smooth user experience for DevOps engineers and SREs, whether they prefer intuitive click-and-filter workflows or fine-grained control through DQL.

Shortening time to value and increasing impact

With the general availability (GA) of the Logs app, another enhancement was introduced that shortens the time to value and increases the positive impact made within organizations.

Filtered log views that were customized by click-to-filter or by leveraging the DQL editor can now be shared using direct links—simply copy and paste the link displayed in your browser’s address bar.

This enables you to

• bookmark and share reoccurring or complex queries.
• share situation or incident-specific views across teams.
• provision new Dynatrace users with relevant queries
• share timeframe-specific and pre-filtered views in a support case

See more with less

During the preview release of the Logs app, many customers provided us with valuable feedback that we incorporated into the design of the app.

User requests, such as the newly added line wrap feature, increase the readability of results and eliminate the extra clicks required to open a detail view or load the results within a notebook. This shortens the time it takes to investigate results and take action.

Speaking of doing more with less, more was released simultaneously with the GA of the Logs app:

• Increased performance of query results returning
• Live search within query results
• Live sorting within query results
• custom column arrangement
• enhanced open-with capabilities to open queries in other platform apps,
  or natively within the Logs app (for example, with Notebooks)

Why Dynatrace

The Dynatrace advantage lies in its ability to set logs in direct context, either automated by Davis® AI, in the context of specific team needs within an application, or as part of the broader observability scope provided by the Logs app itself.

Only Dynatrace provides a comprehensive and accessible log management and analytics experience, helping teams resolve issues faster without compromising on depth. Only Dynatrace Grail is schema-on-read and indexless, built with scaling in mind and built for exabyte scale, leveraging massively parallel processing. With Dynatrace, there is no need to think about schema and indexes, re-hydration, or hot/cold storage concepts.

This architecture also means you’re not required to determine your log data use cases beforehand or while analyzing logs within the new Logs app.

This includes logs collected from your organization’s Hypervisors, Linux and Windows servers, cloud-native logs from Azure, AWS, GCP, or Oracle, alongside the networking signals from Cisco, Juniper, NetScaler, or F5 appliances—to name a few examples.

What’s next

It’s simple, fast, and easy to ingest and gain multi-purpose value from logs—all without the need to be an expert or learn a complex query language first.

Learn how Dynatrace can address your specific needs with a custom live demo. Our observability experts will walk you through our solutions and show you how to deliver excellent customer experiences, foster your application security, and simplify IT operations.

If you’re not yet a Dynatrace customer, start your 15-day free trial.

The post Simplicity meets power: Introducing the all-new Dynatrace Logs app appeared first on Dynatrace news.

You’re in the middle of threat-hunting activities using Dynatrace and discover that some of your assets are trying to resolve a suspicious DNS name to extract data via covert channels like DNS tunneling. You now want to detect such events automatically by creating a custom Dynatrace security event. This blog post explains how to achieve this goal.

Following the Route53 log use case in Dynatrace threat-hunting documentation, a DNS query record is structured in the following way:

{
    "version": "1.100000",
    "account_id": "684399999995",
    "region": "us-east-1",
    "vpc_id": "vpc-026e4d09999995",
    "query_timestamp": "2024-08-17T07:24:57Z",
    "query_name": "250607.56755a335668636d52634969497343694167496e4a6c59584e766269493649.434a4762334a696157526b5a5734694c416f6749434a6b5a5852686157787a.tiitha-maliciousdomain.com.",
    "query_type": "A",
    "query_class": "IN",
    "rcode": "NOERROR",
    "answers": [
        {
            "Rdata": "34.667.67.208",
            "Type": "A",
            "Class": "IN"
        }
    ],
    "srcaddr": "172.31.1.6",
    "srcport": "42241",
    "transport": "UDP",
    "srcids": {
        "instance": "i-0b56150a99995e6d",
        "resolver_endpoint": "rslvr-out-c8e7c49999999fb5a"
    }
}

Let’s say that the meaningful fields you need for the security event are:

• timestamp when the DNS query was executed (query_timestamp);
• source IP where the query was performed (srcaddr);
• DNS query requested (query_name).

The JSON matcher, with its ability to parse out selected members, allows you to use the following simple DQL expression to extract only the fields you need:

json{
  timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp,
  string:query_name,
  ipaddr:srcaddr
}(flat=true)

As a result, you now have three additional fields with their respective types and values.

To add more accuracy to the query you’re about to automate, you need to extract the domain portion from the query_name field. This can be achieved with the following DQL expression:

LD* '.'? ( (LD '.' LD):domain '.' EOS)

That DQL pattern translates to something like this:

“Please extract two line-data (LD) portions from the content separated by a dot LD '.' LD) that is immediately followed by a single dot ('.') and then reaches the end of the string (EOS). A single dot and some line data might precede it.”

So, the pattern will match the highlighted DNS names below:

Adding a filter for the correct domain and limiting the query to only the last 5 minutes gives you the following DQL query to automate:

fetch logs, from: -5m
| filter aws.log_group == "/aws/route53/unguard-secla-demo/resolver-logs"
| parse content, "json{timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp, string:query_name, ipaddr:srcaddr}(flat=true)"
| parse query_name, "ld* '.'? ( (ld '.' ld):domain '.' eos) "
| filter domain == "tiitha-maliciousdomain.com"
| fields query_timestamp, srcaddr, query_name

Automate the query

Dynatrace AutomationEngine provides the ability to automate such periodic query executions. When you’re satisfied with your DQL query and are ready to automate it, select the Open with button in the lower right-hand corner of the query window.

As a result, a modal dialog will display, providing you with different possibilities about where to open the DQL query. In the current use case, you should open the query in Workflows.

Modify the default workflow

To execute the workflow automatically every 5 minutes, change the default on-demand trigger to either a cron or time-interval trigger. You can easily achieve this by selecting the trigger box in Workflows and following the instructions in the right-hand pane. You can read more about workflow triggers in Workflow schedule trigger documentation

Ingest query results as security events

The simplest way to do this is to use Dynatrace OpenPipeline. OpenPipeline allows you to create custom endpoints for data ingestion and process the events in the pipeline (for example, adding custom pipe-dependent fields to simplify data analysis in a later phase).

Set up a custom pipeline

The best way to set up a security event ingestion to Dynatrace is via Dynatrace OpenPipeline. You can create your own ingest endpoint that will then perform custom processing activities, create additional fields, store these events in separate buckets, and much more. For the current use case, we can create a custom security event pipeline to which we can send all the automated security events we have created from the Security Investigator, add respective fields for future analysis, and store them in the default security events bucket.

Now, create a general ingest endpoint for this use case and any automated detections we want to initiate from the Security Investigator in the future.

1. Go to OpenPipeline.
2. Choose Events > Security Events.
3. On the Ingest Sources tab, select + Source.
4. Give your pipeline a name, for example, Automated discovery from Security Investigator.
5. Choose a path to the new ingest source, for example, Signals.
6. Since we currently don’t want to create any dynamic rules to our pipeline, leave Routing set to static and choose the Custom events pipeline as the route.
7. Go to the Pre-processing tab and add a new processor with the type Add Fields.
8. Give the processor a name, for example, My custom processor.
9. Add the fields that you want to be automatically added when an event passes the pipeline. In our case, we will add the following fields:
   – kind == “SECURITY_EVENT”
   – provider == “Automated Detection Script”
   – category == “AUTOMATED_DETECTION”
   – provider_product == “Security Investigator App”
10. To figure out which fields to add for custom security events, refer to the semantic dictionary.
    Following the overall convention here will simplify analyzing this data later, together with other security events created by other sources.
11. You can test the processor with sample data to see what kind of event will be created based on the processor, and if you’re happy with the result, select Save.

For the current demo, this is enough, however, there are lots of things that you could configure additionally. Take a look at the Ingest custom security events via API documentation to learn more about sending security events to OpenPipeline.

Create and secure the token for event ingest

To create the token needed to ingest events, follow the steps in How to ingest data (events) documentation. When you have created the token, we recommend that you protect it in the Credential Vault instead of just using it in plain text in the automation engine.

1. Open Credential Vault.
2. Select Add new credential.
3. Select token as the credential type and give it a meaningful name.
4. Select App Engine as the credential scope.
5. Paste your ingest token into the token field and select Save.

After saving the credentials, an ID is presented to you in the form of CREDENTIALS_VAULT-12345. Write this down for later usage.

Ingest records through the OpenPipeline custom endpoint

Let’s return to our workflow and add a follow-up task for the DQL query. Choose the JavaScript task type and start writing the code to ingest security events.

Consider the following code for your function:

import { execution } from '@dynatrace-sdk/automation-utils';
import { credentialVaultClient } from '@dynatrace-sdk/client-classic-environment-v2';

export default async function ({ execution_id }) {

  const ex = await execution(execution_id);

  // load the token from the credential vault by its' ID
  const token = await credentialVaultClient.getCredentialsDetails({
    id: "CREDENTIALS_VAULT-F12345",
  }).then((credentials)=> credentials.token);

  // loop through the DQL query results and ingest them one by one
  var result = await ex.result('execute_dql_query_1');
  result.records.forEach (record => {
    ingest_event(token, record);
  });
}

async function ingest_event(token, payload) {

  payload["event.description"] = "Suspicious DNS query from AWS DNS Logs";

  fetch('https://<dynatrace_env>.dynatrace.com/platform/ingest/custom/events.security/signals', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json; charset=utf-8",
      "Authorization": `Api-Token ${token}`
    },   
    body: JSON.stringify(payload)
  }).then(function(response) {
    console.log(response);
  })
}

The code loads the token from the Credential Vault that will be used to ingest the events to OpenPipeline. Then, we iterate through the result set we got from the execute_dql_query_1 task and pass each event along with the ingest token to the ingest function, which will POST the event to the OpenPipeline ingest endpoint that we created earlier. The response to the request is shown in the console for debugging purposes.

We’re also adding a custom field called event.description to the event. You can add custom fields to your event from the OpenPipeline processing as well, but since we want to keep the ingest source as general as possible, it makes more sense to add the event-specific fields near the event, not the pipeline.

Now, give it a distinct name and test if it runs perfectly. Ingesting events to Grail via OpenPipeline should result in a status: 202 that is visible in the run_javascript_1 log console.

Fetch the security events from Grail

To view the newly created security events, follow the same principles as with fetching any other security data from Grail. Using the following DQL query, you can see all the events you have started to ingest:

fetch events
| filter event.kind == "SECURITY_EVENT"
| filter event.category == "AUTOMATED_DETECTION"

To feel the power of the semantic dictionary and to illustrate what you gain from using it, consider the following query. If you want to visualize all the security events ingested to Grail over the last 12 hours by event.category, it is quite simple with the following query thanks to the unified field names:

fetch events, from: -12h
| filter event.kind == "SECURITY_EVENT"
| makeTimeseries count(), by: { event.category }, interval:1h

The result could be a line chart like this:

Of course, you can follow your own naming convention, but this would make it much harder to view all the security events (both custom and Dynatrace-created events) in conjunction.

Operationalize the results

You can create additional workflows for your security events in Grail. For example, you can create a workflow that notifies you in Slack when a new event is created or integrate your workflows with Jira to create a ticket every time such events arise.

You can create a custom integration to your own SIEM system (similarly as we did for the OpenPipeline) for events that can enrich your security data in other environments. Consider your use case and work processes to determine the best way to approach security events available in Grail.

The post Generate security events from Dynatrace Security Investigator via OpenPipeline appeared first on Dynatrace news.

What is CVE-2024-6387 (regreSSHion vulnerability)?

The regreSSHion vulnerability (CVE-2024-6387) has been discovered by Qualys Threat Research Unit (TRU) as a Remote Unauthenticated Code Execution (RCE) vulnerability in OpenSSH server (sshd) in glibc-based Linux systems. The vulnerability, a signal handler race condition in OpenSSH server (sshd), allows unauthenticated remote code execution (RCE) as root on glibc-based Linux systems, which is a significant security risk. This race condition affects sshd in its default configuration.

Technical Overview

• Name: CVE-2024-6387
• Description: A security regression (CVE-2006-5051) was discovered in OpenSSH’s server (sshd). There is a race condition which can lead sshd to handle some signals in an unsafe manner. An unauthenticated, remote attacker may be able to trigger it by failing to authenticate within a set time period.
• CVSS Base Score: 8.1 HIGH
• Source: CVE (at NVD; CERT, LWN, oss-sec, fulldisc, Red Hat, Ubuntu, Gentoo, SUSE bugzilla/CVE, GitHub advisories/code/issues, web search, more

What is OpenSSH?

OpenSSH is an essential tool for secure communication and administration in Unix-like operating systems and is also available for Windows. It is part of the broader OpenBSD project and is widely adopted due to its robustness and security features.

What is the potential impact of regreSSHion vulnerability?

Exploiting this vulnerability can lead to full system compromise, where an attacker can execute any commands or code with the highest privileges. This can result in a complete system takeover, malware installation, data manipulation, and the creation of backdoors for persistent access. Moreover, gaining root access will enable attackers to bypass critical security mechanisms such as firewalls, intrusion detection systems, and logging mechanisms, further obscuring their activities. This can result in significant data breaches and leakage, giving attackers access to all data stored on the system, including sensitive or proprietary information that can be stolen or publicly disclosed.

To avoid sounding too alarming, we have to emphasize that this vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. As written in the Qualys white paper, “In our experiments, it takes ~10,000 tries on average to win this race condition. ” To our knowledge as of writing this article, no properly working exploits have been created.

How to mitigate the risk of CVE-2024-6387

The following versions of OpenSSH are impacted by this vulnerability:

• Versions earlier than 4.4p1 (unless patched for CVE-2006-5051 and CVE-2008-4109)
• Versions from 8.5p1 up to, but not including, 9.8p1

OpenBSD systems are unaffected by this, as OpenBSD developed a secure mechanism in 2001 that prevents this vulnerability.

There are three strategic paths to take to mitigate this threat:

• Patch Management: Quickly apply available patches for OpenSSH or upgrade to the relevant version that is not impacted by this vulnerability
• Access Control: Limit SSH access via network-based restrictions
• Network Segmentation: Divide networks to restrict unauthorized access and lateral movements from other systems

How to detect regreSSHion exploitation attempts with Dynatrace

This vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. Thanks to this noisy approach it can be detected from your sshd logs with Dynatrace DQL queries using Dynatrace Security Investigator.

Look for timeout events

Exploitation attempts for this vulnerability can be identified by many lines of “Timeout before authentication” in the logs. The log line in the logs could look something like this:

Jul  1 09:40:39 [localhost] sshd[787404]: fatal: Timeout before authentication for 192.168.12.45 port 17296

To search for such events from your sshd authentication logs stored in Grail, the following DQL query can be used:

fetch logs, from: -48h
| filter contains(content, "Timeout before authentication")

If you don’t get any results, that might be good news! You can further expand the timeframe of your query by changing the from time to -7d. This will get you logs from last week to make sure that no one tried to exploit the system during the last week.

Look for a high volume of log entries

There’s a chance that someone has tried to attack your system with a different attack or that the attack did not succeed as intended. For example, some Proof-of-concept attacks have failed, and these failures write various error messages to the victims’ sshd logs. It’s always good to check the log trend line in general to detect such attacks and when hunting for the unknown.

It’s easy to create a timeseries from your logs using DQL. With the following query, you can create a chart that shows sshd log line count from the previous day, grouped by host and aggregated at an hourly interval.

fetch logs, from: -24h
| filter contains(content, "sshd[")
| makeTimeseries count(), by: { dt.entity.host }, interval:1h

From the chart, it’s easy to see that nothing anomalous was detected using this query: no weird peaks or anomalies could be extracted from the chart.

Look for suspicious IP addresses in sshd logs

To ensure no one has tried to access your services from a suspicious location, it’s valuable to extract all the IP addresses from your sshd logs and see if something suspicious emerges from there. To do that, the following DQL query is helpful:

fetch logs, from: -2d
| filter contains(content, "sshd[")
| parse content, "ARRAY{ ld* ipaddr:ip }{0,10}:source_ip"
| expand source_ip
| summarize count(), by:source_ip
| sort source_ip desc

This query will fetch your sshd logs from the past two days, will extract ALL the IP addresses (both ipv4 and ipv6) from the content and will sort the result based on their occurrances to your results. If you discover any suspicious IP addresses from this list, you can start your investigation based on the discovered threat! If you see only your own addresses, you can rest assured: your sshd has not been accessed by an unknown IP address.

Analyze network flow logs

Last but not least, your network logs are the ultimate source of data. If you’re using AWS for your cloud platform and you have forwarded your VPC flow logs to Dynatrace, you can use DQL to look for your potential victims AND suspicious IP addresses that have tried to access your servers using SSH.

Consider the following query:

fetch logs, from: -24h
| filter aws.log_group == "/aws/vpc/flow_logs"
| parse content, """INT:version ' '
NSPACE:account_id ' '
NSPACE:interface_id ' '
('-' | IPADDR:src_addr) ' '
('-' | IPADDR:dst_addr) ' '
('-' | INT:src_port) ' '
('-' | INT:dst_port) ' '
('-' | INT:protocol) ' '
('-' | LONG:packets) ' '
('-' | LONG:bytes) ' '
TIMESTAMP('s'):start ' '
TIMESTAMP('s'):end ' '
LD:action ' '
LD:log_status"""
| filter dst_port == 22

The query will fetch all the Grail logs ingested from the CloudWatch log group called /aws/vpc/flow_logs. Using the VPC flow log default pattern available in DPL Architect, we can extract the meaningful fields to see only the network traffic targeting the SSH port.

From this dataset, we can create metrics based on the destination IP address to detect all the potential victims by adding the following code snippet to the previous query:

| filter action == "ACCEPT"
| makeTimeseries count(), by: {dst_addr}, interval: 1min

Viewing the results as a graph, you can see that some servers have received an anomalous amount of SSH traffic towards them! From here, you can continue to look further into the specific destinations to understand if this is malicious activity.

To understand who is targeting you, a similar query can be used. Instead of dst_addr use a src_addr to see the most active sources that have been targeting any of your ssh daemons on all the servers in the respective VPC.

| filter action == "ACCEPT"
| makeTimeseries count(), by: {src_addr}, interval: 1min

The results show that of the 725 SSH sources, three stand out with a huge load of requests that require further analysis.

Next steps for mitigating regreSSHion vulnerability

This particular vulnerability is challenging to exploit due to its race condition nature. However, all vulnerabilities should be taken seriously, their potential impact on your environment analyzed, and, where possible, mitigated. If you plan not to take any action for whatever reason, you should at least analyze the potential risk and make an informed decision before taking any action.

The post RegreSSHion vulnerability: Detecting CVE-2024-6387 in OpenSSH appeared first on Dynatrace news.

Dynatrace platform’s unique features are now available on Azure

As of today, Azure customers can leverage the latest Dynatrace core innovations on Microsoft Azure, including:

• Dynatrace Grail™ data lakehouse unifies the massive volume and variety of observability, security, and business data from cloud-native, hybrid, and multicloud environments while retaining data context to deliver instant, cost-efficient, and precise analytics.
• Dynatrace^® AutomationEngine features a no- and low-code toolset and leverages Davis^® AI to empower teams to create and extend customized, intelligent, and secure workflow automation across cloud ecosystems.
• Dynatrace^® AppEngine features a no- and low-code toolset and leverages Davis AI to empower teams to easily create and share custom, intelligent, and secure apps that leverage insights from data generated by their clouds.
• The new Dynatrace user experience, including powerful dashboarding capabilities and interactive Dynatrace Notebooks, drives tighter cross-team collaboration and enables more people within the organization to make data-backed decisions.

Azure Native Dynatrace Service allows easy access to new Dynatrace platform innovations

Dynatrace has long offered deep integration into Azure and Azure Marketplace with its Azure Native Dynatrace Service, developed in collaboration with Microsoft. With the AI-powered Dynatrace platform now generally available on Azure, Azure Native Dynatrace Service customers can now leverage the full AI power of the Dynatrace platform directly from Azure. With Dynatrace directly connected to your Azure subscriptions, Dynatrace OneAgent^® can be deployed directly to VMs and App Services from the Microsoft Azure Portal. One-click activation of log collection and Azure Monitor metric collection in the Microsoft Azure Portal allows instant ingest of Azure Monitor logs and metrics into the Dynatrace platform. For more details, see the blog post, Set up AI-powered observability for your Microsoft Azure cloud resources in just one click.

The following figure shows the benefits of Azure Native Dynatrace Service. For more details, please see the blog post Dynatrace and Microsoft Azure integrate to help accelerate your cloud transformation.

Using the Dynatrace platform on Azure allows Azure Native Dynatrace Service customers to get instant insights into their workloads with a comprehensive set of new innovative features with the best product experience from the Microsoft Azure Portal:

• The Dynatrace platform is automatically provisioned as part of the Azure Marketplace purchase process, allowing you to set up observability for workloads in minutes.
• Configuring the Dynatrace platform and deploying Dynatrace OneAgent from the Microsoft Azure Portal ensures that observability data from Azure resources arrives in Dynatrace within seconds.
• New Dynatrace platform features—with the Grail data lakehouse at their core—allow you to easily query and get insights into your workloads with the new Dynatrace user experience using DQL, Notebooks, and Dashboards, driven by a new Automation Engine, and AppEngine.

Leverage Dynatrace to get observability into all resources in your Azure tenant

This blog post shows you how the Dynatrace platform, together with the Azure Native Dynatrace Service—and unique new capabilities such as Grail, Dashboards, Notebooks, and the Cloud app—can give instant insights into all the Azure resources deployed in your Azure tenant.

Set up complete monitoring for your Azure subscription with Azure Monitor integration

After activating the Azure Native Dynatrace Service (see Dynatrace Documentation for details), the Azure Monitor integration is enabled easily via the Microsoft Azure Portal, as shown in the following screenshot. There’s no need for configuration or setup of any infrastructure.

Enabling the Azure Monitor integration in the Microsoft Azure Portal triggers the linked Dynatrace environment to start polling metrics and resource metadata for the Dynatrace Smartscape^® topology model. Furthermore, as shown in the above screenshot, the collection of resource logs and activity logs is easy to turn on.

Integrate multiple Azure subscriptions under a single Dynatrace environment

If you have multiple Azure subscriptions in your Azure tenant, it’s best practice to integrate the Azure subscriptions with a single Dynatrace environment. Establishing a single source of truth for all observability data reveals the full power of the Dynatrace platform by querying all data with comprehensive new capabilities from the one Dynatrace Grail data lake house for data storage.

To monitor multiple subscriptions within your Azure tenant

1. Create a new Dynatrace environment within the first subscription by creating the first Dynatrace resource (Option Create a new Dynatrace environment as shown in the following screenshot).
2. Then, create a Dynatrace resource in all other Azure subscriptions and link the Dynatrace environment created in Step 1 (the option Link Azure subscription to an existing Dynatrace environment is shown in the following screenshot).

More information can be found at Dynatrace link to existing. Also, see how to automate this process with bicep and how to automate with Azure CLI.

Azure use cases with Dynatrace Apps

The following new Dynatrace^® Apps can be leveraged to get instant insights into all Azure resources of Azure subscriptions connected via the Azure Native Dynatrace Service:

• Clouds enables cross-subscription and cross-region observability in one place.
• Dashboards leverages the power of DQL for Azure monitoring in one place.
• Notebooks offers advanced Azure observability analytics with DQL.

Clouds

Clouds allows easy investigation and troubleshooting of all Azure resources from the subscriptions previously connected to Dynatrace.

The following screenshot shows that you can easily search for resources by name, resource type, region, and tags across all connected Azure subscriptions.

Clouds provides resource properties, metrics, problems, and events in a single view, as shown below.

Davis AI automatically analyzes ongoing problems with your resources. Clouds makes it easy to search for all resources affected by detected problems and navigate directly to the problem details.

Clouds also supports getting the necessary insights for cloud governance. For example, Dynatrace uses automatic tagging to mark the user who created an Azure resource. Searching for all resources with a specific tag (for example, created-by) across your connected Azure subscriptions is easy with Clouds.

Dashboards

Dashboards provides new dashboarding capabilities powered by the new Dynatrace platform query language (DQL), which fully leverages the new Dynatrace experience.

With DQL and Dashboards, combining all observability data in one dashboard is easy. For example, it’s possible to combine resource metadata, resource health from activity logs, Azure Monitor metrics, Davis AI problems, OneAgent process details, Synthetic monitoring results, and much more in a single view, as shown below.

For each section, it’s possible to drill down for detailed context and learn the underlying DQL query.

The following shows a simple DQL summarizing all Azure Virtual Machine cores in the connected Azure subscriptions. By querying the Dynatrace entity model, cores are summed up from entity metadata.

fetch dt.entity.azure_vm
| parse azureVmSize, "KVP{LD[a-zA-Z]+:key'='(LONG:valueLong | STRING:valueStr)','?}:vmsize"
| fieldsAdd cores =   vmsize[`numCores`], memoryMb =   vmsize[`memoryMb`]
| summarize Cores = sum(cores)

Notebooks

Dynatrace Platform offers Notebooks, which allows advanced ad-hoc analysis of Azure environments based on metrics, logs, spans, events, and entities.

The following DQL from Notebooks queries data for all recent Azure health events and aggregates all resource health events from all Azure subscriptions connected via the Azure Native Dynatrace service.

fetch logs //, scanLimitGBytes: 500, samplingRatio: 1000
| filter matchesValue(cloud.provider, "Azure")
| filter azure.resource.type == "MICROSOFT.COMPUTE/VIRTUALMACHINES" or azure.resource.type == "MICROSOFT.COMPUTE/VIRTUALMACHINESCALESETS/VIRTUALMACHINES"
| parse content, "JSON:json"
| fieldsAdd operationName=json[operationName],
category=json[category],
resultType=json[resultType],
callerIpAddress=json[callerIpAddress],
action=json[identity][authorization][action] ,
principalId=json[identity][authorization][evidence][principalId] ,
principalType=json[identity][authorization][evidence][principalType] ,
name=coalesce(json[identity][claims][name],json[identity][claims][xms_mirid]),
time= json[time],
title=json[properties][title],
details=json[properties][details],
current=json[properties][currentHealthStatus],
previous=json[properties][previoustHealthStatus],
type=json[properties][type],
cause=json[properties][cause]
| filter  category== "ResourceHealth"
| fieldsKeep  time, title, details, type, cause, azure.resource.name
| sort  time desc

Utilizing the Dynatrace platform to observe your Azure subscriptions allows you to address compliance requirements beyond the standard; for example, the retention period for Azure activity logs is 90 days. With Dynatrace, you can configure up to a 10-year retention period in Dynatrace Grail (for full details, see data retention periods).

Here are further examples that show the power of DQL and Grail for Azure analysis:

Count the number of VMs per Azure Subscription

fetch dt.entity.azure_vm
| fieldsAdd  azure_subscription = accessible_by[dt.entity.azure_subscription][0], azure_region = belongs_to[dt.entity.azure_region]
| lookup [fetch dt.entity.azure_subscription | fieldsAdd name = entity.name ,  uuid = azureSubscriptionUuid], sourceField:azure_subscription, lookupField:id, prefix:"azure.subscription."
| lookup [fetch dt.entity.azure_region | fieldsAdd name = entity.name], sourceField:azure_region, lookupField:id, prefix:"azure.region."
| summarize  count=count(), by: azure.subscription.name

Count the number of VMs per Azure region

fetch dt.entity.azure_vm
| fieldsAdd  azure_subscription = accessible_by[dt.entity.azure_subscription][0], azure_region = belongs_to[dt.entity.azure_region]
| lookup [fetch dt.entity.azure_subscription | fieldsAdd name = entity.name ,  uuid = azureSubscriptionUuid], sourceField:azure_subscription, lookupField:id, prefix:"azure.subscription."
| lookup [fetch dt.entity.azure_region | fieldsAdd name = entity.name], sourceField:azure_region, lookupField:id, prefix:"azure.region."
| summarize  count=count(), by: azure.region.name

Identify the VMs with top CPU usage

timeseries max= max(dt.cloud.azure.vm.cpu_usage),      
 by:  {dt.entity.azure_vm}
| fieldsAdd maxCpu = arrayMax(max)
| sort maxCpu desc
| lookup [fetch dt.entity.azure_vm | fieldsAdd name = entity.name, azure_subscription=accessible_by[dt.entity.azure_subscription][0],  azure_region=belongs_to[dt.entity.azure_region] ], sourceField:`dt.entity.azure_vm`, lookupField:id, prefix:"azure.vm."
| lookup [fetch dt.entity.azure_subscription | fieldsAdd name = entity.name ,  uuid = azureSubscriptionUuid], sourceField:azure.vm.azure_subscription, lookupField:id, prefix:"azure.subscription."
| lookup [fetch dt.entity.azure_region | fieldsAdd name = entity.name], sourceField:azure.vm.azure_region, lookupField:id, prefix:"azure.region."
| fieldsKeep   maxCpu,azure.vm.name, azure.subscription.name, azure.region.name
| limit 20

Get the latest 10 log lines from Azure

fetch logs | filter cloud.provider == "Azure" or cloud.provider == "azure" | sort  timestamp desc  | limit 10

Upgrade to Dynatrace for Azure

If you’re an existing Dynatrace customer, please contact us to learn how to upgrade to Dynatrace on Azure. An overview of how to upgrade to Dynatrace is available in our guide, Upgrade to Dynatrace SaaS.

Get started with the AI-powered Dynatrace platform and Azure Native Dynatrace Service

Visit the Azure marketplace to start a trial of the Dynatrace platform on Azure and the Azure Native Dynatrace Service.

Please see Data Security Controls in Dynatrace documentation for an overview of the Azure regions currently supported by Dynatrace.

The post Discover the power of the Dynatrace platform on Azure with Azure Native Dynatrace Service appeared first on Dynatrace news.

As a security analyst investigating security incidents or threat hunting, you often must navigate among multiple executed queries and their respective results (in other words, investigation steps). You also need to manage the evidence you gather during investigations and reuse it when building additional queries. All this must be as fast and seamless while maintaining the investigation context.

Navigating investigations and threat hunting can involve saving executed queries and found evidence in unstructured text documents that can quickly lose their context. Manually documenting investigation steps can lead to human errors, like forgotten or wrongly documented queries or evidence, which will result in missed threats and wasted time. As the investigation grows and the different queries accumulate, it becomes harder to maintain the throughlines among the different branches.

The Dynatrace Security Investigator app provides branching navigation, making it easy to track your path through investigations.

Navigate in any direction within your investigation

Investigations in Grail are conducted using the Dynatrace Query Language (DQL). For multiple investigation steps, the Security Investigator provides a representation of the steps in a tree-like view. With this visualization, you can keep track of your investigation and navigate to previous steps and respective results in the investigation history—all in context, with no need to re-execute anything.

Every new DQL query execution creates a new descending node while keeping the integrity and hierarchy of previous queries and results unchanged.

Easily track threat-hunting twists and turns

Threat hunting is a nonlinear process. When looking for the unknown, you often stumble upon an unexpected thread of evidence that sparks a new idea and takes the investigation in a new direction. To support this nonlinear approach, we’ve created the investigation branching feature. When modifying and re-executing a previous step, the app creates a new branch while keeping the original investigation path intact and visible in the query tree.

You can create any new branches for new ideas and hypotheses while retaining the ability to navigate back to your initial train of thought. With the ability to customize and name branches, you can simplify locating historical steps without having to click through them individually.

The final visual representation provides a comprehensive overview of the entire investigation. You can see in a structured way how you reached your conclusions without needing to manually document the investigation flow.

With Security Investigator, you can always find the last working query, which was easy to lose in the past.

Character precision on a petabyte scale

Security Investigator increases the speed of investigation flows and the precision of evidence, leading to higher efficiency and faster results. Both outcomes are game-changers in time-critical situations, where answers are expected “yesterday.”

View raw content details when you need them

Data ingested into Grail is kept in its original format. A schema is applied only during query execution, or as we call it, “schema-on-read.” This makes data immune to format changes and enables you to choose your data structure based on your use cases.

The Security Investigator app enables you to view all data in its original format, regardless of the content, including characters that other tools might hide or not interpret. Enabling the multi-line content details view shows the stack traces with their line breaks in their original form, enabling you to understand the data much faster.

Find the needle in the “needle stack”

Finding a small metallic object in a stack of non-magnetic objects is relatively easy. Finding a particular object among millions of similar objects is trickier. With Security Investigator’s flexible filtering, you can create accurate DQL query filters quickly and efficiently.

If you need to fetch raw content from Grail and filter the results based on a selection of the content, you don’t need to copy partial records and manually paste them into an appropriate filter. Instead, you can easily select the portion of the text string you need and add it to your DQL query directly from the context menu.

To add multiple values to a filter, you can either select individual values or a range of values and apply them as a single filter in your DQL query. Only the unique values from the selection are added to the query, so you don’t have to investigate unneeded values.

Use your findings to enrich your investigation

With Security Investigator evidence management, you can persist relevant findings by attaching them to your investigation. Whether it is a suspicious IP address or a potential indicator of compromise, you can attach it to the investigation and use it later.

You can create type-specific filters to enable fast and accurate results, either focusing only on suspicious activities or filtering out the safe events that don’t require your attention.

Continue hunting for the unknown

Security use cases such as threat hunting and incident analysis are nonlinear, evidence-driven activities in which you don’t know what you’re looking for. You’re not pursuing a specific target; rather, you’re looking for the unknown. With Security Investigator, Dynatrace enables you to be more efficient, investigate faster, and be more precise.

Use Dynatrace Security Investigator for faster and more precise evidence-based security investigations.

The post Speed up evidence-driven security investigations and threat hunting with Dynatrace Security Investigator appeared first on Dynatrace news.

“Great! I have ingested important custom data into Dynatrace, critical to running my applications and making accurate business decisions… but can I trust the accuracy and reliability?”

Welcome to the world of data observability.

The Dynatrace open platform is well-positioned to take advantage of the exponential increase in data generation. However, coupled with the increase of external data sources that can now be ingested, there are new challenges in data management that need to be addressed.

 “Every year, poor data quality costs organizations an average $12.9 million”
– Gartner

Data observability is a practice that helps organizations understand the full lifecycle of data, from ingestion to storage and usage, to ensure data health and reliability. Data observability involves monitoring and managing the internal state of data systems to gain insight into the data pipeline, understand how data evolves, and identify any issues that could compromise data integrity or reliability. At its core, data observability is about ensuring the availability, reliability, and quality of data.

Data observability is crucial to analytics and automation, as business decisions and actions depend on data quality. In the age of AI, data observability has become foundational and complementary to AI observability, data quality being essential for training and testing AI models.

Dynatrace now addresses many of the issues customers experience around the health, quality, freshness, and general usefulness of data that is externally sourced into Dynatrace Grail™, allowing them to make better-informed decisions and optimize their efforts for digital transformation and data-driven operations.

The rise of data observability in DevOps

Data forms the foundation of decision-making processes in companies across the globe. Data is the foundation upon which strategies are built, directions are chosen, and innovations are pursued. Consequently, the importance of continuously observing data quality, and ensuring its reliability, is paramount. Surveys from our recent Automation Pulse Report underscore this sentiment: 57% of C-level executives say the absence of data observability and data flow analysis makes it difficult to drive automation in a compliant way. This not only underscores the universal significance of data, it also hints at its pivotal role within DevOps. For DevOps teams that inform deployment strategies, optimize processes, and drive continuous improvement, the integrity and timeliness of data are of significant importance.

As organizations scale and accelerate their digital transformation journeys, a major hurdle to proper DevOps adoption is the trustworthiness of the massive volume of data coming from various sources, much of which goes into data silos such as log management tools, SIEM solutions, and others.

The rise of data observability needs is where Dynatrace capabilities around Grail, analytics, and Davis® AI are in an outstanding and unmatched position to deliver the currently missing value to the market: a leading and single solution for all data observability analytics needs. This reduces the demand for further data flow analysis tools and clears any hurdles to making data useable for DevOps automation use cases.

Davis AI, Grail, and data observability

By grouping common data observability issues into industry-standard pillars, we can provide tangible examples and showcase current capabilities. The five pillars we focus on are freshness, volume, distribution, schema, and lineage.

Freshness: Timeliness of data

In an ideal ecosystem, actionable data should be as recent as possible, supported by learnings from accurate, historical data. Observing the freshness of data helps to ensure that decisions are based on the most recent and relevant information.

Scenario: Due to an undetected configuration issue, a flight status system from a popular airline had been buffering data for the last two hours before sending it on in one batch. Downstream dashboards and system automations were using outdated data, leading to incorrect statuses of flights in reports.

Solution: After setting up data ingestion into Grail, Dynatrace Query Language (DQL) is used to add a freshness field (Figure 1) which is calculated from the delta between when the signal was written and when it was ingested. This freshness measurement can then be used by out-of-the-box Dynatrace anomaly detection to actively alert on abnormal changes within the data ingest latency to ensure the expected freshness of all the data records. Furthermore, the new Alert on missing data feature in the Anomaly Detector panel can be used to trigger notifications when data is not coming in as expected after being baselined.

Value: The possibility of alerting on data freshness issues, based on a learned baseline through Davis AI, allows for faster time-to-detect where there are seemingly no infrastructure issues. Normally this would have left an issue undetected for much longer, providing a false sense of security, eventually leading to a much bigger customer and monetary impact for the organization.

Volume: Quantity of data generated or processed within a given timeframe

Unexpected increases or drops in the volume of data are often a good indication of an undetected issue.

Scenario: For many B2B SaaS companies, the number of reported customers is an important metric. It heavily influences downstream reports, and dashboards, shaping decisions from daily operations to strategic monthly reviews. In this scenario, a manually triggered run of a production pipeline had the unintended consequence of duplicating the reported customer metric. If left unchecked, this misrepresentation of a single KPI could lead to misguided decision-making processes through multiple layers of the organization.

Solution: Like the freshness example, Dynatrace can monitor the record count over time. Once a DQL query has been set up, it can be used in an automation workflow (Figure 2) where scheduling, prediction, comparison to actual value, and, finally, alerting are all taken care of to enable a fully flexible way to detect anomalies in data volume.

Value: KPIs and metrics such as the number of reported customers are central to an organization’s business and strategic processes. Any issues here will result in a loss of trust in the data, and, if left undetected, they will eventually lead to monetary impact, including loss of reputation for an organization.

Distribution: The statistical spread or ranges of data

The distribution of data is essential in identifying patterns, outliers, or anomalies in the data. Deviation from the expected distribution can signal an issue in data collection or processing.

Scenario: A financial institution processes millions of transactions daily, ranging from credit card purchases and mortgage payments to interbank transfers and ATM withdrawals. An erroneous change in the database system leads to a subset of the data being categorized incorrectly. After several days, the fraud detection system starts triggering on a frequent basis, and liquidity management dashboards begin showing questionable values.

Solution: Baselining and raising alerts on anomalies are core capabilities of Davis AI. After setting up ingestion for the data that you want to monitor, it’s simple to use Dynatrace full AI capabilities to observe and alert on any anomalies in the data. In the example above, ingesting the number of transactions as business events, anomaly detection could be based on this to proactively alert and trigger mitigation activities.

Value: While variations are expected in financial trends, anomalies should be auto-detected, and manual detection should not be relied on. Earlier detection of these issues will keep the fallout as low as possible.

Schema: Structure and relationships of data between entities

Observing the schema can help identify and flag unanticipated changes, such as the addition of new fields or deletion of existing fields.

Scenario: An externally connected database system made an update that inadvertently dropped the account_id column in the customers table. The automated data pipeline propagated these changes, leading to downstream reports, dashboards, and applications breaking as the previous field reference is now missing.

Solution: Using the DQL FieldsSummary command, we can keep track of the number of distinct field keys within a given family of data records. Once confirmed in a notebook, the number of field keys can be used in an automated workflow to continuously monitor the count and write it back to a new metric (Figure 3). Once the new metric is established, out-of-the-box Dynatrace anomaly detection can be used to alert on either a static threshold or a learned baseline.

Value: Observing incoming data Schemas, and thus placing expectations on what the external data should look like and must contain, allows for pro-active alerting and mitigation of issues long before they can lead to widespread business impact such as broken reports, dashboards, or further analytics on top of the data.

Lineage: Journey of data through a system

Data lineage provides insights into where the data came from (upstream) and what is impacted (downstream). It plays a crucial role in root cause analysis as well as informing impacted systems about an issue as quickly as possible.

Scenario: The hourly_consumption table was deprecated and removed by an overzealous database administrator as there were no known downstream consumers of this data, breaking a monthly integration check used for consumption reporting for shareholders.

Solution: In the future, Dynatrace Smartscape® could be used, which already builds a dependency graph, to enable a data lineage view. This would enable faster root cause analysis of any data-related problems, as well as allow for easy notification of downstream consumers who would be impacted.

Value: A proper understanding of the source of the data, as well as where it is used, helps drive down time-to-alert and time-to-repair. Time-to-alert is achieved by quickly and automatically alerting those who are impacted by a data issue by quickly understanding downstream consumers of the data, while time-to-repair informs on the source of where the data originated from, to quickly drill down into those systems.

Data availability: A prerequisite

You could implement the most contemporary, accurate, and useful data observability solution possible, but what good will it be if all the data simply does not arrive as expected? Broken pipelines or missing data sources would mean that there is simply no data to observe and that data may never arrive, forever lost.

A truly valuable data observability solution should be able to alert on data issues as early in the process as possible. This requires monitoring of the upstream infrastructure, applications, or platform supporting those data streams. This is where the power of Dynatrace end-to-end observability comes into play. Dynatrace can leverage existing Infrastructure Monitoring and Application Observability solutions to surface problems that can affect later data observability workstreams—long before a traditional data observability solution would pick up the issue.

Leverage the power of Dynatrace and Davis AI—now and into the Future

Anomaly Detection

Anomaly detection is grounded in the idea of baselining typical patterns of ingested data, designed to alert where a change or deviation from the norm is observed. These patterns typically go beyond simple flat or trend lines, often exhibiting complex seasonal behaviors, such as business hours or weekly patterns related to the industry. Dynatrace is particularly strong in this area: Davis predictive AI has been enriched over the years with a set of advanced machine learning (ML) algorithms optimized for time-series observability datasets to cope with these challenges. Davis AI anomaly detection, leveraging these ML algorithms, can already be used on the results of DQL queries. (Embedding ML algorithms into DQL as functions is on the Dynatrace platform roadmap.)

Considering the examples and solutions provided above, anomaly detection plays a pivotal role in numerous data observability use cases and can be harnessed to effectively address these challenges.

Triage and resolution of a data incident

Triaging requires an ability to identify the root cause of a data incident, which is particularly challenging as an organization scales up the volume and speed of data ingest typical of an enterprise environment. It’s easy to see how Davis causal AI problem detection could be extended in the future to identify root-cause data observability issues.

Depending on the incident, there might be different paths to resolution. One acceptable path could be full auto-remediation, whereby Dynatrace AutomationEngine could be triggered, scripts executed, permissions granted, security checked, and data corrected. A second path might require Jira tickets to be created and human intervention through an approval process. A data problem alert could be used as the event allowing for multiple methods to notify the correct data owners, stewards, governors, or data teams.

An incident requires not only resolution but also understanding and alerting upstream data providers and downstream data subscribers to the potential impact. Dynatrace is strong on the observability of data pipelines ingesting data into Grail and consumers of Grail data, although this is an area that will be enhanced and improved in the future product roadmap.

Prevention of future incidents

Not all data quality incidents can be prevented, especially because ELT/ETL data pipelines typically tend to grow over time and span many different heterogeneous collectors that have different ownerships. There are, however, mitigation techniques you can use, for example:

• Health tracking of key datasets or streams over time—alerting on anomalies
• Monitoring standard query results and changes over time
• Well-designed, data-focused dashboards for monitoring
• Auto remediation where appropriate with built-in audit logging
• Forensic abilities for ad-hoc data analysis

Summary

Dynatrace is uniquely positioned to provide even more value by extending our world-class observability platform into the data observability realm. To achieve this, we leverage Infrastructure Monitoring and Application Observability for early warnings on data pipeline issues and use DQL, Workflows, and Grail for data observability—all enabled by our best-in-class Davis AI engine.

Ensuring the quality and reliability of underlying data is more crucial than ever now that many organizations are deploying Generative AI models. Data observability is becoming a mandatory part of business analytics, automation, and AI. Davis AI and data observability together uniquely ensure the quality and reliability of data at the level of hypermodal AI—predictive, causal, and generative.

You can now monitor sources and incoming data pipelines for freshness, volume, distribution, lineage, and availability issues early on without added noise and in a central location, the Dynatrace platform. This gives your teams additional confidence over data quality, saves time, prevents inaccurate analyses and automation outcomes, leads to more trustworthy AI models, and supports efforts to consolidate or reduce the number of IT tools they rely on.

Ready to get started with Dynatrace data observability? For complete details, best practices, and detailed use cases, see Dynatrace data observability documentation.

The post Introducing Dynatrace built-in data observability on Davis AI and Grail appeared first on Dynatrace news.

VMware Aria Operations for Logs (formerly known as vRealize Log Insight) is used across enterprises to collect logs and provide analytics. The company recently announced a high-severity vulnerability in an earlier version of the tool. Tracked as CVE-2023-34051 (CVSS score: 8.1), the Aria Operations for Logs vulnerability relates to a case of authentication bypass that could lead to remote code execution. Basically, this means that an unauthenticated malicious actor could inject files into the operating system of an impacted appliance, which can result in remote code execution.

James Horseman from http://Horizon3.ai and the Randori Attack Team have been credited for the discovery and reporting. They have made a PoC for the vulnerability available and published the relevant indicators of compromise (IoC).

This vulnerability is a so-called “patch bypass” for a set of critical flaws that were addressed by VMware earlier this year, also discovered by Horizon3.ai. In their report, they presented how an attacker could use three different CVEs to achieve remote code execution. Since the patch only blocks access to Thrift services by IP and does not fix the other CVEs in VMSA-2023-0001, all an attacker needs to do is spoof an IP address and use the published attack again.

In this blog post, we show how to discover the original attacks toward the Aria Operations for Logs vulnerability using Dynatrace and DQL by finding the IoC-s from the log records.

How to detect VMware Aria Operations Logs exploitation with Dynatrace

Since we’re talking about a patch bypass, we’re looking at the original attack vector described in this blog post. To exploit this vulnerability to gain remote code execution (RCE), the following steps have to be taken by the malicious actor:

1. Create a Thrift client to gain unauthenticated access to the Log Insight Thrift server.
2. Spoof the IP address of the known worker.
3. Create a malicious TAR file containing a directory traversal using a valid PAK file.
4. Using remotePakDownloadCommand, upload the malicious PAK file to /tmp/<filename>.pak.
5. Extract the file using pakUpgradeCommand. This writes the file to where it’s needed on the filesystem.

A technical deep dive about the attack can be found in this article, VMware vRealize Log Insight VMSA-2023-0001 Technical Deep Dive.

Discovering the attack with Dynatrace

To discover this attack, you have to go through the log files that Aria Operations for Logs creates. The Log Insight server stores relevant logs in the file /var/log/loginsight/runtime.log. This log file is used to track all runtime information related to Log Insight. To analyze these logs with Dynatrace and DQL, the logs need to be ingested into Dynatrace beforehand.

Forwarding logs away from the main system is beneficial in case of any kind of successful attack by a malicious actor: if an attacker decides to remove all evidence from the system (including log files), then a security investigator wouldn’t have traces left to analyze. Sending logs to Dynatrace at runtime safeguards the log files for later analysis in a remote and secure location.

Security-related log records of Aria Operations for Logs are structured in the following format:

[2023-10-25 11:28:29.709+0000] ["https-jsse-nio-443-exec-9"/10.153.234.136 DEBUG] [com.vmware.loginsight.web.actions.misc.LoginActionBean] [User login success: vIDM: SAM=myusername, Domain=vmware.com, UPN=myusername@vmware.com]

When a new remote PAK file is downloaded as a tarball, the following log lines are created:

[com.vmware.loginsight.daemon.commands.SystemCommands] [PAK download initiated by node f2449ed5-11ee-45fd-a0a0-6225a33a8ac6] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Deleting existing PAK file: /tmp/exploit.pak] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak] 
[com.vmware.loginstght.daemon.commands.SystemCommands] [Max allowed pack size is 1505916450 bytes] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Current downloaded size is 21910]
[com.vmware.loginsight.daemon.commands.SystemCommands] [Total downloaded size is 21910]

From these log lines, the best candidate to look for in the logs is the third populated log, Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak].

Now, let’s fire up the DPL Architect and create a suitable pattern for this line:

With the created DPL pattern, you can extract the timestamp field ts, the logger class name as class that populated the log lines and the action that was written to the log record. These extracted fields give you the option to filter out the relevant records:

However, there’s another goal to keep in mind: you need all the log records that contain the Downloading statement and URL from where the tarball was loaded and the destination path in /tmp/.

As the blog post states: “The URL and filename will likely be different. However, the filename will always have the format /tmp/<filename>.pak. Also, note that this log may be legitimate. To determine if an attack has occurred, you need to have your administrator evaluate the URL to determine if it is a legitimate download URL.”

Let’s continue with our DPL pattern and extract the full URL where the payload was downloaded and the relevant filenames. Again, we’re opening up the DPL Architect and creating the following pattern:

'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file

With the created pattern, we’re extracting the url and the tarball’s name as download_file into separate fields using the look-around functionality of DPL. Additionally, we’re extracting the filename from the destination file path as well as the whole saved_file value for simpler analysis.

DPL pattern extracts the following result set from the log record:

Applying this pattern and filtering the records as described in the blog post gives you the following DQL query:

fetch logs
| filter log.source == "/var/log/loginsight/runtime.log"
| parse content, """'[' timestamp('yyyy-MM-dd HH:mm:ss.SZ'):ts '] [' ld '] [' ld:class '] [' ld:action ']'"""
| filter class == "com.vmware.loginsight.daemon.commands.SystemCommands"
| parse action, """'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file"""
| filter download_file == filename
| fields ts, url, saved_file

The query produces the following result, which you can now pass on to your administrator for manual evaluation.

A longer description of how the VMware Aria Operations for Logs exploitation works can be found in the Horizon3 blog post.

Further reading about how to use DPL Architect for security use-cases can be found at Speed up your security investigations with DPL Architect.

How to mitigate the Aria Operations for Logs vulnerability exploit

To mitigate this exploit, update to the latest version of VMware Aria Operations for Logs by applying the latest patches. Or follow the instructions within the VMSA-2023-0021.

Sources

1. VMware vRealize Log Insight VMSA-2023-0001 IOCs
2. GitHub – horizon3ai/CVE-2023-34051: VMware Aria Operations for Logs CVE-2023-34051
3. VMware Aria Operations for Logs CVE-2023-34051 Technical Deep Dive and IOCs
4. VMSA-2023-0021
5. Alert: PoC Exploits Released for Citrix and VMware Vulnerabilities

The post Detect VMware Aria Operations for Logs exploitation with Dynatrace and DQL appeared first on Dynatrace news.

To help you raise the quality of your investigation results, Dynatrace offers an easy way of structuring data using DPL Architect. This tool lets you quickly extract typed fields from unstructured text (such as log entries) using the Dynatrace Pattern Language (DPL), enabling you to extract timestamps, determine status codes, identify IP addresses, or work with real JSON objects. This allows you to answer even the most complex questions with ultimate precision. The best thing: the whole process is performed on read when the query is executed, which means you have full flexibility and don’t need to define a structure when ingesting data.

>> Scroll down to see Dynatrace DPL Architect in action (24-second video)

Investigating log data with the help of DQL

Let’s look at a practical example. The simplest log analysis use cases can be solved in seconds by applying a simple filter to the query results. If a CISO asks, “Have we seen the IP address 40.30.20.1 in our logs within the last year?” a simple DQL query seems to suffice:

fetch logs, from: -365d
| filter contains(content, “40.30.20.1”)

This query returns all the records that contain the string value 40.30.20.1.

In the next step, adding DQL aggregation functions enables you to answer more complex questions like “How many times in an hour has this IP address visited our website within the last two weeks?” Again, a simple DQL query helps you out:

fetch logs, from: -14d
| filter contains(content, “40.30.20.1”)
| summarize by: bin(timestamp, 1h), count()

However, this kind of simple filtering using a sub-string search is prone to errors and is not precise enough. The mentioned filters return all records that contain 40.30.20.1, but are not necessarily from the source IP portion of the log format. The string 40.30.20.1 might also appear elsewhere, for example, in query parameters, and a simple string filter will return all records containing this search term:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395
40.30.20.109 - - [28/Aug/2023:10:22:04 -0300] "GET / HTTP/1.1" 200 2216

The first log record is matched because of its query parameter—something we don’t care about in our case. The second log record came up because the source IP contains the IP address—this is what we’re really looking for. In a nutshell, if you’re querying terabytes of logs and wish to drill down to specific records, filtering strings from plaintext log content is not enough.

The issue is that questions from CISOs aren’t usually so trivial when it comes to security use cases. Or, the log format where the answers should be looked for is more complex (for example, AWS CloudTrail logs). As a consequence, we need to search structured data to avoid mistakes. To minimize false positives, increase the precision of queries, and get the maximum out of DQL, fields can be extracted from record content.

Extracting patterns using DPL

Dynatrace Pattern Language (DPL) is a pattern language that allows you to describe a schema using matchers, where a matcher is a mini-pattern that matches a certain type of data. These matchers can be used to extract new fields from the schemaless data in Grail to enable more precise log filtering. Consider the same Apache access log example from above:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395

To extract the client IP addresses from the beginning of the log, a simple DPL pattern can be used:

IPADDR:client_ip

The IPADDR matches both versions of IP addresses (IPv4 addresses in dot-decimal notation and IPv6 addresses in hextet notation), leaving all other IP-like strings unmatched. For example, if the log record contains a dot-decimal number that is NOT an IPv4 address (999.999.999.999), it won’t be matched.

DPL patterns can be applied in DQL using the parse command. The simplest way to get started with DPL is to use Dynatrace DPL Architect.

DPL Architect to the rescue

DPL Architect is a handy tool, accessible through the Notebooks app, which supports you in quickly extracting fields from records. It helps create patterns, provides instant feedback, and allows you to save and reuse DPL patterns, for faster access to data analytics use cases.

To open the DQL Architect, you have to execute a DQL query, select the content, and choose Extract fields.

Figure 1: Extract fields in Notebooks using DPL Architect (24-second video)

Starting with preset patterns

The simplest way to extract data is using one of the ready-to-use preset patterns available for the most popular technologies, such as AWS, Microsoft, or GCP. Start exploring AWS VPC Flow logs or analyzing Kubernetes audit logs by choosing the patterns from the panel on the left side. You can also customize the list by adding your own individual patterns.

Developing a new pattern using DPL Architect

DPL Architect helps you by creating your own patterns and provides instant feedback. Start typing a DPL expression in the pattern field, and review the matching data (highlighted below) in the preview editor.

For accessing the extracted fields in the resultset, it’s necessary to add a name. In the example below, we’re only interested in the client_ip and the related response_code. The pattern matches the whole record, but only two fields are being extracted since they have defined extract names. Any other data between those two fields won’t be visible in the results (however, it is very easy to add them later if necessary).

Remember, we started our journey with the DPL Architect by selecting query results in a notebook. The colored bar at the top of the DPL Architect shows how many records from your query result match the current DPL pattern. This enables you to modify the DPL pattern to ensure it matches all records in the resultset.

Records that don’t match the current pattern can easily be added to the Match preview panel by selecting Add to preview.

After changing the DPL pattern, any previously unmatched records will be highlighted and the progress bar on the top informs you that 100% of the records in the Notebooks resultset are matched. It’s now time to insert our pattern into the DQL query by selecting Insert Pattern.

Precise extraction of fields from complex data

DPL also provides matchers for more complex data structures, like key-value pairs, structures, and JSON objects, that enable you to parse individual sub-elements from the whole object. Consider the following log record:

1693230219 230.4.130.168 C "GET / HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" 186 {"username":"james","result":0 }

Imagine your CISO comes with a request: “Give me the list of all IP addresses where the user ‘james’ has successfully logged in from.” Extracting these three fields and building a DQL query is as easy as pie:

LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)

The DPL will extract three fields: an IP address from the record, a string from the JSON object element username, and an integer from the JSON element result.

Summarizing the results based on these fields can be done with the following DQL:

fetch logs
| parse content, "LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)"
| filter username == "james" and result == 0
| summarize by: src_ip, count()

Imagine that you don’t have DPL available and need to filter all log records based on only searching for string values: the result would contain a lot of false positives (which only add “noise” to time-critical investigations). This is especially true with more complex log records containing nested JSON objects. The above example was oversimplified intentionally, but considering complex log records like AWS CloudTrail log records, having precise access to data in a specific node of an object makes a huge difference.

Summary

When performing security investigations or threat-hunting activities, it’s important to have precision in place to get reliable results. Historical data needs to be available, and access to object details is required for precise answers. DPL Architect enables you to quickly create DPL patterns, speeding up the investigation flow and delivering faster results. With the possibility of using Dynatrace-provided patterns for selected technology stacks, investigators can deliver answers even faster!

Check out the following video, where Andreas Grabner and I teamed up for a new episode of Dynatrace Observability Clinic. In this video, we dig deeper into the topic of extracting data via DPL, including a live demonstration of DPL Architect.

Referral links

• Documentation: Dynatrace Pattern Language
• Documentation: Dynatrace Pattern Language Architect
• Blog post: Dynatrace Grail: The data lakehouse for observability, security analysis, and automation
• Blog post: Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail

The post Speed up your security investigations with DPL Architect appeared first on Dynatrace news.

Log forensics—investigating security incidents based on log data—has become more challenging as organizations adopt cloud-native technologies. Organizations are increasingly turning to these cloud environments to stay competitive, remain agile, and grow.

But as organizations rely more on cloud environments, data and complexity have proliferated. Teams struggle to maintain control of and gain visibility into all the applications, microservices and data dependencies these environments generate. Without visibility, application performance and security are easily compromised.

As a result, teams are turning to technologies such as observability to understand events in their cloud environments. Moreover, they have come to recognize that they need to understand data in context. But most observability technologies today provide information in silos. Without unifying these silos, teams miss critical context that can lead to blind spots or application problems—problems that compound when there’s a need to investigate security events.

Modern observability enables log forensics

Dynatrace is a software intelligence platform that provides deep visibility into and understanding of applications and infrastructure. It started as an observability platform; over time, it has expanded to provide real user monitoring, business analytics, and security insights. The recent innovation around log storage, processing, and analysis—Grail—makes Dynatrace a great solution for security use cases such as threat hunting and investigating the who-what-when-where-why-how of an incident.

Grail is a data lakehouse that retains data context without requiring upfront categorization of that data. Unifying data in Grail brings critical security capabilities to bear as teams seek to understand malicious events.

Grail enables organizations to find and analyze security events in the context of their broader cloud environments. Moreover, with capabilities such as log forensics—the analysis of log data to identify when a security-related event occurred—organizations can explore historical application data in its full context.

Grail makes it easy to query historical data without data rehydration or indexing, re-indexing, and up-front schema management. This accessibility gives users quick and precise results about when malicious activity occurred, when reconnaissance was first seen in the systems, what was attempted, and if the attackers were successful.

Demo: “Ludo Clinic” uses log forensics to discover and investigate attacks using Grail

Imagine working as a security analyst for a respectable medical institution called Ludo Clinic. As Ludo Clinic started using Dynatrace, the platform’s Runtime Application Protection feature detected a SQL-injection (SQLI) attack. Thanks to the details provided by the code-level vulnerability functionality, the developers knew where in the code the exploited vulnerability was and were able to address and patch it quickly.

The task now is to investigate whether the system experienced any suspicious activity before the attack so we can determine if any other systems are affected. The good news is there are metrics available a few days before your team detected the attack, and you also ingested three months of application and access logs into Grail. Because these logs are ready for querying with no rehydration, the investigation can start immediately. The bad news is we don’t know exactly what to look for. “Find suspicious activity” can mean anything. So, we’ll start by exploring the data using the hints and context information we already detected with Dynatrace.

Hint one: Blocked SQL injection report details

Here’s the report from Dynatrace on the blocked SQL injection details on 10 February from the IP 104.132.226.34. This report shows details of the attack, such as the entry point, the vulnerability that was exploited, the IP address of the attacker, and so on.

Hint two: Failed logins spike

A quick look at the metrics dashboard dating back to 8 February shows a spike in failed logins metrics before your team detected the attack. Indeed, there’s a spike on 8 February.

It would make sense to see if there’s any activity from that IP before we set up monitoring. Has the attacker been doing reconnaissance from that same IP in our systems even before this? If yes, how? Did they try something else during those three months? Were they successful?

Notebooks, DQL, and DPL: Tools of the Grail log forensics trade

Now that we have some clues about where to look for suspicious activity, we’ll dig into the logs using Notebooks, DQL, and DPL.

Dynatrace Notebooks

Dynatrace Notebooks is a collaborative data exploration feature that operates on data stored in Grail for ad-hoc exploratory analytics. Notebooks enable cross-functional teams, such as IT, development, security, and business analysts, to build, evaluate, and share insights for exploratory analytics using code, text, and rich media. The ability to build insights from the same data using the expertise of different roles helps organizations truly understand everything their data has to say.

Dynatrace Query Language (DQL)

Dynatrace Query Language (DQL) is a piped SQL-like query language, similar to Linux commands executed in sequence. You can look at the queries like a series of building blocks applied in an order you happen to need at this moment. Select fields, summarize, and count a value, apply more filters, select additional fields, extract data from a particular field, and so on. DQL is great for exploring and experimenting with data, which makes it a great ally in log forensics and security analytics.

The first query of our investigation uses DQL in Notebooks to fetch logs from Grail, filter the access log, and limit the result to 1000 records for initial exploration.

Dynatrace Pattern Language (DPL)

In our investigation, we’ll also use DPL. DPL stands for Dynatrace Pattern Language, a parsing language that also consists of intuitive building blocks that help to extract meaningful fields from data on read. That means there is no need to manage indexes and rehydrate archived data; simply specify an ad hoc schema using DPL as part of the query.

What’s more, with DPL, the parsed results return typed fields, so you can be sure that a timestamp is a timestamp and an IP address is an IP address, not some random octets separated by a dot like 320.255.255.586. Working with typed data means excellent quality and precision for investigation results because you can run type-specific queries like calendar operations, calculations on numeric data, working with JSON objects, and so on. Working with typed data means excellent quality and precision for investigation results, as you can run type-specific queries like calendar operations, calculations on numeric data, working with JSON objects, and so on.

Log forensics: Querying the access log

Remember: our task is to investigate whether any other systems are affected. The first step is to query whether the IP address where the SQLI attack came from has been used before. Can we see it in the web application access log months prior to the attack?

The query result shows there is no activity from that IP earlier than records on 10 February, the day Dynatrace detected the SQL injection attack. This means that the attack appeared “out of the blue,” and it is likely the attackers were using other IP addresses to do reconnaissance on our systems.

Because we can’t find the attacker by the IP address, let’s look at abnormalities in login behavior because there is a chance they’ll be related to reconnaissance. This means we’ll investigate the spike in failed logins we saw earlier in the metrics graph. Are there any other failed login spikes three months prior to the attack? Where do the failed logins originate from?

We can see that a failed login attempt takes users to a specific URL:

/ludo-clinic/login?authenticationFailure=true

So, let’s see if and how often this URL appears in the logs by adding the following filter to the query.

| filter contains(content, "/ludo-clinic/login?authenticationFailure=true") 
| limit 10000

Indeed, the query gives us 6531 records containing a failed login URL:

Making sense of the access log

For the next stage of our investigation, let’s make more sense of these ~6,300 records and find out how many unique IP addresses were the origin of failed login attempts. The hypothesis is: some of the IP addresses stand out when it comes to the number of login failures. This means we first need to extract the IP address to run this aggregation.

We can utilize the schema-on-read functionality, that is, extract only the fields we need for a specific query. Taking a closer look at the content field of the access log, we can see a traditional HTTP access log: clientIP, timestamp, requestURL, HTTP response code, and so on.

Notice that the timestamp field (the ingest timestamp) is similar for all log records (15/05/2023 14:09:48). This is because Ludo ingested the historical log records in bulk. To analyze the event time, we need to extract the timestamp from the content field as event time. To count IP addresses, we also need the IP address.

Quick ad-hoc parsing to aggregate login failures

To parse out data (timestamp and IP) from the content field, we’ll select the content field and select Extract fields to open the DPL Architect. To retrieve the timestamp and clientIP, we’ll replace the default DPL pattern with the following:

IPADDR:client_ip LD HTTPDATE:event_time

This matches and extracts the timestamp and the IP address from the content field and gives them a name (event_data and client_ip) and leaves the rest of the pattern unmatched, as we don’t need it for the following query. Clicking Insert pattern brings us back to the query view, adding a parse command to the newly created pattern.

Now with the extracted IP address, we can proceed with queries and use the summarize command to count the number of failed logins per unique IP address to see if there were any failed logins originating from a specific IP. Sorting the result set based on the number of failed logins in descending order gives us the largest outliers.

fetch logs 
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, "IPADDR:client_ip LD HTTPDATE:event_time" 
| fields event_time, client_ip, content 
| summarize total=count(), failed=countIf(contains(content, "/ludo-clinic/login?authenticationFailure=true")), by:client_ip 
| sort failed desc

This pays off! The results reveal that a significant portion of logins (181,774) and failed logins (6161) originate from the IP address: 172.31.24.11. This seems interesting and is worth taking a closer look.

Timing of login failures

Next in our log forensics journey, let’s see when these failed logins from that particular IP address occurred to get more information on the potential reconnaissance activity. Did the requests all occur within a short period or regularly across a longer period?

Because we’re interested in the behavior of a specific IP and investigating the reasons behind failed logins, let’s also extract the session ID from the log line. As the session ID is the only field that occurs both in the access and application weblogs, it will be also useful later when we need to join the two for investigating affected users.

We already extracted the IP and included the timestamp (HTTPDATE). We will now extend the pattern and skip the part of the record we don’t need by not naming the three double-quoted strings (DQS). Finally, we’re extracting the last field that contains the session ID.

IPADDR:client_ip LD HTTPDATE:event_time LD DQS LD DQS LD DQS SPACE LD:session_id

When we select Insert pattern, we again get a parse command populated with the DPL pattern we just created.

Focusing on the suspicious IP

Next, let’s select only the fields we’re interested in and then aggregate fields. These actions reveal more about the extended activity that involves the suspicious IP address responsible for many of the failed logins.

| fields time, client_ip, session_id, content

Filtering the attacker IP and sorting the fields based on the timestamp we just parsed out, it appears this IP address was first seen on 24 December 2022. We now know the start of the suspicious activity. For malicious actors, it is quite common to act during the holiday period.

fetch logs  
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, " IPADDR:client_ip LD HTTPDATE:event_time LD DQS LD DQS LD DQS SPACE LD:session_id" 
| fields event_time, client_ip, session_id, content 
| filter contains(content,"172.31.24.11") 
| sort event_time asc 
| limit 300000

Find the suspicious activity pattern across time

To see the activity pattern of this suspicious IP across time, let’s count the number of failed logins in one-hour time intervals.

fetch logs 
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, "IPADDR:ip LD HTTPDATE:time LD DQS LD DQS LD DQS SPACE LD:sessionID EOS" 
| fields time, ip, sessionID, content 
| filter contains(content,"172.31.24.11") 
| summarize failed=countIf(contains(content, "/ludo-clinic/login?authenticationFailure=true")), by:bin(time, 1h)

It appears as though failed logins from this IP appear to follow a very regular pattern: 24 failed attempts every hour. Looks like this activity is automated and most probably refers to a dictionary attack: regular (automated) attempts from the attacker to try out different usernames and passwords, mostly with failed results.

But to escape the clinic’s countermeasures (failed login attempts velocity check), the attacker also conducts a successful login every now and then. If we count all activity from that IP address (not just the failed logins but successful attempts as well), the results are again very symmetrical:

fetch logs 
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, "IPADDR:ip LD HTTPDATE:time LD DQS LD DQS LD DQS SPACE LD:sessionID EOS" 
| fields time, ip, sessionID, content 
| filter toString(ip) == "172.31.24.11" 
| summarize count=count(), by:bin(time, 1h)

Identify targeted users

Next, it would be useful to know which users the attacker has targeted and whether any attempts have been successful. Let’s aggregate the activity from this IP using sessionIDs:

fetch logs 
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, "IPADDR:ip LD HTTPDATE:event_time LD DQS LD DQS LD DQS SPACE LD:sessionID EOS" 
| fields timestamp, event_time, ip, sessionID, content 
| filter contains(content, "172.31.24.11") 
| summarize count=count(), 
            by:{sessionID 
               } 
| sort count desc 
| limit 10000

The result is again peculiar, suggesting automated activity: 59 log lines per session.

Next log forensics dataset: The web application log

Next, let’s see what was happening based on the web application log, using data from what was going on during those sessions that originated from the suspicious IP address we discovered from the access log dataset.

First, to familiarize ourselves with the content of the webapp log, let’s run a basic query to see what the content field of the web application log looks like:

We can see a timestamp, log severity, traces and spans, a session ID, result, and username. There are plenty of interesting fields to play with, so the next step is to parse the content into fields that are ready for querying. We can extract the fields using the DPL Architect. The following DPL pattern extracts the event time session ID, result, and username from the webapp log.

'[' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss.S'):event_time LD ' - ' LD:sessionIdApp ' ' LD:result_text ': ' LD:username (';' | EOS)

Inserting the pattern, this is what the query looks like when parsing out session IDs and usernames from the web application log.

fetch logs, from:-300d   
| filter contains(log.source, "ludo-clinic-webapp.log") 
| parse content, "'[' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss.S'):event_time LD ' - ' LD:sessionIdApp ' ' LD:result_text ': ' LD:username (';' | EOS) " 
| limit 10000 
| fields event_time, sessionIdApp, result_text, username

Filter out records from the authentication provider

To see which users were targeted and how successful the attacker was, we will continue working with only those web application records that contain authentication responses. First, we filter out the records that originate from the authentication provider, then we skip the responses we’re not interested in:

| filter contains(content, "CustomAuthenticationProvider")  
  AND NOT contains(content, "Starting findUsersByUsernameAndPassword") // we want to see only auth response log records 
  AND NOT contains(content, "retrieved matching list")

The full query now looks like this and returns the following results:

fetch logs  
| filter contains(log.source, "ludo-clinic-webapp.log") 
| filter contains(content, "CustomAuthenticationProvider")  
  AND NOT contains(content, "Starting findUsersByUsernameAndPassword") // we want to see only auth response log records 
  AND NOT contains(content, "retrieved matching list") 
| parse content, "'[' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss.S'):event_time LD ' - ' LD:sessionIdApp ' ' LD:result_text ': ' LD:username (';' | EOS) " 
| limit 10000 
| fields event_time, sessionIdApp, result_text, username

Review the user sessions that originate from the attacker

Next, to see which user sessions in the webapp log originated from the attacker’s activity, we use a lookup query to join aggregated sessions from the attacker IP address we discovered in the access log with sessions in the webapp log. In short, we will see what was happening during the suspicious sessions according to the webapp log.

fetch logs  
| filter contains(log.source, "ludo-clinic-webapp.log") 
| fields content 
| filter contains(content, "CustomAuthenticationProvider")  
  AND NOT contains(content, "Starting findUsersByUsernameAndPassword") // we want to see only auth response log records 
  AND NOT contains(content, "retrieved matching list") 

| limit 10000 
| parse content, "'[' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss.S'):event_time LD ' - ' LD:sessionIdApp ' ' LD:result_text ': ' LD:username (';' | EOS) " 
| lookup [fetch logs, from:-300d 
                | filter contains(log.source, "ludo-clinic-access.log") 
                | filter contains(content,"172.31.24.11") 
                | parse content, "IPADDR:ip LD HTTPDATE:time LD DQS LD DQS LD DQS SPACE LD:sessionID EOS" 
                | filter isNotNull(sessionID) 
                | limit 100000 
                | summarize accesscount=count(), by:{sessionID} 
                | fields sessionID], sourceField:sessionIdApp, lookupField:sessionID 

| fieldsRemove content

The result shows the attacker has achieved both successful authentications as well as failed authentications. Finally, we see which usernames the attack targeted the most by looking for the response “No users found requested username.” The system returns this value when it receives a non-existent user or a wrong password. By aggregating the result based on unique usernames, we get a list of the most (unsuccessfully) targeted users.

| filter result_text == "No users found requested username" 
| summarize count(), by:{username} 
| sort `count()`desc

These results are fascinating – we can see five usernames that the attacker continuously entered and received failed authentication results. We can also see SQL commands instead of regular usernames.

Determine successfully targeted users

Next question: Did they achieve anything besides ‘No users found’ when targeting these users? Let’s have a look by excluding the “No users found requested username” response and concentrating on those five users from the last query result, and adding the following line to the query:

|  filter not matchesPhrase (result_text, "No users found requested username") and in (username, "arnie", "herman", "krzysztofs", "bernice", "sherry")

Indeed, we see a lot of “successfully authenticated” responses in the result text field. This confirms the attackers were successfully conducting a dictionary attack: trying out several usernames and passwords to authenticate as real users of Ludo Clinic. When counting the number of successful authentications per these five users, the results are quite similar:

|  filter not matchesPhrase (result_text, "No users found requested username") and in (username, "arnie", "herman", "krzysztofs", "bernice", "sherry") 
| summarize count(), by:{username} 
| sort `count()`desc

Investigation results from log forensics and metrics with Dynatrace

As a result of Dynatrace detecting a SQL vulnerability, anomalies in metrics, and subsequently running forensic queries on three months of logs prior to the attack, we’ve been able to construct the following timeline:

• As Ludo Clinic started using Dynatrace, they were able to observe a spike in metrics capturing failed logins on 8 February
• The system detected and blocked a SQL injection attack on 10 Feb (Fri)
• There was no other activity from that IP in the access logs (the logs reach back three months)
• When aggregating failed login activity, we discovered the following details: the IP address 172.31.24.11 stands out from the rest, counting to 6161 failed logins based on the access log during the past three months
• This IP was first seen in the logs on 24 December 2022 (the earliest timestamp for this set of logs is 20 November 2022)
• The sessions contain identical activities during identical timeframes, which suggests the attacker was using automated tools
• Joining sessions from the access log to the application log reveal almost ten thousand records originating from the suspicious IP 172.31.24.11
• It looks like the attackers were attempting a dictionary attack because it targeted several users at regular intervals, resulting in failed as well as successful authentications.
• Users stafford, ray, orrel, doug and joby were targeted to discover their passwords. The attacker successfully authenticated 35-37 times per user.
• The attackers also entered SQL statements instead of usernames attempting SQL injection attacks.

The DQL and DPL advantage

For such investigations, DQL and DPL make it convenient to quickly investigate and query logs for security analytics use cases that require drawing broad conclusions from the data one minute and then zooming into the activities of a specific session the next. An interesting find inspires the analyst to parse out yet another field and run aggregations on this data. As historical data is always ready for querying, all hypotheses can be quickly verified or dismissed. A curious mind and the right log forensics tools (DQL and DPL) make a great combination for fighting evil.

The post Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail appeared first on Dynatrace news.