Observability data can tell you a lot, but often not in a language that your business understands. Traces fire, business events flow, and dashboards light up, yet P1 incidents still break out. The first question from the war room is typically, “Who’s impacted?” Too often, the honest answer is, “Let me check the database and get back to you.” This gap between raw telemetry and human-readable business context is exactly what lookup tables in Dynatrace are designed to close.

In our introductory blog post on Dynatrace lookup tables, we explained the fundamentals: what lookup tables are, how to ingest them, and initial use cases spanning security to business context enrichment.

In this blog, we’ll go deeper, with real-world examples of lookup tables, practical patterns, and how teams are already putting them to work.

The problem: Dashboards display cryptic identifiers, not insights

Daniel Adams, Observability Engineer at FreedomPay, put it plainly in a recent Dynatrace Tips & Tricks session:

“A lot of our apps are not coded to show customer names or plain text human-readable stuff. It’s mostly IDs being passed back and forth.”

The result is that dashboards are filled with cryptic identifiers that operations teams and business stakeholders can’t mentally decode. And splitting data by customer, country, or store requires trips back to the SQL database for further contextual configuration.

Many organizations that run distributed systems at scale face the same challenge: telemetry is rich, but the business context is missing.

The solution: Use lookup tables to make your reference data available in Grail

Lookup tables let you upload reference data (CSV, JSON, or XML files) directly into Dynatrace Grail®, where it lives alongside your logs, traces, metrics, and business events. Once uploaded, two DQL commands unlock the enrichment from your lookup tables:

• Load fetches the contents of a lookup table, working just like querying any other data type in Grail.
• lookup joins records from your observability data with matching rows in the lookup table, using a defined key field.

In FreedomPay’s case, the lookup table holds customer names, customer codes, store IDs, and countries. The join key is the store ID, the same cryptic identifier that was previously displayed on their dashboards. One DQL lookup command later, those IDs become readable names, and the data can be split by country, grouped by customer, and surfaced in a dashboard that anyone on the team can use.

Lookup tables reduce user cognitive load

The impact of lookup tables is not just cosmetic. Daniel described the before-and-after in P1 situations:

“Previously, everyone was asking: who’s impacted, which customer? Now it’s just very apparent and real-time. People can share screenshots directly from Dynatrace.”

The shift from “let me go check” to “it’s right here” compresses triage time and reduces the cognitive load on everyone in the room.

Beyond incident response, lookup tables also power dashboard variables. FreedomPay uses dashboard variables to populate dropdown filters (for example, clients and countries) directly from the lookup table itself, keeping the UI dynamic without manual maintenance. The load command behaves like any other fetch command but operates on reference data rather than telemetry, making it a natural fit for driving interactive dashboards.

And the enrichment isn’t limited to business events. The load and lookup commands work across all data types in Grail.

In practice: cost allocation with lookup tables

Cost allocation is a good example of how lookup tables bridge the gap between operational data and business reality. Many organizations already maintain mappings that associate users, teams, or services with products or cost centers, but that context rarely lives inside observability data. Lookup tables allow you to introduce such context.

Consider DPS (Dynatrace Platform Subscription) consumption driven by user activity such as running queries, triggering automation, or executing functions. On its own, such usage is hard to attribute to a specific part of the organization. By associating users with products or cost centers in a lookup table, you can attribute usage to organizational ownership rather than just technical activity, align cost views with existing financial or product structures, and keep that attribution logic centralized and reusable rather than embedding it in individual analyses.

What makes this pattern scale is that business mappings change over time, while usage data continues to flow. Lookup tables allow you to evolve the mapping without rewriting your analyses. The same mechanism used for cost allocation works equally well for mapping users to teams, services to portfolios, or environments to internal programs. Cost allocation simply highlights how powerful this becomes when operational data is interpreted through a business lens.

Importantly, this is just one way to approach cost allocation, not the only way. Some teams rely on host-based attribution, others on pipeline metadata or external financial systems. Lookup tables complement these models by making it easy to incorporate existing reference data wherever it adds clarity.

Keeping lookup data fresh

For lookup tables to deliver lasting value, their data must remain current. FreedomPay currently uses Postman to push updated CSVs via the upload API, but they’re building automation to pull this data from their SQL backend daily and sync changes using the overwrite: true parameter. The lookup table path stays the same; the data underneath gets refreshed. Downstream dashboards and queries are updated automatically.

More broadly, there are several approaches to keeping lookup data up to date:

• Dynatrace Workflows automate extraction and upload from SQL databases or APIs on a schedule.
• Periodic refresh updates your lookup data programmatically as part of CI/CD or data sync processes.
• Manual refresh uses Postman, cURL, or other API-based uploads with the overwrite flag when data changes infrequently.

Dynatrace lookup tables are now generally available

With the release of Dynatrace SaaS version 1.334, lookup tables are generally available for all customers running Dynatrace SaaS with an active Dynatrace Platform Subscription (DPS). GA brings production-readiness, improved query performance for the lookup command, and full integration with Grail’s scalability and access controls.

To get started

1. Identify a data enrichment use case: cost allocation, customer context, error code mapping, or security allow lists.
2. Prepare your reference data as a CSV, JSON, or JSONL (JSON Lines) file.
3. Upload the file to Grail using the Dynatrace API, Workflows, or a custom app.
4. Use the load and lookup commands in your DQL queries, notebooks, and dashboards.

For detailed instructions, visit our lookup data documentation. And to see how FreedomPay built its implementation from end to end, watch the full demo on YouTube.

The post How lookup tables turn observability data into business insight appeared first on Dynatrace news.

Transform how you derive actionable insights from data

Have you ever struggled to make sense of a massive data set or turn raw data into actionable business insights? Investigations handles all this for you—and more—at the scale modern organizations demand.

Whether you’re troubleshooting API call throttling, debugging AWS integration issues, or accelerating root cause analysis, the Investigations app provides a flexible toolkit for exploring your data, allowing you to:

• Analyze large DQL results in their original form at a detailed level
• Maintain your entire investigation flow and historical queries in context via the query tree
• Perform complex investigations on data stored in Dynatrace Grail®
• Build DQL queries quickly and efficiently based on your findings
• Save and reuse evidence to refine queries and uncover answers
• Pivot your queries based on the metainformation attached to log records
• Analyze the observability metrics connected to your log sources

With Investigations, you can easily navigate through investigation history to review queries and results—streamlining incident response, accelerating root cause analysis, and delivering deeper contextual insights from your data.

Get started with practical use cases

As strong advocates of “learning by doing,” we understand you may be curious about how to start using the Investigations app more broadly. You don’t need to start from scratch—there are plenty of documented scenarios you can customize for your own purposes. To help you get started, explore these informative tutorials:

• Investigate API call throttling using DQL and Investigations
• Debug AWS Integration issues
• Creating faster filters for DQL queries to speed up any investigation

If threat hunting is more your style, you can start with these tutorial-style investigation challenges:

• Christmas hands-on: Who stole Santa’s secret file
• Valentine’s Day hands-on: Heartbeat failed

There are, of course, many more use cases to explore—Investigations can be used with any data in Grail: logs, events, metrics, traces, and even performance data.

Drive insights across all your data

Since its launch, the recently renamed Investigations app has been key to unlocking insights in Grail, allowing teams to analyze metrics alongside logs and connect traces during incident response—this is a game-changer for both security and observability. Over time, the capabilities of the Investigations app were extended well beyond security, supporting a wide range of analytical scenarios.

Despite the name change, all the app’s existing features, including application IDs, which are used in intents and in the browser address bar, remain unchanged, so no updates to integrations or bookmarks are needed. And the roadmap will continue to prioritize incident response and accelerations of investigations.

For ease of use, Investigations will remain listed under “Security” on the Dynatrace platform and in Dynatrace Hub.

For complete details, please see Investigations documentation.

Ready to get started?

Investigations is a built-in app available in all SaaS environments beginning with Dynatrace SaaS version 1.330. Best of all, this app requires no additional subscriptions or privileges—so you can start using it immediately without waiting for access permissions and unlock deeper insights for your business.

The post From incident response to everyday analytics: Introducing Dynatrace Investigations appeared first on Dynatrace news.

Managing sensitive data in log files is getting even easier with Dynatrace. Our new Sensitive Data Center unifies privacy request workflow, data cleanup, and the new Sensitive Data Scanner to help you streamline working with sensitive data in Dynatrace. It allows teams to respond to data subject rights with confidence, remove records safely when required, and proactively discover and govern sensitive data ingested into Grail®. Sensitive Data Scanner will be available in a limited preview release by the end of 2025.

Handle sensitive data throughout your growing data ecosystem

Organizations face increasing pressure to demonstrate responsible practices for managing sensitive data while maintaining efficient operations and minimizing downtime. In addition to meeting end users’ data subject rights requests, such as the export or deletion of personal data, organizations must take proactive steps to prevent unnecessary exposure and storage of sensitive information. Achieving these objectives is challenging, especially with fragmented tools, siloed teams, and manual processes. As telemetry volumes increase, these inefficiencies lead to slower response times, higher operational overhead, and increased compliance risks.

Streamline privacy operations in Dynatrace with the Sensitive Data Center

The Sensitive Data Center brings privacy operations and sensitive data management together in a single app on the Dynatrace platform and complements existing privacy controls with an additional layer of control. Aligning scanning, cleanup, and data subject rights workflows with where your data resides helps teams reduce manual work and improve accuracy, all in a transparent process where every scan, cleanup, and request is logged and auditable, supporting your regulatory obligations with clarity and control.

Continuously scan for unintentionally ingested sensitive data in Logs on Grail

Imagine a service administrator who suspects that sensitive data might have been unintentionally ingested in their observability data. They need a quick way to confirm whether it happened and, if so, where the sensitive data resides and what type of sensitive data is affected—ideally without having to build custom scripts or pull engineers off priority work. The Sensitive Data Scanner is a new module in the Sensitive Data Center that helps you discover sensitive data at the time of ingestion, allowing you to govern it more effectively in three steps:

1. Configure the scanner
2. Review the scan results
3. Mitigate any potential findings

Configure scans in the Sensitive Data Scanner

The setup is straightforward. You can choose to monitor specific buckets or the entire environment. Select the sensitive data type or types from built-in rules such as email, credit card, or IP address. You can set up several fine-grained scans with different scopes to accommodate different scan areas. A scan runs at a defined cadence every 6, 12, or 24 hours, depending on your compliance needs, and alerts you when data matching the selected criteria is found.

Review scan results

A dashboard provides a clear overview of scan statuses and highlights when sensitive data is found. From there, you can drill down into a specific scan to review detailed findings and understand exactly what was detected.

You can review the results and examine the data flow from ingestion to the storage location.

Mitigate potential findings

With these results, you can immediately take action. You can configure or adjust masking rules to prevent similar data from being ingested in the future, change access to stored data, update retention periods, or utilize the cleanup functionality to delete the data as needed.

Sensitive Data Scanner preview

The Sensitive Data Scanner will be available in a preview release by the end of 2025. As we gather feedback, we will continue to refine the experience and expand coverage, allowing teams to move confidently from identification to action within the same app.

Act decisively with precise, auditable cleanup

Data cleanup is available directly within the Sensitive Data Center, allowing you to take action when your organization’s regulatory obligations or policies require the removal of data.

The “Cleanup data” workflow in Sensitive Data Center allows you to easily locate, review, and delete an entire time frame of data, as well as any selected individual records that contain sensitive data defined in your DQL search query. To improve accuracy and minimize the risk of accidentally deleting data, you can also select a reviewer who will review and approve deletion requests before the data is deleted. Learn more about deleting data in Grail.

Efficiently locate, export, and delete end users’ personal data

Let’s walk through another common scenario. The services administrator, ensuring Dynatrace is operating smoothly, receives an urgent request from the privacy legal team: “Please locate, compile, and delete all personal data associated with this email address in Dynatrace as part of our end-user’s right to be forgotten.” Privacy requests in the Sensitive Data Center offer an end-to-end experience for managing data subject rights requests within the Dynatrace platform, allowing for quick, compliant, and efficient handling of sensitive data.

Dynatrace empowers you with a ready-made solution for submitting, tracking, and verifying the status of requests. With a user interface designed for compliance needs, you can efficiently manage data export and deletion requests. A dashboard summarizes key details, including the request reference, status, and due date alignment.

Together, scanning and cleanup ensure that only the sensitive data you intend to process is stored in Logs on Grail, while privacy requests provide the workflows and approvals necessary to comply with user privacy rights for lawfully processed personal data.

This blog may contain forward-looking statements about our product plans, upcoming features, and anticipated improvements.  These statements are for informational purposes only and are not promises or guarantees.  The development, release, and timing of any features or functionality described remain at the sole discretion of Dynatrace LLC and may be modified, delayed, or canceled without notice.  We encourage readers to make decisions based on the product’s current capabilities and features.

The post Unified privacy and sensitive data management for logs with the Sensitive Data Center appeared first on Dynatrace news.

Empowering real-time insights through petabyte-scale data processing of all your data

Modern cloud native systems generate massive volumes of telemetry data, spanning logs, metrics, events, and traces. Extracting real-time, actionable insights and enabling meaningful alerting and root cause analysis requires more than just centralized data storage; it requires a scalable data pipeline. Without such a pipeline, even the most advanced analytics tools are constrained by data flow and preparation bottlenecks. To address these challenges, we launched OpenPipeline, our unified data ingest solution for the Dynatrace® platform. OpenPipeline is a customizable, scalable data pipeline for ingesting, transforming, and harmonizing data for reliable analysis and correlation.

With its latest enhancements, OpenPipeline supports real-time data processing even at petabyte scale, by reliably handling high volumes of logs, traces, and other telemetry. This level of scalable pipeline processing is essential for delivering timely insights, enabling Dynatrace to:

• Detect anomalies across distributed systems in real time
• Perform proactive security analytics and real-time vulnerability detection
• Offer real-time insights into your business processes

Scale is not only about the sheer amount of data signals. It’s also represented in the size of individual signals that can be handled. An increasing number of use cases, such as parsing JSON/XML request bodies, capturing detailed audit logs, exploring transactional data dumps, or analyzing input vectors from ML model logs, require the ability to efficiently process large log files. With the release of Dynatrace version 1.311, OpenPipeline now supports log records up to 10 MB in size, empowering you to use Dynatrace for high-volume use cases. For more information, have a look at this Dynatrace support for large log records blog post.

Break silos with unified ingestion across all data types

Alongside scale, OpenPipeline provides unified ingestion across all data types, removing silos, simplifying integration, and ensuring that all data, regardless of format or source, is enriched, contextualized, and processed in the same tool, ready for advanced analytics.

Recent enhancements to data type processing include:

• Spans: You can now configure how spans are processed, including dropping specific fields or entire records, and assign a security context for fine-grained, record-level access control. Spans can also extract metrics and route them into defined target buckets. Full ingest functionality for spans is coming soon.
• RUM data: Ingesting Real User Monitoring (RUM) data, including user events, sessions, and associated metrics, is currently in preview and will soon be made generally available for all Dynatrace SaaS customers.
• Events: OpenPipeline now supports custom processing rules for a broad range of event types, including security events, software development lifecycle (SDLC) events, business events, and Dynatrace internal system-level events.

Simplified pipeline management from setup to optimization

Historically, setting up pipelines for data ingestion involved time-consuming configuration of parsing, field mapping, and transformation logic for each data source. OpenPipeline now streamlines this process with ready-made processor bundles for widely used technologies and formats, accelerating data onboarding and standardizing data handling at scale for ingest sources such as:

• Hyperscaler support, including AWS and Azure
• Web servers like Apache, IIS, JBoss, HAProxy, or Nginx
• Programming languages like .NET, PHP, Java, Python, or NodeJS
• Databases and app frameworks like Elastic or Cassandra

With just a few clicks, you can apply a processor to a source, ensuring all fields are parsed correctly, attributes are renamed, and data structures are accurately standardized. Each bundle includes example records that can be tested interactively, with further customization available to meet specific requirements. This allows your teams to onboard new telemetry streams in minutes instead of hours. We’ll continue to expand our support to cover more technologies in the future. You can also create your own processors to handle any custom format.

Reveal hidden value with data transformation

Upon ingestion, modern observability, security, and business data can be structured in varying, often complex forms. OpenPipeline offers advanced transformation capabilities that allow for easy extraction of relevant data beyond the creation of simple events and metrics:

• Extract values from deeply nested JSON fields. For example, extract the threat identifier from a JSON-formatted security event and convert it into a security metric.
• Create unified metrics from different data types. For example, consolidate error codes from both logs and spans into a single metric for cross-service comparison.

Stay informed: Real-time visibility and alerting for your pipelines

To optimize pipelines, teams need visibility into their performance. OpenPipeline now exposes detailed metrics at key processing stages: ingest, routing, and output, as well as not-stored-records. With these metrics, you can instantly verify any pipeline configuration and detect anomalies early.

These metrics power:

• Real-time microcharts within the OpenPipeline user interface, showing data trends and ratios over the last 30 minutes.
• A ready-made dashboard used to explore daily or weekly summaries, historic volume trends, and detailed routing stats by pipeline or source.
• Smart alerting. Traffic fluctuations on ingest are common, so you can leverage AI-powered dynamic baselining to raise custom alerts and detect anomalies before an issue occurs.
• Automated operations that trigger notifications or perform autonomous remediation steps once an ingest anomaly or traffic volume deviation is detected.

Easy transition to OpenPipeline for existing customers

For Dynatrace SaaS customers using classic pipelines, we’ve simplified the transition to OpenPipeline. Start utilizing OpenPipeline for new data sources while keeping your existing log processing configurations fully operational and uninterrupted. This side-by-side, risk-free approach supports gradual adoption and allows you to modernize your data processing without disrupting ongoing workflows.

Get started today

OpenPipeline is now available to all customers running the latest version of Dynatrace SaaS. Already today, you can:

• Benefit from processing a broad range of data types, including logs, traces, events, and RUM data (currently in preview).
• Take advantage of processing the full payload of large log records, parse embedded XML or JSON structures, parse events or metrics from complex log records, or use them for deep search analytics and forensic use cases—all without splitting individual log lines into separate events. Leverage the built-in processor bundles to get started with popular formats.
• Explore the ready-made OpenPipeline dashboard on the Dynatrace Playground.
• Use real-time pipeline metrics to tune and optimize your configurations.

The post Latest OpenPipeline upgrades simplify high-volume, real-time data processing appeared first on Dynatrace news.

When you’re working with complex, multi-layered data, pinpointing the root cause of an issue or identifying anomalies can feel like trying to find a flickering light in a city skyline. Effective troubleshooting demands a deep understanding of data structures, filters, and query logic. But what happens when you’re staring at a mountain of logs or metrics and don’t even know what to look for? For instance, imagine you’re investigating a sudden spike in error rates across your application. Without knowing which service triggered the cascade or how to filter the noise, you’re left guessing where to begin.

With the introduction of deep search functionality in Grail, we’ve introduced a new way to cut through the noise of complex data. You can now explore data effortlessly without needing to understand every single layer of complexity in your data structure.

Investigating data is a time-consuming process

Difficult data investigations can be taxing. Examining data takes too long, and probing for issues is tricky for those who don’t know the complexities of the data structure. In the query example below, taken before the introduction of DQL search, we attempt to filter within spans where any event sent from an Oracle database contains the phrase “error”.

Search within complex structures

Grail’s unique data warping technology allows for index-free, schema-on-read, high-performance queries, enabling you to flexibly analyze diverse datasets. While Grail has long been capable of dynamically interpreting complex data structures at query time, with the addition of the DQL Search command, you can now deep dive into your metrics, logs, traces, and other data stored in Grail in a way more agile and intuitive way.

Simplified string-based filtering across multiple fields

You can now search across vast and complex data sets without needing to understand the underlying structure. By leveraging string-based search, the system offers intuitive exploration of deeply nested data—no manual filters or query syntax are required. This approach eliminates the traditional barriers of schema knowledge and filter construction, making data investigation accessible to everyone. (See Figure 2 below.)

Surface the needle in the haystack faster

You can search across all fields, not just indexed or predefined ones, making it easy to surface any data without needing to pre-tag or preprocess it. DQL search is optimized to be highly performant, even with terabytes of data.

Search in nested structures

Finding relevant data within nested elements or arrays is cumbersome, requiring a lot of effort to ensure you don’t exclude necessary parts of the data structure. With the new DQL Search command, you can search across complex data structures, including nested elements and arrays, to find exactly what you need.

Imagine a DevOps engineer troubleshooting a failed deployment in a development environment. Suspecting a misconfiguration or runtime error, the engineer needs to find traces containing the term “exception”—buried deep within the span.events array of nested records. The system scans across all nested fields and surfaces relevant spans instantly—saving time, reducing friction, and accelerating root cause analysis.

Now, consider another example with Dynatrace Security Investigator. You’re investigating a security incident in your Amazon cloud environment. You need to track a specific event from your AWS CloudFront logs, and you have the CloudTrail event ID value. You have no idea where to find the event ID. It could be buried in the content, tucked away in header values, or stored in an obscure field. The DQL Search command empowers you to search across all fields instantly—no filters, no field-mapping, just fast, accurate results when you need them the most.

As simple as using a search bar, inside DQL

Experience the flexibility and power of exploratory analytics in Dynatrace without worrying about indexes or schemas. The DQL Search command allows for deep, detailed investigations while maintaining simplicity, making it accessible to a wider range of users.

The post Deep Search: Transform your data exploration and investigation with Dynatrace Grail appeared first on Dynatrace news.

Dynatrace continues to deliver on its commitment to keeping your data secure in the cloud. Enhancing data separation by partitioning each customer’s data on the storage level and encrypting it with a unique encryption key adds an additional layer of protection against unauthorized data access. Separate data storage fulfills the security compliance requirements of many Dynatrace customers operating in highly regulated sectors, making it much easier for them to use Dynatrace SaaS and accelerate their digital business transformation.

Protect data in multi-tenant architectures

To bring you the most value by unifying observability and security in one analytics and automation platform powered by AI, Dynatrace SaaS leverages a multitenancy architecture, enabling efficient and scalable data ingestion, querying, and processing on shared infrastructure. Such infrastructures must implement additional controls to securely separate each customer’s data. Dynatrace ensures that each customer’s tenant data is separated from each other customer’s data throughout its lifecycle using multiple layers of data security controls, such as:

• Authentication
• Separated HTTP requests per tenant
• Logical data separation through data package tracking
• Granular permission control with access policies
• Individual domain per environment

In addition to the data security controls, a rigorous secure development lifecycle (SDL) ensures that data security controls for data separation work as designed and that any potential issues are detected and prevented during development. The Dynatrace SDL includes penetration testing, red teaming, continuous threat modeling and risk assessments, a public bug bounty program, and vulnerability scans.

Dedicated storage and unique encryption keys for each tenant

Dynatrace introduces a fundamental improvement in how each customer’s tenant data is kept separate. By providing dedicated storage and a unique encryption key for each tenant, each Dynatrace tenant’s data is kept separate at rest on the storage level, significantly reducing the risk of unauthorized access to the data.

A unique encryption key is applied to each tenant’s storage and automatically rotated every 365 days. This guarantees that only one tenant is affected in the unlikely case of a compromised encryption key. For further security and convenience, you can easily revoke a key without impacting your data.

This level of data separation and encryption fulfills the security compliance requirements of many Dynatrace customers operating in highly regulated sectors.

Tenant data separation in Dynatrace

Currently, the enhanced tenant data separation and encryption feature is activated by default for all Dynatrace SaaS customers on AWS and Azure using the latest version of Dynatrace SaaS. There is no need to make any changes.

On AWS, each Dynatrace tenant now has a dedicated S3 bucket assigned to it. All new Dynatrace platform data at rest is stored in such a dedicated S3 bucket. S3 buckets are configured to be encrypted with a bucket key stored in the AWS key management system (KMS). Each S3 bucket is assigned its own unique bucket key. All bucket keys are managed by Dynatrace and are configured to rotate automatically after 365 days.

On Azure, each Dynatrace tenant now has a dedicated Azure storage account assigned to it. All new Dynatrace platform data at rest is stored in such a dedicated storage account. Storage accounts are configured to be encrypted with a key stored in the Azure Key Vault. Each storage account is assigned its own unique encryption key. All encryption keys are managed by Dynatrace and are configured to rotate automatically after 365 days. 

What’s next

Next, the enhanced data separation and encryption features are planned for release to all customers on Azure and then to all customers on Google Cloud.

With improved data separation and newly introduced encryption, Dynatrace helps you fulfill data security requirements for highly regulated sectors.

Contact your Dynatrace account manager to learn how Dynatrace meets your organization’s security, privacy, and compliance requirements and to accelerate your journey to Dynatrace SaaS.

The post Dynatrace elevates data security with separated storage and unique encryption keys for each tenant appeared first on Dynatrace news.

An at-a-glance view of your system’s health and performance

Dynatrace guides you in quickly getting the most valuable service level objectives (SLOs) set up in just a few clicks. To provide maximum freedom in selecting the service-level indicators that matter most to your business, Dynatrace combines SLOs with the power of Dynatrace Grail™ data lakehouse, the central data platform with heterogeneous and contextually linked data. Service-level objectives now utilize highly flexible Dynatrace Query Language (DQL) for all Grail data sources. This lets you build your SLOs around the indicators that matter to you and your customers—critical metrics related to availability, failure rates, request response times, or select logs and business events.

Heterogeneous services require heterogeneous indicators

Metrics, logs, and traces are the core ingredients for making your environment observable. Depending on the environment, the different information types provide indicators that reveal potential problems for your customers.

Having insights into the context and dependencies of such heterogeneous data is essential for seeing the forest, not just the trees. This is where Grail, the Dynatrace central data platform, excels. Grail provides access to contextually linked information with just a few query lines.

It doesn’t matter if you need typically used failure-rate or response-time metrics to ensure your system’s availability and performance or if you need to rely on abnormal log drops to gain insights into raising problems—SLOs leveraged with Grail provide all the information you need.

Are you defining SLOs based on the four golden signals of a critical service? No problem—out-of-the-box templates simplify the setup of time series data based on service requests, and templates can quickly be adapted for more fine-grain needs, for example, to consider only single endpoints.

Do you need to see if selected logs or loglines are missing over time? It’s easy—just use log data, extract the desired information, and create an SLO based on the log format.

Are you experiencing an increase or degradation in certain events that indicate a rising problem? It doesn’t matter if the events are ingested business-related events like deal closures or simple login numbers automatically captured by Dynatrace OneAgent®. Fetch your events, target critical components in your environment, and get notified proactively if your required objectives are at risk.

If you can query something using DQL, you can use it as an indicator for your SLO. The only prerequisite is that the result of the SLI needs to be a time series named sli. The make time series command handles how time series are created from other data types.

Grail, the Dynatrace central data platform, provides the needed information and adds context. With the recently released new SLO capability, you can use Grail and all its information to specify selected SLOs to ensure your critical components meet your expectations now and in the future.

Having service-level objectives for leading and lagging indicators provides a more holistic picture of the status of emerging issues. Setting the SLO on the data types that matter provides a more tailored view of the performance stats and helps limit the number of SLOs to critical items and the most impactful indicators.

SLOs on events, metrics, and logs: How to bring SLOs to the next level

Leveraging Grail, SLOs offer an immersive opportunity to focus your SLOs on user expectations, business goals, and core user journeys. When getting started, this can seem overwhelming, and it’s easy to get lost in the sea of opportunities.

With the latest release, Dynatrace introduces convenient ways to give you a head start with SLOs on all data types. If you have already worked with classic Dynatrace SLOs, most steps will be familiar and easy to follow.

The latest Dynatrace SLO implementation relies on the existing SLO creation wizard templates and provides guided workflows for setting up SLOs with the most commonly used SLIs: service availability, service performance, and CPU utilization.

In the coming weeks and months, we will add to the current collection of templates for synthetic monitoring, digital experience management measures, Kubernetes resource optimization, and infrastructure monitoring. However, all of these can be created today using DQL queries. Simply extract and manipulate your desired time series from Grail within a notebook and copy the query into the SLO definition to get your customized SLO.

Service-level objectives come with a dedicated management overview page, where existing SLOs can be reviewed and modified as needed.

The SLO wizard can be called directly from the management overview or the dashboard using a dedicated shortcut.

An added tagging option simplifies how you interact with your SLOs. Adding ownership information or relevant application boundaries as additional metadata to your SLOs makes it easier for all stakeholders to react or reach out to the appropriate teams in case performance degradations or dependencies are identified.

Dedicated management makes it easy to maintain and run your SLOs, while highly customizable dashboard tiles allow you to integrate SLOs with your health and performance overview stats.

SLOs provide immediate information about critical services’ long-term performance and experiences. Hence, having a dedicated dashboard tile visualizing the key parameters of each SLO simplifies the process of evaluating them.

Automate your SLOs

To manage and evaluate your SLOs at scale, Dynatrace allows you to create, manage, and evaluate your SLOs either via the Dynatrace web UI or API. To implement SLOs in your software delivery cycle and consistently add observability measures from the beginning, Dynatrace configuration as code (Monaco and Dynatrace Terraform) will soon support the new API.

The next step will be improved error-budget burn rate alerting, which Dynatrace Davis® AI root cause analysis can recognize, detecting potential threats to your SLOs before they affect your customers. Additionally, the Dynatrace Automation Engine will leverage SLO alerts to create event-triggered workflows to inform relevant stakeholders, provide reports, or automatically kick off remediation activities.

What’s next

Dynatrace SLOs leverage the power of Grail, providing dedicated SLO management, a user-friendly creation wizard with templates, improved dashboarding options, and an API to interact with the new generation of SLOs. While the SLO management web UI and API are already available, the dashboard tile will be released within the next weeks.

Subsequent improvements will include improved error-budget burn rate alerting and a notification option to proactively inform you when an SLO is at risk. Furthermore, SLOs will find their place within all areas in Dynatrace, making it even easier to create SLOs on demand. At the same time, dedicated configuration-as-code support in Monaco and Terraform will provide a scalable, automated solution.

Get started

Measure what matters most to your business and end-users. Set up your first SLOs and gain insights into relevant and important processes.

We’re eager to hear your experience, feedback, and ideas for improvement. A dedicated feedback channel is set up in the Dynatrace Community. We look forward to hearing from you. Feedback channel for the new service-level objectives app – Dynatrace Community

The new SLOs are available within all Dynatrace SaaS environments and ready for you to try out, beginning with Dynatrace SaaS version 1.303. The new dashboard tiles will be available with version 1.304.

Start setting up your SLOs on indicators that matter to you and your customers

We created a few examples on our Dynatrace Playground tenant so that you can experience the potential and flexibility of SLOs leveraging DQL. You’ll find SLO examples for different data types, such as metrics, events, or logs. You’ll also find examples for entity types like services, synthetic monitors, Kubernetes clusters, and namespaces, and you’ll learn how to use DQL to filter for a selected group of entities or remove specific time periods from SLO-status calculations. Additionally, we set up a sample dashboard that presents an overview of selected critical services and their combined health and performance status aggregated over time.

The post Reliability indicators that matter to your business: SLOs for all data types appeared first on Dynatrace news.

Unrealized optimization potential of business processes due to monitoring gaps

Imagine a retail company facing gaps in its business process monitoring due to disparate data sources. Due to separated systems that handle different parts of the process, the view of the process is fragmented. On top of that, the data sources are inconsistent. While some data comes from modern systems with APIs, other data stems from older systems that generate log files, and some data originates from external vendors. This inconsistency complicates uniform gathering and data analysis, resulting in incomplete or inaccurate insights. This scenario is shared among large organizations that rely on multiple internal and external systems for data collection.

Get business process observability across data silos with Dynatrace OpenPipeline

Traditional observability platforms often focus on technical metrics and logs, which are essential for troubleshooting, but they don’t take business value into consideration. Dynatrace OpenPipeline goes beyond this by offering the possibility of extracting business-relevant data from logs and using them to monitor end-to-end business observability.

In our retail company example, older systems are involved in shipping the order. They only produce logs that have not yet been monitored from a business perspective, although they contain valuable business information.

By leveraging Dynatrace OpenPipeline, the retail company can integrate data from all sources across the whole process, including Dynatrace OneAgent^®, logs, and external business tools. This approach ensures that the company can easily track the entire journey from online order to its successful delivery.

Benefits of capturing business events

Logs often contain valuable insights into your business; however, this information can be difficult to process, particularly as you probably only need data from some specific log lines.

Dynatrace OpenPipeline extracts this business information from logs and stores it as a separate data type, so-called business events. This has several advantages:

• Separation of concerns: Logs are often used for technical troubleshooting. Extracting business events allows for a clear separation between technical and business-relevant data.
• Access control: Different teams can access different types of data. For example, support teams might access logs for troubleshooting, while business teams access business events for analytics.
• Ease of access: Having business events in a uniform format simplifies querying and visualization, making it easier to analyze and derive insights.

How to find valuable business information in logs

In our example of a retail company, we need to extract business information from shipping logs to know if our order has already been shipped. See a typical log file with shipping details below.

From this log file, we can identify and capture the following business information:

1. Correlation ID – The Correlation ID is part of the log payload and identifies this event in the context of a business process.

2. Message – The message is also part of the log payload and contains information on which part of the business process it represents.

3. Context – The context contains information on which IT system handled this part of the business process.

4. Status—The status information tells us if something was successful or if we hit an issue.

Use OpenPipeline to identify relevant log events

To turn this information into a business event for analytics, we utilize the capabilities of Dynatrace OpenPipeline to identify relevant log events, parse the data, and create a business event out of it. The approach is the following:

1. Go to the OpenPipeline app (available by default in your Dynatrace tenant)

1. Create a route that specifies which logs should be processed. For example, you can only process log events coming from a specific ingest source or containing a specific phrase in its log message.
2. Define a pipeline that will process these logs.
3. Configure the data extraction rules within the pipeline. This involves:

1. • Extracting correlation IDs: Identify and extract correlation IDs from the log content. Rename these IDs to make them uniform with other events (for example, rename “order ID”).
   • Extracting the message: The relevant message will be extracted from the log content and used as part of the business event.
   • Parse out contextual information such as the context (which is the related host) or status information in our case.
   • Defining event types: Specify the event type using the extracted message and provider information. For example, you might name the provider retail.logs for consistency.

The extracted information is re-ingested into the pipeline as a new business event. The event is then stored in Grail™ datalake house and can be used for further analysis or process visualization.

Do you want to dig deeper into extracting the data? Watch the dedicated Dynatrace Lab episode with Andreas Grabner and Alistair Emslie for a step-by-step guide on how this is done:

Turn your business data into tangible value

The extracted business data enables you to optimize your processes and gain valuable new insights. You can easily create a dashboard like the one below: it provides the example retail company with real-time data and KPIs, such as the success rate of shipping orders. Unlike traditional dashboards focusing on specific applications or technical aspects, this dashboard tracks the performance of the complete business journey with business KPIs and metrics, utilizing the information extracted from the log files.

Of course, you can also leverage the full power of the Dynatrace platform, like AI-powered monitoring of your business KPIs through Davis^® AI. You can also visualize the end-to-end process flow in our Business Flow app, which enables you to identify delays, errors, and exceptions, providing insights into IT and business-related issues.

Learn more about the capabilities of Dynatrace OpenPipeline

If you’re struggling with gaps in your business process monitoring caused by disparate data sources and outdated systems, consider how Dynatrace can transform your approach.

The post Leverage logs for an end-to-end view of your business processes via Dynatrace OpenPipeline appeared first on Dynatrace news.

Data deletion might seem simple, but it’s critical in data management and privacy laws. It’s not just about hitting the Delete button; it’s about securely meeting compliance requirements while ensuring data integrity. Dynatrace Grail™ is a data lakehouse optimized for high performance, automated data collection and processing, and queries of petabytes of data in real time. A data lakehouse is schema-on-read and indexless, making deletion operations complex. Adding to the technical challenges, effective deletion involves a combination of policies, procedures, and technologies to ensure data is appropriately managed throughout its lifecycle.

Strategically handle end-to-end data deletion

Two key elements form the backbone of an effective deletion strategy in Dynatrace SaaS data management: retention-based and on-demand deletion. Retention-based deletion is governed by a policy outlining the duration for which data is stored in the database before it’s deleted automatically. The retention period can vary based on the nature of the data, regulatory requirements, and the business’s specific needs. With Dynatrace Grail™, you can customize data retention periods with day-level granularity for up to 10 years.

On-demand deletions are initiated in response to specific events or requests. For instance, if data is mistakenly ingested into the database, it may need to be deleted to prevent inaccuracies or sensitive data from being stored. Another consideration is compliance with end-user privacy rights to delete personal data processed about them in line with data protection laws like GDPR and CCPA.

Hard deletion on the record level is the gold standard

On-demand data deletion in a Dynatrace SaaS environment relies on two core characteristics: granularity and hard deletion. Granularity, specifically at the record level, allows for precise control over what data is deleted. This means that individual records can be targeted for deletion without affecting the rest of the dataset. Hard deletion refers to the permanent and secure erasure of data. These characteristics are crucial in ensuring data deletion is effective, secure, and compliant with data privacy regulations.

Many other SaaS vendors mistakenly assume that a soft delete—merely hiding or marking data as deleted while retaining it elsewhere—is sufficient. However, this approach leaves room for accidental exposure, unauthorized access, or data breaches, jeopardizing compliance and customer trust. Industry standards refer to hard deletion by overwriting data as the appropriate data sanitization method.^[1]

In addition, as with some other solutions, data deletion is often limited to a timeframe or requires removing all data in an index. This approach lacks flexibility and results in losing large volumes of important business data.

_{[1] NIST Special Publication 800-88 Revision 1, Guidelines for Media Sanitization}

Keep valuable data in Grail with targeted deletion

With record-level hard deletion, Dynatrace gives you control and transparency over your data. It enables you to effectively identify and promptly delete targeted data in a thorough and irreversible manner. So, while the relevant data is completely erased, all other valuable data remains intact. This ensures that your business operations continue smoothly, and you can focus on extracting value from Dynatrace.

In addition to the possibility of deleting data in buckets in Grail, record deletion in Grail now enables you to identify and select the records to be removed by leveraging DQL. You can use the Grail Storage Record Deletion API to trigger a deletion request.

Here are some tips to consider when deleting records in Grail:

• Leverage Notebooks: Use Notebooks to create a DQL query to filter all the records you intend to delete. With Notebooks, you can easily query data from Grail and visualize the results. This step also helps you determine the timeframe within which the relevant records are stored. To delete the records, use the Storage Record Deletion API. You might need to invoke the API multiple times if your data spans multiple days. Be aware that only one deletion process can be executed at a time.
• Verify your permissions: Before proceeding with any data deletion tasks, ensure that you have the necessary permissions, specifically storage:records:delete.
• Pause data ingestion: To avoid mistakenly deleting any recent data, stop ingesting new data that contains information you plan to delete. Only data older than 4 hours can be deleted.
• Keep track of deletion: Use the status command to gain insights into the status of the current deletion process. If necessary, use the cancel command to cancel a running process.

An audit trail is maintained to provide a clear and transparent record of what data was deleted, when, and by whom. By following these best practices, you can ensure efficient and safe data management, allowing you to focus on extracting value from Dynatrace while maintaining smooth and compliant business operations.

Cleanup data in Grail with Privacy Rights

You can also hard delete on the record level in Grail using the out-of-the-box deletion functionalities in Privacy Rights. With an interface designed for your compliance needs, you can efficiently manage data deletion requests and cleanup tasks. A dashboard provides you with a summary of key details, including the request reference, status, and due date alignment. This streamlined process—from request creation to record deletion—is logged and auditable within the app, ensuring transparency and accountability.

The cleanup workflow in Privacy Rights allows you to refine your query to locate all relevant data points, guaranteeing a comprehensive and precise deletion process. In addition, a multi-user approval process with an assignee and a reviewer minimizes deletion risks and ensures accuracy. With Privacy Rights, what once seemed like a daunting compliance task has transformed into a simple and transparent workflow.

Get started

• New to Dynatrace? Open a free trial environment right now and see Dynatrace in action.
• In the Dynatrace Trust Center, learn more about our commitment to giving you control and transparency over your customers’ personal data.
• Share your feedback, suggestions, and ideas with us in the Dynatrace Community.

What’s next

As we continue to innovate and enhance our privacy features, we remain committed to providing you with the industry’s most robust and transparent data protection solutions. We’re working on making data deletion even easier by onboarding log deletion in Grail to the Privacy Rights app. Check our Privacy Rights documentation to stay tuned to our continuous improvements.

The post Privacy spotlight: Ensure compliance by hard deleting individual records in Grail appeared first on Dynatrace news.

What is CVE-2024-6387 (regreSSHion vulnerability)?

The regreSSHion vulnerability (CVE-2024-6387) has been discovered by Qualys Threat Research Unit (TRU) as a Remote Unauthenticated Code Execution (RCE) vulnerability in OpenSSH server (sshd) in glibc-based Linux systems. The vulnerability, a signal handler race condition in OpenSSH server (sshd), allows unauthenticated remote code execution (RCE) as root on glibc-based Linux systems, which is a significant security risk. This race condition affects sshd in its default configuration.

Technical Overview

• Name: CVE-2024-6387
• Description: A security regression (CVE-2006-5051) was discovered in OpenSSH’s server (sshd). There is a race condition which can lead sshd to handle some signals in an unsafe manner. An unauthenticated, remote attacker may be able to trigger it by failing to authenticate within a set time period.
• CVSS Base Score: 8.1 HIGH
• Source: CVE (at NVD; CERT, LWN, oss-sec, fulldisc, Red Hat, Ubuntu, Gentoo, SUSE bugzilla/CVE, GitHub advisories/code/issues, web search, more

What is OpenSSH?

OpenSSH is an essential tool for secure communication and administration in Unix-like operating systems and is also available for Windows. It is part of the broader OpenBSD project and is widely adopted due to its robustness and security features.

What is the potential impact of regreSSHion vulnerability?

Exploiting this vulnerability can lead to full system compromise, where an attacker can execute any commands or code with the highest privileges. This can result in a complete system takeover, malware installation, data manipulation, and the creation of backdoors for persistent access. Moreover, gaining root access will enable attackers to bypass critical security mechanisms such as firewalls, intrusion detection systems, and logging mechanisms, further obscuring their activities. This can result in significant data breaches and leakage, giving attackers access to all data stored on the system, including sensitive or proprietary information that can be stolen or publicly disclosed.

To avoid sounding too alarming, we have to emphasize that this vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. As written in the Qualys white paper, “In our experiments, it takes ~10,000 tries on average to win this race condition. ” To our knowledge as of writing this article, no properly working exploits have been created.

How to mitigate the risk of CVE-2024-6387

The following versions of OpenSSH are impacted by this vulnerability:

• Versions earlier than 4.4p1 (unless patched for CVE-2006-5051 and CVE-2008-4109)
• Versions from 8.5p1 up to, but not including, 9.8p1

OpenBSD systems are unaffected by this, as OpenBSD developed a secure mechanism in 2001 that prevents this vulnerability.

There are three strategic paths to take to mitigate this threat:

• Patch Management: Quickly apply available patches for OpenSSH or upgrade to the relevant version that is not impacted by this vulnerability
• Access Control: Limit SSH access via network-based restrictions
• Network Segmentation: Divide networks to restrict unauthorized access and lateral movements from other systems

How to detect regreSSHion exploitation attempts with Dynatrace

This vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. Thanks to this noisy approach it can be detected from your sshd logs with Dynatrace DQL queries using Dynatrace Security Investigator.

Look for timeout events

Exploitation attempts for this vulnerability can be identified by many lines of “Timeout before authentication” in the logs. The log line in the logs could look something like this:

Jul  1 09:40:39 [localhost] sshd[787404]: fatal: Timeout before authentication for 192.168.12.45 port 17296

To search for such events from your sshd authentication logs stored in Grail, the following DQL query can be used:

fetch logs, from: -48h
| filter contains(content, "Timeout before authentication")

If you don’t get any results, that might be good news! You can further expand the timeframe of your query by changing the from time to -7d. This will get you logs from last week to make sure that no one tried to exploit the system during the last week.

Look for a high volume of log entries

There’s a chance that someone has tried to attack your system with a different attack or that the attack did not succeed as intended. For example, some Proof-of-concept attacks have failed, and these failures write various error messages to the victims’ sshd logs. It’s always good to check the log trend line in general to detect such attacks and when hunting for the unknown.

It’s easy to create a timeseries from your logs using DQL. With the following query, you can create a chart that shows sshd log line count from the previous day, grouped by host and aggregated at an hourly interval.

fetch logs, from: -24h
| filter contains(content, "sshd[")
| makeTimeseries count(), by: { dt.entity.host }, interval:1h

From the chart, it’s easy to see that nothing anomalous was detected using this query: no weird peaks or anomalies could be extracted from the chart.

Look for suspicious IP addresses in sshd logs

To ensure no one has tried to access your services from a suspicious location, it’s valuable to extract all the IP addresses from your sshd logs and see if something suspicious emerges from there. To do that, the following DQL query is helpful:

fetch logs, from: -2d
| filter contains(content, "sshd[")
| parse content, "ARRAY{ ld* ipaddr:ip }{0,10}:source_ip"
| expand source_ip
| summarize count(), by:source_ip
| sort source_ip desc

This query will fetch your sshd logs from the past two days, will extract ALL the IP addresses (both ipv4 and ipv6) from the content and will sort the result based on their occurrances to your results. If you discover any suspicious IP addresses from this list, you can start your investigation based on the discovered threat! If you see only your own addresses, you can rest assured: your sshd has not been accessed by an unknown IP address.

Analyze network flow logs

Last but not least, your network logs are the ultimate source of data. If you’re using AWS for your cloud platform and you have forwarded your VPC flow logs to Dynatrace, you can use DQL to look for your potential victims AND suspicious IP addresses that have tried to access your servers using SSH.

Consider the following query:

fetch logs, from: -24h
| filter aws.log_group == "/aws/vpc/flow_logs"
| parse content, """INT:version ' '
NSPACE:account_id ' '
NSPACE:interface_id ' '
('-' | IPADDR:src_addr) ' '
('-' | IPADDR:dst_addr) ' '
('-' | INT:src_port) ' '
('-' | INT:dst_port) ' '
('-' | INT:protocol) ' '
('-' | LONG:packets) ' '
('-' | LONG:bytes) ' '
TIMESTAMP('s'):start ' '
TIMESTAMP('s'):end ' '
LD:action ' '
LD:log_status"""
| filter dst_port == 22

The query will fetch all the Grail logs ingested from the CloudWatch log group called /aws/vpc/flow_logs. Using the VPC flow log default pattern available in DPL Architect, we can extract the meaningful fields to see only the network traffic targeting the SSH port.

From this dataset, we can create metrics based on the destination IP address to detect all the potential victims by adding the following code snippet to the previous query:

| filter action == "ACCEPT"
| makeTimeseries count(), by: {dst_addr}, interval: 1min

Viewing the results as a graph, you can see that some servers have received an anomalous amount of SSH traffic towards them! From here, you can continue to look further into the specific destinations to understand if this is malicious activity.

To understand who is targeting you, a similar query can be used. Instead of dst_addr use a src_addr to see the most active sources that have been targeting any of your ssh daemons on all the servers in the respective VPC.

| filter action == "ACCEPT"
| makeTimeseries count(), by: {src_addr}, interval: 1min

The results show that of the 725 SSH sources, three stand out with a huge load of requests that require further analysis.

Next steps for mitigating regreSSHion vulnerability

This particular vulnerability is challenging to exploit due to its race condition nature. However, all vulnerabilities should be taken seriously, their potential impact on your environment analyzed, and, where possible, mitigated. If you plan not to take any action for whatever reason, you should at least analyze the potential risk and make an informed decision before taking any action.

The post RegreSSHion vulnerability: Detecting CVE-2024-6387 in OpenSSH appeared first on Dynatrace news.

Dynatrace Grail™ data lakehouse provides contextual analytics across unified observability, security, and business data. It allows you to query and combine data anytime using the Dynatrace Query Language (DQL). This enables exploratory data analysis and the ability to collaborate visually on the results with your colleagues.

Anomaly detection in Notebooks

You likely encounter “why” questions in your daily work. Why did we have an outage? Why did the system behave differently? Why did I receive an alert? These questions can be effectively investigated in Dynatrace Notebooks, where you can easily compile the necessary data and break it down into a time series. However, in the time series example below, we must determine whether the number of access attempts to our example Travel Mobile app is normal or abnormal.

In many cases, it’s evident, based on your past experiences looking at time series data, whether or not something is an anomaly. But how can you automate your expertise? Such automation could ensure that you and your colleagues don’t have to manually monitor time series to identify whether or not they include anomalies.

Davis^® AI provides such automated anomaly detection out of the box. Still, your business requires the flexibility of Davis AI to detect anomalies based on your specific requirements, for example, to automatically generate a Davis problem based on a detected anomaly. For this purpose, we provide the Davis AI Analyzer, which allows you to select a specific analyzer. Three anomaly detection analyzers are available, each equipped with unique mechanisms to detect anomalies in your data that significantly deviate from the norm.

• Static threshold anomaly detection
• Auto-adaptive threshold anomaly detection
• Seasonal baseline anomaly detection

One unique feature of the Davis AI Analyzer is that it works on any time series, regardless of its origin—whether generated with makeTimeseries from events, business events, logs, or other sources or the joining of different time series. As you can see in the screenshot below, Davis AI Analyzer gains the full power of DQL, making Davis anomaly detection even more flexible and stronger than ever. This power can be easily experienced by selecting the desired Davis anomaly detection analyzer in Notebooks or Dashboards.

By selecting the seasonal baseline analyzer, Davis AI recognizes that the number of attempted accesses to the app in this example doesn’t deviate from the norm based on the past data during the same period. The time series falls within the seasonal green confidence band. A potential alert would be visually simulated if the time series fell outside this band.

This anomaly detector observes the number of attempted accesses per minute and triggers an event when anomalies are detected. You can create a similar Davis anomaly detector in a few simple steps.

Automate your experience with Davis Anomaly Detection

In Notebooks, select open with and choose Davis Anomaly Detection; all settings required for creating an anomaly detector will be carried over.

The new anomaly detector is created in four steps; the first two steps are carried over automatically from Notebooks. Let’s start with the most straightforward step, Get started, where you define a title for your anomaly detector and a description for the configuration.

The next two steps, as mentioned, have already been prefilled from Notebooks. In the Configure your query step, you’ll find the DQL query you predefined, and in the Customize parameters step, you’ll find your selected anomaly detection analyzer. The last significant step, the Create an event template step, remains. Here, you can define the template for your event and describe all essential information for the subsequent process.

What makes this template exceptional is that you can use {placeholder} hints to add additional context to the text about the event. For example, the value of the violation or the source entity where the anomaly was detected. This ensures that all essential information about the event is immediately visible to the Site Reliability Engineer (SRE). After completing all four steps, we can create the Davis anomaly detector by selecting Create. The anomaly detector will automatically monitor your defined time series every minute and trigger your specified event upon detection of an anomaly.

The new anomaly detector is now listed in Davis Anomaly Detection. Here, you’ll find all anomaly detector configurations, and you can filter them according to your specific criteria. Additionally, you can expand this table with extra information about the configurations, such as when the anomaly detectors were last modified.

Of course, you always have the option to reopen an anomaly detector directly in Notebooks, where all configuration settings are carried over. You also have the option to display a preview of your anomaly detector directly in Davis Anomaly Detection.

The exciting challenge is finding answers to your everyday “why” questions using Grail and DQL analytics capabilities. If the answer is successfully identified in a time series and you want to automate the result with anomaly detection, this can be done in just a few steps. We recommend you explore the new Davis Anomaly Detection analyzer in Notebooks; we’re confident you’ll quickly discover its many uses.

Try out Davis Anomaly Detection

Want to know more? Check out the following video, in which Andreas Grabner and I collaborated on a new episode of the Dynatrace Observability Clinic. Here, we share a live introduction to Anomaly Detection based on DQL.

We also recommend watching the exciting use case for Anomaly Detection and the 5 Pillars of Data Observability.

What’s next

Davis Anomaly Detection is automatically enabled for all Dynatrace SaaS environments with the release of Dynatrace version 1.291. No effort is needed from your side. We’re, of course, highly interested in your feedback. So, please head to the Dynatrace Community and share your suggestions and product ideas to help us continuously improve Dynatrace Anomaly Detection.

Are you interested in learning more? In Dynatrace Documentation, you can learn more about Davis Anomaly Detection and how to use anomaly detection within Notebooks, or look at our Playground, where you can explore practical examples of how to utilize Davis AI Analyzer in your anomaly detection.

The post Learn how to create a Davis AI anomaly detector on Grail appeared first on Dynatrace news.

Logs provide answers, but monitoring is a challenge

Manual tagging is error-prone

Making sure your required logs are monitored is a task distributed between the data owner and the monitoring administrator. Often, it comes down to provisioning YAML configuration files and listing the files or log sources required for monitoring. This manual, error-prone approach can lead to monitoring gaps, which become critical when a host or service has an outage or incident.

Finding the right logs is cumbersome

Even if your logs are monitored, you need to make sense of the vast data volume. As the scale and complexity of your tech stack grows, you might need to navigate the maze of hosts or Kubernetes clusters, apps, and microservices and understand the relevance and risks associated with logs originating from these entities. Challenges compound: Manual tagging of log sources has long been difficult regarding monitoring coverage. And you can’t assume the tagging is 100% correct to pinpoint the correct logs.

In the past, more work was needed to understand the context of log data. What about correlated trace data, host metrics, real-time vulnerability scanning results, or log messages captured just before an incident occurs? This context is vital to understanding issues.

Dynatrace automatically puts logs into context

Dynatrace Log Management and Analytics directly addresses these challenges. First, OneAgent takes care of log autodiscovery. Once logs are selected for monitoring, OneAgent enriches log data with the topological context you need. For example, OneAgent helps you monitor the logs from a Kubernetes environment with automatic enrichment that identifies the right cluster, namespace, container, and pod ID.

Once logs are stored in Dynatrace Grail™, our purpose-built data lakehouse for observability data, the logs are automatically shown in the right context. Finding answers begins with opening the right app for your use case.

You can easily pivot between a hot Kubernetes cluster and the log file related to the issue in 2-3 clicks in these Dynatrace® Apps: Infrastructure & Observability (I&O), Databases, Clouds, and Kubernetes.

Open a host, cluster, cloud service, or database view in one of these apps, and you immediately see logs alongside other relevant metrics, processes, SLOs, events, vulnerabilities, and data offered by the app.

By eliminating slow and manual correlation, lack of context, and getting visibility into the surrounding data, you reduce the risk of prolonged outages, mean time to repair, and tool sprawl.

Get quicker answers

Let’s look at how logs in context can make your teams more effective.

Log histograms: Insight into log volumes and patterns

Open one of these Dynatrace Apps and select Logs for any listed entity (host, Kubernetes workload, cloud service, or database instance):

• Infrastructure & Operations
• Kubernetes
• Databases
• Clouds

You’ll see a histogram chart of log data with various severity levels (such as Error, Info, or Warning) relevant to the selected Dynatrace entity, giving you a clear understanding of log patterns and volumes over time. Is there a sudden spike in errors? A sudden drop in received log data? Depending on which app is in use, one glance at a histogram provides invaluable insight into managing clouds, databases, Kubernetes environments, and infrastructure.

Log analytics simplified: Deeper insights, no DQL required

Your team will immediately notice the streamlined log analysis capabilities below the histogram. Jump directly into log insights by selecting a recommended query, for example, to see the errors related to a problem detected by Davis® AI during the selected timeframe. Furthermore, your team can easily access all error logs within the specified timeframe displayed on the histogram or view all logs within that timeframe, all without writing any queries from scratch.

Surrounding logs display: Effortlessly navigate log context

You can see the result after opening a recommended query without leaving an app’s context. Upon expanding a single log entry, all relevant context provided by OneAgent during the ingestion process is displayed, making it easy to expand your analysis to the infrastructure or entity related to the error logs. For a single log record found, you can easily see the surrounding logs.

Look at this example of an online store payment service generating errors. The application owner found error logs related to unsupported credit cards. Select Surrounding logs to view the log messages for the whole transaction, based on the trace ID, that ended up with an error and a failed order.

In Infrastructure & Operations, surrounding logs can also be displayed based on other criteria, like the host file or log source from which logs are collected. This allows quick and easy troubleshooting without writing or editing queries.

Logs in context across Dynatrace Apps

• Infrastructure & Operations leverages advanced AI capabilities that automatically discover and map all components within your infrastructure, including hosts, virtual machines, containers, and cloud instances.
• Databases offers comprehensive database monitoring capabilities, providing organizations with real-time visibility into the performance and health of their database environments.
• Clouds is a central hub for monitoring and managing multicloud environments, providing organizations with a unified view of their cloud infrastructure and services.
• Kubernetes delivers comprehensive monitoring and management capabilities for Kubernetes environments, enabling organizations to ensure the performance, availability, and scalability of their containerized workloads.

Stay tuned for even wider support of log data embedded seamlessly into the context of Dynatrace Apps, and better ways to get answers from logs without writing queries.

See for yourself

• Watch a demo of logs in context within various Dynatrace Apps in this Dynatrace University course.
• Open your free Dynatrace trial now to get started.

The post Unlock log analytics: Seamless insights without writing queries appeared first on Dynatrace news.

Dynatrace platform’s unique features are now available on Azure

As of today, Azure customers can leverage the latest Dynatrace core innovations on Microsoft Azure, including:

• Dynatrace Grail™ data lakehouse unifies the massive volume and variety of observability, security, and business data from cloud-native, hybrid, and multicloud environments while retaining data context to deliver instant, cost-efficient, and precise analytics.
• Dynatrace^® AutomationEngine features a no- and low-code toolset and leverages Davis^® AI to empower teams to create and extend customized, intelligent, and secure workflow automation across cloud ecosystems.
• Dynatrace^® AppEngine features a no- and low-code toolset and leverages Davis AI to empower teams to easily create and share custom, intelligent, and secure apps that leverage insights from data generated by their clouds.
• The new Dynatrace user experience, including powerful dashboarding capabilities and interactive Dynatrace Notebooks, drives tighter cross-team collaboration and enables more people within the organization to make data-backed decisions.

Azure Native Dynatrace Service allows easy access to new Dynatrace platform innovations

Dynatrace has long offered deep integration into Azure and Azure Marketplace with its Azure Native Dynatrace Service, developed in collaboration with Microsoft. With the AI-powered Dynatrace platform now generally available on Azure, Azure Native Dynatrace Service customers can now leverage the full AI power of the Dynatrace platform directly from Azure. With Dynatrace directly connected to your Azure subscriptions, Dynatrace OneAgent^® can be deployed directly to VMs and App Services from the Microsoft Azure Portal. One-click activation of log collection and Azure Monitor metric collection in the Microsoft Azure Portal allows instant ingest of Azure Monitor logs and metrics into the Dynatrace platform. For more details, see the blog post, Set up AI-powered observability for your Microsoft Azure cloud resources in just one click.

The following figure shows the benefits of Azure Native Dynatrace Service. For more details, please see the blog post Dynatrace and Microsoft Azure integrate to help accelerate your cloud transformation.

Using the Dynatrace platform on Azure allows Azure Native Dynatrace Service customers to get instant insights into their workloads with a comprehensive set of new innovative features with the best product experience from the Microsoft Azure Portal:

• The Dynatrace platform is automatically provisioned as part of the Azure Marketplace purchase process, allowing you to set up observability for workloads in minutes.
• Configuring the Dynatrace platform and deploying Dynatrace OneAgent from the Microsoft Azure Portal ensures that observability data from Azure resources arrives in Dynatrace within seconds.
• New Dynatrace platform features—with the Grail data lakehouse at their core—allow you to easily query and get insights into your workloads with the new Dynatrace user experience using DQL, Notebooks, and Dashboards, driven by a new Automation Engine, and AppEngine.

Leverage Dynatrace to get observability into all resources in your Azure tenant

This blog post shows you how the Dynatrace platform, together with the Azure Native Dynatrace Service—and unique new capabilities such as Grail, Dashboards, Notebooks, and the Cloud app—can give instant insights into all the Azure resources deployed in your Azure tenant.

Set up complete monitoring for your Azure subscription with Azure Monitor integration

After activating the Azure Native Dynatrace Service (see Dynatrace Documentation for details), the Azure Monitor integration is enabled easily via the Microsoft Azure Portal, as shown in the following screenshot. There’s no need for configuration or setup of any infrastructure.

Enabling the Azure Monitor integration in the Microsoft Azure Portal triggers the linked Dynatrace environment to start polling metrics and resource metadata for the Dynatrace Smartscape^® topology model. Furthermore, as shown in the above screenshot, the collection of resource logs and activity logs is easy to turn on.

Integrate multiple Azure subscriptions under a single Dynatrace environment

If you have multiple Azure subscriptions in your Azure tenant, it’s best practice to integrate the Azure subscriptions with a single Dynatrace environment. Establishing a single source of truth for all observability data reveals the full power of the Dynatrace platform by querying all data with comprehensive new capabilities from the one Dynatrace Grail data lake house for data storage.

To monitor multiple subscriptions within your Azure tenant

1. Create a new Dynatrace environment within the first subscription by creating the first Dynatrace resource (Option Create a new Dynatrace environment as shown in the following screenshot).
2. Then, create a Dynatrace resource in all other Azure subscriptions and link the Dynatrace environment created in Step 1 (the option Link Azure subscription to an existing Dynatrace environment is shown in the following screenshot).

More information can be found at Dynatrace link to existing. Also, see how to automate this process with bicep and how to automate with Azure CLI.

Azure use cases with Dynatrace Apps

The following new Dynatrace^® Apps can be leveraged to get instant insights into all Azure resources of Azure subscriptions connected via the Azure Native Dynatrace Service:

• Clouds enables cross-subscription and cross-region observability in one place.
• Dashboards leverages the power of DQL for Azure monitoring in one place.
• Notebooks offers advanced Azure observability analytics with DQL.

Clouds

Clouds allows easy investigation and troubleshooting of all Azure resources from the subscriptions previously connected to Dynatrace.

The following screenshot shows that you can easily search for resources by name, resource type, region, and tags across all connected Azure subscriptions.

Clouds provides resource properties, metrics, problems, and events in a single view, as shown below.

Davis AI automatically analyzes ongoing problems with your resources. Clouds makes it easy to search for all resources affected by detected problems and navigate directly to the problem details.

Clouds also supports getting the necessary insights for cloud governance. For example, Dynatrace uses automatic tagging to mark the user who created an Azure resource. Searching for all resources with a specific tag (for example, created-by) across your connected Azure subscriptions is easy with Clouds.

Dashboards

Dashboards provides new dashboarding capabilities powered by the new Dynatrace platform query language (DQL), which fully leverages the new Dynatrace experience.

With DQL and Dashboards, combining all observability data in one dashboard is easy. For example, it’s possible to combine resource metadata, resource health from activity logs, Azure Monitor metrics, Davis AI problems, OneAgent process details, Synthetic monitoring results, and much more in a single view, as shown below.

For each section, it’s possible to drill down for detailed context and learn the underlying DQL query.

The following shows a simple DQL summarizing all Azure Virtual Machine cores in the connected Azure subscriptions. By querying the Dynatrace entity model, cores are summed up from entity metadata.

fetch dt.entity.azure_vm
| parse azureVmSize, "KVP{LD[a-zA-Z]+:key'='(LONG:valueLong | STRING:valueStr)','?}:vmsize"
| fieldsAdd cores =   vmsize[`numCores`], memoryMb =   vmsize[`memoryMb`]
| summarize Cores = sum(cores)

Notebooks

Dynatrace Platform offers Notebooks, which allows advanced ad-hoc analysis of Azure environments based on metrics, logs, spans, events, and entities.

The following DQL from Notebooks queries data for all recent Azure health events and aggregates all resource health events from all Azure subscriptions connected via the Azure Native Dynatrace service.

fetch logs //, scanLimitGBytes: 500, samplingRatio: 1000
| filter matchesValue(cloud.provider, "Azure")
| filter azure.resource.type == "MICROSOFT.COMPUTE/VIRTUALMACHINES" or azure.resource.type == "MICROSOFT.COMPUTE/VIRTUALMACHINESCALESETS/VIRTUALMACHINES"
| parse content, "JSON:json"
| fieldsAdd operationName=json[operationName],
category=json[category],
resultType=json[resultType],
callerIpAddress=json[callerIpAddress],
action=json[identity][authorization][action] ,
principalId=json[identity][authorization][evidence][principalId] ,
principalType=json[identity][authorization][evidence][principalType] ,
name=coalesce(json[identity][claims][name],json[identity][claims][xms_mirid]),
time= json[time],
title=json[properties][title],
details=json[properties][details],
current=json[properties][currentHealthStatus],
previous=json[properties][previoustHealthStatus],
type=json[properties][type],
cause=json[properties][cause]
| filter  category== "ResourceHealth"
| fieldsKeep  time, title, details, type, cause, azure.resource.name
| sort  time desc

Utilizing the Dynatrace platform to observe your Azure subscriptions allows you to address compliance requirements beyond the standard; for example, the retention period for Azure activity logs is 90 days. With Dynatrace, you can configure up to a 10-year retention period in Dynatrace Grail (for full details, see data retention periods).

Here are further examples that show the power of DQL and Grail for Azure analysis:

Count the number of VMs per Azure Subscription

fetch dt.entity.azure_vm
| fieldsAdd  azure_subscription = accessible_by[dt.entity.azure_subscription][0], azure_region = belongs_to[dt.entity.azure_region]
| lookup [fetch dt.entity.azure_subscription | fieldsAdd name = entity.name ,  uuid = azureSubscriptionUuid], sourceField:azure_subscription, lookupField:id, prefix:"azure.subscription."
| lookup [fetch dt.entity.azure_region | fieldsAdd name = entity.name], sourceField:azure_region, lookupField:id, prefix:"azure.region."
| summarize  count=count(), by: azure.subscription.name

Count the number of VMs per Azure region

fetch dt.entity.azure_vm
| fieldsAdd  azure_subscription = accessible_by[dt.entity.azure_subscription][0], azure_region = belongs_to[dt.entity.azure_region]
| lookup [fetch dt.entity.azure_subscription | fieldsAdd name = entity.name ,  uuid = azureSubscriptionUuid], sourceField:azure_subscription, lookupField:id, prefix:"azure.subscription."
| lookup [fetch dt.entity.azure_region | fieldsAdd name = entity.name], sourceField:azure_region, lookupField:id, prefix:"azure.region."
| summarize  count=count(), by: azure.region.name

Identify the VMs with top CPU usage

timeseries max= max(dt.cloud.azure.vm.cpu_usage),      
 by:  {dt.entity.azure_vm}
| fieldsAdd maxCpu = arrayMax(max)
| sort maxCpu desc
| lookup [fetch dt.entity.azure_vm | fieldsAdd name = entity.name, azure_subscription=accessible_by[dt.entity.azure_subscription][0],  azure_region=belongs_to[dt.entity.azure_region] ], sourceField:`dt.entity.azure_vm`, lookupField:id, prefix:"azure.vm."
| lookup [fetch dt.entity.azure_subscription | fieldsAdd name = entity.name ,  uuid = azureSubscriptionUuid], sourceField:azure.vm.azure_subscription, lookupField:id, prefix:"azure.subscription."
| lookup [fetch dt.entity.azure_region | fieldsAdd name = entity.name], sourceField:azure.vm.azure_region, lookupField:id, prefix:"azure.region."
| fieldsKeep   maxCpu,azure.vm.name, azure.subscription.name, azure.region.name
| limit 20

Get the latest 10 log lines from Azure

fetch logs | filter cloud.provider == "Azure" or cloud.provider == "azure" | sort  timestamp desc  | limit 10

Upgrade to Dynatrace for Azure

If you’re an existing Dynatrace customer, please contact us to learn how to upgrade to Dynatrace on Azure. An overview of how to upgrade to Dynatrace is available in our guide, Upgrade to Dynatrace SaaS.

Get started with the AI-powered Dynatrace platform and Azure Native Dynatrace Service

Visit the Azure marketplace to start a trial of the Dynatrace platform on Azure and the Azure Native Dynatrace Service.

Please see Data Security Controls in Dynatrace documentation for an overview of the Azure regions currently supported by Dynatrace.

The post Discover the power of the Dynatrace platform on Azure with Azure Native Dynatrace Service appeared first on Dynatrace news.

Across the globe, privacy laws grant individuals data subject rights, such as the right to access and delete personal data processed about them. Rising consumer expectations for transparency and control over their data, combined with increasing data volumes, contribute to the importance of swift and efficient management of privacy rights requests.

“By 2026, fines due to mismanagement of subject rights will have increased tenfold from 2022 to total over $1 billion.” [1]
–Gartner^®

These drivers and the growing complexity of data privacy regulations make manual handling of these requests unsustainable, necessitating automated and scalable solutions.

Handling privacy rights throughout your complex data ecosystem

Your obligation to uphold privacy rights extends beyond the boundaries of your own organization to the vendors in your supply chain. Because of that, SaaS providers, such as Dynatrace, play a crucial role in facilitating compliance with privacy rights. When a privacy request is initiated, it cascades through every layer of your supply chain. Successful compliance with privacy rights requests involves tracking and verifying requests across the entire data ecosystem, including third-party services.

Lack of control over the management of privacy rights can lead to increased administrative burdens, higher risk of errors, and overall frustration for organizations who fear a loss of customer trust in case of non-compliance with data protection regulations like GDPR and CCPA.

“While the need for scalable subject rights delivery and fulfillment will not go away, the demand for more automation will lead to a faster move toward a zero-touch model.” [2]
— Nader Henein, VP Analyst, Gartner

The Privacy Rights app is designed to streamline this process in Dynatrace.

Efficiently locate and export your end users’ personal data

Let’s walk through a fictitious example to explore privacy rights handling in Dynatrace. A diligent services administrator is busy making sure that Dynatrace is deployed and working correctly. But today, something new awaits in their inbox: an urgent data-subject rights request from the privacy legal team. It states: “Please locate, compile, and delete all personal data associated with this email address in Dynatrace as part of our end-user’s right to access their data.” How can this services administrator meet this request in a quick, compliant, and efficient way?

Unlike many other SaaS vendors, Dynatrace empowers you by providing a tool for submitting, tracking, and verifying the status of requests. Dynatrace overcomes the challenge of manual, error-prone processes with workflows that enhance transparency and accountability in privacy rights handling. With an out-of-the-box solution, Privacy Rights enables your team to focus on deriving value from Dynatrace rather than being stuck with time-consuming compliance tasks.

Privacy Rights in action

With an interface tailored to your compliance needs, Dynatrace users can track privacy rights requests. A dashboard summarizes relevant details, such as reference to the request and its status and alignment with any due date. This approach effortlessly keeps you and your privacy team informed about end-user requests without requiring additional follow-up through support for status updates. The entire process, from the creation of the request to the export or deletion of the relevant data, is logged and auditable within the app.

When creating an export or deletion request, an authorized user specifies the end user’s details. The details are used to search Grail for any personal data related to that user. This step lets you fine-tune your query to identify all matching data points, ensuring a thorough and accurate retrieval process.

Once the data in Grail that matches the run query is returned, a second authorized user reviews the results in terms of volume (the number of log records, volume, data residency, and the number of systems). Based on this overview, the export of the matching logs is either approved or rejected. Similarly, in the case of personal data deletion, the reviewer has the opportunity to accept or reject the deletion request. This multi-user approval process reduces export and deletion risks and serves as an accuracy check.

With just a few clicks, what might have initially seemed like a cumbersome compliance task becomes a seamless and transparent workflow. Privacy Rights helps organizations confidently navigate the complexities of data subject rights handling in Dynatrace.

Get started with Privacy Rights today

• New to Dynatrace? Open a free trial environment right now and see Dynatrace in action.
• Check out the documentation for the Privacy Rights app.
• Learn more about our commitment to providing you with control and transparency over your customers’ personal data in the Dynatrace Trust Center.
• Share your feedback, suggestions, and ideas with us in the Dynatrace Community.

What’s next

We’re working on making Dynatrace privacy management even easier. Check out Privacy Rights documentation to stay informed about our continuous improvements.

^{[1] Gartner Press Release, “Gartner Predicts Fines Related to Mismanagement of Data Subject Rights Will Exceed $1 Billion by 2026,” 24 August 2023, https://www.gartner.com/en/newsroom/press-releases/2023-08-24-gartner-predicts-fines-related-to-mismanagement-of-data-subject-rights-will-exceed-1-billion-dollars-by-2026. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.}

^{[2] Ibid.}

The post Privacy Spotlight: Easily comply with data subject rights in Dynatrace appeared first on Dynatrace news.

As a security analyst investigating security incidents or threat hunting, you often must navigate among multiple executed queries and their respective results (in other words, investigation steps). You also need to manage the evidence you gather during investigations and reuse it when building additional queries. All this must be as fast and seamless while maintaining the investigation context.

Navigating investigations and threat hunting can involve saving executed queries and found evidence in unstructured text documents that can quickly lose their context. Manually documenting investigation steps can lead to human errors, like forgotten or wrongly documented queries or evidence, which will result in missed threats and wasted time. As the investigation grows and the different queries accumulate, it becomes harder to maintain the throughlines among the different branches.

The Dynatrace Security Investigator app provides branching navigation, making it easy to track your path through investigations.

Navigate in any direction within your investigation

Investigations in Grail are conducted using the Dynatrace Query Language (DQL). For multiple investigation steps, the Security Investigator provides a representation of the steps in a tree-like view. With this visualization, you can keep track of your investigation and navigate to previous steps and respective results in the investigation history—all in context, with no need to re-execute anything.

Every new DQL query execution creates a new descending node while keeping the integrity and hierarchy of previous queries and results unchanged.

Easily track threat-hunting twists and turns

Threat hunting is a nonlinear process. When looking for the unknown, you often stumble upon an unexpected thread of evidence that sparks a new idea and takes the investigation in a new direction. To support this nonlinear approach, we’ve created the investigation branching feature. When modifying and re-executing a previous step, the app creates a new branch while keeping the original investigation path intact and visible in the query tree.

You can create any new branches for new ideas and hypotheses while retaining the ability to navigate back to your initial train of thought. With the ability to customize and name branches, you can simplify locating historical steps without having to click through them individually.

The final visual representation provides a comprehensive overview of the entire investigation. You can see in a structured way how you reached your conclusions without needing to manually document the investigation flow.

With Security Investigator, you can always find the last working query, which was easy to lose in the past.

Character precision on a petabyte scale

Security Investigator increases the speed of investigation flows and the precision of evidence, leading to higher efficiency and faster results. Both outcomes are game-changers in time-critical situations, where answers are expected “yesterday.”

View raw content details when you need them

Data ingested into Grail is kept in its original format. A schema is applied only during query execution, or as we call it, “schema-on-read.” This makes data immune to format changes and enables you to choose your data structure based on your use cases.

The Security Investigator app enables you to view all data in its original format, regardless of the content, including characters that other tools might hide or not interpret. Enabling the multi-line content details view shows the stack traces with their line breaks in their original form, enabling you to understand the data much faster.

Find the needle in the “needle stack”

Finding a small metallic object in a stack of non-magnetic objects is relatively easy. Finding a particular object among millions of similar objects is trickier. With Security Investigator’s flexible filtering, you can create accurate DQL query filters quickly and efficiently.

If you need to fetch raw content from Grail and filter the results based on a selection of the content, you don’t need to copy partial records and manually paste them into an appropriate filter. Instead, you can easily select the portion of the text string you need and add it to your DQL query directly from the context menu.

To add multiple values to a filter, you can either select individual values or a range of values and apply them as a single filter in your DQL query. Only the unique values from the selection are added to the query, so you don’t have to investigate unneeded values.

Use your findings to enrich your investigation

With Security Investigator evidence management, you can persist relevant findings by attaching them to your investigation. Whether it is a suspicious IP address or a potential indicator of compromise, you can attach it to the investigation and use it later.

You can create type-specific filters to enable fast and accurate results, either focusing only on suspicious activities or filtering out the safe events that don’t require your attention.

Continue hunting for the unknown

Security use cases such as threat hunting and incident analysis are nonlinear, evidence-driven activities in which you don’t know what you’re looking for. You’re not pursuing a specific target; rather, you’re looking for the unknown. With Security Investigator, Dynatrace enables you to be more efficient, investigate faster, and be more precise.

Use Dynatrace Security Investigator for faster and more precise evidence-based security investigations.

The post Speed up evidence-driven security investigations and threat hunting with Dynatrace Security Investigator appeared first on Dynatrace news.

Organizations choose data-driven approaches to maximize the value of their data, achieve better business outcomes, and realize cost savings by improving their products, services, and processes. However, there are many obstacles and limitations along the way to becoming a data-driven organization. The following are some of the most pressing challenges:

• Managing cost and scale of large data. The exponential growth of data volume—including observability, security, software lifecycle, and business data—forces organizations to deal with cost increases while providing flexible, robust, and scalable ingest.
• Understanding the context. With siloed data sources, heterogeneous data types—including metrics, traces, logs, user behavior, business events, vulnerabilities, threats, lifecycle events, and more—and increasing tool sprawl, it’s next to impossible to offer users real-time access to data in a unified, contextualized view.
• Addressing security requirements. Organizations need to ensure their solutions meet security and privacy requirements through certified high-performance filtering, masking, routing, and encryption technologies while remaining easy to configure and operate.

Dynatrace is addressing these challenges with a single, built-in data ingest functionality: Dynatrace OpenPipeline™, the ultimate addition for data-driven organizations.

Introducing Dynatrace OpenPipeline

OpenPipeline is a stream-processing technology that transforms how the Dynatrace platform ingests data from any source, at any scale, and in any format. With OpenPipeline, you can easily collect data from Dynatrace OneAgent®, open source collectors such as OpenTelemetry, or other third-party tools. OpenPipeline then filters and preprocesses that data to manage and reduce costs.

OpenPipeline also includes data contextualization technology, which enriches data with metadata and links it to other relevant data sources. By putting data in context, OpenPipeline enables the Dynatrace platform to deliver AI-driven insights, analytics, and automation for customers across observability, security, software lifecycle, and business domains.

Furthermore, OpenPipeline is a data security and privacy technology that ensures data is collected and processed securely and compliantly, with high-performance filtering, masking, routing, and encryption capabilities that are easy to configure and operate.

OpenPipeline works seamlessly with Dynatrace Grail™ to handle data ingest at an unparalleled scale. During the data ingestion process, OpenPipeline enriches data and data signals with their context and simultaneously interconnects them with Smartscape®, a real-time interactive map that reflects the topology and dependencies of all data signals. This “data in context” feeds Davis® AI, the Dynatrace hypermodal AI, and enables schema-less and index-free analytics. Unlike other pipelining solutions that serve to consolidate and move data from one place to another, OpenPipeline enhances the Dynatrace platform and allows users to extract even more value from their data.

Scale beyond petabytes

Dynatrace can collect data from the full application stack without configuration—including metrics, traces, logs, user sessions, security events, business events, and more. OpenPipeline unifies the ingestion of data sent to Dynatrace from any source in any format. During transport, data is prioritized, compressed and encrypted, ensuring data integrity and protection. Data is then dynamically routed into pipelines for further processing.

Designed to reach beyond petabyte (PB) scale, OpenPipeline will—at its launch—quintuple the ingest throughput from 100 TB a day per tenant to 500 TB a day per tenant. Further scaling, to and beyond 1 PB per day, will be announced in the near future.

This massive increase in data-ingestion throughput is possible thanks to several patent-pending high-performance stream-processing technologies, including a breakthrough in the simultaneous processing of thousands of data-processing rules. OpenPipeline rule processing outperforms most rule-processing algorithms by magnitudes (by a factor of 6-10 compared to prior art), with lower memory consumption and no degradation in data throughput.

Manage the cost of data with ease

With the exponential growth of data, and the need to retain data longer for business and security reasons, it’s important to preprocess and manage data appropriately to maximize value and minimize cost. OpenPipeline high-performance filtering and preprocessing provides full ingest and storage control for the Dynatrace platform. As a result, dedicated data pipeline tools are unnecessary for preprocessing data before ingest.

Filtering data is crucial for privacy and compliance to minimize the exposure of sensitive data. Additionally, it helps to reduce data volume and keep the cost of storing and querying data under control by eliminating duplicates, redundancies, and dropping unnecessary data fields. As an example, in early preview usage, AWS GuardDuty events were reduced by 84% by filtering out security-irrelevant events, which reduced cost and alert noise at the same time.

Transformations preprocess data and reduce data volume further, especially in situations where raw data is not required. OpenPipeline extracts data with context and transforms it into more efficient formats, for example, logs to metrics. Such transformations can reduce storage costs by 99%.

Routing of data to specific Grail buckets of varying retention durations lets you decide which data to keep and for how long. It also separates data organizationally for improved access control and focused query scope.

Protect your sensitive data

• Privacy by design. One essential aspect of OpenPipeline is the ability to mask data at capture using OneAgent® and automatic, rule-based processing. This approach suppresses sensitive data capture entirely before the data leaves the process, service, or customer environment. This ensures compliance and protects sensitive information from the start.
• Commitment to privacy. Dynatrace Trust Center demonstrates the Dynatrace commitment to privacy (and security) by design. Masking personal and sensitive data is of vital importance. With OpenPipeline, Dynatrace users can configure filtering and masking to their specific needs.

Bring data into context for improved analytics, automation, and AI

As OpenPipeline processes data streams in real time and it retains context during data normalization. At the same time, it performs contextual enrichment to ensure high-fidelity analytics, automation, and AI. With OpenPipeline processing power, you can:

• Enrich data and improve its value and quality by adding supplemental attributes (such as IP address geolocation or the related trace ID to a log line).
• Normalize data without losing context, as it detects known data structures automatically and provides optional rules for custom data structures.
• Transform contents into well-defined fields, convert raw data to time series, calculate metrics, or create business events from log lines.
• Converge heterogeneous data sources with ease, as data normalization and contextual enrichment can also happen on read, thanks to the schemaless and indexless approach of Grail.
• Contextualize and map data and get a unified view by identifying and connecting data points to the topology and dependencies within a software environment using the Dynatrace Semantic Dictionary. This contextualization provides Grail with additional semantic information in real time.
• Prioritize business data for a desired quality of service (QoS). Dynatrace provides unmatched accuracy by treating relevant business data (for example, real-time consumption or revenue dashboarding and analytics) with a higher priority and ensuring that data is not only in context but also not dropped or sampled.

Such contextually enriched data is the key to unlocking the full potential of your data for improved analytics, automation, and AI. By adding metadata and linking data to other relevant data sources, you enhance the quality, accuracy, and value of your data.

Davis—the unique Dynatrace hypermodal AI—builds on more than a decade of AI expertise in predictive and causal AI, and as announced recently, generative AI technologies. Automation and business observability require precise results. Contextually enriched data enables unrivaled predictive and causal AI power that provides real-time risk and root-cause analysis, enabling immediate insights and automated, AI-powered IT operations.

OpenPipeline in action

Let’s take a look at a concrete example—a logline that contains the following information:

2024-01-31 15:08:12 INFO [user-service] User john.doe@example.com logged in from 192.168.0.1

OpenPipeline can parse this logline and extract the following fields:

• Timestamp: 2024-01-31 15:08:12
• Log level: INFO
• Service name: user-service
• User email: john.doe@example.com
• User IP: 192.168.0.1

From here, OpenPipeline can convert this log entry into a time series metric that counts the number of logins per service, or create a business event that triggers an alert or notification when a user logs in.

With no schema or indexing, Grail even handles such data normalization and contextual enrichment “on read” as well, which makes it even easier to converge heterogeneous data sources. This additional power is enabled by the Dynatrace Semantic Dictionary, which provides Grail additional semantic information in real-time for the mapping of topology and dependencies within a software environment. Let’s look at another example, where a data source contains the following information:

{

"user_id": "123456789",

"user_email": "john.doe@example.com",

"user_name": "John Doe",

"user_role": "admin",

"user_location": "Linz, Austria"

}

Grail can use the Dynatrace Semantic Dictionary to map the user_email field to the User email field extracted above by OpenPipeline and enrich the data with additional context, such as user_id, user_name, user_role, and user_location. This way, Grail can provide a holistic view of the user and their activities across different data sources.

OpenPipeline is part of the unified Settings experience, inside the built-in Settings app 

OpenPipeline offers an easy way to create and configure routes and pipelines at scale, comfortably situated within the Settings app. This app simplifies the configuration process by utilizing Dynatrace query language (DQL) for matching and processing routes. Of course, configuration-as-code using an application programming interface (API) is also available.

Visual guidance within the user interface is available to administrators for managing rules, setting up ingestion configurations, dynamic routing into individual pipelines, controlling data enrichment and transformation policies, and sending data from a source to its destination (such as specific Grail buckets).

You can read more about setting up and managing pipelines in our documentation. 

Get more value from your data with OpenPipeline

OpenPipeline complements Grail with high-performance stream processing to maximize security and ease the management of large heterogeneous data, while minimizing cost. It manages parallel pipelines to ingest, transport, mask, filter, enrich, normalize, transform, contextualize, route, and persist data. OpenPipeline is available to Dynatrace customers at no additional cost.

• OpenPipeline enables cost-effective and scalable data ingest, with up to 500 TB per day per tenant, with plans to go beyond the petabyte-per-day level in the near future. It also offers built-in data transformation features that reduce storage costs by up to 99%.
• OpenPipeline contextualizes data in real time with enrichment, discovered topology, and tags. Using its patent-pending stream-processing technologies, OpenPipeline optimizes data for Dynatrace analytics and AI.
• OpenPipeline ensures data security and privacy with source-side masking and encryption at the source, additional filtering, and masking at ingest. In combination with Grail, this ensures data privacy and compliance at the storage and query level. Such a centralized data management approach has substantial potential to reduce security and audit efforts.

The first release of Dynatrace OpenPipeline, supporting logs, business events, and generic events, will be released within 90 days. Once available, Dynatrace SaaS customers running on AWS or Azure, and using the latest version of Dynatrace, can start using OpenPipeline without installing anything: the OpenPipeline Configuration app will be pre-installed on all eligible tenants. Existing processing rules for logs and business events will be automatically migrated to new pipelines, so existing customers will benefit from the new OpenPipeline functionality from day one.

The post Dynatrace OpenPipeline: Stream processing data ingestion converges observability, security, and business data at massive scale for analytics and automation in context appeared first on Dynatrace news.

“Great! I have ingested important custom data into Dynatrace, critical to running my applications and making accurate business decisions… but can I trust the accuracy and reliability?”

Welcome to the world of data observability.

The Dynatrace open platform is well-positioned to take advantage of the exponential increase in data generation. However, coupled with the increase of external data sources that can now be ingested, there are new challenges in data management that need to be addressed.

 “Every year, poor data quality costs organizations an average $12.9 million”
– Gartner

Data observability is a practice that helps organizations understand the full lifecycle of data, from ingestion to storage and usage, to ensure data health and reliability. Data observability involves monitoring and managing the internal state of data systems to gain insight into the data pipeline, understand how data evolves, and identify any issues that could compromise data integrity or reliability. At its core, data observability is about ensuring the availability, reliability, and quality of data.

Data observability is crucial to analytics and automation, as business decisions and actions depend on data quality. In the age of AI, data observability has become foundational and complementary to AI observability, data quality being essential for training and testing AI models.

Dynatrace now addresses many of the issues customers experience around the health, quality, freshness, and general usefulness of data that is externally sourced into Dynatrace Grail™, allowing them to make better-informed decisions and optimize their efforts for digital transformation and data-driven operations.

The rise of data observability in DevOps

Data forms the foundation of decision-making processes in companies across the globe. Data is the foundation upon which strategies are built, directions are chosen, and innovations are pursued. Consequently, the importance of continuously observing data quality, and ensuring its reliability, is paramount. Surveys from our recent Automation Pulse Report underscore this sentiment: 57% of C-level executives say the absence of data observability and data flow analysis makes it difficult to drive automation in a compliant way. This not only underscores the universal significance of data, it also hints at its pivotal role within DevOps. For DevOps teams that inform deployment strategies, optimize processes, and drive continuous improvement, the integrity and timeliness of data are of significant importance.

As organizations scale and accelerate their digital transformation journeys, a major hurdle to proper DevOps adoption is the trustworthiness of the massive volume of data coming from various sources, much of which goes into data silos such as log management tools, SIEM solutions, and others.

The rise of data observability needs is where Dynatrace capabilities around Grail, analytics, and Davis® AI are in an outstanding and unmatched position to deliver the currently missing value to the market: a leading and single solution for all data observability analytics needs. This reduces the demand for further data flow analysis tools and clears any hurdles to making data useable for DevOps automation use cases.

Davis AI, Grail, and data observability

By grouping common data observability issues into industry-standard pillars, we can provide tangible examples and showcase current capabilities. The five pillars we focus on are freshness, volume, distribution, schema, and lineage.

Freshness: Timeliness of data

In an ideal ecosystem, actionable data should be as recent as possible, supported by learnings from accurate, historical data. Observing the freshness of data helps to ensure that decisions are based on the most recent and relevant information.

Scenario: Due to an undetected configuration issue, a flight status system from a popular airline had been buffering data for the last two hours before sending it on in one batch. Downstream dashboards and system automations were using outdated data, leading to incorrect statuses of flights in reports.

Solution: After setting up data ingestion into Grail, Dynatrace Query Language (DQL) is used to add a freshness field (Figure 1) which is calculated from the delta between when the signal was written and when it was ingested. This freshness measurement can then be used by out-of-the-box Dynatrace anomaly detection to actively alert on abnormal changes within the data ingest latency to ensure the expected freshness of all the data records. Furthermore, the new Alert on missing data feature in the Anomaly Detector panel can be used to trigger notifications when data is not coming in as expected after being baselined.

Value: The possibility of alerting on data freshness issues, based on a learned baseline through Davis AI, allows for faster time-to-detect where there are seemingly no infrastructure issues. Normally this would have left an issue undetected for much longer, providing a false sense of security, eventually leading to a much bigger customer and monetary impact for the organization.

Volume: Quantity of data generated or processed within a given timeframe

Unexpected increases or drops in the volume of data are often a good indication of an undetected issue.

Scenario: For many B2B SaaS companies, the number of reported customers is an important metric. It heavily influences downstream reports, and dashboards, shaping decisions from daily operations to strategic monthly reviews. In this scenario, a manually triggered run of a production pipeline had the unintended consequence of duplicating the reported customer metric. If left unchecked, this misrepresentation of a single KPI could lead to misguided decision-making processes through multiple layers of the organization.

Solution: Like the freshness example, Dynatrace can monitor the record count over time. Once a DQL query has been set up, it can be used in an automation workflow (Figure 2) where scheduling, prediction, comparison to actual value, and, finally, alerting are all taken care of to enable a fully flexible way to detect anomalies in data volume.

Value: KPIs and metrics such as the number of reported customers are central to an organization’s business and strategic processes. Any issues here will result in a loss of trust in the data, and, if left undetected, they will eventually lead to monetary impact, including loss of reputation for an organization.

Distribution: The statistical spread or ranges of data

The distribution of data is essential in identifying patterns, outliers, or anomalies in the data. Deviation from the expected distribution can signal an issue in data collection or processing.

Scenario: A financial institution processes millions of transactions daily, ranging from credit card purchases and mortgage payments to interbank transfers and ATM withdrawals. An erroneous change in the database system leads to a subset of the data being categorized incorrectly. After several days, the fraud detection system starts triggering on a frequent basis, and liquidity management dashboards begin showing questionable values.

Solution: Baselining and raising alerts on anomalies are core capabilities of Davis AI. After setting up ingestion for the data that you want to monitor, it’s simple to use Dynatrace full AI capabilities to observe and alert on any anomalies in the data. In the example above, ingesting the number of transactions as business events, anomaly detection could be based on this to proactively alert and trigger mitigation activities.

Value: While variations are expected in financial trends, anomalies should be auto-detected, and manual detection should not be relied on. Earlier detection of these issues will keep the fallout as low as possible.

Schema: Structure and relationships of data between entities

Observing the schema can help identify and flag unanticipated changes, such as the addition of new fields or deletion of existing fields.

Scenario: An externally connected database system made an update that inadvertently dropped the account_id column in the customers table. The automated data pipeline propagated these changes, leading to downstream reports, dashboards, and applications breaking as the previous field reference is now missing.

Solution: Using the DQL FieldsSummary command, we can keep track of the number of distinct field keys within a given family of data records. Once confirmed in a notebook, the number of field keys can be used in an automated workflow to continuously monitor the count and write it back to a new metric (Figure 3). Once the new metric is established, out-of-the-box Dynatrace anomaly detection can be used to alert on either a static threshold or a learned baseline.

Value: Observing incoming data Schemas, and thus placing expectations on what the external data should look like and must contain, allows for pro-active alerting and mitigation of issues long before they can lead to widespread business impact such as broken reports, dashboards, or further analytics on top of the data.

Lineage: Journey of data through a system

Data lineage provides insights into where the data came from (upstream) and what is impacted (downstream). It plays a crucial role in root cause analysis as well as informing impacted systems about an issue as quickly as possible.

Scenario: The hourly_consumption table was deprecated and removed by an overzealous database administrator as there were no known downstream consumers of this data, breaking a monthly integration check used for consumption reporting for shareholders.

Solution: In the future, Dynatrace Smartscape® could be used, which already builds a dependency graph, to enable a data lineage view. This would enable faster root cause analysis of any data-related problems, as well as allow for easy notification of downstream consumers who would be impacted.

Value: A proper understanding of the source of the data, as well as where it is used, helps drive down time-to-alert and time-to-repair. Time-to-alert is achieved by quickly and automatically alerting those who are impacted by a data issue by quickly understanding downstream consumers of the data, while time-to-repair informs on the source of where the data originated from, to quickly drill down into those systems.

Data availability: A prerequisite

You could implement the most contemporary, accurate, and useful data observability solution possible, but what good will it be if all the data simply does not arrive as expected? Broken pipelines or missing data sources would mean that there is simply no data to observe and that data may never arrive, forever lost.

A truly valuable data observability solution should be able to alert on data issues as early in the process as possible. This requires monitoring of the upstream infrastructure, applications, or platform supporting those data streams. This is where the power of Dynatrace end-to-end observability comes into play. Dynatrace can leverage existing Infrastructure Monitoring and Application Observability solutions to surface problems that can affect later data observability workstreams—long before a traditional data observability solution would pick up the issue.

Leverage the power of Dynatrace and Davis AI—now and into the Future

Anomaly Detection

Anomaly detection is grounded in the idea of baselining typical patterns of ingested data, designed to alert where a change or deviation from the norm is observed. These patterns typically go beyond simple flat or trend lines, often exhibiting complex seasonal behaviors, such as business hours or weekly patterns related to the industry. Dynatrace is particularly strong in this area: Davis predictive AI has been enriched over the years with a set of advanced machine learning (ML) algorithms optimized for time-series observability datasets to cope with these challenges. Davis AI anomaly detection, leveraging these ML algorithms, can already be used on the results of DQL queries. (Embedding ML algorithms into DQL as functions is on the Dynatrace platform roadmap.)

Considering the examples and solutions provided above, anomaly detection plays a pivotal role in numerous data observability use cases and can be harnessed to effectively address these challenges.

Triage and resolution of a data incident

Triaging requires an ability to identify the root cause of a data incident, which is particularly challenging as an organization scales up the volume and speed of data ingest typical of an enterprise environment. It’s easy to see how Davis causal AI problem detection could be extended in the future to identify root-cause data observability issues.

Depending on the incident, there might be different paths to resolution. One acceptable path could be full auto-remediation, whereby Dynatrace AutomationEngine could be triggered, scripts executed, permissions granted, security checked, and data corrected. A second path might require Jira tickets to be created and human intervention through an approval process. A data problem alert could be used as the event allowing for multiple methods to notify the correct data owners, stewards, governors, or data teams.

An incident requires not only resolution but also understanding and alerting upstream data providers and downstream data subscribers to the potential impact. Dynatrace is strong on the observability of data pipelines ingesting data into Grail and consumers of Grail data, although this is an area that will be enhanced and improved in the future product roadmap.

Prevention of future incidents

Not all data quality incidents can be prevented, especially because ELT/ETL data pipelines typically tend to grow over time and span many different heterogeneous collectors that have different ownerships. There are, however, mitigation techniques you can use, for example:

• Health tracking of key datasets or streams over time—alerting on anomalies
• Monitoring standard query results and changes over time
• Well-designed, data-focused dashboards for monitoring
• Auto remediation where appropriate with built-in audit logging
• Forensic abilities for ad-hoc data analysis

Summary

Dynatrace is uniquely positioned to provide even more value by extending our world-class observability platform into the data observability realm. To achieve this, we leverage Infrastructure Monitoring and Application Observability for early warnings on data pipeline issues and use DQL, Workflows, and Grail for data observability—all enabled by our best-in-class Davis AI engine.

Ensuring the quality and reliability of underlying data is more crucial than ever now that many organizations are deploying Generative AI models. Data observability is becoming a mandatory part of business analytics, automation, and AI. Davis AI and data observability together uniquely ensure the quality and reliability of data at the level of hypermodal AI—predictive, causal, and generative.

You can now monitor sources and incoming data pipelines for freshness, volume, distribution, lineage, and availability issues early on without added noise and in a central location, the Dynatrace platform. This gives your teams additional confidence over data quality, saves time, prevents inaccurate analyses and automation outcomes, leads to more trustworthy AI models, and supports efforts to consolidate or reduce the number of IT tools they rely on.

Ready to get started with Dynatrace data observability? For complete details, best practices, and detailed use cases, see Dynatrace data observability documentation.

The post Introducing Dynatrace built-in data observability on Davis AI and Grail appeared first on Dynatrace news.

Without observability, the benefits of ARM are lost

Over the last decade and a half, a new wave of computer architecture has overtaken the world. ARM architecture, based on a processor type optimized for cloud and hyperscale computing, has become the most prevalent on the planet, with billions of ARM devices currently in use. This growth was spurred by mobile ecosystems with Android and iOS operating systems, where ARM has a unique advantage in energy efficiency while offering high performance.

While ARM processors have been humming in consumers’ pockets for over a decade, enterprise IT adoption has been slower. Legacy data center infrastructure and software support have kept all the benefits of ARM at, well… arm’s length. Still, we at Dynatrace speak to customers who recognize this value and want to implement it.

Energy efficiency and carbon footprint outshine x86 architectures

The first clear benefit of ARM in the enterprise IT landscape is energy efficiency. This is a crucial factor for all data centers, cloud or managed, where power consumption and cooling costs are reflected in the total cost of ownership. Traditional x86 architecture needs more power, so switching to ARM can offer a clear advantage.

Energy efficiency is coupled with the total enterprise carbon footprint. As organizations look to take ownership of their total ecological footprint and help mitigate climate change, it’s critically important for organizations to measure, monitor, and reduce their IT carbon footprints. Initiatives like the Carbon Impact app can be used to measure the footprint of monitored ARM-based hosts compared to x86 hosts.

Huge performance leaps in recent years

The top priority is often performance, where ARM resources have improved significantly. You would have been unimpressed if you evaluated public cloud vendor ARM-based offerings five or more years ago. But take a look at the latest iterations of, for example, AWS Graviton2, which delivered a 40% price/performance boost, and Graviton3, which had an additional 27% price/performance improvement over Graviton2.

These Improvements and other optimizations show that mature service offerings can rely on ARM architecture. You’re no longer required to use a single offering or choose from a few instance families; Graviton includes general-purpose and accelerated-computing offerings, plus compute-, memory-, and storage-optimized instances.

No observability, no gains

All these factors have made ARM an attractive computing architecture for innovative companies. However, the lack of end-to-end integrated observability with full support for log collection is a clear blocker for many organizations looking to adopt ARM-based services.

Without an observability platform to collect and process signals, and provide AI-powered answers for problem detection, root cause identification, and impact analysis, the migration to ARM remains only a roadmap item for many AIOps teams.

Even if some part of your codebase can be instrumented to collect observability data, having all three signal types (metrics, traces, and logs) is crucial. Without collecting logs from the observed platform in a scalable AI-powered data lakehouse like Grail, it’s more of a challenge to identify the root cause of problems and provide details for troubleshooting or security incidents.

Having no access to logs or relying on a siloed approach to logging poses a risk of blind spots in your IT landscape, prolonged outages, and an increase in Mean Time To Identify (MTTI) for incidents, all of which can nullify the benefits of ARM.

Hassle-free ARM deployment with automated log collection

With automated log observability, the Dynatrace platform enables your AIOps team to reap the full benefits of ARM architecture. Dynatrace OneAgent can automatically detect common logs for ARM-based hosts and ingest them on a massive scale out of the box.

As soon as you install just a single OneAgent on a host, for example, an ARM64 (AArch64) based Linux host, the OneAgent scans and autodiscovers logs every 60 seconds.

After installation, you can control OneAgent behavior using a powerful central configuration mechanism in your Dynatrace environment. You can tune the granularity of OneAgent filtering for each host to meet your requirements for discovery and ingestion, share configuration details with other hosts in the same host group, or apply global settings for your whole environment.

As OneAgent collects logs from your ARM-based host, it automatically ties the discovered data to your environment topology. In this way, log data is always associated with the host, service, or other entity that generated it.

AI-powered problem detection can then connect the log data, which often contains the source of truth, to detected issues in your environment, thereby speeding up troubleshooting and remediation by orders of magnitude compared to manual correlation.

In addition, the integration with auto-baselined metrics and automatic connection between distributed traces and logs puts you in complete control of the software you run on ARM-based architectures.

Immediately see the root cause in ARM host logs

You can enable the full value of Dynatrace with log monitoring on ARM in just a few steps.

1. Install OneAgent version 1.269+ on an ARM host (Go to Apps In the Dynatrace web UI and search for Deploy OneAgent to access the installer). For full details, see OneAgent installation on Linux.
2. Following OneAgent installation, you can verify on the host page that the ARM 64-bit instruction set is used.
   
   Next, go to Settings > Log monitoring > Set up log ingest and review the log ingestion rules in the Hosts section. For the simplest quickstart, select Show rules.
   
3. Turn on the rule [Built-in] Ingest all logs to enable all rules and gain complete visibility into the host.
   

Now you have the additional value of log data informing your Dynatrace root cause analysis should a problem arise on this host.

In the following example, a synthetic monitor is set up for webpage uptime monitoring. Dynatrace Davis^® AI has automatically discovered a problem on the host, and the root cause analysis points to a PHP web service as the source of the problem.

The problem page gives you a direct link to the service causing the disruption, and you can quickly access all related logs.

Error logs for the process reveal a PHP syntax error with code-level information that caused the outage.

As you can see, with the Dynatrace AI-powered observability platform at your disposal, you can benefit from resource-efficient ARM architectures in your organization without the risk of blindspots or guesswork when you encounter incidents or outages.

Start monitoring ARM and see what’s next

• New to Dynatrace? Open a free trial environment right now and see Dynatrace in action.
• See the OneAgent support matrix for your operating system and deploy OneAgent in your ARM environment today.
• Recently released improvements to the central configuration of all your OneAgents now include:
  • Renamed advanced log configuration pages for simpler navigation
  • Log filtering based on log level
  • Enforced auto-timestamping for logs, in case your log source includes varied log formats that prevent correct timestamp recognition
• Browse the Dynatrace Hub to see how you can bring in logs from different sources and technologies, for example, ingesting Syslog via OpenTelemetry Collector and Oracle log monitoring.
• Share your feedback and expectations with us in the Dynatrace Community

Coming next

We’re working on making Dynatrace log monitoring even easier to use at scale with a focus on areas highlighted by you and other customers:

• Are you ingesting logs across multiple sources with a wildcard (*) ingest rule? Soon, you can enhance log records with custom file names or log path attributes for later analysis in Dynatrace.
• Get automatic log ingest recommendation rules relevant to your environment based on the log sources discovered by OneAgent.
• Tools to troubleshoot the status of your log sources and instructions for resolving issues in cases where a log file is inaccessible.
• See the complete list of log sources on each host page, even if the sources are tied to another process group.

The post Dynatrace log collection for ARM unlocks power-efficient architecture for your enterprise appeared first on Dynatrace news.

Streamline privacy requirements with flexible retention periods

Data retention is a critical aspect of data handling, and it’s not just about privacy compliance—it’s about having the flexibility to optimize data storage times in Grail for your Dynatrace use cases. Most observability solutions rely on fixed, default retention periods, or provide only limited configurability with pre-defined retention periods to choose from (number of days, weeks, or months), or require you to set up rules to archive and extract logs.

Holding onto personal data beyond the minimum required timeframe can introduce privacy risks, inefficiency, and excessive storage costs. Implementing clear data deletion policies is also closely linked with the legal requirement of personal data minimization. The core privacy requirement to consider when defining retention periods is to keep data no longer than required for a specific purpose. While this might seem straightforward, the challenge lies in determining precisely what this means in diverse scenarios. Deciding on the correct duration for retaining data is context-specific; it depends on the nature of the data, its intended use, legal requirements, and your evolving business needs.

For example, log data, which can include personal data, should have a shorter retention period when used to troubleshoot and debug performance issues in your testing environment (compared to log data used for audit logs) because the data might not be needed after a few days. Fixed or predetermined timeframes lack the adaptability needed for diverse data management scenarios. Archiving options fail to provide a dynamic, user-friendly solution for tailoring retention periods.

Customize retention periods with buckets in Grail

You can define retention periods in custom buckets for all the data you store in Grail. Grail buckets function like folders in a file system. Each bucket should contain only those records that should be handled together as a set. You can learn more about custom bucket retention periods in our recent blog post, which explains how to enhance data management and includes best practices for setting up buckets with security context in Grail. Besides improving query speed, reducing related costs, and enabling organizations to target a specific business unit or production stage, custom buckets enable you to segment your data for specific use cases.

Segmentation of use cases establishes the perfect landscape for tailoring retention periods to the purposes for which data is collected. You might have specific data retention needs based on your industry and use cases. For example, industries such as finance and healthcare have specific regulations that dictate audit log retention periods ranging from months to several years.

With Dynatrace, you have full control over Grail retention periods—you choose how long to store each portion of data in line with your requirements. You can select the desired retention period for your data in bucket configuration settings, with the available retention periods ranging from 1 day to 10 years, plus an additional week (3,657 days). A hard deletion process is initiated when there’s no legal or business reason to continue retaining data in alignment with your configuration settings. The system reviews the timestamp of all data segments to see if any have reached the end of their specified retention period. Data that has exceeded its retention period is flagged for deletion, and the data is permanently removed from Grail, making it unrecoverable.

The Storage Management app makes it simple to set data retention periods

While the Grail bucket management API is a powerful approach you can apply to your data management policies in an enterprise setting at scale, a simpler method is also available. The Storage Management app on the Dynatrace® platform lets you manage all custom buckets for Grail in a visual user interface. This makes the data retention rules easy to grasp and implement even for non-technical users and auditors.

The Storage Management app provides an intuitive overview of Grail buckets for different types of data you store with Dynatrace (for example, logs and business events. Other data types will be available soon). For each type you can inspect existing buckets along with their defined retention periods and current status. The app also provides a quick way to extend or shorten a retention period for existing buckets, rename buckets, add new buckets, or delete buckets to completely wipe the data they contain.

What’s next?

• Sign up for a Dynatrace free trial account to start using Dynatrace today.
• Start managing your Grail buckets with the Storage Management app, to be released with Dynatrace version 1.281.
• Explore our documentation, which explains data retention periods and how to manage custom Grail buckets.
• Learn more about our commitment to providing you with control and transparency over your customers’ personal data in the Dynatrace Trust Center.

The post Privacy spotlight: Retain data in Grail with 1-day precision to better meet your compliance requirements appeared first on Dynatrace news.

VMware Aria Operations for Logs (formerly known as vRealize Log Insight) is used across enterprises to collect logs and provide analytics. The company recently announced a high-severity vulnerability in an earlier version of the tool. Tracked as CVE-2023-34051 (CVSS score: 8.1), the Aria Operations for Logs vulnerability relates to a case of authentication bypass that could lead to remote code execution. Basically, this means that an unauthenticated malicious actor could inject files into the operating system of an impacted appliance, which can result in remote code execution.

James Horseman from http://Horizon3.ai and the Randori Attack Team have been credited for the discovery and reporting. They have made a PoC for the vulnerability available and published the relevant indicators of compromise (IoC).

This vulnerability is a so-called “patch bypass” for a set of critical flaws that were addressed by VMware earlier this year, also discovered by Horizon3.ai. In their report, they presented how an attacker could use three different CVEs to achieve remote code execution. Since the patch only blocks access to Thrift services by IP and does not fix the other CVEs in VMSA-2023-0001, all an attacker needs to do is spoof an IP address and use the published attack again.

In this blog post, we show how to discover the original attacks toward the Aria Operations for Logs vulnerability using Dynatrace and DQL by finding the IoC-s from the log records.

How to detect VMware Aria Operations Logs exploitation with Dynatrace

Since we’re talking about a patch bypass, we’re looking at the original attack vector described in this blog post. To exploit this vulnerability to gain remote code execution (RCE), the following steps have to be taken by the malicious actor:

1. Create a Thrift client to gain unauthenticated access to the Log Insight Thrift server.
2. Spoof the IP address of the known worker.
3. Create a malicious TAR file containing a directory traversal using a valid PAK file.
4. Using remotePakDownloadCommand, upload the malicious PAK file to /tmp/<filename>.pak.
5. Extract the file using pakUpgradeCommand. This writes the file to where it’s needed on the filesystem.

A technical deep dive about the attack can be found in this article, VMware vRealize Log Insight VMSA-2023-0001 Technical Deep Dive.

Discovering the attack with Dynatrace

To discover this attack, you have to go through the log files that Aria Operations for Logs creates. The Log Insight server stores relevant logs in the file /var/log/loginsight/runtime.log. This log file is used to track all runtime information related to Log Insight. To analyze these logs with Dynatrace and DQL, the logs need to be ingested into Dynatrace beforehand.

Forwarding logs away from the main system is beneficial in case of any kind of successful attack by a malicious actor: if an attacker decides to remove all evidence from the system (including log files), then a security investigator wouldn’t have traces left to analyze. Sending logs to Dynatrace at runtime safeguards the log files for later analysis in a remote and secure location.

Security-related log records of Aria Operations for Logs are structured in the following format:

[2023-10-25 11:28:29.709+0000] ["https-jsse-nio-443-exec-9"/10.153.234.136 DEBUG] [com.vmware.loginsight.web.actions.misc.LoginActionBean] [User login success: vIDM: SAM=myusername, Domain=vmware.com, UPN=myusername@vmware.com]

When a new remote PAK file is downloaded as a tarball, the following log lines are created:

[com.vmware.loginsight.daemon.commands.SystemCommands] [PAK download initiated by node f2449ed5-11ee-45fd-a0a0-6225a33a8ac6] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Deleting existing PAK file: /tmp/exploit.pak] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak] 
[com.vmware.loginstght.daemon.commands.SystemCommands] [Max allowed pack size is 1505916450 bytes] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Current downloaded size is 21910]
[com.vmware.loginsight.daemon.commands.SystemCommands] [Total downloaded size is 21910]

From these log lines, the best candidate to look for in the logs is the third populated log, Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak].

Now, let’s fire up the DPL Architect and create a suitable pattern for this line:

With the created DPL pattern, you can extract the timestamp field ts, the logger class name as class that populated the log lines and the action that was written to the log record. These extracted fields give you the option to filter out the relevant records:

However, there’s another goal to keep in mind: you need all the log records that contain the Downloading statement and URL from where the tarball was loaded and the destination path in /tmp/.

As the blog post states: “The URL and filename will likely be different. However, the filename will always have the format /tmp/<filename>.pak. Also, note that this log may be legitimate. To determine if an attack has occurred, you need to have your administrator evaluate the URL to determine if it is a legitimate download URL.”

Let’s continue with our DPL pattern and extract the full URL where the payload was downloaded and the relevant filenames. Again, we’re opening up the DPL Architect and creating the following pattern:

'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file

With the created pattern, we’re extracting the url and the tarball’s name as download_file into separate fields using the look-around functionality of DPL. Additionally, we’re extracting the filename from the destination file path as well as the whole saved_file value for simpler analysis.

DPL pattern extracts the following result set from the log record:

Applying this pattern and filtering the records as described in the blog post gives you the following DQL query:

fetch logs
| filter log.source == "/var/log/loginsight/runtime.log"
| parse content, """'[' timestamp('yyyy-MM-dd HH:mm:ss.SZ'):ts '] [' ld '] [' ld:class '] [' ld:action ']'"""
| filter class == "com.vmware.loginsight.daemon.commands.SystemCommands"
| parse action, """'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file"""
| filter download_file == filename
| fields ts, url, saved_file

The query produces the following result, which you can now pass on to your administrator for manual evaluation.

A longer description of how the VMware Aria Operations for Logs exploitation works can be found in the Horizon3 blog post.

Further reading about how to use DPL Architect for security use-cases can be found at Speed up your security investigations with DPL Architect.

How to mitigate the Aria Operations for Logs vulnerability exploit

To mitigate this exploit, update to the latest version of VMware Aria Operations for Logs by applying the latest patches. Or follow the instructions within the VMSA-2023-0021.

Sources

1. VMware vRealize Log Insight VMSA-2023-0001 IOCs
2. GitHub – horizon3ai/CVE-2023-34051: VMware Aria Operations for Logs CVE-2023-34051
3. VMware Aria Operations for Logs CVE-2023-34051 Technical Deep Dive and IOCs
4. VMSA-2023-0021
5. Alert: PoC Exploits Released for Citrix and VMware Vulnerabilities

The post Detect VMware Aria Operations for Logs exploitation with Dynatrace and DQL appeared first on Dynatrace news.

To help you raise the quality of your investigation results, Dynatrace offers an easy way of structuring data using DPL Architect. This tool lets you quickly extract typed fields from unstructured text (such as log entries) using the Dynatrace Pattern Language (DPL), enabling you to extract timestamps, determine status codes, identify IP addresses, or work with real JSON objects. This allows you to answer even the most complex questions with ultimate precision. The best thing: the whole process is performed on read when the query is executed, which means you have full flexibility and don’t need to define a structure when ingesting data.

>> Scroll down to see Dynatrace DPL Architect in action (24-second video)

Investigating log data with the help of DQL

Let’s look at a practical example. The simplest log analysis use cases can be solved in seconds by applying a simple filter to the query results. If a CISO asks, “Have we seen the IP address 40.30.20.1 in our logs within the last year?” a simple DQL query seems to suffice:

fetch logs, from: -365d
| filter contains(content, “40.30.20.1”)

This query returns all the records that contain the string value 40.30.20.1.

In the next step, adding DQL aggregation functions enables you to answer more complex questions like “How many times in an hour has this IP address visited our website within the last two weeks?” Again, a simple DQL query helps you out:

fetch logs, from: -14d
| filter contains(content, “40.30.20.1”)
| summarize by: bin(timestamp, 1h), count()

However, this kind of simple filtering using a sub-string search is prone to errors and is not precise enough. The mentioned filters return all records that contain 40.30.20.1, but are not necessarily from the source IP portion of the log format. The string 40.30.20.1 might also appear elsewhere, for example, in query parameters, and a simple string filter will return all records containing this search term:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395
40.30.20.109 - - [28/Aug/2023:10:22:04 -0300] "GET / HTTP/1.1" 200 2216

The first log record is matched because of its query parameter—something we don’t care about in our case. The second log record came up because the source IP contains the IP address—this is what we’re really looking for. In a nutshell, if you’re querying terabytes of logs and wish to drill down to specific records, filtering strings from plaintext log content is not enough.

The issue is that questions from CISOs aren’t usually so trivial when it comes to security use cases. Or, the log format where the answers should be looked for is more complex (for example, AWS CloudTrail logs). As a consequence, we need to search structured data to avoid mistakes. To minimize false positives, increase the precision of queries, and get the maximum out of DQL, fields can be extracted from record content.

Extracting patterns using DPL

Dynatrace Pattern Language (DPL) is a pattern language that allows you to describe a schema using matchers, where a matcher is a mini-pattern that matches a certain type of data. These matchers can be used to extract new fields from the schemaless data in Grail to enable more precise log filtering. Consider the same Apache access log example from above:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395

To extract the client IP addresses from the beginning of the log, a simple DPL pattern can be used:

IPADDR:client_ip

The IPADDR matches both versions of IP addresses (IPv4 addresses in dot-decimal notation and IPv6 addresses in hextet notation), leaving all other IP-like strings unmatched. For example, if the log record contains a dot-decimal number that is NOT an IPv4 address (999.999.999.999), it won’t be matched.

DPL patterns can be applied in DQL using the parse command. The simplest way to get started with DPL is to use Dynatrace DPL Architect.

DPL Architect to the rescue

DPL Architect is a handy tool, accessible through the Notebooks app, which supports you in quickly extracting fields from records. It helps create patterns, provides instant feedback, and allows you to save and reuse DPL patterns, for faster access to data analytics use cases.

To open the DQL Architect, you have to execute a DQL query, select the content, and choose Extract fields.

Figure 1: Extract fields in Notebooks using DPL Architect (24-second video)

Starting with preset patterns

The simplest way to extract data is using one of the ready-to-use preset patterns available for the most popular technologies, such as AWS, Microsoft, or GCP. Start exploring AWS VPC Flow logs or analyzing Kubernetes audit logs by choosing the patterns from the panel on the left side. You can also customize the list by adding your own individual patterns.

Developing a new pattern using DPL Architect

DPL Architect helps you by creating your own patterns and provides instant feedback. Start typing a DPL expression in the pattern field, and review the matching data (highlighted below) in the preview editor.

For accessing the extracted fields in the resultset, it’s necessary to add a name. In the example below, we’re only interested in the client_ip and the related response_code. The pattern matches the whole record, but only two fields are being extracted since they have defined extract names. Any other data between those two fields won’t be visible in the results (however, it is very easy to add them later if necessary).

Remember, we started our journey with the DPL Architect by selecting query results in a notebook. The colored bar at the top of the DPL Architect shows how many records from your query result match the current DPL pattern. This enables you to modify the DPL pattern to ensure it matches all records in the resultset.

Records that don’t match the current pattern can easily be added to the Match preview panel by selecting Add to preview.

After changing the DPL pattern, any previously unmatched records will be highlighted and the progress bar on the top informs you that 100% of the records in the Notebooks resultset are matched. It’s now time to insert our pattern into the DQL query by selecting Insert Pattern.

Precise extraction of fields from complex data

DPL also provides matchers for more complex data structures, like key-value pairs, structures, and JSON objects, that enable you to parse individual sub-elements from the whole object. Consider the following log record:

1693230219 230.4.130.168 C "GET / HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" 186 {"username":"james","result":0 }

Imagine your CISO comes with a request: “Give me the list of all IP addresses where the user ‘james’ has successfully logged in from.” Extracting these three fields and building a DQL query is as easy as pie:

LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)

The DPL will extract three fields: an IP address from the record, a string from the JSON object element username, and an integer from the JSON element result.

Summarizing the results based on these fields can be done with the following DQL:

fetch logs
| parse content, "LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)"
| filter username == "james" and result == 0
| summarize by: src_ip, count()

Imagine that you don’t have DPL available and need to filter all log records based on only searching for string values: the result would contain a lot of false positives (which only add “noise” to time-critical investigations). This is especially true with more complex log records containing nested JSON objects. The above example was oversimplified intentionally, but considering complex log records like AWS CloudTrail log records, having precise access to data in a specific node of an object makes a huge difference.

Summary

When performing security investigations or threat-hunting activities, it’s important to have precision in place to get reliable results. Historical data needs to be available, and access to object details is required for precise answers. DPL Architect enables you to quickly create DPL patterns, speeding up the investigation flow and delivering faster results. With the possibility of using Dynatrace-provided patterns for selected technology stacks, investigators can deliver answers even faster!

Check out the following video, where Andreas Grabner and I teamed up for a new episode of Dynatrace Observability Clinic. In this video, we dig deeper into the topic of extracting data via DPL, including a live demonstration of DPL Architect.

Referral links

• Documentation: Dynatrace Pattern Language
• Documentation: Dynatrace Pattern Language Architect
• Blog post: Dynatrace Grail: The data lakehouse for observability, security analysis, and automation
• Blog post: Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail

The post Speed up your security investigations with DPL Architect appeared first on Dynatrace news.

Grail: Enterprise-ready data lakehouse

Grail, the Dynatrace causational data lakehouse, was explicitly designed for observability and security data, with artificial intelligence integrated into its foundation. We’ve further enhanced its capabilities to meet the high standards of large enterprises by incorporating record-level permission policies.

To fully utilize Grail features, it’s recommended that you incorporate its unique buckets and security policies at the beginning of your observability journey. Utilizing built-in mechanisms and customizing organization-specific policies maximizes the benefits of Grail capabilities.

Custom data buckets for faster queries, increased control, and custom retention periods

>> Scroll down to the bottom of this blog post to view a ~7-minute video demonstration of custom data buckets.

The first layer in the Grail data model consists of buckets and tables (and views for entities, which is outside this blog post’s scope).

Tables are a physical data model, essentially the type of observability data that you can store. Buckets are similar to folders, a physical storage location.

There is a default bucket for each table. Here is the list of tables and corresponding default buckets in Grail.

The default buckets let you ingest data immediately, but you can also create additional custom buckets to make the most of Grail.

Address specific use cases with custom buckets

It’s logical to segregate high-volume data into its own bucket. This allows the data to be frequently queried and used separately from other scenarios.

For example, a separate bucket could be used for detailed logs from Dynatrace Synthetic nodes. Debug-level logs, which also generate high volumes and have a shorter lifespan or value period than other logs, could similarly benefit from dedicated storage. Keeping these logs separate decreases the data volume for other troubleshooting logs. This improves query speeds and reduces related costs for all other teams and apps.

Address organizational structure with custom buckets

Depending on your organization’s structure, you may find it beneficial to keep logs used by specific business units or departments in separate buckets. This approach makes queries faster for individual units, as they only query relevant logs, and it ensures distinct access separation.

Suppose a single Grail environment is central storage for pre-production and production systems. In that case, the use of separate buckets makes it easier to distinguish different stages, keeping production data separate from development or staging data.

Adopting this level of data segmentation helps to maximize Grail’s performance potential. If your typical queries only target a specific use case, business unit, or production stage, ensuring they don’t include unrelated buckets helps maintain efficiency and relevance.

Custom buckets unlock different retention periods.

Segmenting your data into multiple buckets also puts you in control of the data retention period. The simplicity of storing data in Grail is reflected in its retention policies; you choose how long to store each portion of your data, and you never have to think about managing archives or retrieving archived data. Coupled with the transparent pricing of GiB/day, you can set up buckets to exactly match your business needs.

While the built-in default_logs bucket has a retention period of 35 days, you have more options to choose from. Use Grail’s public bucket management API to create new buckets for which you can select data retention periods of 1 day to 10 years , or use the new Storage Management app to that with just a few clicks.

In conjunction with the previous example of keeping high-volume and short-lived logs separate, you might also need to keep your application data longer. For example, transaction data and user-profile logs might need to be retained for 12 or 18 months.

Use buckets to query only the log data you need

Whether looking for a “needle in a haystack” or reporting on data stored in Grail, you can start an advanced query with DQL by fetching data from one of the tables. At this point, you should familiarize yourself with the blog post Tailored access management, Part 2: Onboard users to Grail and AppEngine, which covers access to Grail tables and buckets.

Now, let’s take a look at a query example that puts this all into use.

fetch logs

| filter loglevel=="ERROR"

In this example, we query a certain table (logs) and filter the results by a field (loglevel) with a certain value (ERROR). Note that with such a query, you fetch logs from all the buckets the end-user can access.

Although this initially only includes the default bucket, you might also include other buckets (if these are available to the user). As you bring in more data and users to Grail, relying just on the default buckets is not the optimal setup.

This is where filtering on custom buckets comes in. This allows you to query data from a specific bucket.

fetch logs

| filter dt.system.bucket=="prod_infra_logs" and loglevel=="ERROR"

This example now includes an additional filter that restricts data retrieval from a certain bucket (prod_infra_logs).

Using buckets to query only the data you need significantly speeds up queries and reduces query costs.

Buckets for data with high-security requirements

Buckets can also be used for managing high-level access control of data.

You can store access control logs or payment provider events in separate buckets to grant only limited SecOps or business administrators access to these logs.

Grail and AppEngine’s new policy-based access management provides a way to do this with buckets. For example, by adding a WHERE clause to the policy statement, you can define a specific bucket (=) or a range of buckets (STARTSWITH). In this example, the policy grants access to all buckets that have names starting with prod_infra_.

ALLOW storage:buckets:read WHERE storage:bucket-name STARTSWITH "prod_infra_";

However, creating access policies solely on the bucket and table level is not scalable in a enterprise landscape, as one Dynatrace tenant can have a limited number of custom buckets. Instead, access control based on specific attributes like host groups or team assignments can be achieved using different policies that are based on supported attributes or security context.

Record-level permissions and security context

As covered in the previously linked blog post about access management, Dynatrace Grail brings a new architecture to permissions management. The new approach that uses security policies provides you with new dynamic controls for user authorization.

As using custom buckets opened up a basic approach to access where users could get access to a whole bucket, record-level permissions allow you to take a fine-grained approach.

This means that whenever you run a DQL query to fetch data from Grail, your policy-based access rights are evaluated, and records without defined access are filtered out.

This means your teams’ permissions are not constrained by data management decisions on a bucket level. Let’s say it makes sense to consolidate all short-living app debug logs to a bucket that has a short retention period. With record-level permissions, you can now ensure multiple app owners can see only their data in that bucket.

Another example would be a business unit admin who needs to have access to departmental data across buckets.

Permissions based on DQL fields and security context

To implement this in the Log Management and Analytics context, you can create policies with additional clauses that provide access.

ALLOW storage:logs:read
WHERE storage:k8s.namespace.name="abc"

 AND storage:dt.host_group.id STARTSWITH "org1-";

In this example, the policy allows access to logs that have a certain Kubernetes namespace (abc) and originate from a host group whose name must start with a specific string (org1-).

The list of standard table fields used in security policies provides flexibility for defining individual policies.

When you look at the list of DQL fields, you’ll notice one reserved field, dt.security_context.

You can assign a value to dt.security_context during data ingest for use in a security policy, which is not covered by the list of previous DQL fields. You can explicitly set a value for dt.security_context for some logs, or take the value of an existing field.

In this example, logs from a particular source (dsfm) are enriched with a literal value for dt.security_context (sec-lvl-7) during log ingest. A policy with a specific clause can provide access to only logs with this security context. Security context rule management is available via settings In the Dynatrace web UI and API.

This approach to granular record-level permissions opens up the flexibility needed in enterprise environments. You can craft policies based on existing fields like Kubernetes cluster or namespace, a host or a host group, a service name, or a log source. Or you can use security context for any other use cases, like granting access based on an AWS account, a GCP project, an Azure subscription, a username, or a team name.

Take the first step now

Many organizations have found immediate value in working with logs in Grail. Starting from optimizing their business and opening revenue streams based on data in logs to unlocking real-time insights from observability data and eliminating hours of manual work per process, as the Bank of Montreal did recently. Utilizing the core aspects of Grail, like buckets and permissions, sets organizations on the path to success.

Next steps

• Start a Dynatrace free trial and explore Log Management and Analytics powered by Grail
• Read our documentation explaining buckets, permissions in Grail, security context for logs, and IAM
• Record-level permissions for Grail are generally available with Dynatrace version 1.277. Custom buckets, in addition to bucket and table permissions, are available with Dynatrace version 1.265.
• Use Storage Management app to create and manage custom Grail buckets and unlock custom retention times for your data since Dynatrace version 1.281.

Special thanks to Dominik Punz and Christian Kiesewetter for contributing to this blog post.

Video demo

Watch this 7-minute video to see how you can use custom data buckets to separate use cases, data retention, and access permissions in Dynatrace.

The post Enhance data management with Grail: Ultimate guide to custom buckets and security policies appeared first on Dynatrace news.

Today’s security analysts have no easy job. Not only are cyberattacks increasing, but they’re also becoming more sophisticated, with tools such as WormGPT putting generative AI technology in the hands of attackers. As a result, analysts are turning to AI and TTP-based threat-hunting techniques to uncover how attackers are trying to exploit their environments.

While AIOps with generative AI will certainly empower security teams to mitigate threats faster and with greater precision, attackers will just as certainly utilize the same technology to create novel malware, more convincing phishing campaigns, and uncover high-risk zero-day vulnerabilities quicker.

Not only that, teams struggle to correlate events and alerts from a wide range of security tools, need to put them into context, and infer their risk for the business. But the industry as a whole is still hampered by a ubiquitous tool sprawl to achieve that critical mission under a barrage of alert noise.

In this blog post, we’ll use Dynatrace Security Analytics to go threat hunting, bringing together logs, traces, metrics, and, crucially, threat alerts. We use the power of DQL on Grail to derive high-level attacker tactics, techniques, and procedures (TTPs), which are much easier to interpret and act upon.

TTP-based threat hunting: Tactics, techniques, procedures

At Dynatrace, we don’t want to bombard you with alert noise and uncorrelated warnings. Instead, we want to focus on detecting and stopping attacks before they happen: In your applications, in context, at the exact line of code that is vulnerable and in use. But even when an attack happens, Dynatrace detects and blocks them in real time while providing you with rich technical details on the concrete attack procedure. Procedures describe the specific technical details that an adversary used to carry out an attack, for example, what script they ran to exploit a weakness.

When investigating advanced cyberattacks, it’s helpful to map attack procedures to attack techniques. Techniques describe the tactical goal an adversary is pursuing by executing a specific procedure. One of the most critical attack techniques within the MITRE ATT&CK® knowledge base of adversary tactics and techniques — and one example of what Dynatrace can prevent, detect, and block in real-time — is attack technique T1190, “Exploit Public-Facing Application”.

Public-facing applications can be an initial access vector an attacker could exploit to gain entry into a system. Attack tactics describe why an attacker performs an action, for example, to get that first foothold into your network.

Thinking in terms of tactics, techniques, and procedures (TTPs) brings many benefits. For example, security analysts can more easily stitch together advanced cyberattacks on an abstract level. Likewise, operation specialists can prioritize their efforts on monitoring the highest-risk tactics, and executives can better communicate the business risk.

Threat hunting and analyzing threat alerts with Dynatrace Security Analytics and Grail

Dynatrace offers Runtime Application Protection to detect a wide range of injection attacks in your applications. However, our customers often want to augment the data Dynatrace provides with data from third-party tools. Customers also want to carry out their own analysis tailored to specific use cases and forensic needs.

Dynatrace Grail is a data lakehouse that provides context-rich analytics capabilities for observability, security, and business data. You may also ingest additional data into our unified intelligence platform: One popular choice to gather fine-grained security data is Falco. Falco is an open-source, cloud-native security tool that utilizes the Linux kernel technology eBPF, to generate fine-grained networking, security, and observability events.

In the following sections, we demo the following:

1. Introduce Unguard, our insecure cloud-native microservices demo application.
2. Install Falco in AWS EKS to gather security-relevant events from all the happenings in Unguard.
3. Ingest those Falco events into Dynatrace Grail using falcosidekick.
4. Query Falco events in Dynatrace Grail, map them to TTPs, and conduct structural multi-step attack detection.

In other words, we find attacks that are composed of multiple steps by using TTPs and Dyntrace Smartscape for DQL in a way that eliminates alert noise.

First, Dynatrace OneAgent will automatically monitor and trace our infrastructure and communicate with Dynatrace. Second, we will enrich our data in the Grail data lakehouse by also ingesting Falco events using falcosidekick.

Setting up our TTP-based threat-hunting demo environment

Before we start threat hunting, we’ll first walk through how to set up the demo environment.

Introducing Unguard, our insecure cloud-native demo app

As our playground, we introduce Unguard, a microblogging demo application that embodies the challenges of modern cloud-native environments. It consists of eight services, written in at least four different languages, with countless vulnerabilities and misconfigurations. To keep it real, we have a load generator that creates benign traffic. It also generates OpenTelemetry traces.

Unguard was first introduced at DEFCON 31 by our colleagues Simon Ammer and Christoph Wedenig.

For the demonstration in this blog post, we want to deploy Unguard in AWS EKS and hunt for attacks within that environment. You can easily play around with Unguard by installing its Helm chart:

helm install unguard \ 
  oci://ghcr.io/dynatrace-oss/unguard/chart/unguard \ 
  --wait --namespace unguard --create-namespace

(Please read the Unguard README for detailed and up-to-date instructions)

This demo assumes your Kubernetes cluster is already monitored by Dynatrace. For instructions, see Set up Dynatrace on Kubernetes.

Deploy Falco and falcosidekick in AWS EKS

You can install Falco in various ways. For this demo, we installed it with the Helm chart in our AWS EKS cluster:

helm repo add falcosecurity https://falcosecurity.github.io/charts 
helm repo update 
helm install falco falcosecurity/falco --namespace falco --create-namespace

(See the Falco README for detailed and up-to-date instructions)

Next, we set up falcosidekick, which is a daemon that forwards Falco events to many possible outputs. We’re proud to announce that, with Falco version 2.29, currently in pre-release, you can now also use Dynatrace as an output.

You can use this minimal values.yaml configuration file for the Helm chart:

# values.yaml 
 
falcosidekick: 
  enabled: true 
  image: 
    tag: 2.29.0-rc.1 
  config: 
    # as of 2023-08-02, this feature is still a pre-release so we 
    # have to manually override the environment variables for now 
    extraEnv: 
      - name: DYNATRACE_APITOKEN 
        value: dt0c01.EXAMPLE_TOKEN_REPLACE_THE_ENTIRE_STRING 
      - name: DYNATRACE_APIURL 
        value: https://ENVIRONMENTID.live.dynatrace.com/api

(Please read the Helm chart README for detailed and up-to-date instructions)

We insert the apitoken we generated within Dynatrace and grant the token the scope logs.ingest. See the topic Dynatrace API – Tokens and authentication to learn more about creating tokens. As the apiurl, use the following:

Dynatrace SaaS:

https://ENVIRONMENTID.live.dynatrace.com/api

Dynatrace Managed:

https://YOURDOMAIN/e/ENVIRONMENTID/api

See the topic Environment ID to learn more about environment IDs.

Finally, we update the Falco Helm chart with this new configuration:

helm upgrade falco falcosecurity/falco -f values.yaml

If everything worked out well (check the pod logs otherwise), we are now able to successfully query Falco events with DQL. To verify, we open a new Notebook and see how Dynatrace automatically infers the fields from our events already:

fetch logs, from:now() - 5m 
| filter (event.provider == "Falco")

Observing TTPs using Dynatrace Security Analytics

For the sake of this demonstration, our internal red team unleashed a novel attack on our Unguard application. The attack lit up our Falco deployment with more than 100,000 events in 24 hours, more than 3,000 of them critical.

As security analysts, we know we can’t find sophisticated attacks by manually scrolling through thousands of audit logs and events. We need automation, full contextual knowledge of our infrastructure, and very often, domain-specific expertise from security analysts.

To get an initial overview, we can use DQL on Grail to visualize what MITRE techniques Falco observed in our infrastructure over the past 72 hours. We can summarize events using mitre.tactic or mitre.technique. These fields exist on many Falco alerts and are automatically ingested by the Dynatrace output of falcosidekick. We can explore the distribution of techniques with this query:

fetch logs, from:now() - 72h 
| filter event.provider == "Falco" and isNotNull(mitre.technique) 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| summarize event_count = count(), by:{mitre.technique}

In this example, we also observe that we can attribute most events to the following MITRE techniques:

• T1548.001
• T1083
• T1565
• T1055.008

After manually investigating these alerts, however, we conclude they’re noisy false positives. Some of our applications were treating environment variables in an insecure way or communicating with the Kubernetes API server with improperly configured service accounts. Therefore, we filtered them out with DQL.

Observability and context: Attributing reconnaissance activity to TTPs using distributed traces

So far in our TTP-based threat hunting, we’ve utilized Dyntrace Security Analytics to visualize ingested alerts from third-party tools.

But truly magical things arise when we combine this with the rich and high-quality observability data that our customers have valued since the beginning of Dynatrace. Using observability data, we can close an important security-relevant gap. Attackers often probe systems using automated scanning tools. Their many access attempts leave behind a lot of traces. Dynatrace PurePath is one of the core platform technologies that captures and analyzes those distributed traces across an entire infrastructure.

Attack sub-technique T1595.003 “Active Scanning: Wordlist Scanning” describes how attackers use scanners to learn about the many endpoints an application might expose to the internet. Typically, they use large lists with well-known path names, where many of them could be potentially vulnerable. Such wordlists often contain common path names, such as wp-admin, .git, or .htaccess. The following query looks for five indicative files and expresses how many of them match with the new recon.confidence field we set up to track wordlist-based scanners. The more matches, the more confident we can be that these requests came from a wordlist-based scanner:

fetch spans, from:now() - 72h 
| filter in(http.target, {"/wp-admin", "/.git", "/.htaccess", "/.ssh", "/cgi"}) 
| summarize { 
    recon.confidence = countDistinct(http.target) / 5, 
    recon.first_seen = min(timestamp), 
    recon.last_seen = max(timestamp) 
  }, by:{host.name, k8s.container.name, k8s.namespace.name, k8s.pod.name} 
| fieldsAdd mitre.technique = "T1595.003", mitre.tactic = "mitre_reconnaissance" 
| filter recon.confidence > 0.5

Indeed, it did find some reconnaissance attempts! This query scanned 2.5 million spans in less than 50 ms and reduced them to three comprehensible TTP records. With a conventional database, a query that scans millions of records would take many seconds to complete and require that we structure our queries up front. But Grail completes the search across millions of records in milliseconds, and automatically parses and infers the structure for us so we can just start writing queries directly.

We now know that the Envoy proxy in our environment most likely got scanned by an attacker. Let us bring all the bits and pieces together in the next sequence.

Structural multi-step attack detection with Dynatrace Security Analytics

Attackers typically perform many small steps to achieve their mission. Security experts like to think in terms of so-called kill chains, which describe the many stages of an attack. When you work with TTPs, the attack tactics represent those stages. While the MITRE ATT&CK® knowledge base describes as many as 14 tactics, we can distill this into three broad categories:

1. Land – First, hackers investigate their target, looking for an initial way to gain access and establish a foothold in your system.
2. Expand – Then, hackers typically try to escalate their privileges and move laterally within your system, compromising neighboring hosts.
3. Execute – Finally, they find their target and execute their mission.

Next in our demonstration of TTP-based threat hunting with Dynatrace Security Analytics, we’re going to show you a simple but effective strategy that can uncover such advanced attacks: Structural multi-step attack detection. This strategy is structural since it utilizes Smartscape for DQL to take the topological relationship of events into account when hunting for attacks that are composed of multiple steps.

The following DQL query looks for the filtered Falco alerts, and for each Kubernetes pod, it records how many distinct tactics and techniques we just observed. This way, we’re not just looking at whatever pod was the noisiest, but instead, which pod generated alerts from the most tactics and techniques. The more tactics and techniques, the higher the chances that an attacker carried out a full kill chain on that pod.

fetch logs, from:now() - 72h 
| filter (event.provider == "Falco") 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| summarize { 
    num_tactics = countDistinct(mitre.tactic), 
    num_techniques = countDistinct(mitre.technique), 
    tactics = collectDistinct(mitre.tactic), 
    techniques = collectDistinct(mitre.technique) 
  }, by:{k8s.pod.name} 
| filter num_tactics > 1 and num_techniques > 1 
| sort num_tactics desc, num_techniques desc

This result is highly interesting and confirms our previous suspicion. There is one instance of the Envoy proxy that captured alerts for the following techniques:

• TA0002 “Execution”
• TA0003 “Persistence”
• TA0005 “Defensive Evasion”
• TA0006 “Credential Access”
• TA0011 “Command and Control”
• TA0010 “Exfiltration”

Further, the Dynatrace spans we looked at in the previous sequence that explored wordlist scanning indicated TA0043 “Reconnaissance”. This query just scanned through more than 23 million records in 300 ms, providing us with an abstract description of a full kill chain.

But we don’t stop here. We can drill down and observe the individual steps our attacker has taken:

fetch logs, from:now() - 72h 
| filter (event.provider == "Falco") 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| filter k8s.pod.name == "unguard-envoy-proxy-666464f76d-5p26f" 
| fields timestamp, event.name, mitre.technique, content.output_fields.proc.cmdline 
| sort timestamp asc

In the result, we see the records that explain the attack procedure in detail. The above screenshot shows only an excerpt of all 47 records. Here’s what we learned about our attacker’s steps:

• Scanned our Envoy proxy with a well-known wordlist, as we learned by mining the traces for wordlist entries.
• Launched a Perl-based reverse shell on Envoy, indicated by the perl command opening a socket, giving them full code execution access.
• Downloaded a couple of binaries like nmap and nc, indicated by the curl command that pulled them from the internet.
• Scanned our internal network with nmap.
• Exfiltrated large volumes of data from our Redis database, indicated by the queries to Redis that request all keys with the KEYS * command
• Tried to cover their tracks by deleting the shell history, indicated by the rm /home/envoy/.bash_history command

Isn’t this a truly elegant way to hunt for attacks?

TTP-based threat hunting with context-rich observability and security analytics

This demonstration shows how modern attack detection strategies become a reality with context-rich security analytics on a unified observability and security platform. With this approach, you can do the following:

• Utilize observability data to capture security-relevant reconnaissance alerts and map them to TTPs.
• Enrich the Dynatrace platform with more data of your own, such as ingesting Falco alerts into Grail.
• Use DQL and Grail to find the needle in the haystack to scan tens of millions of records in milliseconds to identify the chain of only a handful of events that exposed an attacker and the exact methods they used.

For another great demonstration, we recommend reading the blog post Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail by Liisa Tallinn.

The post TTP-based threat hunting with Dynatrace Security Analytics and Falco Alerts solves alert noise appeared first on Dynatrace news.

Our recent blog post, Stay ahead of the game: Forecast IT capacity with Dynatrace Grail and Davis AI, showed how Dynatrace Notebooks are used to predict the future behavior of time series data stored in Grail™. This follow-up post introduces Davis^® AI for Workflows, showing you how to fully automate prediction and remediation of your future capacity demands. The anticipation of future capacity demands makes it possible to completely avoid critical outages by notifying you days in advance, well before incidents arise.

Predictive capacity management starts within a Dynatrace Notebook, where the operations team explores important capacity indicators, such as the percentage of free disks, as shown below.

After exploring and selecting the most important capacity indicators for your environment, a workflow triggers forecast reporting at regular intervals. The example workflow below is triggered every Monday at 8:00 AM to provide a capacity report for all the disks that will likely run out of space within the next week.

Define the forecast

The workflow uses the Davis for Workflows action to automatically trigger a forecast for a selected set of disks. The forecast operation is selected within the Davis action, and a DQL query is used to specify the set of disks and the capacity indicator metric that should be predicted. Note that you can use any time series data you can fetch from Grail using DQL within the forecast action.

While this example uses the metric dt.host.disk.free, you can choose any kind of capacity metric, such as host CPU, memory, or network load—you can even extract a metric value from a given log line.

The forecast is trained on a relative timeframe (for example, the last seven days) which is specified in the configured DQL query. The DQL query example below trains forecasting on a relative timeframe of the last seven days:

timeseries avg(dt.host.disk.free), by:{dt.entity.host, dt.entity.disk}, bins: 120, from:now()-7d, to:now()

The configuration below shows that a forecast horizon of 100 data points is requested, which means that 100 additional predicted points will expand the initially fetched 120 data bins of the source DQL query. This predicts one week into the future.

The prediction action returns all its forecasted time series lines, which can include hundreds or even thousands of individual disk predictions.

Evaluate the forecast results

Within the following TypeScript action, each disk prediction is tested against a threshold to determine if the disk will run out of space in the next week. The TypeScript code snippet below is responsible for checking for threshold violations and for preparing all the violations in a result object for subsequent actions to follow up on:

The TypeScript action returns a custom object that uses a Boolean flag (violation) to tell the follow-up actions about violations and an array of all the violation details (violations).

const predictionSummary = { violation: false, violations: new Array<Record<string, string>>() };

Tip: Download the TypeScript template from our documentation.

Trigger remediation actions

A collection of remediation actions can be used to follow up on predicted capacity shortages. In this example, two parallel actions are defined. One action sends out an email notification; the other raises a Davis problem for each violating disk. All remediation actions use the Boolean violation flag of the previous workflow action to avoid invocations when there are no violations.

Here you can see the invocation condition used in the follow-up actions that control the invocation.

Raise events in case of disk capacity shortage!

A TypeScript remediation action is used to iterate through all the predicted disk shortages and to raise individual alarm events. Each alarm event has custom event properties that can be used to deliver further details about the situation and to further identify the disk or host.

Tip: Download the TypeScript template from our documentation.

Review all Davis-predicted capacity problems

Navigating to the Davis problems feed, the operations team can review all the predicted disk capacity shortages. Remember, raising events and problems is an optional remediation step that can be skipped entirely by directly sending emails or Slack messages to the responsible teams.

The creation of alerting events within this workflow example highlights the flexibility and power of the Dynatrace AutomationEngine combined with the analytical capabilities of Davis AI and Grail.

Summary

The combination of Davis AI forecasts with Dynatrace AutomationEngine and Grail opens the door for many valuable use cases—anticipative management of capacity being the most prominent of these. Predicting future capacity shortages for thousands of disks or hosts allows operations teams to anticipate critical situations weeks before incidents occur. The flexibility and power of the Dynatrace AutomationEngine allow operations teams to react to detected shortages flexibly and to customize and implement their remediation flows.

You can install Davis® for Workflows via the Dynatrace Hub. As a starting point for implementing your own anticipative capacity management workflow, you can download all the TypeScript code used in this example from our documentation:

• Evaluate the forecast results
• Raise a Davis problem

For full details, see Davis AI analysis in workflows documentation.

Predictive capacity management in action (14-second video)

The post Automate predictive capacity management with Davis AI for Workflows appeared first on Dynatrace news.