logs in context | Dynatrace news The tech industry is moving fast and our customers are as well. Stay up-to-date with the latest trends, best practices, thought leadership, and our solution's biweekly feature releases. Wed, 05 Mar 2025 11:49:10 +0000 en hourly 1 Unlock the power of contextual log analytics https://www.dynatrace.com/news/blog/unlock-the-power-of-contextual-log-analytics/ https://www.dynatrace.com/news/blog/unlock-the-power-of-contextual-log-analytics/#respond Wed, 02 Oct 2024 16:29:18 +0000 https://www.dynatrace.com/news/?p=65826 Observability graphic

In the ever-evolving landscape of IT operations and software development, logs are a critical data source for understanding system behavior, diagnosing issues, and maximizing business value. However, different teams often rely on a variety of monitoring and troubleshooting tools that use different data types, leading to fragmented data and inconsistent analytics. Dynatrace addresses this challenge by providing unified analytics and automation for logs, integrating them with all other observability, security, and business data types.

The post Unlock the power of contextual log analytics appeared first on Dynatrace news.

]]>
Observability graphic

Dynatrace enables various teams, such as developers, threat hunters, business analysts, and DevOps, to effortlessly consume advanced log insights within a single platform. Dynatrace Grail™ and Davis® AI act as the foundation, eliminating the need for manual log correlation or analysis while enabling you to take proactive action.

Dynatrace unified observability and security help enterprises, like our customer, BMO, save time and money, fostering collaboration across business, development, and operations teams.

In this blog post, we’ll provide an overview of the following log-related topics:

  • Logs in the context of applications
  • Easy yet powerful access to any log with the all-new Logs app
  • How logs are ingested
  • Dynatrace Application Security with logs
  • Dynatrace Davis CoPilot™ integration and AI-powered Application Security
  • Schemaless: Instantly gain business event insights from logs

Simplicity is key to success

As IT responsibilities shift left and expand, such as when developers take on application security duties, simplicity in toolsets becomes essential. Existing siloed tools lead to inefficient workflows, fragmented data, and increased troubleshooting times.

Tool consolidation is becoming a priority for C-level decision-makers in 2025. Enterprises are turning to Dynatrace for its unified observability approach for cloud-native, on-premises, and hybrid resources.

Rather than relying on disparate tools for each environment and team, Dynatrace integrates all data into one cohesive platform. Davis AI contextually aligns all relevant data points—such as logs, traces, and metrics—enabling teams to act quickly and accurately while still providing power users with the flexibility and depth they desire and need.

The Clouds app provides a view of all available cloud-native services. Logs in context, along with other details, are instantly available after selecting a resource.
Figure 1. The Clouds app provides a view of all available cloud-native services. Logs in context, along with other details, are instantly available after selecting a resource.

Logs in context with applications

Applications are provided within the Dynatrace platform to address the various needs of different teams and specific use cases. DevOps teams operating, maintaining, and troubleshooting Azure, AWS, GCP, or other cloud environments are provided with an app focused on their daily routines and tasks.

For instance, in a Kubernetes environment, if an application fails, logs in context not only highlight the error alongside corresponding log entries but also provide correlated logs from surrounding services and infrastructure components. This shortens root cause analysis dramatically, as explained in our recent blog post Full Kubernetes logging in context from Fluent Bit to Dynatrace.

The Kubernetes app provides an overview of the current log volume, criticality, and security status. Automatic log correlation for the selected Kubernetes node happens in the backend and is visualized when selecting Run query.
Figure 2. The Kubernetes app provides an overview of the current log volume, criticality, and security status. Automatic log correlation for the selected Kubernetes node happens in the backend and is visualized when selecting Run query.

The show surrounding logs function provides Dynatrace users with the ability to dive deeper and surface context-specific log lines of the components and services linked to the problem—all without a single line of code or complex query language knowledge. This is explained in detail in our blog post, Unlock log analytics: Seamless insights without writing queries.

For advanced analysis, there is a direct “open with” path, allowing you to load the current view in Notebooks for manual analysis, the Logs app, or other apps capable of visualizing context-specific log lines.

Screen video of Dynatrace platform when Davis AI automatically correlates Amazon AWS EC2 and business backend logs
Figure 3. A Service Reliability Engineer (SRE) manually reviews cloud-native front-end application warnings. Davis AI automatically correlates Amazon AWS EC2 and business backend logs. The platform offers the flexibility to dive deeper or filter views at any time by selecting highlighted components.

While the way that logs in context are interconnected and provided by Dynatrace is unique, as also Gartner® recognized for the 14th consecutive time in their Magic Quadrant™, there are use cases where this is not sufficient and raw access to the logs is required.

Easy yet powerful access to any log with the all-new Logs app

Developers love Dynatrace, without a doubt, and are one of the many teams next to DevOps or SREs making use of our all-new Logs app. The reasons are easy to find, looking at the latest improvements that went live along with the general availability of the Logs app.

In our product news blog post, Simplicity meets power: Introducing the all-new Dynatrace Logs app, we examine these features, which make life easy for new Dynatrace users, along with the newly introduced DQL Editor for power users.

­­Screenshot with unfiltered query results, where the newly introduced ’search in results’ feature has been used, to locally filter for log lines containing the word ’product‘.
­­Figure 4. Screenshot with unfiltered query results, where the newly introduced ’search in results’ feature has been used, to locally filter for log lines containing the word ’product‘.

Directly from individual log line results, you can filter simply by selecting corresponding items in the details pane or by loading the surrounding logs when selecting Show surrounding logs.

Keep in mind that Dynatrace Grail is schema-on-read and indexless, built with scaling in mind. There is no need to think about schema and indexes, re-hydration, or hot/cold storage. This architecture also means you are not required to determine your log data use cases beforehand or while analyzing logs within the new logs app.

How logs are ingested

Dynatrace offers OpenPipeline to ingest, process, and persist any data from any source at any scale. OpenPipeline ensures data security and privacy—data is collected and processed securely and compliantly, with high-performance filtering, masking, routing, and encryption—and contextualizes incoming data in real time. Using patent-pending high ingest stream-processing technologies, OpenPipeline currently optimizes data for Dynatrace analytics and AI at 0.5 Petabyte per day and tenant; this will soon increase to one Petabyte per day and tenant.

OpenPipeline’s high-performance filtering and preprocessing provide full ingest and storage control for the Dynatrace platform. As a result, dedicated data pipeline tools are unnecessary for preprocessing data before ingestion.

OpenPipeline architecture log flow
Figure 5. OpenPipeline architecture log flow

Dynatrace meets your teams where they are, with your preferred ingest routes and methods—be they Fluent Bit, OpenTelemetry, SysLog, or automatic log collection leveraging OneAgent, to name a few options.

If your team deploys applications cloud-natively, we meet you there, too, as we recently covered in our blog post, Dynatrace log management innovations: Syslog, AWS Firehose. We covered in rich detail how Dynatrace supports log ingestion for cloud-native workloads and simplified log ingestion also for hybrid environments.

In any case, at the heart of the Dynatrace Platform, Grail enables contextual analytics across unified observability, security, and business data. Grail is built for exabyte scale and leverages massively parallel processing (MPP) as well as advanced automated cold/hot data management to ensure that data remains fully accessible at all times, with zero latency, and full hydration.

Figure 6. Dynatrace marketecture – Logs in context
Figure 6. Dynatrace marketecture with logs in context

With no index or schema boundaries in place, paired with long-term data retention ranging from 1 day up to 10 years, you can leverage metrics, logs, and traces for a variety of additional use cases, such as business analytics or security analytics.

Dynatrace Application Security with logs

While most enterprises have Application Firewalls (AppFW), Intrusion Detection Solutions (IDS), and Static Code Analysis (SCA) in place for applications that will be deployed to production, it’s still relevant to understand if anomalies occur during runtime. Monitoring known vulnerabilities within the service hosting the application itself is just another puzzle piece to be considered for full end-to-end observability.

Instead of relying on static patterns, Dynatrace Causal AI understands the desired outcome of the triggered action and the context of the environment hosting the service. For example, deleting the database is not an expected outcome when the function provided is to update a user profile.

Dynatrace Security Investigator app visualizes the results of a shared incident investigation. The right-hand pane provides a query tree view with manually saved patterns in the evidence collection pane.
Figure 7. Dynatrace Security Investigator app visualizes the results of a shared incident investigation. The right-hand pane provides a query tree view with manually saved patterns in the evidence collection pane.

With these sophisticated security analytics, simple use cases such as authentication failure anomalies or password spraying attacks, along with more technical HTTP RST statistics, can be visualized in simple and sharable views in Security Investigator, leveraging logs.

Advanced analytics are not limited to use-case-specific apps. Dynatrace offers Notebooks and Dashboards to build views and reports—without the need to write a single line of code or Dynatrace Query Language, all supported by Dynatrace Davis CoPilot™ integration.

Dynatrace Davis CoPilot integration and AI-powered Application Security

Davis CoPilot™ assists occasionally visiting Dynatrace operators throughout the platform in a variety of applications, including Dynatrace Notebooks.

With natural language input in the example displayed in the screenshot below, “Show me the most recurring log lines and add a column with the log source and AWS region,” Davis CoPilot will evaluate the input, translate it into a corresponding DQL Query, and fetch the results accordingly.

Guardrails are in place and can be altered to prevent a high volume of unwanted material from being scanned or returned to Notebooks.

The Dynatrace operator defined a question in natural language that Davis CoPilot translated into Dynatrace Query Language
Figure 8. The Dynatrace operator defined a question in natural language that Davis CoPilot translated into Dynatrace Query Language

In the same way, Davis CoPilot can recommend remediation strategies and simplify security analysis across all data by translating natural language into the Dynatrace Query Language (DQL) to drive attack protection, security investigations, and forensics.

As mentioned, when ingesting relevant logs into Grail, including firewall and authentication gateway logs, Davis AI can provide end-to-end security insights. At the same time,

Davis AI not only visualizes or assesses risks automatically; it also detects and provides the option to block such threats when the corresponding features have been activated and configured platform-wide.
Figure 9. Davis AI not only visualizes or assesses risks automatically; it also detects and provides the option to block such threats when the corresponding features have been activated and configured platform-wide.

Davis CoPilot is integrated into the Dynatrace platform as an intelligent assistant to ease the effort of configuring and finding relevant details and options. By simply asking CoPilot the question stated above, you’re provided with the required configuration steps and product documentation references.

Davis CoPilot Assistant was asked to provide guidance for the example provided in this blog post and swiftly replied with the required configuration, referencing the source Application Security FAQ.
Figure 10. Davis CoPilot Assistant was asked to provide guidance for the example provided in this blog post and swiftly replied with the required configuration, referencing the source Application Security FAQ.

Schemaless: Instantly gain business event insights from logs

A unique picture can be drawn when logs from business-critical LoB applications are collected, which is an underestimated value that can be gained when using Dynatrace.

There are many customer examples and use cases, like room bookings in hotel portals, shopping cart statistics from online shops, or customer metrics from finance platforms, where customers can gain additional business value by translating logs to metrics within Dynatrace.

A pipeline health dashboard showing growth, paired with security-related information, is just one of the many examples of what you can build, either on your own or with the help of the Dynatrace Services team.

A custom business dashboard provides a holistic view of business process performance
Figure 11. A custom business dashboard provides a holistic view of business process performance

In our blog post, Leverage logs for an end-to-end view of your business processes via Dynatrace OpenPipeline, we demonstrated the retail company dashboard example above at a higher granularity, including the necessary steps for JSON log ingestion via Dynatrace OpenPipeline.

Conclusion

It’s simple, fast, and easy to ingest and gain multi-purpose value from logs—all without the need to be an expert or the requirement to first learn a complex query language.

Davis AI and CoPilot make it easy for casual contributors and power users to gain meaningful insights and build notebooks or dashboards while simultaneously increasing application security and reliability.

Learn how Dynatrace can address your specific needs with a custom live demo. Our observability experts will walk you through our solutions and show you how to deliver excellent customer experiences, foster your application security, and simplify IT operations.

If you’re not yet a Dynatrace customer, start your 15-day free trial.

If you want to learn more about Dynatrace and Logs in context, join us for a demo.

The post Unlock the power of contextual log analytics appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/unlock-the-power-of-contextual-log-analytics/feed/ 0
Unlock log analytics: Seamless insights without writing queries https://www.dynatrace.com/news/blog/log-analytics-seamless-insights-without-writing-queries/ https://www.dynatrace.com/news/blog/log-analytics-seamless-insights-without-writing-queries/#respond Tue, 28 May 2024 14:48:22 +0000 https://www.dynatrace.com/news/?p=64183

Logs are an integral part of the daily workflow for your DevOps and SRE teams to understand what’s happening in your tech stack. No matter the industry you operate in or the scale of your business, getting value from log data is often slowed down by challenges: making sure the right logs are monitored, finding the relevant logs when you need answers, and making sense of logs in the context of other data like traces, events, and metrics.

The post Unlock log analytics: Seamless insights without writing queries appeared first on Dynatrace news.

]]>

Logs provide answers, but monitoring is a challenge

Manual tagging is error-prone

Making sure your required logs are monitored is a task distributed between the data owner and the monitoring administrator. Often, it comes down to provisioning YAML configuration files and listing the files or log sources required for monitoring. This manual, error-prone approach can lead to monitoring gaps, which become critical when a host or service has an outage or incident.

Finding the right logs is cumbersome

Even if your logs are monitored, you need to make sense of the vast data volume. As the scale and complexity of your tech stack grows, you might need to navigate the maze of hosts or Kubernetes clusters, apps, and microservices and understand the relevance and risks associated with logs originating from these entities. Challenges compound: Manual tagging of log sources has long been difficult regarding monitoring coverage. And you can’t assume the tagging is 100% correct to pinpoint the correct logs.

In the past, more work was needed to understand the context of log data. What about correlated trace data, host metrics, real-time vulnerability scanning results, or log messages captured just before an incident occurs? This context is vital to understanding issues.

Dynatrace automatically puts logs into context

Dynatrace Log Management and Analytics directly addresses these challenges. First, OneAgent takes care of log autodiscovery. Once logs are selected for monitoring, OneAgent enriches log data with the topological context you need. For example, OneAgent helps you monitor the logs from a Kubernetes environment with automatic enrichment that identifies the right cluster, namespace, container, and pod ID.

Once logs are stored in Dynatrace Grail™, our purpose-built data lakehouse for observability data, the logs are automatically shown in the right context. Finding answers begins with opening the right app for your use case.

Kubernetes logs in context in Dynatrace screenshot

You can easily pivot between a hot Kubernetes cluster and the log file related to the issue in 2-3 clicks in these Dynatrace® Apps: Infrastructure & Observability (I&O), Databases, Clouds, and Kubernetes.

Open a host, cluster, cloud service, or database view in one of these apps, and you immediately see logs alongside other relevant metrics, processes, SLOs, events, vulnerabilities, and data offered by the app.

By eliminating slow and manual correlation, lack of context, and getting visibility into the surrounding data, you reduce the risk of prolonged outages, mean time to repair, and tool sprawl.

Log data in Dynatrace

Get quicker answers

Let’s look at how logs in context can make your teams more effective.

Video thumbnail

Log histograms: Insight into log volumes and patterns

Open one of these Dynatrace Apps and select Logs for any listed entity (host, Kubernetes workload, cloud service, or database instance):

  • Infrastructure & Operations
  • Kubernetes
  • Databases
  • Clouds

You’ll see a histogram chart of log data with various severity levels (such as Error, Info, or Warning) relevant to the selected Dynatrace entity, giving you a clear understanding of log patterns and volumes over time. Is there a sudden spike in errors? A sudden drop in received log data? Depending on which app is in use, one glance at a histogram provides invaluable insight into managing clouds, databases, Kubernetes environments, and infrastructure.

hosts logs in context

Log analytics simplified: Deeper insights, no DQL required

Your team will immediately notice the streamlined log analysis capabilities below the histogram. Jump directly into log insights by selecting a recommended query, for example, to see the errors related to a problem detected by Davis® AI during the selected timeframe. Furthermore, your team can easily access all error logs within the specified timeframe displayed on the histogram or view all logs within that timeframe, all without writing any queries from scratch.

Surrounding logs display: Effortlessly navigate log context

You can see the result after opening a recommended query without leaving an app’s context. Upon expanding a single log entry, all relevant context provided by OneAgent during the ingestion process is displayed, making it easy to expand your analysis to the infrastructure or entity related to the error logs. For a single log record found, you can easily see the surrounding logs.

Look at this example of an online store payment service generating errors. The application owner found error logs related to unsupported credit cards. Select Surrounding logs to view the log messages for the whole transaction, based on the trace ID, that ended up with an error and a failed order.

Surrounding logs

In Infrastructure & Operations, surrounding logs can also be displayed based on other criteria, like the host file or log source from which logs are collected. This allows quick and easy troubleshooting without writing or editing queries.

Logs in context across Dynatrace Apps

  • Infrastructure & Operations leverages advanced AI capabilities that automatically discover and map all components within your infrastructure, including hosts, virtual machines, containers, and cloud instances.
  • Databases offers comprehensive database monitoring capabilities, providing organizations with real-time visibility into the performance and health of their database environments.
  • Clouds is a central hub for monitoring and managing multicloud environments, providing organizations with a unified view of their cloud infrastructure and services.
  • Kubernetes delivers comprehensive monitoring and management capabilities for Kubernetes environments, enabling organizations to ensure the performance, availability, and scalability of their containerized workloads.

Stay tuned for even wider support of log data embedded seamlessly into the context of Dynatrace Apps, and better ways to get answers from logs without writing queries.

See for yourself

Already have a Dynatrace account? See logs in context for yourself in the Dynatrace Playground.

The post Unlock log analytics: Seamless insights without writing queries appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/log-analytics-seamless-insights-without-writing-queries/feed/ 0