Logs are essential observability telemetry

Organizations are increasingly challenged to deliver seamless digital services, an essential component for achieving business-critical objectives. These challenges are amplified by complex hybrid cloud environments, where managing diverse technologies across cloud providers and platforms like IBM Z Mainframe becomes particularly demanding.

To address this complexity, it’s vital to unify observability telemetry and its signals across all layers of the application delivery chain, including the mainframe platform, within a single AI-powered observability solution.

Dynatrace enrichment capabilities enhance ingested log records by adding contextual metadata such as trace IDs, span IDs, and process group instance IDs automatically and without any manual tagging efforts. This allows seamless correlation between logs, application performance metrics, and traces. This is essential for accelerating AI-driven root cause analysis and significantly reducing time spent on shortening RCA and MTTx, as many of our customers can confirm.

Simplify and automate z/OS log collection

Dynatrace Log Management & Analytics extends beyond distributed technology stacks to include support for the IBM Z Mainframe platform. It automatically captures and ingests logs from monitored IBM CICS and IBM IMS regions and offers you advanced ingest rules. All collected logs are enriched automatically with topological metadata, allowing seamless mapping to Dynatrace’s topology and entity model for z/OS Hosts (LPARs) and z/OS Processes (regions).

Additionally, logs can be enriched with Trace IDs and Span IDs to precisely correlate each log line with the corresponding CICS or IMS trace or span that generated it.

This dramatically simplifies navigation for any user in your organization. By selecting the log line containing the Trace ID and Span ID, you can directly navigate to the related trace while understanding the load times and delay.

Let’s summarize what we’ve seen so far. Log enrichment significantly enhances and accelerates:

• Correlation with distributed traces, enabling end-to-end visibility across systems.
• Troubleshooting, by linking logs to specific spans or transactions—accelerating troubleshooting.
• Observability for both structured and unstructured log data, ensuring comprehensive insights regardless of log format.

Beyond the agent: Stream mainframe logs to Dynatrace with OpenTelemetry

Dynatrace OneAgent® already supports ingestion of logs out of the box. However, when it comes to mainframe environments, the story is more nuanced.

Why all logs are not created equal

Some logs—especially those on mainframes—are proprietary, customer-specific, and deeply embedded in legacy workflows. And while Dynatrace is constantly adding additional and automated coverage for additional log types, some might never be supported natively by OneAgent, simply because their structure and relevance are unique to each customer.

But that doesn’t mean they’re out of reach.

OpenTelemetry to the rescue

For logs that fall outside OneAgent’s native scope, OpenTelemetry offers a powerful alternative. By deploying an OpenTelemetry Collector, customers can stream log data from their LPARs to a distributed host—preferably to Linux, Windows, or zLinux to save MSU consumption. But even z/OS itself is an option for hosting an OpenTelemetry Collector.

The Collector supports:

• Filelog receiver for arbitrary text files
• Syslog receiver for structured system logs
• Filter processor for preprocessing and enrichment
• Concurrent export to multiple backends, including Dynatrace via OTLP

Getting logs off the mainframe

There are several ways to move logs from LPARs to distributed systems:

• SFTP: A blunt but reliable method
• z/OSMF: Offers REST API access to SMF records
• z/OS Data Gatherer: SMF REST Services
• Custom scripts or processes: Tailored to specific datasets
• Streaming frameworks: Kafka, MQ, or even FTP-to-Collector bridges

The bottom line: Just transfer log data from the mainframe to the host where the OpenTelemetry Collector is located, and it will handle everything for you from there.

There’s no strict requirement to provision a dedicated host to run your OpenTelemetry Collector. If Dynatrace OneAgent is already monitoring one of your LPARs, the ActiveGate hosting the Dynatrace zRemote component is a perfectly suitable environment for the Collector.

Preprocessing and enrichment

Both the Dynatrace Distro and the Contrib Distro of the OpenTelemetry Collector support advanced preprocessing:

• Timestamp normalization (for example, converting z/OS timestamps to Unix time)
• Resource attribute extraction (for example, job name, LPAR ID, subsystem)
• Sensitive data masking and filtering

This ensures that even complex logs are transformed into structured telemetry before reaching the backend.

The cherry on top: Dynatrace OpenPipeline

While OpenTelemetry handles ingestion and transformation, Dynatrace OpenPipeline® adds another layer of intelligence during ingestion and processing:

• Further enrich logs with business context
• Extract metrics, events, and business observability events
• Apply AI-driven baselining and anomaly detection
• Transform, mask, or drop data
• Some of these capabilities overlap with the Collector, giving users flexibility to choose where to apply logic based on performance, cost, and control.

Operlog: A prime candidate

Let’s take Operlog as an example—a system log that many customers are keen to stream. While it’s not a simple text dataset, creative solutions can bridge the gap. If you can extract Operlog records and store them as text on a Linux host, the Collector can ingest them immediately. From there, Dynatrace visualizations and alerting kick in.

Here’s a snapshot of how Operlog looks once streamed and processed in Dynatrace:

Not satisfied with only logs?

You’re not limited to ingesting only proprietary log files via OpenTelemetry.

Do you have access to metrics that are relevant for tracking the health of the subsystems on your mainframe? Would you rather feed in certain data as events instead of logs?

Just as the OpenTelemetry Collector is highly customizable, Dynatrace offers multiple ways of ingesting all these signals.

Conclusion

Mainframe logs may be complex, but they’re not unreachable. With OpenTelemetry and Dynatrace working in tandem, even the most proprietary datasets can be brought into the fold. Whether you’re using OneAgent, OpenTelemetry, or a hybrid approach, the key is creativity—and the right tooling.

What’s next

Dynatrace currently supports CICS MSGUSR and IMS Master Terminal Logs via its z/OS Agents, and remains committed to enhancing these capabilities. This includes exploring support for additional z/OS log types and expanding the scope of information captured through the z/OS Agents.

Log monitoring is also available for Linux on IBM Z and LinuxONE. For more details, see the blog post, Enable full observability for Linux on IBM Z mainframe now with logs.

Get started with Dynatrace log observability

If you’re looking to elevate your end-to-end observability and explore tailored possibilities within your specific z/OS environment, we’d be happy to connect. Reach out to us to request a demo and dive deeper into what Dynatrace can offer.

The post Significantly improve your Mainframe availability by connecting logs with traces appeared first on Dynatrace news.

Mobile teams struggle to solve what they can’t see

For many businesses, mobile is now the most critical channel for engaging users. Mobile apps are where purchases happen, accounts are created, and first impressions are made. But it’s also where users are most likely to abandon their journeys if something feels broken, slow, or unintuitive.

All practitioners struggle to understand what actually happened during a user’s session when something goes wrong or when a user behaves in a way that deviates from the expected journey. This can get particularly puzzling when there are no apparent errors or performance blockers hindering a user’s experience. Traditional tools provide fragmentary details: logs, events, crash reports, or aggregated analytics. However, none of these capture the full context. Mobile engineers are unable to reliably reproduce bugs without seeing exact user actions. Support teams can’t respond effectively to vague or incomplete user reports.

Full visibility into mobile user behavior

This is where Session Replay comes in. This capability recreates real user behavior in a video-like playback, so you can visually step through a session just as the user experienced it: taps, swipes, scrolls, page transitions, and all. And now, for the first time, this powerful capability extends beyond the browser and crashes on mobile apps to include full sessions for native mobile apps.

With Dynatrace Session Replay for mobile apps, teams get a complete, contextual view of how users interact with mobile apps in the wild by collecting and visualizing key signals like gestures, views, UI states, and navigation paths. While Session Replay is often seen as a frontend-focused tool, it can significantly help identify, triage, and resolve backend issues by connecting user actions to system behavior. This helps contextualize backend failures, correlate user actions to backend events, and, in a nutshell, identify misalignment between front and backends.

How to make the most of Dynatrace Session Replay for Mobile Apps

Let’s dive into key use cases where Session Replay delivers tangible value across teams:

Faster debugging

Understanding the problem is often impossible with only data from Logs or Real User Monitoring. In some cases, observing the sessions where the error occurs can provide valuable context and lead to a more accurate solution.

Use case: Imagine your team can’t reproduce a performance issue affecting only a small segment of users still using iOS 17. Logs are inconclusive, and RUM data doesn’t tell the whole story.

With Session Replay, you can:

• Watch the exact gestures and actions related to the issue.
• See which page or interaction triggered the problem.
• Identify whether it’s a device-specific issue, a network hiccup, or a UI bug.

Result: Your dev team fixes the issue in hours, not days, without needing to reproduce it manually.

UX optimization and boosting conversion

The best way to understand users is to observe how they engage with your applications. Session Replay allows you to select a sample of mobile sessions based on various criteria. Watching these sessions helps you learn about annoyances and understand patterns in user behavior, which can then be streamlined into a better user experience.

Use case: Your analytics show a drop-off in the final step of your checkout funnel, and you don’t know why. Is it a confusing UI? A broken button? A slow-loading view?

With Session Replay, you can:

• Watch real user sessions to see where they pause, scroll, or back out.
• Identify if users struggle with form validation, payment options, or navigation.
• Spot rage taps or repeated attempts to interact with unresponsive elements.

Result: You optimize the checkout flow, reduce abandonment, and increase conversions—backed by real user behavior.

Improved customer support

Allowing help desk personnel to review the session that prompted a user’s complaint provides common ground for the conversation and helps ensure user satisfaction.

Use case: A user submits a support ticket: “The app didn’t work when I tried to book a ride.” No screenshots, no steps to reproduce, and the issue can’t be replicated.

With Session Replay, your support team can:

• Instantly view the user’s session to see what they saw.
• Understand the context: which screen they were on, what they tapped, and what went wrong.
• Provide a fast, accurate response, or escalate with full context to engineering.

Result: Faster resolution, happier users, and fewer back-and-forth emails.

Data privacy

Dynatrace adheres to world-class standards for data protection. Mobile session data is secured just like any other captured telemetry. However, privacy configurations must still be reviewed by each customer to ensure compliance with internal and legal requirements. Like web applications, Session Replay for mobile applications offers three levels of privacy controls:

• Safest – Masks all editable text fields, labels, images, switches, and web views
• Safe – Masks all editable text fields only
• Custom – By default, behaves like Safest, but allows you to define exactly which elements or views to mask.

See configure custom masking in the preferred documentation below.

Turn on Dynatrace Session Replay for Mobile Apps

Cluster-side activation

Session Replay is now supported for both iOS (including those built in SwiftUI) and Android apps (including those using Jetpack Compose). To get started, you’ll first need to turn it on for each mobile app, as it’s managed separately at the cluster level.

To activate the feature:

1. Navigate to: Frontends → Select a mobile application → Edit → Enablement and cost control
2. Set the Session Replay Mobile toggle to on.

Once turned on, you can define the percentage of sessions to capture. By default, Dynatrace captures 100% of sessions, but you can reduce this based on your data volume and storage needs.

Session Replay for Mobile integrates seamlessly with Session Replay for Crashes. This allows you to:

• Automatically capture 100% of crash sessions
• Sample a configurable percentage of all other sessions

If you’re hosting Dynatrace on-premises, make sure to prepare for increased storage requirements before enabling this feature.

Instrumentation and configuration of a mobile application

Scenarios differ depending on the operating system, so please follow our documentation:

• Configure Session Replay for iOS
• Configure Session Replay for Android

The instrumentation and configuration process is simplified for customers who have already used Session Replay for Crashes or mobile Real User Monitoring.

The post Dynatrace Session Replay for Mobile Apps: see what your users see appeared first on Dynatrace news.

We have released Dynatrace Managed version 1.324. To learn what’s new, have a look at the release notes.

The post Dynatrace Managed release notes version 1.324 appeared first on Dynatrace news.

We have released Dynatrace Managed version 1.322. To learn what’s new, have a look at the release notes.

The post Dynatrace Managed release notes version 1.322 appeared first on Dynatrace news.

OpenTelemetry excels at capturing data from any environment and service: traces flowing from microservices, metrics streaming from containers and infrastructure hosts, and logs capturing the application and service lifecycles. OTel does exactly what it was designed to do: standardize telemetry collection. But here’s what OpenTelemetry doesn’t do by design: unified observability that analyzes that data, correlates it across services, or turns it into actionable, intelligent insights.

This is where most organizations hit a wall and require lots of expert knowledge. Organizations today have more observability data than ever before, but somehow less visibility into what’s actually happening in their systems. Raw telemetry data becomes a burden rather than an asset. Engineers spend more time hunting through dashboards than solving actual problems.

This is the “analytics gap” that Dynatrace was built to solve, transforming your telemetry data from scattered signals into unified, AI-powered intelligence.

Why OpenTelemetry + Dynatrace changes everything

Here’s what makes this combination powerful: OpenTelemetry gives you standardized data collection. Dynatrace gives you intelligent analysis that goes far beyond the static dashboards and manual correlation work that other platforms require.

While many observability solutions leave it to you to build custom dashboards and manually connect the dots between your telemetry signals, Dynatrace transforms your OpenTelemetry data into insights that actually drive decisions. Your traces, metrics, and logs aren’t just stored; they’re automatically correlated, analyzed, and contextualized as they flow through Dynatrace OpenPipeline®.

When a trace shows latency spikes, you immediately see related log entries and metric anomalies automatically contextualized and correlated. Lightning-fast queries via Dynatrace Grail® data lakehouse process millions of spans at the speed of thought, making observability accessible to your entire team, not just the experts who know how to build complex queries and visualizations.

The result? Your OpenTelemetry investment becomes a competitive advantage, not just another data collection project that requires a team of dashboard architects to maintain.

Complete OpenTelemetry coverage

Here’s how Dynatrace helps you to get the most out of your telemetry data, without requiring additional agents or complex configurations:

Distributed tracing excellence

Native OpenTelemetry tracing delivers superior span and trace processing with dynamic visualization tools that transform complex distributed architectures into complete end-to-end visibility. But it doesn’t stop there; all your telemetry signals (logs, traces, and metrics) correlate seamlessly, giving you clear, actionable insights within the full context of your traces and services. Get simple answers to advanced questions by expanding your investigations with DQL for powerful analytics, including correlation of logs and traces.

Service monitoring that understands your Architecture

Comprehensive service health monitoring built on OpenTelemetry standards. Dynatrace provides intelligent service analysis, anomaly detection, and visualization that work seamlessly with your OpenTelemetry-instrumented applications. Our service monitoring goes beyond simple health checks.

When issues arise, you see exactly which services are affected and how problems cascade through your architecture, all without manual tagging, configuration, or service discovery setup with YAML files.

Intelligent metrics with full context

You get flexible metric ingestion for custom business metrics and standard application performance indicators. Your metrics connect directly to the services and traces that generated them. But here’s where Dynatrace takes it further: we allow you to unify all your OpenTelemetry signals into comprehensive service intelligence. Instead of analyzing metrics in isolation, you see how they connect to actual service behavior, request flows, and application logs. Every metric becomes part of a complete service story.

Complete log processing

Your OpenTelemetry logs are transformed from noise to narrative. Instead of searching through endless log streams and manually created dashboards, Dynatrace supports a comprehensive log ingestion and analysis pipeline, allowing you to go big with Dynatrace.

Every log event becomes part of a larger story about user journeys, interactions, services, and app behavior, all focused on your desired business outcomes and incident investigations.

Here’s where it gets powerful: you automatically get additional contextual enrichment when you direct all your telemetry signals to Dynatrace. By creating bi-directional relationships between logs and traces, where logs provide context to traces and traces illuminate relevant logs, Dynatrace evolves troubleshooting from detective power-user work into AI-driven, streamlined, and intuitive investigations.

Video: See the full story behind every trace with correlated logs.

Kubernetes native support

For teams running OpenTelemetry in Kubernetes, Dynatrace delivers enterprise-grade support that scales with your cloud native operations. Native Kubernetes handling of spans, metrics, and logs from your Kubernetes OpenTelemetry deployments automatically collects Kubernetes context for automated enrichment: namespace, cluster, and workload relationships, all without any additional instrumentation. Your existing Kubernetes labels, AWS tags, and Azure tags become first-class filtering dimensions for all OpenTelemetry data, enabling automatic cost attribution and comprehensive data permissions using your existing RBAC patterns.

The result is that your OpenTelemetry observability inherits the same operational patterns, security boundaries, and cost structures as your Kubernetes infrastructure.

Why this matters for your team

Every organization adopting OpenTelemetry faces the same challenge: turning data collection into intelligent insights. The engineering teams that succeed are those that choose analytics platforms built specifically for OpenTelemetry data.

Dynatrace transforms your OpenTelemetry investment from a data collection project into a competitive advantage. We meet you where you are. We respect your choice to standardize OpenTelemetry by simplifying its operational complexity and enhancing it with analytics that actually deliver value.

Ready to transform your OpenTelemetry data?

Open standards have clear benefits. Industry standardization makes it easier to make sense of data coming from multiple different sources, whether it’s traces, metrics, logs, or telemetry from third-party tools. Your analytics platform can deliver intelligent insights across your entire technology stack. Your OpenTelemetry investment deserves analytics that reveal its full potential.

• Want to explore specific OpenTelemetry capabilities with Dynatrace? Try them out on the Dynatrace Playground
• Boost your productivity with these quick video guides for service owners working with OpenTelemetry:

Video: Easy access to your OTel Logs and Traces

Video: Analyze Service Failure from OTel Data

Video: Easy access to your OTel & Prometheus Service Metrics

Join us at OpenSource Summit. We’ll be in Amsterdam August 25-27. Stop by our booth to see the magic in action!

The post OpenTelemetry and Dynatrace: Complete unified observability analytics for modern applications appeared first on Dynatrace news.

Dynatrace has enhanced its runtime vulnerability analytics by integrating data from the Cybersecurity & Infrastructure Security Agency’s (CISA) Known Exploited Vulnerabilities (KEV) catalog, complete with critical remediation due dates. This powerful addition allows organizations to prioritize actively exploited vulnerabilities, align with federal security standards, and focus remediation efforts where they matter most—on threats that attackers are weaponizing right now.

The challenge of vulnerability prioritization

In today’s threat landscape, security teams face an overwhelming challenge in determining which vulnerabilities pose the greatest risk to their organization. With thousands of new vulnerabilities discovered each year, traditional scoring systems like the Common Vulnerability Scoring System (CVSS) often fall short of providing the real-world context needed for effective prioritization.

The reality is that not all vulnerabilities are created equal. While a vulnerability might receive a high CVSS score based on its theoretical impact, it might never be exploited in the wild. Conversely, some vulnerabilities with moderate scores are favorite tools of threat actors and are often involved in widespread attacks and significant business impact.

This disconnect between theoretical risks and actual threats creates several challenges for organizations:

• Resource allocation inefficiencies: Teams spend valuable time patching vulnerabilities that might never be exploited, while more dangerous threats remain unaddressed.
• Alert fatigue: Security professionals become overwhelmed by the sheer volume of vulnerability notifications, which leads to critical issues being missed.
• Compliance gaps: Organizations struggle to align their security practices with federal standards and regulatory requirements.

Enhanced Vulnerability Management with CISA KEV

Dynatrace already provides a Davis Security Score, an enhanced risk-calculation score that’s based on the industry-standard CVSS. Davis® AI provides this precise risk-assessment score by considering additional parameters such as public internet exposure and whether or not data assets are reachable from an affected entity.

Now, Dynatrace has further enhanced its runtime vulnerability analytics by integrating data from the CISA KEV catalog, including the critical due dates for remediation. This addition brings a new layer of actionable intelligence to vulnerability management by unlocking the ability to prioritize actively exploited vulnerabilities.

Unlike general vulnerability databases, KEV is based on real-world exploitation data and expert analysis. Each entry is carefully vetted and includes a due date by which federal agencies must remediate the issue. This due date is not arbitrary—it reflects the urgency of the threat and the need for swift action. By aligning with KEV, Dynatrace helps organizations stay ahead of attackers by focusing on vulnerabilities that are not just theoretical but actively weaponized.

The KEV catalog is especially valuable because it’s curated by cybersecurity experts at CISA using intelligence from government, industry, and open source reporting. Customers integrating KEV into their security workflows can align with federal standards, improve risk-based prioritization, and reduce exposure to high-impact attacks.

KEV filtering and prioritization

Once you’ve turned on runtime vulnerability analytics in Dynatrace, you can filter KEV vulnerabilities in the Vulnerabilities app. The resulting view is sorted based on the remediation due date, starting with the most urgent vulnerability.

Expanding KEV support

CISA KEV data is available directly within the Vulnerabilities app, giving your teams immediate visibility into the known exploited vulnerabilities that affect their environment and the remediation date by which they must be resolved. In the future, Dynatrace will expand support for KEV data to vulnerability dashboards and automation workflows so you can visualize KEV-related risks in your custom dashboards or assign remediation tasks based on KEV due dates or exploit status.

Try KEV filtering today

This integration helps organizations align with federal security standards, improve risk-based prioritization, and reduce exposure to high-impact attacks. It’s a practical way to stay ahead of cybersecurity threats by focusing on vulnerabilities that are actively exploited.

The post Prioritize vulnerabilities based on the CISA Known Exploited Vulnerabilities Catalog appeared first on Dynatrace news.

Python is a popular programming language with a clear and readable syntax. Python’s versatility allows it to be applied in various fields, from web development to data science. Detecting vulnerabilities in Python is crucial due to its widespread use in critical applications, which makes it a prime target for attackers. Additionally, Python projects often rely on third-party libraries, which can introduce risks if not properly monitored. Proactive vulnerability monitoring ensures compliance with security standards and helps prevent costly security incidents.

Real-time Python vulnerability detection in production

With Dynatrace Runtime Vulnerability Analytics extended to monitor Python, organizations can use the newest Python libraries while making sure that all code running in production and pre-production environments is subject to continuous, stringent security monitoring. Newly published CVEs are detected immediately, allowing security teams and champions to quickly analyze the actual risk, triage the most impactful vulnerabilities, and have them remediated. RVA analyzes if an application uses vulnerable Python libraries at runtime or a vulnerable Python runtime to execute the application code.

Similar to other monitored technologies, we provide full visibility into all affected processes, including related services, applications, and hosts, as well as Kubernetes workloads, nodes, and clusters. Mitigators can quickly prioritize vulnerabilities based on network exposure and understand which data is at risk and how easily vulnerabilities can be exploited by an attacker.

Python vulnerability monitoring is easily set up for all hosts monitored by Dynatrace OneAgent®. Dynatrace detects if you’re using vulnerable third-party libraries or runtimes to execute your code. Activate Python monitoring in the security settings of the Vulnerabilities app. The app allows you to filter, sort, and connect vulnerabilities to specific remediation tickets. Python vulnerability monitoring is also available for Dynatrace Managed customers.

Get started

Activate Python vulnerability monitoring in Vulnerabilities app settings or explore it in the Dynatrace Playground.

The post Dynatrace launches Python Vulnerability Monitoring for enhanced customer security appeared first on Dynatrace news.

In this blog post, you’ll learn how Dynatrace OneAgent® automatically identifies Journald and ingests structured logs into Dynatrace while enriching them with topology and infrastructure context.

Why migrate from Syslog to Journald

Journald provides a more modern alternative that addresses the limitations of existing Syslog implementations. It offers structured logging, fast indexing for search, access controls, and signed messages. Thanks to its structured and binary format, Journald is quick and efficient.

Journald provides unified structured logging for systems, services, and applications, eliminating the need for custom parsing for severity or details. It supports multi-line logs, handles log rotation, and even includes mechanisms to check for data corruption.

Overcoming native Journald limitations with Dynatrace

Journald doesn’t natively support easy log collection and centralization. Unlike rsyslog, which requires minimal configuration for centralization, Journald’s approach, using systemd-based operating systems, necessitates more advanced configurations and additional components.

Dynatrace OneAgent solves this problem by effortlessly detecting and centralizing logs while providing unmatched flexibility, thanks to the Dynatrace platform’s centralized and dynamic ingest rule controls.

Dynatrace comes with log ingest rules, including a built-in Journald log rule. When activated, this rule facilitates the OneAgent Journald log detector’s collection and ingestion of Journald-related logs.

The configuration is managed centrally within Dynatrace settings (Settings > Log ingest > Log ingest rules).

The Dynatrace platform and settings also allow you to define custom rules, such as defining which Journald units and logs should additionally be included or excluded.

Why unified observability boosts productivity

While journalctl is a powerful local tool with local filtering capabilities, it doesn’t scale well, especially considering the globally distributed components of today’s hybrid/cloud-hosted environments. When using Dynatrace, in addition to automatic log collection, you gain full infrastructure context and access to powerful, advanced log analytics tools such as the Logs, Notebooks, and Dashboards apps. For forensic log analytics use cases, the Security Investigator app benefits from the scalability and analytics power of Dynatrace Grail™.

Grail, the Dynatrace schema on-read data lakehouse, is at the heart of the Dynatrace platform. It provides unified observability by automatically correlating logs and placing them in the context of traces and metrics.

Unlike traditional setups that require predefined schemas, Grail allows you to store diverse data types without schema definitions at any point, providing greater flexibility in any analytic situation.

The Grail architecture ensures scalability, making log data accessible for detailed analysis regardless of volume. With real-time analysis, you gain faster data-driven decisions and simplified data ingestion. Dynatrace Grail lets you focus on extracting insights rather than managing complex schemas or index and storage concepts.

Davis® AI works with Grail as part of the Dynatrace platform’s integrated automation and AI capabilities.

System health, performance troubleshooting, and debugging situations no longer require manual correlation of logs across multiple disconnected tools or servers. Thanks to Davis AI’s automatic problem detection capabilities with logs in the context of entities, services, applications, and traces, your root cause analysis efforts are greatly reduced.

Why use Dynatrace OneAgent for Journald

Dynatrace OneAgent automatically ingests logs from Journald, eliminating the need for manual configuration and ensuring comprehensive log coverage while offering centralized and granular rules for log collection. In addition, OneAgent handles log rotation, decompression, and parsing automatically, reducing the overhead associated with manual log management.

With built-in log enrichment, OneAgent adds valuable attributes such as trace and span IDs to log records, facilitating deeper analysis and correlation with distributed traces. Journald logs are integrated into the Dynatrace platform this way, providing a unified view of system and application logs alongside other observability data.

Since you can ingest Journald logs using Dynatrace OneAgent, this feature is available to customers of both the latest Dynatrace SaaS and Dynatrace Managed.

Start collecting Journald logs with Dynatrace today

If you haven’t started collecting Journald logs with Dynatrace, install OneAgent on your systemd-based operating systems and consult Dynatrace Documentation for additional details if you want to configure custom rules to include or exclude specific Journald logs.

The post How to observe logs with Journald and Dynatrace appeared first on Dynatrace news.

Dynatrace Managed customers are increasingly seeking to access the latest innovations from Dynatrace. By transitioning to the Dynatrace SaaS platform, customers can benefit from recent innovations like Grail™, AppEngine, and Workflows. Our partner community plays a vital role in facilitating this transition by effectively communicating the benefits of SaaS and ensuring a seamless migration process.

During a recent webinar, Dynatrace partners had the opportunity to learn how they deliver greater value to their customers while expanding their account footprints. Peter Vinh, Senior Director of Product Management at Dynatrace was joined by Andre van der Veen, Technical Consultant for Dynatrace at Mediro ICT, to share valuable insights that can support Dynatrace partners in guiding their customers from Managed to SaaS.

Customer benefits of transitioning from Dynatrace Managed to SaaS

By helping customers understand the benefits of migrating to Dynatrace SaaS, partners are providing enhanced value to their customers. During the webinar, Peter Vinh highlighted a crucial point for partners to convey: the latest innovations on the Dynatrace platform, including Grail, Davis CoPilot™, OpenPipeline™️, and Workflows, are exclusively available to SaaS customers.

Beyond the benefits these capabilities unlock, there are also significant commercial drivers. Dynatrace SaaS presents a lower total cost of ownership (TCO), enabling customers to consolidate various tools, thereby optimizing costs and enhancing internal user experiences. This sentiment was later echoed by Andre van der Veen, who explained that TCO is one of the primary drivers for Mediro customers, alongside higher availability. By migrating to SaaS, customers can reduce hardware expenses, enabling them to concentrate on accelerating innovation with Dynatrace.

Key insights for partners to enhance customer engagement and accelerate SaaS migration

Vinh also highlighted the benefits of customer SaaS migration for the Dynatrace partner community. Moving customers from Managed to SaaS presents a valuable opportunity to grow customer accounts by delivering enhanced end-to-end value. Rather than maintaining a transactional relationship selling licenses, partners can provide continuous support. With quicker access to customers’ SaaS tenants, they can view issues from the customers’ perspective and collaboratively resolve challenges. Additionally, there is the potential for increased revenue through the sale of migration services and supplemental modules exclusive to SaaS.

Vinh provided valuable insights on how partners can enhance their conversations with customers, acknowledging the various roadblocks they might face. Here are his key recommendations:

Don’t wait for renewals. The conversation could take months, freeze windows could prevent new sign-ups in a certain period, and security teams could have a long list of questions that must be answered. It’s important to start early, working backwards from these dates to identify when to initiate the discussion.

Align to strategic initiatives. It’s important to engage senior business leaders and position the value of Dynatrace SaaS in a way that aligns to their strategy and objectives. Vinh highlighted the Dynatrace for Executives webpage as a great jumping off point for these conversations.

Reassure customers about their data. Van der Veen discussed that a key objection is often customer concerns about putting monitoring data in the cloud, and whether they will be able to access historical data from their Managed instances. He explained how those concerns can be allayed by discussing types of data the customer already stores in the cloud, showing how Managed clusters can be kept online for long term storage, and outlining the security certifications and standards that Dynatrace adheres to.

Start small if you need to. Some customers may have a complex environment, or multiple business lines that operate individually, making them reluctant to change. Starting with a small SaaS footprint can help to alleviate concerns, focusing on areas where capabilities like Grail can deliver the greatest immediate impact. This helps to prove the value, building the customer’s confidence, so the footprint can expand over time.

Use Dynatrace resources. Vinh also reminded partners of the support resources available to them, including the Dynatrace ONE team, customer success managers, Dynatrace University certifications, the community support, documentation, and assets on the partner portal.

Learnings from the frontline

Finally, van der Veen shared his first-hand experiences delivering Dynatrace Managed to SaaS migrations for Mediro customers. He was amongst the earliest adopters of the Dynatrace SaaS Upgrade Assistant, and provided helpful insights for other partners. His key points included:

• Communication is key, ensuring all stakeholders are bought into the value of moving to SaaS.
• Cleanse the environment prior to migration; removing unused dashboards, making sure the OneAgent version is compatible with SaaS, and ensuring only one person has administrative access.
• Restart processes after the migration to ensure it has been fully completed. Verify with the customer that the dashboards are fully operational, and data is coming through as expected.

Offering his closing thoughts, van der Veen highlighted that the value and benefits of upgrading from Dynatrace Managed to SaaS is huge for both partners and their customers. While the process may seem daunting, the tooling that is now available makes it much easier.

For further insights on how Dynatrace partners can help their customers move from Managed to SaaS, and for a demo of the Migration Assistant, watch the webinar on-demand.

The post Empowering partners to drive customer success through Dynatrace SaaS migration appeared first on Dynatrace news.

Modern software intelligence needs a new approach. It should be open by design to accelerate innovation, enable powerful integration with other tools, and purposefully unify data and analytics. Enter Grail-powered data and analytics.

Grail™ is a purpose-built data lakehouse for observability, security, and AIOps. Grail makes converging real-time, historical, and predictive analytics possible on a single platform. All innovations based on Grail are only available in Dynatrace SaaS environments. Customers using Dynatrace Managed, the on-premises market-leading observability solution, need to upgrade their deployments to leverage Grail. The SaaS Upgrade Assistant reduces the manual effort and imports Dynatrace Managed environment configuration automatically and seamlessly into the target SaaS environment, saving precious time and aiding a seamless upgrade.

As a pioneer in the industry, Dynatrace is rewriting cloud observability with Grail data lakehouse and all Grail-based innovations. Grail makes converging real-time, historical, and predictive analytics possible on a single platform designed to accelerate innovation, enable powerful integration with third-party tools, and unify data and analytics.

Dynatrace Managed is the market-leading observability solution for customers with mandatory on-premises requirements. Grail-based innovations are, however, only available in Dynatrace SaaS environments. Customers currently using Dynatrace Managed need to upgrade their deployments to leverage Grail. While upgrading to Dynatrace SaaS involves numerous considerations, SaaS Upgrade Assistant reduces manual effort by automatically and seamlessly importing Dynatrace Managed environment configurations and even migrating OneAgents to your new Dynatrace SaaS environment, saving you precious time and assuring a smooth upgrade.

Effective upgrade of large environments requires automation

Upgrading your Dynatrace Managed environment to Dynatrace SaaS takes time and careful planning, depending on the size and complexity of your existing environment. We recommend investing time in planning to make the upgrade experience seamless.

• Larger environments have potentially hundreds of configurations that need to be migrated, creating a real need for automation.
• Tracking the upgrade process, which can take weeks, is daunting and requires automation to ensure nothing is missed.
• Incorrect environment configurations can lead to disruptions for Dynatrace users.
• Every custom Dynatrace dashboard needs updated ownership information.

Dynatrace SaaS Upgrade Assistant automatically imports your environment configurations

SaaS Upgrade Assistant is a Dynatrace® app that imports your Dynatrace Managed environment configuration. It enables Dynatrace Managed cluster administrators to navigate through a frictionless upgrade from their Dynatrace Managed on-premises deployment to the Dynatrace SaaS deployment model. A cluster administrator can import the configuration to the target SaaS environment and update it to align with SaaS environment requirements. The app ensures a faster migration and minimizes disruption to Dynatrace users that might be caused by incorrect environment configuration. Automation eliminates time-consuming manual tasks, such as updating dashboard ownership or adjusting entity IDs that have changed between environments.

Before deployment in your new Dynatrace SaaS environment, you can update the imported configuration from Dynatrace Managed directly in SaaS Upgrade Assistant. Two editing modes are available: single and bulk. Once imported, you receive a report showing how much of your Dynatrace Managed configuration was replicated in the new environment.

The intuitive web UI of SaaS Upgrade Assistant helps ensure that OneAgent® and ActiveGate® migrations align with your desired configuration. You can also update all dashboard owners to match the user identifiers in your new Dynatrace SaaS environment. Preview mode gives you confidence that your migration is on track.

Get started

SaaS Upgrade Assistant is generally available with configuration export from Dynatrace Managed, starting with Dynatrace Managed version 1.294 on Dynatrace Hub. Simply install it in your SaaS environment and follow the instructions.

• For a demonstration of SaaS Upgrade Assistant, see SaaS Upgrade Assistant – Dynatrace App Spotlight.
• For a closer look into incorporating the new app into your upgrade to Dynatrace SaaS, see How to upgrade from Dynatrace Managed to Dynatrace SaaS effectively.
• For more information on migrating configurations, see Migrate configurations documentation.
• Are you ready to upgrade to SaaS? Answer the questions in our SaaS readiness assessment to find out.

Don’t forget to share your feedback in the Upgrade to Dynatrace SaaS community.

The post Saas Upgrade Assistant offers smooth migration of configurations from Dynatrace Managed to Dynatrace SaaS appeared first on Dynatrace news.

Stay compliant in an ever-changing environment

As a platform for unified observability and security, Dynatrace provides intelligent answers that help you accelerate digital business transformation. Observing complex environments involves handling regulatory, compliance, and data governance requirements. This continuously evolving landscape requires careful management and clarity regarding how sensitive data is used. This is particularly important when dealing with large volumes of data.

Dynatrace multi-layered masking capabilities allow you to conveniently stay on top of these requirements. They ensure you get answers from your invaluable data while effortlessly taking care of sensitive data points related to digital compliance requirements for GDPR, CCPA, HIPAA, LGPD, PCI-DSS, and more—at scale throughout the Dynatrace data lifecycle!

Control how sensitive data is used

Empower yourself with complete control as you effortlessly select the data you want to capture with Dynatrace. The whole experience is designed to fit into your individual data governance and compliance framework. To reach this granular level of control, Dynatrace offers data masking features on multiple levels that can be applied in layers:

Masking at capture: Data is persistently masked at first contact with Dynatrace. When using Dynatrace OneAgent^®, captured data doesn’t leave the monitored environment.

Masking at storage: Data is persistently masked upon ingestion into Dynatrace. (Besides masking, you can also set up other customized operations via the ingest pipeline configuration).

Masking at display: Sensitive data points are stored in their original form but are only readable by users with the right permissions.

With this three-layer design, you enjoy maximum flexibility in managing sensitive data points end-to-end.

Leverage three masking layers

Masking at capture and masking at storage operations exclude targeted sensitive data points. Data points that undergo this operation are not recoverable or re-engineerable, making it the perfect solution for controlling which sensitive data points are ingested by Dynatrace. Other tools typically rely on attributes and markers that require unnecessary manual effort. This includes digging through each monitored data source and adding tags to the sensitive data points; this process is usually expensive, exhausting, error-prone, and unscalable.

Dynatrace offers a more user-friendly approach, leveraging a mix of pre-defined patterns to be recognized and excluding selected sensitive data points like credit card numbers, email addresses, end-user IP addresses, and more.

Sometimes, you might need to configure additional masking rules; this is possible using the respective settings (see the illustration below). Especially when working with log data, keeping the context of the original data points is often crucial. For such cases, Dynatrace offers the option to replace sensitive data points with a hash or a self-defined string, calling out the replaced data point type (for example, <emailaddress>).
Read more about these options in Log Monitoring documentation.

Masking at display hides data points from the readers who don’t have sufficient permissions, giving you granular control down to the record level. Instead of limiting access to all the valuable information about a monitor or a whole webpage, this approach hides selected data points based on fully configurable permission policies, helping your teams stay productive while enforcing your data governance rules.

Upgraded masking at capture for OneAgent

With recent upgrades to OneAgent*, we introduced new extended at-capture masking capabilities and now provide masking rules for the most commonly encountered sensitive data points in URIs and exception messages. You can select to mask email addresses, query parameters, financial and payment card numbers, IDs, and other numeric values that might not be needed. The selected rules can be configured for a whole environment or, more granularly, for specific process groups. When enabled, such data points are masked directly at first contact with OneAgent and never leave your monitored environment.

*Does not include RUM JavaScript, which will be delivered in an upcoming release.

Granular masking configuration made simple

Dynatrace masking settings can be configured for a whole environment or granularly for specific monitored entities like process groups, log sources, etc.

Configuring masking at capture settings for a whole environment has never been more straightforward. Go to Settings > Preferences > Data privacy. Choose your preferred configuration for masking at first contact using the settings on the OneAgent Side Masking and IP Masking tabs.

The settings on the General tab remain unchanged and govern the privacy settings executed on the server side (masking at storage), offering a second layer of control. Emerging industry standards indicate that relying solely on one layer of protection is no longer sufficient. Dynatrace provides multiple-layer protection out-of-the-box to help ensure your sensitive data is safeguarded.

Open the available tabs to explore and easily tailor your data privacy settings.

While environment-level settings effectively ensure a base standard, specific use cases may require special handling of sensitive data points. To fine-tune your masking settings, select the entity you want to adjust and leverage the entity-specific settings. See the process-group settings example in the screengrab below.

Why the Dynatrace solution is unique

Masking at capture

A single toggle lets you turn on the masking of selected sensitive data points at first contact with Dynatrace, so sensitive data never leaves your environment.

Multiple layers of protection

In addition to masking at capture, you can leverage:

Masking at storage to manage data directly ingested into Dynatrace or have an additional safety layer of masking.

Masking at display to effortlessly govern access to sensitive data points.

Easily configurable masking

• Control your sensitive data points with simple and granular configuration.
• Choose a setup that applies to your whole Dynatrace environment or configure masking for specific process groups, or log data sources.
• Dynatrace masking capabilities scale with your ingested data volume.

What’s next

We’re working on further improvements to our easy-to-use masking toolkit and will soon extend the masking at-capture capabilities to RUM JavaScript. Stay tuned to this Privacy Spotlight blog series to learn more about these continuous improvements.

Check out Dynatrace Documentation for data masking and other privacy controls.

Learn more about our commitment to providing you with control and transparency over your customers’ personal data in the Dynatrace Trust Center.

Share your feedback, suggestions, and ideas with us in the Dynatrace Community.

The post Privacy spotlight: Control compliance in Dynatrace with multiple layers of sensitive data masking appeared first on Dynatrace news.

Real-time streaming needs real-time analytics

As enterprises move their workloads to cloud service providers like Amazon Web Services, the complexity of observing their workloads increases. Log data—the most verbose form of observability data, complementing other standardized signals like metrics and traces—is especially critical. As cloud complexity grows, it brings more volume, velocity, and variety of log data.

Managing this change is difficult. Without the ability to see the logs that are relevant to your service, infrastructure, or cloud function—at exactly the right time and in exactly the right format—your cloud or DevOps engineers lose the ability to find the root causes of the issues they troubleshoot. Even the AIOps approach doesn’t cut it if you don’t have proper logs in your observability platform.

Amazon CloudWatch is the most common method of collecting logs across your AWS footprint. As a native tool used by many enterprises, CloudWatch supports a wide range of AWS resources, applications, and services.

Amazon Data Firehose helps stream logs to the right destination

But your SREs and DevOps engineers know CloudWatch is not the terminal destination for data but rather an intermediate station. Their job is to find out the root cause of any SLO violations, ensure visibility into the application landscape to fix problems efficiently and minimize production costs by reducing errors. SREs and DevOps engineers need cloud logs in an integrated observability platform to monitor the whole software development lifecycle.

When trying to address this challenge, your cloud architects will likely choose Amazon Data Firehose. This fully managed native service is indispensable for streaming high-frequency logs collected by CloudWatch.

In some deployment scenarios, you might skip CloudWatch altogether. Take the example of Amazon Virtual Private Cloud (VPC) flow logs, which provide insights into the IP traffic of your network interfaces. VPC flow logs can be used as the source for troubleshooting connectivity issues, implementing security incident investigations, detecting intrusions, or managing access control issues. VPC flow logs can be massive in volume as your cloud deployment footprint grows, and directly streaming these logs with Amazon Data Firehose can be the most cost-effective method.

After configuring Amazon Data Firehose, your teams discover they have completed only the first part of the observability jigsaw puzzle. They also need a high-performance, real-time analytics platform to make that data actionable.

Direct link from AWS to Dynatrace

Dynatrace delivers the missing piece for AWS cloud observability with native Firehose integration. This complements our existing AWS logging integrations like S3 log forwarder, Lambda layer log forwarding, or direct log ingest API. These already provide a common integration with AWS log sources. The new Firehose integration removes intermediary components that previously required additional maintenance and provides a direct link from AWS to Grail data lakehouse.

This means high-frequency streamed logs from Firehose can be captured in your Dynatrace environment, automatically processed, stored in Grail for the retention period of your choice, and included in the full observability automation suite of the Dynatrace® platform, apps, and Davis^® AI problem detection.

With this out-of-the-box support for scalable data ingest, log data is immediately available to your teams for troubleshooting and observability, investigating security issues, or auditing. As logs are first-class citizens alongside traces, metrics, business events, and other data types, you have an observability platform ready to scale with you in your cloud-native journey.

Easy setup takes just a few steps

Setting up a direct ingest of Firehose log data is quick and easy.

First, you need to generate an API key to ingest logs. In the Dynatrace web UI, go to Access tokens and select Generate new token. Select ingest logs as the scope of the token. Then, generate the token.

Next, go to the AWS console to configure the forwarding of data streams defined in your log groups. Data Firehose stream requires a trusted relationship with CloudWatch through an IAM role. Follow the instructions available in Dynatrace documentation to allow proper access and configure Firehose settings.

Now, you can set up your Firehose stream. The preferred way is to use a CloudFormation template that streamlines and automates the process. See CloudFormation template documentation for details.

Alternatively, you can configure the stream in the AWS web console. Choose Dynatrace as the Destination in the AWS console and complete the other fields with the correct parameters.

Now, you can view your cloud logs in Dynatrace!

For example, open the Clouds app with integrated logs in the context of your Lambda functions observability for one-click access to error logs.

When Dynatrace Davis AI detects a problem in your environment, you can also see relevant logs streamed via AWS Firehose that are related to the problem. When analyzing a problem, look at the related service, which displays related log data. This lets you jump right to the error that provides details of the problem.

When doing proactive health checks or analysis, you can inspect log data in Notebooks. For example, pick a template to explore data or write your own DQL query and chart incoming error rates from logs streamed via AWS Firehose.

Try it out today

• Direct integration with Amazon Data Firehose is generally available in Dynatrace version 1.290 for Dynatrace SaaS (Log Management and Analytics) and Dynatrace Managed (Log Monitoring Classic).
• Read the Dynatrace SaaS documentation for streaming logs via Amazon Data Firehose (or see our Dynatrace Managed Documentation).
• Need to support an alternative cloud log setup? Explore these other integrations:
  • AWS S3 log forwarder
  • AWS Lambda log forwarding
  • Direct log ingest API
  • Fluent Bit log forwarder

Share your experience

We’d love to hear from you. Share your use cases for Amazon Data Firehose integration with the Dynatrace Community.

The post Stream logs to Dynatrace with Amazon Data Firehose to boost your cloud-native journey appeared first on Dynatrace news.

This blog post is for both novice and seasoned audiences alike. If you’re interested in the relevance and utility of SLOs and how they might be helpful for you, you’ve come to the right place.

The first part of this blog post briefly explores the integration of SLO events with AI. The second part delves into the approach to be adopted with SLOs based on a client’s context and environment, considering whether SLAs (Service Level Agreements) have already been historically defined. Finally, this post addresses the practical aspects, focusing on cultivating the right leading indicator instincts for problem detection through SLOs and extracting their full value to meet business and functional requirements.

Strategic approach to root cause detection

Two frequently used SLOs are the Apdex score and failure rate. For a more proactive approach and to gain further visibility, other SLOs focusing on performance can be implemented.

Every problem identified through Dynatrace Davis® AI indicates an issue with potential user impact. However, understanding the precise impact on the end user can be challenging. For instance, consider how fine-tuning failure detection can provide insights for comprehensive understanding.

Davis AI uses Dynatrace Smartscape® topology to map out the application chain. As a result, the AI operates on the related events and detection parameters (threshold, period, analysis interval, frequent detection, and so on) an issue represents. Using an SLO, you can target the desired entity and map the incoming and outgoing interactions against it.

For example, envision an apple tree that drops an apple. Can the AI discern exactly where and when the apple dropped? And what caused the apple to fall? Was it the wind, a natural progression, a leaping cat, or something else entirely? Likewise, what is the result of the apple falling? Did it split open on the ground or dent a car?

These conditions are pivotal for pinpointing the root cause and facilitating an expansion of the AI’s detection capabilities by incorporating statistical and forecasting models. Consequently, we recognize the value in augmenting AI capabilities through a Dynatrace generative AI model that incorporates user feedback, enhancing its functional dimension.

Further in this article, we will explore which empirical approach to adopt in order to best align with the service-level objectives (SLOs). Specifically, we can initially divide the approach into two parts. We either know the entities to apply the SLO, or we don’t.

Which approach should you take?

SLO methodology is based on empirical approaches with some catalyzers. The approach we will adopt for implementing SLOs follows the diagram illustrated below. Either you have no immediate functional or business visibility (path 1), and you will implement your initial SLOs, or you have business assistance (path 2) and can strategically position your SLOs in key areas.

There are two possible scenarios: You either have defined SLAs, or you do not.

This approach can be accurately validated through empirical means. From my experience, a month of monitoring is the optimal duration to gain statistically significant insights into “how my entity behaves with the configured SLO.” Consequently, by understanding the intricacies involved, this enables us to swiftly confirm or refute the functional/business case of the business teams involved, including project managers, top management, developers, and tech leads.

Path 1: SRE is siloed and lacks familiarity with the instrumented application.

The suggested approach involves utilizing Frontend SLOs to swiftly draw attention to business-related issues. This method generates interest and prompt action by promptly highlighting pertinent business concerns.

Next, a pragmatic approach involves examining the backend, focusing on Service type entities prominently exposed to the frontend (for example, Apache Tomcat in a Linux environment). Additionally, meaningful functional names should be considered, and the number of calls (throughput) should be analyzed for applying Service Level Objectives (SLOs).

Path 2 (easier): I’m familiar with my application and the development team.

The recommendation emphasizes promptly adopting Backend Service Level Objectives (SLOs). In today’s landscape, we lack a clear understanding of properly creating frontend SLOs (for example, RUM application type entities) based on key user actions. Often, businesses find themselves in disagreement or simply lost. The most user-friendly and effective approach to pinpointing weaknesses involves establishing a “Service SLO” (dt.entity.service, the “backend”). Given that the implementation of backend SLOs is generally better understood by the business/development and allows for a straightforward and rapid placement of SLOs on services, controllers, or APIs that are generally comprehensible to the in-house developers of the client. That said, the guiding thread, predominantly under the client’s control, remains the backend perspective. In other words, where the application code resides.

However, it’s essential to exercise caution: Limit the quantity of SLOs while ensuring they are well-defined and aligned with business and functional objectives. For example, in a specific application type like e-commerce, select significant Service names (such as Basket, CardPayment, or UserController).

When the SLO status converges to an optimal value of 100%, and there’s substantial traffic (calls/min), BurnRate becomes more relevant for anomaly detection.

What characterizes a weak SLO?

Let’s assume we created a service-availability SLO, monitoring the request failure count against the overall request counts. Now, let’s try to explain what the reasons for no AI detection could be related to this SLO event

AI does not correctly detect that two relevant criteria are involved in disadvantaging AI detection:

When the status of the SLO falls below the specified target by a certain threshold (for example, below 90%), it signifies a failure rate event, indicating an average error rate of 10%.

This implies that when the status is unfavorable, implementing sophisticated alerting methods like error budget burn rate alerting presents challenges and is therefore not applicable. Bad status indicates constant violations of the threshold, akin to the state of a broken door that requires fixing before defining the Service Level Objective (SLO).

For instance, if we maintain an average SLO status of 80%, indicating an average error rate of 20%, utilizing the error budget burn rate in this scenario might be challenging. A simple ratio of 2 implies a 40% error rate, a situation that rarely occurs unless there’s a service interruption leading to observed consequences.

Therefore, these considerations outline two types of coupling for the SLO: Rapid adoption of backend SLOs and cautious application of error budget burn rate alerting, aligning with the traffic and error rate observations for effective anomaly detection.

When there isn’t enough traffic (requests/min) for an SLO

Detection becomes sporadic, resulting in insufficient data points for establishing the sampling interval necessary for generating “Event duration.”

This necessitates a subsequent description of how we should apply default transformation in this specific case, only if this SLO is deemed worthy of tracking.

Strong SLOs

With a view to becoming highly sensitive to detection and adopting a proactive approach, we need to fulfill this condition. If we have constant traffic (meaning sufficient data points to feed the entity) and an averaged baseline, thus exhibiting a regular trend, then it would be easier to detect significant deviations from the behavior of the entity. Therefore, configuring SLOs for this type of entity is considered “strong”, and the error budget burn rate makes sense in detecting these deviations. This proactive stance allows us to maximize our focus on the actual impact experienced by the end user in real time. See the following example with BurnRate formula for Failure rate event.

Error budget burn rate = Error Rate / (1 – Target)

Best practices in SLO configuration

To detect if an entity is a good candidate for strong SLO, test your SLO. SLOs must be evaluated at 100%, even when there is currently no traffic.

If the targeted entity is validated as relevant for the SLO and occasionally experiences a lack of traffic, then using the “default” transformation in the SLO expression is advisable to prevent misleading SLO status!

Use the default transformation.

This example shows the SLI of a performance SLO, targeting response times/loading times of a key-user-action/DOMload.

((
(builtin:apps.web.action.domInteractive.load.browser
:splitBy("dt.entity.application_method")
:avg
:filter(and(or(in("dt.entity.application_method", entitySelector("type(~"APPLICATION_METHOD~"),entityId(~"APPLICATION_METHOD-XXX~")")))))
:partition("perf",value("good",lt(100))) 
:splitBy()
:count
:default(0))
/
(builtin:apps.web.action.domInteractive.load.browser
:splitBy("dt.entity.application_method")
:auto
:filter(and(or(in("dt.entity.application_method", entitySelector("type(~"APPLICATION_METHOD~"),entityId(~"APPLICATION_METHOD-XXX ~")")))))
:splitBy("dt.entity.application_method")
:count
:default(1)
))
:default(1)
*
(100))

The outer default(1) is employed in this context to specify how any data should be handled. This assumes that if there’s an absence of data, the Service Level Objective (SLO) must be assessed at 100%. This approach prevents the truncation of the SLO health state in cases where no data is received, ensuring that the SLO status remains intact even without data.

• Data Explorer “test your Metric Expression” for info result coming from the above metric.
  Following the previous metric (above) used for the SLO, the threshold employed is an average of 100 ms for the Key Performance Indicator (KPI) of DOM Interactive.
  
  If this threshold is exceeded during THE test within Data Explorer, I will have the tendency or a preemptive glimpse of the impending alert. This allows me to anticipate the threshold and future adjustments to the alert taken by the SLO. In other words, the peaks here indicate that the condition of the SLO is met, resulting in a status of 100%, meaning we adhere to the set threshold. Outside of these peaks, the threshold is violated! On the other hand, if the threshold is violated, it decreases the error budget.Spikes indicate that the conditions for meeting the Service Level Objective (SLO) are fulfilled. However, the gap and space between these impactful SLO statuses, on the other hand, contribute to an increase in the error budget.
• Service type (General Parameters Exceptions / mute request)

To maintain SLO integrity, we employ various strategies, such as muting requests, ignoring certain elements, or enforcing exceptions only when a problem is identified. This approach ensures that SLO degradation is prevented unnecessarily, with the aim of preserving performance until a patch release can be implemented.

As mentioned earlier, it is necessary to have a minimum observation period to determine the SLO behavior of a targeted entity. It is reiterated that the positioning of this SLO includes the selection of strategic factors (entity positioning within the application architecture = ServiceFlow, sufficient traffic in requests per minute, etc).

During this observation phase, business stakeholders will quickly validate this case and thus define a waiting period for the implementation of the application fix. It is understood that during this waiting period, we will mute the actual exception/error impacting the health of the SLO.

Caution is advised, as by doing this, it should not come as a surprise to observe a better health state (score) of the SLO. This is normal because we have bypassed a parasitic part constituted by the exceptions and errors that actually cause user impact.

Find below the way to detect potential user-impacted cases thanks to SLO.

Details of the root cause

The developer deems it appropriate to either exclude or designate this error as acceptable during the patch release to prevent being overwhelmed with false positive alerts. Most importantly, this should not impede the health status of the SLO, as this is a recognized issue.

• The metric selector must be split by service

(100)*(builtin:service.errors.server.successCount:splitBy("dt.entity.service"))/(builtin:service.requestCount.server:splitBy())

Splitting with the appropriate entity dimension enables the AI to more accurately pinpoint the created Service Level Objective (SLO). Consequently, when triggering an alert, this approach facilitates drilling down into the affected entity, thereby enabling a deeper understanding of the root cause behind the issue.

• Smart alerting approach: BurnRate/ Status and AlertingProfile

E-Commerce Use Case

In this instance, upon reviewing the week’s alerts, it’s evident that parasitic noise has significantly decreased due to implementing good practices. There were 441 issues without intelligent configuration, whereas with intelligent configuration, only 29 were identified as potentially impacting the end user.

The kinetics of the impact on the Service Level Objective (SLO) are represented by the red arrow, illustrating the concept of error budget burn rate, which indicates how rapidly the error rate escalates.

Based on the IT incident indicators, our MTTD is 3 min for this event below, degrading the SLO.

For more info on how to configure and use BurnRate, see SLO monitoring alerting on SLOs error budget burn rates.

Conclusion

An effective Service Level Objective (SLO) holds more value than numerous alerts, reducing unnecessary noise in monitoring systems. The crucial final step is to highlight strong signals that genuinely impact the end user. Validating and integrating this approach into an intelligent ITSM tool like ServiceNow, for example, will optimize service management by aligning IT services with business needs, ensuring efficient delivery and improved user experiences.

The post Efficient SLO event integration powers successful AIOps appeared first on Dynatrace news.

Knowledge is power. Knowing who is responsible for specific areas and services is a vital skill that saves time and hassle. This can quickly become overwhelming and hard to manage in today’s complex microservice environments. Combining services and responsible teams increases transparency, simplifies the identification of the right people, enables automation, and saves time and money.

As always, work must be done before you can reap the rewards. This includes preparing the environment with the required metadata to provide the ownership information transparently and when needed. These efforts include:

• Extending entities with ownership metadata.
• Enriching ownership information with desired metadata.
• Ensuring ownership coverage in the environment.
• Automating incident triage via targeted remediation or notification tasks.

What to consider when adding ownership information

Introducing ownership information requires several simple steps and typically depends on the desired application. In any case, three building blocks are required to connect the right information to software artifacts.

• Linking team ownership with the specific service or component they own.
• Enriching team ownership information with desired metadata.
• Providing easy access to team ownership information.

The first challenge is the assignment of team ownerships to the appropriate entities. At first glance, this sounds easy: Team A takes care of component B. However, as software changes continuously due to new deployments and releases, so can responsibilities. Revisiting an environment to ensure the assignments are still correct is almost impossible when done manually. The automated extraction of ownership information, for example, from Kubernetes annotations, is therefore essential.

Secondly, knowing who is responsible is essential but not sufficient, especially if you want to automate your triage process. Essential metadata about team ownership helps in addressing the right team with the appriopriate responsibilities via the desired channels. As teams and their structure and metadata are often maintained in a dedicated database, such as Microsoft Entra ID (formerly Azure Active Directory) or ServiceNow. This information needs to be reusable, avoiding the need to maintain the same information twice and ensuring data is not out of sync.

Keeping ownership teams and their properties up to date is essential, as is having the right contact information available when needed.

Finally, the best information is still useless if users can’t retrieve it quickly when needed and use it accordingly. Be it a visual representation or an automated task in a workflow, easily resolving ownership information for services is where the full value becomes noticeable.

How to efficiently introduce team ownerships

Dynatrace provides different ways of associating team ownership with entities and adding desired team metadata, such as contact details, to your environments.

Import teams

It is necessary to get ownership team information into the system and keep it updated. Dynatrace offers several ways to ingest ownership team information. Besides supporting UI and API input for ownership teams, a dedicated workflow action for importing, storing, and updating ownership teams is available.

The import_teams workflow action can be used in either an on-demand or a regularly triggered workflow to get ownership team information and store it accordingly. As data sources can vary, the import_teams action supports different options. Besides the generic import option of accepting JSON objects in the ownership schema (find details on how to use the import teams workflow actions here), two often-used databases for storing and maintaining team information are available to you:

• Microsoft Entra (formerly Azure Active Directory)
• ServiceNow (sys_user_groups)

The ownership team information can be queried regularly from the databases, maintaining them as the source of truth for the team and its properties. A dedicated workflow allows automatic synchronization of the database information with Dynatrace and keeps the ownership team information up-to-date.

Dynatrace ownership functionality supports configuration-as-code via its proprietary Monaco (Monitoring as code) CLI or Terraform. A workflow template for importing teams can be found in this public GitHub repository.

Assign teams to services

This correlation between people and software is crucial; if a problem occurs or a new security vulnerability is detected, it’s critical to quickly and easily know who is responsible and owns each specific software artifact.

As the ownership of certain components is highly dependent on the organization structure and possibly quite dynamic or heterogeneous, the linking is realized via key-value pairs added to the entities. This ensures flexibility in adding the ownership information when needed.

Since adding key-value pairs to entities is highly flexible, there are several ways of adding ownership metadata to an environment. However, to increase reliability, it’s recommended to add the ownership information depending on the environment characteristics. In cloud-native environments Kubernetes annotations or labels are recommended; these can be further used to propagate certain information within the software topology. Dedicated environment variables or custom properties are additional options to add ownership related metadata. More details on the supported ways of enriching your environment are described in Best practices for ownership information documentation.

When adding ownership teams, the key must start with one of five customizable indicators to be recognized as ownership metadata. By default, Dynatrace supports dt.owner or owner as the key prefix, while the value needs to reflect the unique team identifier. This guarantees an unambiguous identification of the right owners per component. The team identifier can be set only when creating the ownership team for the first time, either through the previously explained ownership importer or manually via the web UI or API.

If certain naming conventions or key-value pairs are already used to tag services or apps with responsible teams, it’s possible to reuse the same keys easily. For example, if team:myTeamName is already used to mark selected components, then team can be added as a supported key. Dynatrace will automatically recognize the existing tags as ownership metadata. Otherwise, it’s recommended that you reuse the default key prefix to stay consistent throughout the environment.

To keep an overview of which areas in an environment are already covered and where there might be potential blind spots, ownership information (or the lack thereof) can be easily visualized in notebooks and dashboards. The dashboard shown in figure 4 below can be found in the publicly accessible Github Repository:
https://github.com/dynatrace-perfclinics/platform-engineering-demo/blob/main/dynatraceassets/dashboards/team-ownership-dashboard.json

The dashboard serves as an example and likely needs to be adapted to your specific user needs.

Ownership  

If it comes to the worst case, and a security vulnerability is introduced or a severe incident happens, it’s important to automatically inform the responsible teams and start remediation actions.

With Dynatrace Workflows, triaging time can be minimized and eliminated. Based on the detected issue or event, a workflow can automatically retrieve the associated ownership teams during the execution. The dedicated get_owner workflow action, queries all teams of affected entities. Further the ingested ownership metadata, such as contact details can be used to prepare Jira tickets or notify the right teams with all the crucial information they need to immediately start issue resolution.

Some examples of how ownership information enables automated notification and remediation workflows are listed below.

• Automated Release Validation and notification of the responsible teams using the Site Reliability Guardian
  • https://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.automations/workflows/b9305953-0512-33be-9504-75039b00f5d5

• Problem Remediation Automation utilizing Red Hat Ansible Automation Platform

• Kubernetes Workload Optimization

• Security Vulnerability Processing

Another example of utilizing targeted notifications via Dynatrace Workflows can be found in this public GitHub repository. These examples can be extended to cover similar use cases as above.

What’s next

Dynatrace already provides the possibility to reference ownership team information from entities, ingest crucial ownership team metadata such as contact information in an efficient way, and offers automated access to this information either via the user interface or Dynatrace Workflows. This enables a wide possibility of use cases where automated and targeted notifications play a significant role in reducing MTTR and increasing efficiency and reliability.

To further improve the user handling of the ownership functionality, we are currently working on making this information easily accessible via DQL (Dynatrace Query Language). Making use of all relations and dependencies will further simplify the retrieval of necessary information reliably when needed.

Follow these steps to increase an environment’s transparency and prepare environments to automatically inform and notify the right people at the right time:

• Ingest ownership metadata, such as contact details, into a Dynatrace environment.
  • Using the teams-importer to keep ownership data automatically in sync.
  • An example of how to set up an ownership import workflow is given in our workflow-samples repository.
• Extend components and services with ownership information.
  • Use object-specific and reliable association methods such as Kubernetes annotations, environment variables, host metadata, etc. More details can be found in Dynatrace Documentation.
• Detect blind spots and ownership gaps in environments.
  • Use dashboards and notebooks to keep an overview of the current state and proactively detect blind spots.
    • An example can be seen in our platform engineering demo (If you want to try out the platform engineering demo, please look at this repository, where you can run the demo within GitHub codespaces.
  • Set up automation workflows, tailoring and targeting ticketing and remediation activities based on the affected entities and occurring events, problems, or security vulnerabilities. Checkout the already available examples within our publicly available Configuration as Code GitHub repository or our Dynatrace Discover tenant.

The post The right person at the right time makes all the difference: Best practices for ownership information appeared first on Dynatrace news.

Getting insights into the health and disruptions of your networking or infrastructure is fundamental to enterprise observability. Syslog is the go-to protocol that delivers infrastructure administrators, network engineers, and security team logs that tell them all they need to know about their systems’ delivery, performance, availability, and security.

Without syslog, you’re blind to what happens on your infrastructure

While syslog is a common way to gain insights into enterprise infrastructure operations, integrating it with other signals into an observability overview is often a painful experience.

Syslog is a protocol with clear specifications that require a dedicated syslog server. This is needed to collect messages across your systems because many different types of devices and applications can produce logs in the syslog format.

However, enterprise adoption at scale typically has much higher requirements for components than for supported features—components must have proper vendor support. Without vendor support, you’re betting your business on goodwill. Even for a supported component, delivering logs from applications and infrastructure to DevSecBizOps workflows requires significant manual configuration.

For example, a supported syslog component must support the masking of sensitive data at capture to avoid transmitting personally identifiable information or other confidential data over the network. Log batching, enrichment, transformation, log source distinction, and application offloading are also regular requirements.

As enterprise environments scale enormously, filtering and dropping data “at the edge” before transmission to a central collection point must be a supported option.

Compliance, retention, archiving, or data governance regulations often require multicasting logs from the original source to multiple destinations, like an observability platform with long-term log storage.

In the end, site reliability engineering (SRE) and security teams need to have data delivered via syslog to their observability platform, in the context of other data types.

Syslog will remain a proven log solution because without understanding why connections are dropped, server starts or stops, or which requests your firewall blocked, your organization runs like a ship where the captain on the bridge has no understanding of what’s going on at the lower levels of the ship. However the challenges in maintaining syslog in a cloud-native era create a maze of requirements that SRE teams and infrastructure administrators must navigate, often finding themselves maintaining multiple tools and components.

This increases the risk of multiple points of failure, adds overhead, and ultimately fractures observability overview with prolonged time needed to recover from potential outages.

Start monitoring syslog using OpenTelemetry under the Dynatrace umbrella of support

OpenTelemetry has been a rising star in the observability landscape and is often a preferred way to achieve end-to-end visibility with a vendor-agnostic footprint. Dynatrace has been a part of the OpenTelemetry journey for years and has contributed to its rise.

With the new Dynatrace OTel Collector distribution, we provide a streamlined and supported way to collect logs using the syslog protocol. This fills all the requirements enterprises have and makes it hassle-free to stream syslog to Grail data lakehouse integrating logs with other observability data.

The Dynatrace OTel Collector for syslog has numerous benefits. Our approach is to understand what components our customers need and value. We then integrate them with our observability platform and offer support, so you don’t have to worry about unsupported bugs or lack of ownership.

We also provide security updates and patches to critical vulnerabilities that may arise in the components. This alleviates the risk of open source components with unpatched vulnerabilities remaining open to exploitation long after they have been revealed.

Ultimately this combination of Dynatrace support and the OpenTelemetry standard gives you the best of both worlds—enterprise-grade software support with open source community contributions.

Dynatrace OTel Collector fits with your existing setup

The new Dynatrace OTel Collector fits nicely into your existing Dynatrace setup to bring in syslog data. Our existing log ingest API already supports your logs using the OpenTelemetry protocol, so you just need to deploy the collector and point your syslog producers to it.

To start using the Dynatrace OTel Collector, take the following steps:

1. Generate an API token for the OTLP endpoint in your environment.
2. Find our newly released Dynatrace OTel Collector, deploy it, and configure the exporter with your API key and environment ID.
3. Configure receivers to enable different log sources for your syslog producers.
4. Point your syslog sources to the collector and you’re done!

This diagram explains how the components communicate with each other.

Take a look at this example for configuration. After generating an API token and deploying the collector, configure your instance. You need to configure each component (receiver, optional processor, and exporter) individually in a YAML file and enable them via pipelines. Follow the examples below or refer to Collector configuration documentation.

To point the exporter to your environment’s OTLP endpoint, add the following configuration:

exporters:
  logging:
    verbosity: detailed

  otlphttp/tenant_1:
    endpoint: "https://{your-tenant}.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token {your-api-token}"

Next, you can add receivers to your collectors, for example, F5 BIG-IP systems to log to a remote syslog server (version 11.x-17.x). Refer to F5 BIG-IP documentation for detailed and up-to-date instructions regarding remote Syslog configuration. Take a look at Syslog (Dynatrace OTel Collector) in Dynatrace Hub for an example configuration file for the receiver, so you can enable two separate syslog endpoints for F5 and host syslogs. This allows you to differentiate log sources (attribute.device.type) for analysis in Dynatrace.

You can also make the Dynatrace OTel Collector multicast incoming syslog messages to multiple destinations. For example, you can set up exporters for your Dynatrace production environment and sandbox environment:

service:
  pipelines:
    logs:
      receivers: [syslog/f5, syslog/host]
      processors: [batch]
      exporters: [logging, otlphttp/tenant_1, otlphttp/tenant_2]

As a result, you should see logs in Dynatrace with corresponding log.source and device.type attributes:

Deploy Dynatrace OTel Collector for syslog now

• Go to Syslog (Dynatrace OTel Collector) in Dynatrace Hub to see examples and continue to the installation.
• Learn how to ingest syslog data into Dynatrace with this self-service Observability Lab.
• Dynatrace OTel Collector was released in Preview in January and is available generally from March 2024.
• Get more guidance for syslog configuration in documentation.

What’s next

• Stay tuned for direct syslog ingestion into Dynatrace, which brings syslog endpoints to an Environment ActiveGate, fully configurable from the cluster. This will enable you to use Dynatrace ActiveGate to ingest syslog data.

The post Bring syslog into Dynatrace using OpenTelemetry to get open source value with enterprise support appeared first on Dynatrace news.

As organizations strive for observability and data democratization, OpenTelemetry emerges as a key technology to create and transfer observability data. OpenTelemetry is gaining popularity because it’s considered a standard, and that’s why it’s a common choice for creating future-proof solutions for years to come.

To answer the growing demand for OpenTelemetry, Dynatrace is proud to announce the release of the Dynatrace OTel Collector distribution. This collector, fully supported and maintained by Dynatrace, is entirely open source. Before we get into the specifics, let’s first recap the benefits OpenTelemetry offers and why using collectors is a best practice.

Understanding OpenTelemetry

OpenTelemetry is an open, vendor-neutral standard for creating, collecting, and transferring telemetry data, like traces, metrics, and logs. Developers and operators can gain insights into their applications and infrastructure without fear of vendor lock-in because OpenTelemetry is fully open source and owned by CNCF. The OpenTelemetry project is supported and maintained by representatives from Microsoft, Google, Amazon, and many others, including Dynatrace.

Why do I need an OpenTelemetry collector?

As the name suggests, an OpenTelemetry collector gathers data from multiple sources and sends it to observability backends, like Dynatrace, for analysis. A collector helps developers control their telemetry data streams for each signal. Different data streams can be directed to different backends or even multicast to multiple backends simultaneously. The configuration is highly flexible in solving various user needs.

A collector is also a powerful component for data processing. It removes the burden of managing retries, batching, and sampling from monitored applications, which can reduce the CPU and memory requirements of applications. A collector can also transform and enrich the data with additional context. For example, in a Kubernetes environment, a collector can automatically attach metadata about pods and namespaces to all observability data. This ensures that application telemetry is contextualized with the infrastructure, enabling the observability backend to link the application and infrastructure for enhanced insights and root cause analysis.

From a user perspective, a collector can also serve as an open source platform that can be extended with custom components. You can create internal collector components of your own, for example, to receive telemetry data in a special format or to process it in a certain way. Using a collector as a telemetry processing platform can be much easier than creating an entirely new application.

Why the Dynatrace OTel Collector

The OpenTelemetry community releases different distributions of the collector, many of which our customers use to send OpenTelemetry data to Dynatrace. So, why should you consider using the Dynatrace Otel Collector? Quite simply, support and stability.

We have seen many customers identify collectors as a potential solution for their needs. Still, they haven’t been able to deploy collectors in production due to the lack of support. Deploying an open source component without external support and the needed expertise is undoubtedly a risk. That’s why we provide Dynatrace customers with a Dynatrace-supported solution.

The Dynatrace Otel Collector comes with collector components that have been verified by Dynatrace for seamless operation. This removes the burden of manually validating each component and use case. To further help you with your collector journey, we publish configuration examples of typical Dynatrace use cases and best practices to provide a good starting point.

The Dynatrace Otel Collector includes components that we know run stably in production, which means we can offer full Dynatrace support. At the same time, innovations from the OpenTelemetry community can be added to the Dynatrace Otel Collector only after they are mature enough and have proven their stability.

Deployment and the typical use cases

The Dynatrace Otel Collector can be deployed on Kubernetes or Docker using a provided container image or directly on a host with the published binary. For more details, refer to our Dynatrace OTel Collector deployment guide.

In the initial release, the Dynatrace Otel Collector comes with components for:

• Receiving, processing, and exporting OpenTelemetry protocol (OTLP) for traces, metrics, and logs
• Integrating Jaeger, Prometheus, and different log protocols, such as syslog, Fluentd, and log files
• To learn more about ingesting log data over syslog with the new Dynatrace OTel Collector, see the blog post Bring syslog into Dynatrace over OpenTelemetry to get open source value with enterprise support.

Additionally, the Dynatrace Otel Collector includes a rich toolset for data processing to enrich, filter, transform, sample, and batch. The complete list of the components is available in our GitHub repository.

What’s next

After the initial release, we’ll continue enhancing the Dynatrace OTel Collector with new features and capabilities to make it even easier to integrate with Dynatrace. We also intend to offer more automated methods to deploy the collector with pre-configurations. So, stay tuned.

As a significant contributor to the OpenTelemetry project, Dynatrace remains committed to working with the community and other vendors to enrich its capabilities and make it user-friendly for everyone.

The post Enhance data collection with Dynatrace OTel Collector distribution appeared first on Dynatrace news.

Cloud-native CI/CD pipelines and build processes often expose Kubernetes to attack vectors via internet-sourced container images. Despite scanning, these container images can still be susceptible to supply chain attacks if not properly verified. Ensuring immutability—coupled with thorough scanning and strict verification—is crucial for any container entering Kubernetes clusters. This is particularly vital for securing observability solutions like Dynatrace® Kubernetes infrastructure observability, application observability, and Application Security.

The Dynatrace Operator is responsible for the secure lifecycle of components necessary for Kubernetes cluster monitoring. Dynatrace Operator ensures secure download and rollout of components via protected connections to the Dynatrace platform. Introducing Cosign-signed immutable images, Dynatrace further empowers you to independently verify images, maintaining observability free from supply chain attacks.

The benefits of independently verifiable container images begin, but do not end, with enhanced security.

• Security: Signing and immutability of container images significantly reduce the risk of security breaches, ensuring that only verified, tamper-proof observability tools are deployed.
• Compliance: Adhering to stringent security standards helps meet regulatory and compliance requirements for cloud-native environments.
• Reliability: Immutable images guarantee consistent performance and behavior, enhancing stability.

Incorporating signed Dynatrace containers into your pipeline

To enhance security in CI/CD processes, Dynatrace customers can integrate verified Dynatrace container images into their deployment pipelines. This process, illustrated below using Amazon’s Elastic Container Registry (ECR), is also applicable to Docker Hub and will be extended to other cloud platforms in the future.

How it works

Begin by browsing Dynatrace images on the Amazon Elastic Container Registry (ECR) Public Container Gallery. Signed and immutable container images are available for the entire Dynatrace observability stack.

• Dynatrace OneAgent®, which offers Application Security, host, network, and infrastructure monitoring
• Dynatrace OneAgent code modules, which offer application observability and security for Go, Java, Node, .Net, and PHP
• Dynatrace Operator
• Dynatrace ActiveGate for Kubernetes monitoring, routing, and signal compression

Drilling down on a container image reveals tags for both signatures as well as versions of the binaries. The version numbers of these binaries correlate 1:1 with the container image’s tags. This correlation ensures that Dynatrace software components are versioned exactly the same way for both containerized and non-containerized workloads.

Using public images

The recommended way of using Dynatrace signed immutable images is to use a private registry, which is typical for most organizations looking to prioritize security in their CI/CD workflows. This approach offers potentially improved performance and reliability, as the registry can be optimized for specific network environments.

This process involves a few steps:

1. Query your cluster on latest available OneAgent, code module, and ActiveGate tag information.
2. Copy container image to private registry from public registry.
3. Check that the images are valid and secure.
4. Reference the container image in the DynaKube.

To determine the latest supported version of the Dynatrace OneAgent container image execute the following command to get the JSON output from the REST API:

curl https://{your_tenant}.live.dynatrace.com/api/v1/deployment/image/agent/oneAgent/latest -
-header 'Authorization: Api-Token {your_API_token}'

Required token scope is the following: InstallerDownload

Here is the example of the the reply to the provided REST API call is in the JSON format:

{
"source": "{public_ecr_address}/dynatrace/dynatrace-oneagent",
"tag": "1.279.242.20240108-114943"
}

The retrieved image tag can now be used to copy the container image from the provided ECR to your private image registry.

To copy all image architectures and Cosign signatures for verification, make sure to use the --all flag and set use-sigstore-attachments to true in Skopeo’s container registry configuration.

skopeo copy --all docker://public.ecr.aws/dynatrace/dynatrace-oneagent:<tag> \
docker://registry.my-company.com/dynatrace-oneagent:<tag>

Finally, verify the container image signature to ensure authenticity and integrity.

cosign verify --insecure-ignore-tlog --key https://ca.dynatrace.com/v1/cosign.pub \

  registry.my-company.com/dynatrace-oneagent:<tag>

Once the image has been copied over to your private registry, it can be scanned for vulnerabilities.

Finally, reference the verified private registry images in the DynaKube. The example below shows all three images: OneAgent, CodeModules, and ActiveGate. Note the inclusion of a pull secret, required for protected private registries.

apiVersion: dynatrace.com/v1beta1

kind: DynaKube

metadata:

  name: private-registry

  namespace: dynatrace

spec:

  apiUrl: https://(your environment)/api

  tokens: api-tokens

  customPullSecret: pull-secret

  oneAgent:

    cloudNativeFullStack:

      image: (your-registry)/dynatrace-oneagent:1.279.242.20240108-114943

      codeModulesImage: (your-registry)/dynatrace-codemodules:1.279.242.20240108-114943

  activeGate:

    capabilities:

      - kubernetes-monitoring

      - routing

    image: (your-registry)/dynatrace-activegate:1.279.116.20231206-155926

Take cloud-native security to new heights

The latest release of Dynatrace signed immutable container images marks a step forward in securing cloud-native observability stacks. By ensuring the integrity and security of containers at every step, from public registry to Kubernetes deployment, Dynatrace sets a new standard in cloud-native security. This gets to the heart of the Dynatrace mission to provide unparalleled observability and security in an ever-evolving digital landscape.

What’s next

New users can explore these advanced features with a free 15-day trial, experiencing firsthand how Dynatrace is transforming cloud-native security.

The post Dynatrace strengthens container security across popular cloud-based registries appeared first on Dynatrace news.

What is IBM i?

IBM i, formerly known as iSeries, is an operating system developed by IBM for its line of IBM i Power Systems servers. It is based on the IBM AS/400 system and is known for its reliability, scalability, and security features. IBM i is designed to integrate seamlessly with legacy and modern applications, allowing businesses to run critical workloads and applications. It also includes various built-in software components for database management, security, and application development.

Gaining knowledge about IBM i performance can be a challenging and pricey task. Some tools demand the installation of agents on those systems and provide complex, disconnected views. Additionally, certain tools require auxiliary services to gather performance data before it can be examined and queried.

Effortlessly analyze IBM i Performance with the new Dynatrace extension

Dynatrace has created a new version of its popular extension that is faster, offers better interactive pages, and includes more metrics, metadata, and analytics without having to install anything on your mainframe infrastructure.

Find out how your LPARs are doing overall, which jobs are consuming the most resources, and which users are holding up your output queues. Or perhaps you want to be notified when specific critical system or completion messages appear in your message queues.

Maybe you’d like to know when your jobs are in a waiting status, when an application is spawning too few or too many job instances, or when all the disks in your pool are being utilized evenly.

Get answers to all of these questions with the new Dynatrace IBM i extension.

How does the extension work?

The extension runs remotely from your Dynatrace ActiveGates and connects to your IBM i system. It then collects performance data using existing database services running on your system. The entire topology is built on Dynatrace based on the components it finds configured on your IBM i system, all acting cohesively to accelerate root cause and impact analysis.

Nothing is installed on your IBM i systems. It’s all monitored remotely!

More metrics, more data

We have improved performance KPI collection and added new metrics and entities, like:

• System
• Memory pools
• Jobs
• Job queues
• ASP and disks
• Output queues and spooling files
• Message queues
• Network
• Subsystems

Each entity has more metadata to help you identify and understand its configuration.

All systems at a glance

A ready-to-use dashboard, straight out of the box, is the starting point, showing important information in a unified view.

The default dashboard provides an overview of all monitored systems and how many different entities are created by IBM i components.

Starting at this dashboard, you can drill down into any component to see how it performs on more detailed dashboards.

Get a health overview of each system

Monitor your system’s performance and detect unexpected events such as IPLs, CPU spikes, and exceeded total job limits. Identify users who are consuming the most spooling files.

Monitor your jobs and their instances

On IBM i, jobs are tasks that are triggered to run by the system and applications. It’s crucial to monitor the performance of these jobs, including their CPU usage, number of instances, and status. Additionally, it’s important to identify when a job is locked or hanging by observing various indicators.

Critical messages to your teams

Monitoring the main message queues can provide critical information about system failures, application failures, completion tasks, and any other message type you want to know about.

Monitor disks and disk pool utilization

One of the most important functions of your mainframe infrastructure is reading and writing data at high speeds while making it readily available. It’s critical to know the performance of your disks and ensure their optimum utilization to maintain balance in I/O operations. If you observe any disk running out of space more quickly than others—or underperforming—this type of analysis can help prevent disastrous outages or potential data loss.

As this extension evolves, your valued feedback is important to ensure its continuous improvement. Please visit our Product Ideas community forum, let your voice be heard, and give others a chance to upvote your ideas.

The post Easily monitor IBM i with updated Dynatrace extension appeared first on Dynatrace news.

Companies face mounting pressure to digitally transform and deliver seamless cross-channel experiences. These experiences are critical not only for attracting new prospects and driving satisfaction but also for providing more value to customers and enabling business growth. Adobe Experience Manager empowers businesses to optimize and deliver engaging experiences across websites, mobile apps, and other touchpoints. As part of delivering a compelling experience, it is vital that companies develop a deeper understanding of their customers. As a result, user experiences can be optimized, and issues can be proactively addressed before customers become frustrated with errors or broken journeys.

End-to-end observability across every tier powering your digital experiences using RUM

Application owners and SREs often struggle to get a complete picture of the customer journey and are unable to pinpoint issues due to a lack of comprehensive visibility. As a result, it is difficult to ensure application efficiency and ensure accurate delivery across touchpoints. This lack of end-to-end visibility leads to blind spots that impact customer satisfaction.

The Dynatrace platform offers comprehensive observability and monitoring across the entire technology stack. By going beyond application performance monitoring with AI-powered, full stack observability, Dynatrace enables seamless end-to-end visibility. Furthermore, by integrating with Adobe Experience Manager, you get quick visibility, ensuring application efficiency across all channels and the ability to scale in support of even the largest Adobe Experience Manager instances.

Dynatrace automatically detects all Adobe Experience Manager applications and visualizes their dependencies—from the website, to the container, to the cloud service. This enables you to enrich your Adobe analytics data with end-to-end traces across all tiers and to get precise, real-time user insights. You can monitor Adobe Experience Manager instances intelligently and provide personalized, content-driven experiences without gaps or blind spots.

By relying on Dynatrace® Digital Experience Monitoring (DEM) integrated with Adobe Experience Manager—specifically Real User Monitoring, Synthetic Monitoring, and Session Replay—you gain real-time insight into the user experience of their sites and applications. This capability extends across devices and geographies and includes critical front-end monitoring use cases like performance and availability monitoring, troubleshooting, root cause analysis, complaint resolution, and understanding user behavior. This allows teams to understand exactly where to make improvements in the Adobe Experience Manager stack and deliver a smoother, more personalized experience to customers.

Furthermore, Dynatrace turns analytics into real-time answers, enabling you to better understand how applications’ outages impact critical business metrics like conversions, revenue, and engagement. This actionable insight allows you to accelerate time to value, optimize customer experience, and meet your business needs.

Deliver excellent customer experience with the powerful Davis AI engine

The Dynatrace Davis® AI engine allows you to diagnose anomalies when they arise in real-time and pinpoint the root cause down to the broken code before your customers are affected. You can now proactively detect availability and performance issues across the stack.

By enabling faster identification of potential issues, you can reduce mean time to repair and minimize customer impact. With AI-powered observability, you can quickly troubleshoot complex environments like Adobe Experience Manager to maximize uptime and customer satisfaction. The result is a smooth, uninterrupted user experience.

Visit Adobe Experience Manager documentation to learn more about the Dynatrace integration

Want to it out for yourself? Sign up for your Dynatrace free trial.

The post Dynatrace and Adobe Experience Manager: Seamless end-to-end observability appeared first on Dynatrace news.

Multicloud strategy: Balancing potential with complexity in modern IT ecosystems

In the ever-changing digital world, cloud technologies are crucial in driving business innovation and adaptability. They offer unmatched flexibility and scalability to meet the fluctuating demands of the market. While cloud deployments offer benefits, they also pose management challenges—especially in multicloud strategies that use various cloud providers. Organizations are increasingly adopting these multicloud approaches to bolster resilience, access specialized features, control costs, adhere to regulations, boost user performance, and remain agile in a technology-centric competitive landscape.

Much like cloud deployments, a multicloud strategy offers numerous benefits. However, a multicloud strategy also introduces new challenges for platform teams. Those teams need to be able to delve quickly into resources across all cloud providers to identify a problem or investigate a trend.

Unified observability is the key to tackling multicloud integration challenges

The rise of multicloud environments necessitates a tool capable of seamlessly integrating and providing insights across various cloud services. Many of our customers face challenges due to using various tools that are not integrated, making it difficult to track their cloud services and correlate data from different sources. Platform teams facing this challenge might compromise on tools, settling for limited capabilities and a weaker security posture. For these teams, it’s not just about the inventory of resources across clouds; it’s about the ability to correlate across clouds, tracking multiple data points across different resources.

Streamline multicloud observability with the Dynatrace Clouds app

Enter the Dynatrace Clouds app, a novel way for observing multiple resources across multiple clouds. It provides a single, centralized dashboard that displays all resources across multiple clouds, and significantly enhances multicloud resource tracking and governance.

Centralized oversight and insights

A unified view eliminates the need to switch between different cloud service dashboards, saving time and reducing the risk of oversight errors. Centralization brings all the critical metrics and logs into one place, providing a holistic perspective over your cloud environment.

Logs and metrics in context

The Clouds app simplifies cloud operations by providing detailed logs and metrics in context. With a single click, the dashboard shows an analysis of logs and metrics for a selected resource in its appropriate context, making it easier to understand the complexities inherent in cloud infrastructure. Consolidating data from multiple clouds is now more achievable and straightforward.

Observing performance metrics from multiple cloud providers side by side allows CloudOps teams to compare and contrast the efficiency and response times of different services, which is critical to ensure optimal application performance.

Detailed logs offer insights into application behavior and system operations, helping developers and platform teams to pinpoint the root causes of errors, crashes, or performance issues.

Enhanced troubleshooting

With data from various clouds correlated, it becomes easier to identify an issue’s root cause. Troubleshooting is more effective when teams can see patterns and anomalies across all cloud environments, rather than analyzing them in isolation.

Correlate data from multiple clouds in a single view

Analyzing data from multiple cloud providers in a single view also allows for comparative analytics, where insights gained from one cloud can inform strategies in another, effectively leveraging each provider’s strengths.

Incorporating these advantages into cloud operations greatly enhances an organization’s agility, efficiency, and resilience, driving a more proactive and informed approach to observing multicloud environments.

Start using the Clouds app

Dynatrace will continuously improve the Clouds app, making it an even more powerful tool.

Embrace the opportunity to explore the Dynatrace Clouds app and let your voice be heard. Join us on our community channel and be a part of shaping the future of the Dynatrace Clouds app.

The post Observe and optimize multicloud environments with the Dynatrace Clouds app appeared first on Dynatrace news.

We’ve all been there. An important fix is waiting to be delivered, and a manager asks you why it’s taking so long and says that the pipeline was much faster a few months ago. Although you disagree with the assessment, you can’t dispute it because you don’t have the data to show that the pipeline is not slower.

At Dynatrace, Jenkins is the most used Continuous Integration (CI) tool. It’s open source, allows for great scalability with the possibility to run workloads on the cloud and on-premises Kubernetes clusters, and is easily extensible via plugins.

To monitor these pipelines, OpenTelemetry can be used to trace every step of the build and send the traces to various backends. You’re probably already familiar with the view of a trace in Dynatrace, as shown below. If you’re also familiar with Jenkins, you’ll notice that the trace steps are the same as in your Jenkins builds—you get a one-to-one mapping.

In this trace, we see that the checkout took up a large portion of the build. The question is, what can be done with this data? How can you show things like the failure rate of a pipeline during the last two weeks with as little effort as possible?

What we measure at Dynatrace

It’s important to agree upon what should be measured; there is no right or wrong answer here. At Dynatrace, we measure the DORA metrics from Google.

These include:

• Deployment frequency: How often is code deployed?
• Lead time for changes: How long does it take to get committed code into production?
• Time to restore service: How long does it take to restore a service?
• Change failure rate: How often does the service fail following changes?

While tracking the time to restore a service and the change failure rate require an issue management system, the first two metrics can be acquired by looking at the pipelines.

Unified services: Easy, out-of-the-box service visualization

If you use GitHub or GitLab, you can use the Four Keys project (https://github.com/dora-team/fourkeys), which can be integrated into your project, or you can write your own service that takes the OpenTelemetry data and executes the necessary operations on it.

But if you want to get everything out of the box, you can use Dynatrace unified services. This service visualization experience for OpenTelemetry was introduced with  Dynatrace version 1.274. As soon as you have unified services enabled on your Dynatrace tenant, your OpenTelemetry API-ingested services will look like the image below.

There’s an endpoint for each branch that exists within Jenkins. You get the number of failures, the throughput, and the response time, which maps to the duration of the build.

Now, let’s jump to a real-world example. Let’s say that your developers believe that their pull requests sometimes take longer than usual, not a lot, but a noticeable amount. And sometimes, the pull requests aren’t triggered at all. In such a scenario, check the durations of all the pull requests. In this case, we can see a clustering of pull requests and some outliers. With this information, we can guess that the webhook to Jenkins isn’t correctly configured, and only when the pipeline is scanned on Jenkins are the PRs picked up.

Normally, you get one endpoint per pull request, but as you can see, we summarized all the pull requests into one chart. This can be achieved by adapting the query a bit.

builtin:service.request.response_time_chart:filter(and(eq("dt.entity.service",SERVICE-AE4EB9E449B856CA),eq("endpoint.name","EP/backstage/PR-156"))):splitBy("endpoint.name"):median

You can adapt the equals to a contains and remove the PR number from the endpoint name and you get data for all pull requests in one chart.

builtin:service.request.response_time_chart:filter(and(eq("dt.entity.service",SERVICE-AE4EB9E449B856CA),contains("endpoint.name","EP/backstage/PR-"))):splitBy("endpoint.name"):median

Now, let’s say we want to check the Backstage pipeline. The first DORA metric is rather easy; we simply look at the throughput of the main branch. You can view the throughput in the Dynatrace Data Explorer, which translates to the number of builds that ran. In the view below, you can see the number of builds in the last two hours.

To learn the amount of time it takes until a commit appears in production, you need the duration of pull requests and the duration of the main branch. The chart below shows you the duration of builds on the main branch.

The chart for the main branch works out of the box. For the pull requests, we must adapt the query a bit, because based on the built-in rules that come with the unified services, they create a new endpoint for every pull request, which allows for exact control, but this is not what we want in this case. So, we will leave the query as-is .

One last thing that you might be interested in is the failure rate of the main branch. As with the other metrics, simply search your main branch, view the data in the Dynatrace Data Explorer and be happy that your main branch only broke twice in the last seven days.

Sneak-peek time!

While Traces powered by Grail™ is still in development, we already use it for internal use cases and can give you a sneak peek. With a simple Dynatrace Query Language (DQL) query, we can filter for each and every step of a trace, or as is the case in this example, a Jenkins pipeline. As you can see, the allocation times look fine, but there is a huge outlier. This outlier points us to an overloaded cloud; too many workloads were running, and the cluster ran out of resources. Therefore, the pods had to wait a long time to be spawned. With this data in hand, we don’t need to rely on a gut feeling—you can now back up your report with data.

See the current state of your pipeline at a glance

With Dynatrace unified services, you can assess the state of your pipelines at a glance and can make confident statements based on data and not your gut feelings. You can now provide answers to tough questions like “Are the pipelines getting slower?” or “Are my builds breaking more often?” Another big advantage is the fact that once you have the services enabled, you have the data available all the time and no longer need to search when new questions arise. If you want a deeper look into this topic, watch Dynatrace Tips & Tricks #18 – Jenkins Pipeline Analytics with OpenTelemetry.

Try it yourself

Using Jenkins? The following links help you to export the traces from Jenkins and to ingest them into Dynatrace.

• Jenkins OpenTelemetry Plugin
• OpenTelemetry Collector
• Dynatrace Tips & Tricks #18 – Jenkins Pipeline Analytics with OpenTelemetry

The post Unified services: Easily keep track of your pipelines appeared first on Dynatrace news.

Without observability, the benefits of ARM are lost

Over the last decade and a half, a new wave of computer architecture has overtaken the world. ARM architecture, based on a processor type optimized for cloud and hyperscale computing, has become the most prevalent on the planet, with billions of ARM devices currently in use. This growth was spurred by mobile ecosystems with Android and iOS operating systems, where ARM has a unique advantage in energy efficiency while offering high performance.

While ARM processors have been humming in consumers’ pockets for over a decade, enterprise IT adoption has been slower. Legacy data center infrastructure and software support have kept all the benefits of ARM at, well… arm’s length. Still, we at Dynatrace speak to customers who recognize this value and want to implement it.

Energy efficiency and carbon footprint outshine x86 architectures

The first clear benefit of ARM in the enterprise IT landscape is energy efficiency. This is a crucial factor for all data centers, cloud or managed, where power consumption and cooling costs are reflected in the total cost of ownership. Traditional x86 architecture needs more power, so switching to ARM can offer a clear advantage.

Energy efficiency is coupled with the total enterprise carbon footprint. As organizations look to take ownership of their total ecological footprint and help mitigate climate change, it’s critically important for organizations to measure, monitor, and reduce their IT carbon footprints. Initiatives like the Carbon Impact app can be used to measure the footprint of monitored ARM-based hosts compared to x86 hosts.

Huge performance leaps in recent years

The top priority is often performance, where ARM resources have improved significantly. You would have been unimpressed if you evaluated public cloud vendor ARM-based offerings five or more years ago. But take a look at the latest iterations of, for example, AWS Graviton2, which delivered a 40% price/performance boost, and Graviton3, which had an additional 27% price/performance improvement over Graviton2.

These Improvements and other optimizations show that mature service offerings can rely on ARM architecture. You’re no longer required to use a single offering or choose from a few instance families; Graviton includes general-purpose and accelerated-computing offerings, plus compute-, memory-, and storage-optimized instances.

No observability, no gains

All these factors have made ARM an attractive computing architecture for innovative companies. However, the lack of end-to-end integrated observability with full support for log collection is a clear blocker for many organizations looking to adopt ARM-based services.

Without an observability platform to collect and process signals, and provide AI-powered answers for problem detection, root cause identification, and impact analysis, the migration to ARM remains only a roadmap item for many AIOps teams.

Even if some part of your codebase can be instrumented to collect observability data, having all three signal types (metrics, traces, and logs) is crucial. Without collecting logs from the observed platform in a scalable AI-powered data lakehouse like Grail, it’s more of a challenge to identify the root cause of problems and provide details for troubleshooting or security incidents.

Having no access to logs or relying on a siloed approach to logging poses a risk of blind spots in your IT landscape, prolonged outages, and an increase in Mean Time To Identify (MTTI) for incidents, all of which can nullify the benefits of ARM.

Hassle-free ARM deployment with automated log collection

With automated log observability, the Dynatrace platform enables your AIOps team to reap the full benefits of ARM architecture. Dynatrace OneAgent can automatically detect common logs for ARM-based hosts and ingest them on a massive scale out of the box.

As soon as you install just a single OneAgent on a host, for example, an ARM64 (AArch64) based Linux host, the OneAgent scans and autodiscovers logs every 60 seconds.

After installation, you can control OneAgent behavior using a powerful central configuration mechanism in your Dynatrace environment. You can tune the granularity of OneAgent filtering for each host to meet your requirements for discovery and ingestion, share configuration details with other hosts in the same host group, or apply global settings for your whole environment.

As OneAgent collects logs from your ARM-based host, it automatically ties the discovered data to your environment topology. In this way, log data is always associated with the host, service, or other entity that generated it.

AI-powered problem detection can then connect the log data, which often contains the source of truth, to detected issues in your environment, thereby speeding up troubleshooting and remediation by orders of magnitude compared to manual correlation.

In addition, the integration with auto-baselined metrics and automatic connection between distributed traces and logs puts you in complete control of the software you run on ARM-based architectures.

Immediately see the root cause in ARM host logs

You can enable the full value of Dynatrace with log monitoring on ARM in just a few steps.

1. Install OneAgent version 1.269+ on an ARM host (Go to Apps In the Dynatrace web UI and search for Deploy OneAgent to access the installer). For full details, see OneAgent installation on Linux.
2. Following OneAgent installation, you can verify on the host page that the ARM 64-bit instruction set is used.
   
   Next, go to Settings > Log monitoring > Set up log ingest and review the log ingestion rules in the Hosts section. For the simplest quickstart, select Show rules.
   
3. Turn on the rule [Built-in] Ingest all logs to enable all rules and gain complete visibility into the host.
   

Now you have the additional value of log data informing your Dynatrace root cause analysis should a problem arise on this host.

In the following example, a synthetic monitor is set up for webpage uptime monitoring. Dynatrace Davis^® AI has automatically discovered a problem on the host, and the root cause analysis points to a PHP web service as the source of the problem.

The problem page gives you a direct link to the service causing the disruption, and you can quickly access all related logs.

Error logs for the process reveal a PHP syntax error with code-level information that caused the outage.

As you can see, with the Dynatrace AI-powered observability platform at your disposal, you can benefit from resource-efficient ARM architectures in your organization without the risk of blindspots or guesswork when you encounter incidents or outages.

Start monitoring ARM and see what’s next

• New to Dynatrace? Open a free trial environment right now and see Dynatrace in action.
• See the OneAgent support matrix for your operating system and deploy OneAgent in your ARM environment today.
• Recently released improvements to the central configuration of all your OneAgents now include:
  • Renamed advanced log configuration pages for simpler navigation
  • Log filtering based on log level
  • Enforced auto-timestamping for logs, in case your log source includes varied log formats that prevent correct timestamp recognition
• Browse the Dynatrace Hub to see how you can bring in logs from different sources and technologies, for example, ingesting Syslog via OpenTelemetry Collector and Oracle log monitoring.
• Share your feedback and expectations with us in the Dynatrace Community

Coming next

We’re working on making Dynatrace log monitoring even easier to use at scale with a focus on areas highlighted by you and other customers:

• Are you ingesting logs across multiple sources with a wildcard (*) ingest rule? Soon, you can enhance log records with custom file names or log path attributes for later analysis in Dynatrace.
• Get automatic log ingest recommendation rules relevant to your environment based on the log sources discovered by OneAgent.
• Tools to troubleshoot the status of your log sources and instructions for resolving issues in cases where a log file is inaccessible.
• See the complete list of log sources on each host page, even if the sources are tied to another process group.

The post Dynatrace log collection for ARM unlocks power-efficient architecture for your enterprise appeared first on Dynatrace news.

“Engineers today lack an easy way to track the tokens and prompt usage of their LLM applications in production. By using OpenLLMetry and Dynatrace, anyone can get complete visibility into their system, including gen-AI parts with 5 minutes of work.”

Nir Gazit, CEO and Co-Founder Traceloop

Why AI model observability matters

The adoption of LLMs has surged across various industries, particularly since the introduction of OpenAI’s GPT model. While these models yield impressive results, the challenge of maintaining their operation within defined boundaries has increased.

AI model observability plays a crucial role in achieving this by addressing these key aspects:

1. Model performance and reliability: Evaluating the model’s ability to provide accurate and timely responses, ensuring stability, and assessing domain-specific semantic accuracy.
2. Resource consumption: Observing computational resource availability and saturation, whether deployed in cloud-native environments like Kubernetes or CPU-enabled servers.
3. Data quality and drift: Monitoring the quality and characteristics of training and runtime data to detect significant changes that might impact model accuracy.
4. Explainability and interpretability: Providing information on model versions, parameters, and deployment schedules, which is essential for interpreting and understanding model answers.
5. Security and compliance: Actively preventing security threats at both the application and model levels to ensure responsible and compliant AI usage.

The challenge of AI model observability

One challenge in AI model observability is the diverse tooling landscape required to gain critical insights. OpenTelemetry has become a standard for collecting traces, metrics, and logs. However, seamless support for various SDKs and AI model frameworks, such as LangChain and Pinecone, remains essential.

Combining Dynatrace with Traceloop’s OpenLLMetry addresses the heterogeneity challenge by supporting a range of popular LLMs, prompt engineering, and chaining frameworks. OpenLLMetry, an open source SDK built on OpenTelemetry, offers standardized data collection for AI Model observability.

How OpenLLMetry works

OpenLLMetry supports AI model observability by capturing and normalizing key performance indicators (KPIs) from diverse AI frameworks. Utilizing an additional OpenTelemetry SDK layer, this data seamlessly flows into the Dynatrace environment, offering advanced analytics and a holistic view of the AI deployment stack.

Given the prevalence of Python in AI model development, OpenTelemetry serves as a robust standard for collecting observability data, including traces, metrics, and logs. While OpenTelemetry’s auto-instrumentation provides valuable insights into spans and basic resource attributes, it falls short in capturing specific KPIs crucial for AI models, such as model name, version, prompt and completion tokens, and temperature parameters.

OpenLLMetry bridges this gap by supporting popular AI frameworks like OpenAI, HuggingFace, Pinecone, and LangChain. Standardizing the collection of essential model KPIs through OpenTelemetry ensures comprehensive observability. The open source OpenLLMetry SDK, built atop OpenTelemetry, enables thorough insights into your Large Language Model (LLM) applications.

As the collected data seamlessly integrates with your Dynatrace environment, you can analyze LLM metrics, spans, and logs in the context of all traces and code-level information. Maintained under the Apache 2.0 license by Traceloop, OpenLLMetry is a valuable asset for product owners, providing a transparent view of AI model performance.

The diagram below illustrates how OpenLLMetry captures and transmits AI model KPIs to your Dynatrace environment, empowering your business with unparalleled insights into your AI deployment landscape.

Dynatrace OneAgent® is perfectly capable of automatically injecting and tracing code-level information for many technologies, such as Java, .NET, Golang, and NodeJS. However, Python models are trickier.

In the Dynatrace web UI, you can track your AI model in real time, examine its model attributes, and assess the reliability and latency of each specific LangChain task, as demonstrated below.

The captured span by Traceloop automatically displays vital details, including the mode utilized by our LangChain model gpt-3-5-turbo, the model’s invocation with a temperature parameter of 0.7, and the utilization of 53 completion tokens for this individual request.

With the growth of AI, maintaining transparency is essential

Observing AI models like Large Language Models (LLMs) in production is crucial for enhancing performance, reliability, security, and user trust. This includes the monitoring of AI-related costs to ensure they remain within acceptable margins. The Dynatrace platform, coupled with Traceloop’s OpenLLMetry OpenTelemetry SDK, offers comprehensive visibility from model inception to completion.

As AI adoption grows, maintaining transparency is essential for regulatory compliance. While Dynatrace automates tracing for various technologies, Python-based AI models require OpenTelemetry. OpenLLMetry bridges this gap, supporting popular AI frameworks and vendors to ensure standardized data collection. OpenLLMetry provides an open source SDK for LLM observability, seamlessly integrating with Dynatrace for in-depth analysis.

References

• Dynatrace Documentation: AI model observability
• Traceloop documentation: OpenLLMetry

The post Enhanced AI model observability with Dynatrace and Traceloop OpenLLMetry appeared first on Dynatrace news.

Establishing SLOs for Kubernetes clusters can help organizations optimize resource utilization. A good Kubernetes SLO strategy helps teams manage and make containerized workloads more efficient.

Kubernetes is a widely used open source system for container orchestration. It allows for seamless running of containerized workloads across different environments. However, managing Kubernetes optimally can be a daunting task due to its complex architecture. Properly monitoring a Kubernetes cluster or any related environment can be difficult.

Effective resource provisioning and management is a critical aspect of a Kubernetes cluster. It involves a coordinated effort among various stakeholders, including cluster owners, application teams, and business owners. The primary goal is to allocate sufficient resources to keep the applications running smoothly without overprovisioning and incurring unnecessary costs. This delicate balance requires a collaborative approach and transparent communication among all parties involved. Service-level objectives (SLOs) play a vital role in ensuring that all stakeholders have visibility into the resources being used and the performance of the applications.

SLOs are often used to monitor business-critical services and applications for customers. However, they can also be used to monitor optimization processes effectively. Essentially, SLOs track a selected service-level indicator (SLI) and continuously evaluate its behavior over a given timeframe against a fixed threshold. This feature is valuable for platform owners who want to monitor and optimize their Kubernetes environment. By considering historical behavior, SLOs provide an excellent way to track and evaluate optimization tasks. Users can continuously evaluate the system’s performance against predefined quality criteria, making SLOs for Kubernetes clusters a good option for monitoring and improving the system’s overall performance.

Efficient coordination among resource usage, requests, and allocation is critical. To optimize resource utilization in a Kubernetes cluster, all users must work together seamlessly. A Kubernetes SLO can serve as a transparent and trackable collaboration tool for different teams and collaborators.

Optimize memory utilization of your Kubernetes Namespaces

Monitoring your Kubernetes cluster allows for proactively identifying and resolving resource constraints, failures, and crashes before they impact the end-user experience and your business.

Proper Kubernetes monitoring includes utilizing observability information to optimize your environment. By gaining insights into how your Kubernetes workloads utilize computing and memory resources, you can make informed decisions about how to size and plan your infrastructure, leading to reduced costs. A Kubernetes SLO that continuously evaluates CPU, memory usage, and capacity and compares these available resources to the requested and utilized memory by Kubernetes workloads makes potential waste of resources visible, revealing opportunities for countermeasures.

When it comes to resource utilization in a Kubernetes environment, there are two main perspectives to consider based on the primary stakeholders involved. One perspective focuses on the potential for optimization at the interface between the team responsible for managing the Kubernetes cluster and the teams responsible for developing and deploying applications.

When setting up SLOs for Kubernetes clusters, it is important to choose the right metrics to track based on your objective. If your team is responsible for setting up the Kubernetes cluster, you might want to monitor and optimize the workload performance. However, if you’re part of the application team, the usage of reserved resources may differ significantly from the blocked resources.

Requests represent the number of resources being reserved or blocked for a container. Tracking the ratio between request and usage can provide valuable insights into optimization potential. As every container has defined requests for CPU and memory, these indicators are well-suited for efficiency monitoring.

One option is to continuously track the memory utilization efficiency of existing Kubernetes objects, such as namespaces or workloads. Since teams typically run multiple workloads on namespaces, using this level of abstraction is a suitable option for a Kubernetes SLO. However, if you require more granular information, you can adjust the levels for resource utilization monitoring accordingly.

This Kubernetes SLO measures the ratio between the requested memory and the memory used for an entire namespace. It provides insights into how efficiently the blocked resources are being utilized. Since resources that have been requested cannot be used elsewhere, the objective is to keep the difference between requested and used resources as small as possible. Utilizing such SLOs for Kubernetes clusters makes it possible to track efficiency transparently over time. The teams responsible for the cluster and the application teams that run their containers on the cluster can agree on the intended ratio between used and requested memory.

Optimize Kubernetes cluster’s resource allocation

One aspect of managing cloud resources is tracking and adjusting the ratio between the requested and used memory resources. On the other hand, the cluster owners must allocate cloud resources to meet the application team’s resource requests.

For cluster owners, monitoring resource usage at the node level provides better insights and information for taking sound actions. By tracking resource usage at each node, teams can gain insights into how many resources the entire cluster uses and whether the nodes are working correctly and efficiently.

Teams should implement suitable SLOs to continuously monitor resource usage for Kubernetes cluster nodes at the node level. These SLOs should cover metrics like node memory utilization, which involves monitoring the ratio of requested versus allocated resources or the ratio of desired versus running pods per node.

For instance, if a node’s memory or CPU utilization is high, it can lead to the undesired deletion of pods. This, in turn, can disrupt an application or service and incur additional costs for potential cluster upscaling.

A possible Kubernetes SLO for monitoring and evaluating the ratio of requested versus allocated memory can be expressed as follows:

In an ideal environment, the requested resources would almost match the allocated ones, and hence, cloud resources would be optimally used, reducing costs of invoiced but unused resources.

Ensure overall Kubernetes resource utilization efficiency

The two SLOs mentioned above provide valuable insights into the overall utilization of resources in a Kubernetes cluster and are a good starting point for improving efficiency. Although both interfaces (usage vs. request and request vs. allocation) must be analyzed together to obtain a holistic view, the split is extremely helpful in ensuring accountability. While the application teams have complete control over the usage/request SLO, the cluster owner can influence the request vs. allocation SLO.

Setting up proper SLOs to make these ratios visible and transparent does not reduce the need for collaboration and alignment, but it provides a solid foundation for optimizing resource utilization efficiency.

The outlined SLOs for Kubernetes clusters guide you in implementing SRE best practices in monitoring your Kubernetes environment. By recognizing the insights provided, you can optimize processes and improve overall efficiency. This makes it an excellent tool for collaboration between different contributors while also holding the respective parties accountable.

What’s next with SLOs for Kubernetes clusters?

Increase your resource utilization efficiency with Dynatrace using the SLOs mentioned in the blog. If you want to learn how to set up SLOs with Dynatrace, have a look here:

• Service level objective examples: 5 SLO examples for faster, more reliable apps
• Lessons learned from enterprise service-level objective management
• SLOs done right: how DevOps teams can build better service-level objectives

Or check the following Kubernetes blogs if you’d like to learn more about Dynatrace capabilities to increase efficiency and reliability in your Kubernetes environment:

• Dynatrace memory analysis helps Product Architects identify unknown unknowns
• How to maximize Kubernetes monitoring with AI-driven automation
• Kubernetes in the wild report 2023

The post SLOs for Kubernetes clusters: Optimize resource utilization of Kubernetes clusters with service level objectives appeared first on Dynatrace news.