At Dynatrace, we believe the future of observability and cloud-native operations is open. Not “open” as a slide-deck buzzword, but open as in showing up every day to write code, review PRs, chair working groups, and build tools the community can use, extend, and make their own.

We’re proud to be an active contributor to the cloud-native ecosystem. We build in the open because we’ve seen firsthand that open collaboration produces better software for our customers, the community, and the industry.

This post looks at some of our most significant open source contributions — why we built them, why we open sourced them, and how they’ve grown into thriving community projects. It’s not exhaustive; for the full set, explore our projects on GitHub.

The OpenTelemetry community, setting the standard for observability

What is the OpenTelemetry ecosystem?

The OpenTelemetry® (OTel) ecosystem provides an open source, vendor-neutral way to generate, collect, process, and export telemetry data, specifically traces, metrics, and logs. Born from the merger of the OpenTracing and OpenCensus projects in 2019, the OTel community is now the second most active Cloud Native Computing Foundation® (CNCF) community after Kubernetes and the de facto standard for instrumenting cloud-native applications.

Before OpenTelemetry standards, every observability vendor had its own proprietary agent, data format, and instrumentation libraries. Switching vendors meant ripping out instrumentation and starting over. The standards change that by providing a single set of APIs, SDKs, and a Collector that work with any backend. You instrument once, and you can send your telemetry data wherever you need it.

How Dynatrace contributes to the OpenTelemetry ecosystem

Dynatrace has been involved in the OpenTelemetry ecosystem since the project’s earliest days, and our engagement goes well beyond code commits. Our engineers hold leadership positions across the project:

• Governance Committee: Daniel Dyla, a Senior Open Source Architect at Dynatrace, serves on the OpenTelemetry Governance Committee, helping shape the project’s strategic direction.
• Technical Committee: Armin Ruech, a Team Lead and Product Owner at Dynatrace, sits on the Technical Committee, guiding the cross-language specification that ensures consistency across all OTel implementations.
• SIG Maintainers and Approvers: Dynatrace engineers serve as maintainers or approvers across multiple Special Interest Groups. Contributors like Mario Jonke, Marc Pichler, Evan Bradley, Giovanni Liva, Joao Grassi, and many more are active across these SIGs.
• Community Management: Adriana Villela, a Principal Developer Advocate at Dynatrace, recently took on the role of OpenTelemetry Community Manager, working to grow the contributor and end-user community and raise the OTel community’s profile within the CNCF and the broader open source world.

Dynatrace also ships its own distribution of the OpenTelemetry Collector, with enterprise-grade support and curated components so teams can confidently run OTel in production. Our acquisition of Bindplane, a telemetry pipeline built on OpenTelemetry standards, extends that commitment.

How the community benefits

The OpenTelemetry community has fundamentally changed how the industry approaches observability. Developers no longer commit to a single vendor’s instrumentation at the code level, SREs can standardize on one set of libraries across polyglot environments. Organizations can evolve their observability strategy, switching or combining backends without re-instrumenting their applications.

Get involved

Check out OpenTelemetry.io and find an interest group that matches your interests. Also check out the first episode of the Observation Deck to learn more about the community and how to get involved.

W3C® Trace Context: Making distributed tracing actually work

What is W3C Trace Context?

W3C Trace Context is a specification that defines a standardized set of HTTP headers for propagating distributed tracing context between services. Before it existed, every tracing tool—both commercial and open source—used its own proprietary headers, which resulted in a high volume of broken traces. We saw the problem firsthand every day across thousands of customer environments: Traces would vanish whenever a request crossed a boundary between two monitoring tools or passed through a cloud provider’s managed service. The root cause was always the same: no agreed-upon header format.

Trace Context solves this by defining a universal traceparent header that all compliant tools and middleware recognize and forward. It’s a deceptively simple idea with a massive practical impact on end-to-end tracing in production.

How Dynatrace contributes

Dynatrace helped lead W3C Trace Context. Alois Reitbauer, now Dynatrace’s Chief Technology Strategist, chaired the W3C Distributed Tracing Working Group; colleagues including Christoph Neumüller and Daniel Khan worked alongside engineers from Google, Microsoft, and New Relic to iterate on the standard, and Daniel Dyla contributed as a co-author alongside his OpenTelemetry work. Dynatrace was a founding sponsor of the working group when it moved into the W3C in 2018, and the first vendor to ship production support for the standard.

How the community benefits

W3C Trace Context has been adopted by the OpenTelemetry project (which uses it as its default propagation format), Microsoft Azure services, Google Cloud Platform™ services, Elastic® services, and many others. Middleware, service meshes, and cloud-managed services now forward traceparent headers automatically, making end-to-end visibility achievable in environments where it was previously impossible. It’s one of those standards that, once adopted, makes you wonder how anything worked without it.

OpenFeature: An open standard for feature flags

What is the OpenFeature standard?

The OpenFeature project, a CNCF Incubating project, defines a vendor-neutral open standard for feature flag management. It provides a unified API and SDK developers can code against regardless of the backend — an in-house solution, a commercial platform, or an open source tool.

Feature flags are a core practice in modern software delivery: They let teams decouple deployments from releases, roll out features to targeted user groups, run A/B experiments, and instantly kill-switch problematic features — all without redeploying code. But the ecosystem has historically been fragmented: Every vendor and homegrown solution has its own API, so migrating between solutions requires rewriting application code.

The OpenFeature project remedies this with a standardized abstraction layer. You write your evaluations once against the OpenFeature API; to switch backends — say, from an in-house solution to a commercial vendor — you swap out the “provider” (a thin adapter), and your application code doesn’t change.

How Dynatrace contributes

Dynatrace initiated and led the creation of the OpenFeature project. In 2022, we assembled an industry consortium of top feature flag management solutions and submitted the standard to the CNCF as a sandbox project. We remain one of the project’s key drivers, contributing to the specification, SDKs, and provider ecosystem.

Dynatrace engineers play central roles in the project’s governance and day-to-day development:

Governance Committee: Michael Beemer, a Product Manager at Dynatrace, serves on the OpenFeature Governance Committee and has been instrumental in driving the project from its earliest days. Alois Reitbauer, Dynatrace’s Chief Technology Strategist and a driving force behind many of our open source initiatives, has recently passed leadership to Jonathan Norris following our acquisition of DevCycle.

Project Maintainer: Todd Baert, a Software Engineer at Dynatrace, works almost entirely as an OpenFeature project maintainer, focusing on SDK development, integration points, extensibility, and the OpenTelemetry hook that bridges feature flags with observability data.

Observability is built in by design: The OpenTelemetry API supports OpenFeature flags, so feature flag evaluations are automatically captured as part of distributed traces. Observability platforms like Dynatrace can show you not just what happened in a request, but which flags were evaluated, what values were returned, and how those flags affected performance and user experience.

In early 2026, Dynatrace acquired the DevCycle® platform, a feature management solution built natively on the OpenFeature standard, deepening our investment in open, standards-based progressive delivery.

How the community benefits

The OpenFeature standard lets developers adopt feature flags without vendor lock-in. Outgrow an in-house solution? Move to a commercial provider without rewriting code. Vendor sunsets their product? Your application code is unaffected; just swap providers. And because feature flag evaluations are captured as part of distributed traces, teams can correlate flag states with performance data, error rates, and user experience metrics — making progressive delivery measurably safer.

Get involved

Visit openfeature.dev to learn about the standard, try a provider, or contribute to an SDK. The community is active on the CNCF Slack.

Keptn toolkit: Cloud-native application lifecycle orchestration

What is Keptn toolkit?

Keptn toolkit is a CNCF Incubating project that provides event-driven orchestration for cloud-native application delivery and operations. It connects observability to operations through a declarative approach to automation for deployment pipelines, quality gates, and auto-remediation workflows.

In practical terms, Keptn toolkit lets you define Service Level Objectives (SLOs) and automatically evaluate them at every stage of your delivery pipeline. Did that new deployment meet its performance targets? Keptn toolkit checks your SLIs (Service Level Indicators) against your SLOs and decides automatically whether to promote the release or roll it back. It’s often described as “SRE in a Box.”

Field teams at Dynatrace kept seeing the same pattern across customer engagements: every organization was writing the same Jenkins scripts, the same ServiceNow® integrations, the same dashboard-checking routines, just with slight variations.

Keptn toolkit is tool-agnostic by design: it integrates with whatever CI/CD, testing, monitoring, and notification tools you already use. You define the “what” — your delivery and operations strategy — and Keptn toolkit orchestrates the “how” using your existing toolchain.

Keptn toolkit’s unique value in the CNCF landscape is its focus on observability-driven operations. SLOs drive its orchestration decisions, bridging the gap between “we deployed it” and “it’s actually working well for users.” For SRE and platform engineering teams, it’s a structured, declarative way to codify operational best practices without being tied to any specific tool.

How Dynatrace contributes

Dynatrace created Keptn toolkit internally in 2019 and donated it to the CNCF as a sandbox project in 2020. It was promoted to CNCF Incubating status in 2022 after significant community growth.

Key Dynatrace contributors include Alois Reitbauer (Keptn toolkit creator and Dynatrace CTS), Andreas Grabner (co-founder and DevOps Activist at Dynatrace), and Johannes Bräuer (co-founder). The project was born directly out of our work helping customers implement autonomous cloud operations.

Get involved

Head to keptn.sh to learn about the project, try the quickstart, or build an integration for your favorite tool.

How the community benefits

Since joining the CNCF, Keptn has grown tenfold in adoption, with more than 100 production instances and more than 40 companies using it in production including Citrix, T-Systems, and others. The project has attracted dozens of community-built integrations and inspired the CDEvents standard in the Continuous Delivery Foundation.

OffOn.dev: Building the Open Source Community of tomorrow

Every project in this post depends on something that doesn’t show up in a GitHub® commit graph: the people who sustain them — maintainers, contributors, mentors, and the next generation of developers who haven’t written their first PR yet. They’re often overlooked. Most platforms, foundations, and content formats are built with enterprises, legal teams, or marketing goals in mind, not for individual maintainers or attracting new contributors. Yet these are the people who keep open source alive.

OffOn.dev was created to address that gap. Too many community spaces are built to react to problems — they grow, but stay focused on troubleshooting rather than adoption, innovation, and exploration. The Open Ecosystem takes a different approach: a practical, vendor-agnostic space for builders working with open standards and cloud-native tools, focused on reproducible, hands-on knowledge — tutorials with real code and configs, Q&A with accepted solutions, technical challenges, and a Community Voices space for members’ projects and demos.

This is not a sales channel. Companies like Dynatrace may sponsor with resources or funding, but sponsors don’t influence the content or the mission. The purpose is to sustain open source momentum and help create the maintainers of tomorrow’s digital landscape — a need made more urgent by digital sovereignty and emerging regulations like the CRA and EU AI Acts.

Whether you’re an SRE wiring up an OpenTelemetry pipeline, a developer experimenting with OpenFeature flags, a student writing your first contribution, or a maintainer mentoring the next generation — the Open Ecosystem is built for you. Introduce yourself, jump into a challenge, share what you know. Head to OffOn.dev and come build with us.

The post Building in the open: How Dynatrace invests in open source to move the industry forward appeared first on Dynatrace news.

In May 2026, OpenTelemetry (OTel) officially graduated from Cloud Native Computing Foundation (CNCF). This milestone marks the cloud native ecosystem’s achievement of production readiness and maturity, thanks to the efforts of hundreds of companies and thousands of developers who believed in an open standard for observability and built it together.

OpenTelemetry was already the standard. Now it’s official.

If you’ve been shipping production code over the last few years, OpenTelemetry has almost certainly touched your tech stack, whether through its SDKs and collectors, or traces, metrics, and logs. For many teams, OTel has quietly become part of the default toolbox for building and operating modern applications. It solves a real problem: the industry needed a common language for telemetry data, and OpenTelemetry became that language.

CNCF graduation reflects the strength of the ecosystem: a diverse contributor base, widespread vendor support with proven production readiness, comprehensive security audits and a governance model built by the community, for the community and for future sustainability.

Why a shared standard changes everything

Graduation formalizes OpenTelemetry as the common protocol and shared language for observability:

1. A standard protocol allows different tools, open source and commercial, to work together.
2. Semantic conventions define how telemetry is named and structured. When all systems speak the same language, correlation becomes possible at scale.
3. OpenTelemetry decouples instrumentation from backend analytics, enabling teams to export telemetry data to any backend system and switch analytics platforms without rewriting code.
4. Standardized, high-quality telemetry data allows automation, anomaly detection, and AI-driven insights. A consistent protocol becomes critical as systems grow more complex and autonomous.

Dynatrace loves OpenTelemetry and open source

Dynatrace has been involved in shaping OpenTelemetry from its early days, contributing to the specification, semantic conventions, Collector, and many other areas, ensuring the standard works at enterprise scale. With over 46,000 contributions and 54,000 commits, Dynatrace is one of the top contributors to the project.

Our focus has always been clear: make OpenTelemetry production-ready without compromising its open, vendor-neutral model.

Beyond OpenTelemetry, Dynatrace actively contributes to over 30 open source projects, including W3C Trace Context, and integrations with Kubernetes, JMeter, and more.

Frequently asked questions

Is OpenTelemetry stable after CNCF graduation?

Yes. Graduation confirms that the core specifications, APIs, and data model are stable and suitable for long-term production use. Teams can adopt OpenTelemetry with confidence across environments and use cases.

Does OpenTelemetry lock teams into a specific vendor or backend?

No. OpenTelemetry decouples instrumentation from backend analytics. Teams can export telemetry data to any backend and switch platforms without rewriting instrumentation code.

How does Dynatrace support OpenTelemetry?

Dynatrace supports you wherever you are. You can ingest pure OpenTelemetry data natively; no proprietary agents are required. From there, you can query billions of spans in seconds, correlate metrics, logs, and traces automatically across petabytes of data, and get the full observability context.

Want to see OpenTelemetry in action?

Learn more about OpenTelemetry at Dynatrace Hub.

The post OpenTelemetry graduates: A milestone for the observability Open Source community appeared first on Dynatrace news.

If you’ve ever tried to track down a weird bug or security issue in a mountain of log files, you know how important it is to be able to search for any string no matter how random or unexpected. Log search engines need to be flexible, letting you look for anything from error codes to user IDs, or even fragments of stack traces. But with modern systems generating massive amounts of logs every day, storing all that data quickly becomes a problem.

That’s where compression comes in. By squeezing down the size of your logs, you save on storage costs and make it easier to keep more history around. The trick is to compress your data without making searches painfully slow. Enter the FM-Index: a clever data structure that combines powerful compression with fast search.

With the FM-Index, you can have a practical balance between compact storage and the freedom to search for anything you need. And if you are working in Java, we’ve got you covered with the open source index4j, created by the Real-Time Analytics lab of Dynatrace Research, an internal research group at Dynatrace where we work on researching methods for real-time processing and analysis of massive data streams.

When should you use the FM-Index?

The FM-Index is a relatively complex data structure that shines in specific scenarios:

• Your data is immutable or changes infrequently. The FM-Index is built for static datasets, so if your logs are constantly changing, you might want to look at other options.
• You need to perform a lot of queries. If you only need to search your logs occasionally, the overhead of building the FM-Index might not be worth it.
• Your queries are likely to be arbitrary substrings or they don’t fit a predefined pattern. The FM-Index does not care about the structure of your queries, making it ideal for searching through unstructured log data.
• You care about storage efficiency: It might not be your top priority, but you still want to save space without sacrificing search speed.
• You like the idea of combining compression and indexing into a single data structure, simplifying your architecture.

If any of the above sounds about right for your use case, then the FM-Index could be a strong fit.

In the next sections, we look at how to get started with the Dynatrace Open Source FM-Index library and how to apply it to your own log data!

What is the FM-Index?

So, what exactly is this FM-Index thing? In simple terms, the FM-Index is a supercharged search tool that lets you look for any substring inside your data, even if you never split your logs into words or tokens. It’s built on some clever computer science tricks: suffix arrays, bit vectors, wavelet trees, and the Burrows–Wheeler transform (BWT). Don’t worry if you are not familiar with them, the important part is that they work together to make searching and compressing possible at the same time.

With the FM-Index, you can:

• Count how many times a pattern appears in your logs, even if it’s just a random string of characters.
• Jump straight to the position of each match, so you can see exactly where something happened.
• Retrieve (or decompress) the matching lines or the context around them, all without having to decompress the entire log file upfront.

What’s really cool is that the FM-Index does all this while keeping storage and search times relatively low in practice, even as your logs grow huge. The magic is in its sublinear complexity: as your data gets bigger, the time and space needed for searches don’t grow nearly as fast. That’s why the FM-Index is a favorite in fields like bioinformatics, where people need to search through massive DNA sequences. But it’s just as handy for anyone dealing with mountains of log data and looking for fast, flexible search without the storage bloat.

And the coolest part? You can trade off between how much space you use and how fast your searches are by adjusting only a single parameter when building the index. A higher sample rate means better compression but slower queries, while a lower sample rate gives you faster searches at the cost of more storage space. You can choose what works best for your system!

Let’s get into it

Imagine you have collected ~1.5 million logs from an Android framework. Here’s a sample data set you can use to follow along (around 180 MB of raw text).

Let’s say you suspect one app is doing some questionable activity. Notice that:

1. The logs are static
2. You want to keep them around for a long time (maybe you want to do some forensics later)
3. You need to run many queries to find the root cause

The above constraints make it a a strong candidate for using an FM-Index. By using index4j, you can build an FM-Index over these logs with just a few lines of Java code. The index4j library is available in the Maven Central repository, so you can add it to your Java project as follows.

For maven projects, use:

<dependency> 
    <groupId>com.dynatrace.index4j</groupId> 
    <artifactId>indices</artifactId> 
    <version>0.3.1</version> 
</dependency>

For gradle projects, use:

implementation group: 'com.dynatrace.index4j', name: 'indices', version: '0.3.1'

Then, building your index is as simple as:

char[] text = Files.readString(Path.of("Android.log"), StandardCharsets.UTF_8) 
        .toCharArray(); // catch the exception as needed 
FmIndex fmi = new FmIndexBuilder() 
        .setSampleRate(64) 
        .setEnableExtraction(true) 
        .build(text); 

This will build an index in about one minute in our test environment while at the same time compressing the data down to roughly ~70 MB. From ~180 MB; that’s a reduction of more than 60%! And recall that this also includes the actual index. No additional storage is needed for searching.

Now that you have your index, searching is a breeze. Let’s say you want to find if your logs contain the string Permission Denial: which shows that an app is attempting to access resources for which it does not have the required permissions and may indicate suspicious behavior. Since we only care about whether this string exists and how many times it appears, we can use the count method:

char[] pattern = "Permission Denial:".toCharArray();
int counts = fmi.count(pattern);

This query will return the number of occurrences of the pattern in your logs, which should be 283 for the Android.log. Running it on a desktop processor (Intel Xeon W-10885M with the clock fixed at 2.40GHz) takes:

• If cold (no caching, no optimization done by the compiler yet), around 200 microseconds (or 0.2 milliseconds and ~5,000 queries per second)
• If warm (with caching and optimizations), around 30 microseconds (or 0.03 milliseconds and ~33,000 queries per second)

These results are based on a single benchmark setup; actual performance may vary depending on hardware, dataset, and workload. Note that if you run the same query naively, this would require a brute-force scan of the entire log file, which takes around ~100 milliseconds (~10 queries per second). With index4j, that represents a substantial performance improvement compared to a naïve scan in this benchmark.

You are probably wondering now: “What if I want to see the actual log lines that contain this pattern?” Counting is useful, but most likely you want to see the context around the matches. For that you can use the extract methods that index4j provides. Note that for the index there is no notion of “lines” anymore. The FM-Index just sees a big array of characters that is shuffled in some obscure way (that’s also where the compression comes from). However, since we enabled extraction when building the index (recall the line .setEnableExtraction(true)), index4j can still retrieve the original log lines. There are two steps to do this:

First, we need to find the positions of all occurrences of the pattern. We can do this using the locate method:

int[] locations = new int[counts]; 
int found = fmi.locate(pattern, 0, pattern.length, locations, counts); // find all matches 

Where counts is the number of matches we obtained in the previous count query. Note that we can also restrict how many positions we want to find by changing the last parameter of locate.

Second, we can extract the actual log lines, either in its entirety (using extractUntilBoundary) or partially using one of its variants. In this case, we want to see what is to the right side of each match (i.e., the context after the pattern), so we can use extractUntilBoundaryRight:

char[] destination = new char[1024]; // maximum length per extraction 
for (int i = 0; i < found; i++) { 
    // extract previous locations until the end of line 
    int length = fmi.extractUntilBoundaryRight(locations[i] - 1, destination, 0, '\n');  
    System.out.println(new String(destination, 0, length)); // print the extracted string 
} 

This will print each match and the remaining of the log line for all of the 283 lines containing Permission Denial:. For example:

Permission Denial: reading com.android.providers.media.MediaProvider uri content://media/external/audio/media from pid=12127, uid=10146 requires android.permission.READ_EXTERNAL_STORAGE, or grantUriPermission() 
 
Permission Denial: receiving Intent { act=android.intent.action.PHONE_STATE flg=0x10 (has extras) } to ProcessRecord{9ef8336 4492:com.tencent.mm/u0a105} (pid=4492, uid=10105) requires android.permission.READ_PRIVILEGED_PHONE_STATE due to sender android (uid 1000) 
 
Permission Denial: broadcasting Intent { act=android.media.SCO_AUDIO_STATE_CHANGED flg=0x10000010 (has extras) } from null (pid=-1, uid=-1) requires com.huawei.permission due to registered receiver BroadcastFilter{14b6ff9 u0 ReceiverList{c9f82e3 633 com.huawei.espacev2/10102/u0 remote:497ab12}} 
 
... 

Note that we can also stop extracting at a certain character (for example, a space) if we know there is some structure to the log line itself. The runtime to extract depends only on the number of characters and is roughly ~10 microseconds per character, regardless of how arbitrary the pattern may be or its frequency. That means that the extraction of 100 log lines of average length 100 characters will take around 100 * 100 * 10 microseconds = 100 milliseconds. And all of this while your data and index take only ~30% to ~40% of the original log size.

How do I store my index?

Imagine you have already run a few queries and found some interesting patterns in your logs, but you wish to save the index to disk so you can load it later without having to rebuild it from scratch. That is easy because the whole index is serializable. You can simply write it to a byte array like this:

// Save 'serialized' to disk as needed 
byte[] serialized = Serialization.writeToByteArray(FmIndex::write, fmi); 

And read it back later like this:

// read from disk into 'serialized' byte array as needed 
FmIndex deserialized = Serialization.readFromByteArray(FmIndex::read, serialized); 

And that’s it! You can now save and load your FM-Index as needed, making it easy to persist your work and pick up where you left off.

Conclusion

As we saw, the FM-Index is a powerful tool for anyone dealing with large amounts of log data. It combines competitive compression with fast and arbitrary search capabilities (usually in microseconds-time), making it ideal for scenarios where logs are static and queries are changing. While still a relatively niche data structure, state-of-the-art research articles on log search engines have already demonstrated the capabilities of the FM-Index.

By using the index4j library, you can easily build an FM-Index over your logs in your customized log processing pipeline. Whether you’re hunting for specific error messages or performing recurrent queries, the FM-Index provides a flexible and efficient solution that can be tailored to your use case. If you are interested in other cool projects that we work at Dynatrace Research, make sure to check out our research homepage and our open source GitHub repository!

The post index4j: Open-source FM-Index for fast queries on compressed logs appeared first on Dynatrace news.

Hashing shows up everywhere in computer science, from quick lookups in hash tables to verifying data with checksums and fingerprints. A hash table works by turning each key into an integer so it can be retrieved fast—ideally with each distinct key producing its own unique value. When two different inputs produce the same output, you get a collision.

While hash tables usually handle collisions through strategies like chaining or open addressing, it is still best to avoid them wherever possible, especially for use cases such as calculating checksums or fingerprinting.

Hashing raw byte arrays is simple enough, but real‑world data structures often contain fields of different lengths, and that’s where collisions become surprisingly easy to introduce.

The Real-Time Analytics lab of Dynatrace Research has developed an open-source library called Hash4j that implements some best practices for hashing, including our preferred approach to variable-length fields.

This article walks through why variable‑length fields make hashing tricky and how to handle them cleanly.

A common pitfall: Concatenating variable-length fields

Let’s consider a simple example of a data structure with two variable-length string fields:

firstName and lastName.

record Person(String firstName, String lastName) {} 

A naive approach to hashing this structure might involve concatenating the two strings and then applying a hash function to the resulting byte sequence. However, this method can lead to collisions. For instance, the records ("ANNA", "BELL") and ("ANN", "ABELL") would both produce the same concatenated string "ANNABELL". This occurs because the boundaries between the variable-length fields are lost, making it impossible to distinguish between different combinations of field values.

When serializing data structures for hashing, good practice is to ensure that the serialized byte sequence allows for unambiguous reconstruction of the original structure, which you can’t do in the example above unless additional information is provided.
Let’s explore different strategies to handle variable-length fields properly by adding extra information when mapping the data structure to a byte sequence.

Strategy 1: Delimiters

One effective strategy to avoid this issue is to use delimiters between variable-length fields. For example, we could serialize the data structure as:

firstName ‖ ";" ‖ lastName 

where ‖ denotes concatenation and ";" is a chosen delimiter. This way, the records ("ANNA", "BELL") and ("ANN", "ABELL") would serialize to "ANNA;BELL" and "ANN;ABELL", respectively, thus producing distinct byte sequences and avoiding collisions.

The limitation of this approach is the need to ensure that the chosen delimiter does not appear in the actual data. For strings like names, this might be manageable as certain byte patterns are not used. For example, the null character is often used as a delimiter in C-style strings as it does not appear in regular text.

Escape sequences are needed to tell delimiters apart from real data, but that means scanning everything and replacing conflicts, which adds overhead. Nested structures like List<List<String>> make this even harder, since each level needs its own delimiter rules, quickly becoming complex and error‑prone.

Strategy 2: Length prefixes

A better approach is to use length prefixes for each variable-length field. This method involves prefixing each field with its length before concatenation. For example, we could serialize the data structure as:

len(firstName) ‖ firstName ‖ len(lastName) ‖ lastName

where len() represents the length of the string. This way, the records ("ANNA", "BELL") and ("ANN", "ABELL") would serialize to 4 ‖ "ANNA" ‖ 4 ‖ "BELL" and 3 ‖ "ANN" ‖ 5 ‖ "ABELL", respectively, resulting in distinct byte sequences. This approach eliminates the need for delimiters and escaping. Additionally, it scales better for nested data structures, as each level can independently use length prefixes without concern for delimiter conflicts.

This method is the standard practice in many serialization protocols, such as Protocol Buffers and Thrift. Length prefixes make serialized data easy to parse because the parser can read the length first and then extract exactly that many bytes. But for hashing, this doesn’t matter because you only need a one‑way conversion to a byte sequence, not full reconstruction.

Strategy 3: Length postfixes

Another approach is to use length postfixes, where the length of each variable-length field is appended after the field value itself. For example, we could serialize the data structure as:

firstName ‖ len(firstName) ‖ lastName ‖ len(lastName)


This method also ensures that different combinations of variable-length fields produce distinct byte sequences. It may be less intuitive than using length prefixes, as the length information appears after the data, which is also a less common practice in serialization, because it would require parsing starting from the end of the byte sequence to reconstruct the original data structure. However, for hashing purposes, this method is equally effective in preventing collisions, because, for a high-quality hash function, the hash-quality does not change when reordering the input bytes.

The method is even better suited for streaming scenarios or data structures, where the length of the data may not be known in advance. For example, consider null-terminated strings, where the end of the string is indicated by a special null character or linked lists, where the end of the list is indicated by a null pointer or null reference. This would require double scanning the data: once to compute the length, once to write it. In contrast, the postfix approach allows writing the data first and then appending the length, which can be determined during the same pass.

What about mixing strategies?

Though not recommended, strategies can also be mixed, provided that this is done with careful consideration. For example, one could use length prefixes for some fields and delimiters for others. However, this approach can introduce complexity and potential pitfalls if not managed carefully. It is generally advisable to stick to a single consistent strategy throughout the data structure to maintain clarity and reduce the risk of errors resulting in a higher collision probability.

To demonstrate this, consider a record with two variable-length integer arrays:

record Record(int[] v1, int[] v2) {}

Assume that we serialize the first array using length prefixes and the second array using a postfix length. The serialization would look like this:

len(v1) ‖ v1 ‖ v2 ‖ len(v2)

Obviously, this serialization is unambiguous, because the record can be reconstructed without any issue if we assume that the lengths are encoded with a fixed amount of bytes following these steps:

1. Take the length of v1 from the beginning of the byte sequence
2. Read the corresponding number of integer values for the first array
3. Take the length of v2 from the end of the byte sequence
4. Read the corresponding number of integer values for the second array from the remaining bytes in the middle

Now, let’s see if we serialize the first array with a length postfix and the second array using a length prefix. The serialization would look like this:

v1 ‖ len(v1) ‖ len(v2) ‖ v2 

In this case, the serialization can be ambiguous. Consider the
records ([5,1,4], [6,7]) and ([5], [3,2,6,7]). Both records would produce the same serialized byte sequence:

5 ‖ 1 ‖ 4 ‖ 3 ‖ 2 ‖ 6 ‖ 7 

This example shows that mixing strategies can lead to ambiguities and potential hash collisions if not handled carefully. Applying a single strategy consistently across all variable-length fields in a data structure is a safer and more reliable approach.

Is the length information needed for all variable-length fields?

You might have noticed that we could skip the length of one field in the example above. When using length-postfixes, the length of the first variable-length field can be omitted because it can be inferred from the total length of the byte sequence minus the lengths of the other fields.

v1 ‖ v2 ‖ len(v2) 

Similarly, when using length-prefixes, the length of the last variable-length field can be omitted for the same reason.

It’s safe to omit certain length values if you can still determine the size of every variable‑length field without ambiguity, whether by reading from the beginning, the end, or by deriving it from other fields.

For example, a data structure consisting only of a single variable-length field does not require any length information, as the length can be derived from the total length of the byte sequence minus the lengths of all fixed-length fields. Also, in a data structure with multiple variable-length fields, whose lengths are known to be equal or can be derived from other fields, some length information can be theoretically omitted.

However, these performance optimizations are not recommended as a general practice as they introduce complexity and risk for errors, in particular, when dealing with nested and more complex data structures.

How to serialize the length information

When serializing the length information for variable-length fields, we recommend using fixed-size integers that are appropriate for the expected maximum length of the fields. For example, if you expect that the length of a string will not exceed 65,535 characters, you can use a 2-byte unsigned integer to represent the length. If the lengths can be larger, consider using 4-byte or 8-byte integers accordingly. In Java, the size of Arrays and Collections cannot exceed Integer.MAX_VALUE, so a 4-byte integer is usually sufficient there.

Variable‑length integer encodings like VarInt are great for saving space during serialization, but they’re not recommended for hashing. They require branching logic to encode, which is slower than handling fixed‑size integers. Since modern hash functions can process each byte in just a few CPU cycles, the extra overhead from variable‑length encodings simply isn’t worth it.

Moreover, using variable-length encodings for the length information also introduces additional complexity and potential for errors. For example, VarInt is not compatible with the length-postfix strategy without reversing the byte order, because the byte sequence cannot be unambiguously deserialized when reading from the end to the beginning, which is the underlying idea behind length-postfixes.

Hash4j’s approach

Hash4j, the Dynatrace hash library for Java, provides a comprehensive set of hashing utilities, including support for hashing data structures with variable-length fields. Due to the advantages discussed above, Hash4j implements the length-postfix strategy for serializing variable-length fields. This allows hashing of collections, arrays, optionals, and other composed data structures with variable-length fields effectively while minimizing the risk of collisions. Let’s consider the following data structure to store information about a personal contact:

record Contact( 
  String firstName, 
  String lastName, 
  OptionalInt age, 
  byte[] utf8Email, 
  List<String> sortedSocialMediaHandles 
  ) {} 

In this example, Contact has five variable-length fields:

1. firstName – a variable-length string representing the person’s first name.
2. lastName – a variable-length string representing the person’s last name.
3. age – an optional integer representing the person’s age if known. Optionals can be considered and treated like lists with lengths restricted to zero and one. Thus, a single byte will be sufficient to represent the length information.
4. utf8Email – a variable-length byte array containing the person’s email address encoded in UTF-8. (Arrays in Java records are discouraged, but this is just to demonstrate Hash4j’s capability of handling arrays.)
5. sortedSocialMediaHandles – a variable-length list of strings representing the person’s social media handles. This field nests variable-length strings inside a variable-length list. The list is sorted to impose a deterministic order before hashing so that the same set of handles yields the same hash regardless of insertion order. Unordered collections like sets would require special handling to yield order-independent hash values, which is beyond the scope of this blog post.

With Hash4j Contact instances can be simply hashed by adding a member method to the record that describes how the individual fields contribute to the hash computation:

void contribute(HashSink hashSink) { 
  hashSink 
    .putString(firstName) 
    .putString(lastName) 
    .putOptionalInt(age) 
    .putByteArray(utf8Email) 
    .putOrderedIterable(sortedSocialMediaHandles,
      (handle, hs) -> hs.putString(handle)); 
} 

The putString, putOptionalInt, putByteArray, and putOrderedIterable methods all handle the serialization of variable-length fields using the length-postfix strategy internally, ensuring that the hash computation minimizes the risk of collisions. All methods append some fixed-size information about the length to the data to the hash sink. For Strings, byte arrays, and Iterables, the length is represented as a 4-byte integer, while for Optionals, the length is represented as a 1-byte value.

After defining the contribute method, the hash of a Contact instance c can then finally be computed as follows:

long hash = Hashing.komihash5_0().hashToLong(c, Contact::contribute); 

Here komihash5_0() is only one of the high-quality hash functions provided by Hash4j. The library also offers other hash functions like XXH3 and MurmurHash3, all of which can be used interchangeably in the example above.

Effective serialization strategies for hashing variable-length fields

When hashing data structures with variable-length fields, it is crucial to use serialization strategies that preserve the boundaries between fields to minimize the risk of hash collisions. Applying the length‑postfix strategy the way Hash4j does gives you a robust approach to hashing variable‑length fields. It avoids the ambiguity problem with naive concatenation and is simpler and better suited for nested data structures than delimiter-based methods. Unlike prefixes, it does not require a second pass over fields whose length is not known in advance.

The post Best practices for hashing variable-length data structures appeared first on Dynatrace news.

Dynatrace principal developer advocate Adriana Villela is stepping up her involvement with the OpenTelemetry® community. As of today, she will be taking on the role of OpenTelemetry community manager, alongside Reese Lee of New Relic. They are joined by Julia Furst Morgado, of Dash0, who will serve as associate community manager, and will be taking over the role from Austin Parker of Honeycomb.

Community managers are appointed by the OpenTelemetry Governance Committee and act as public stewards of the contributor and end-user community. They’re responsible for organizing events, coordinating efforts to improve contributor experience, managing the OpenTelemetry presence on social media, and working to nurture and grow the OpenTelemetry community overall. Effectively, this is a project-maintainer role whose project is the OpenTelemetry community itself.

“I’m thrilled, then, that Adriana and Reese have stepped up to not only continue growing our community management role, but invest time in helping scope and shape this function for the project as it enters into its next era of growth,” said Austin in the official announcement, which was also made at OTel Unplugged in Brussels, Belgium, on February 2nd, 2026.

“As a community manager, I want to continue to raise the OpenTelemetry community profile within the Cloud Native Computing Foundation and the wider open-source world,” Adriana says.

The path to OpenTelemetry community leadership

Adriana’s journey with the OpenTelemetry community began in 2021 when she worked for Tucows, where she managed the platform engineering and observability teams. “I had dabbled a bit in observability when I was at my previous job as a release engineer at Ceridian, but when I stepped in as manager of the observability team at Tucows, to do right by my team and the organization, so I could lead them in the right direction,” she says. “I learned as much as I could about observability and OpenTelemetry projects, and I did it in public.” She documented her learnings in her Unpacking Observability series on Medium.

While at Tucows, Adriana made it her mission to move away from vendor lock-in and towards OpenTelemetry tools. But 2021 was still early days for the project. For example, traces weren’t yet generally available. To learn more about how to expand OpenTelemetry project use in a large enterprise, she connected with OpenTelemetry project co-founder Ted Young, then director of open source development at Lightstep, and Honeycomb field CTO Liz Fong-Jones, then developer advocate at Honeycomb. “Both Liz and Ted put aside their competitor differences to jump on a Q&A call with Tucows developers to answer their questions and concerns about OTel,” Adriana recalls.

Adriana’s writings caught Austin’s eye in 2022, and they hired her for her first developer relations role. As part of the role, she was encouraged to contribute to the OpenTelemetry community. When one of the original founders of the OpenTelemetry End User Working Group (later converted to a special interest group or SIG) left the project in 2023, Ted and Austin asked Adriana to help lead the group alongside Reese.

Together, Adriana and Reese raised the End User SIG’s profile by running monthly Open Telemetry livestreams, launched the Humans of Open Telemetry series, and collaborated with various other SIGs to gather end-user feedback through surveys. She also contributed to the CNCF OpenTelemetry Certified Associate (OTCA) certification.

The OpenTelemetry community has advanced significantly since Adriana began her exploration of the project, and it only continues to evolve. She points to the work around the Open Agent Management Protocol (OpAMP) for managing data collection agents such as the OpenTelemetry Collector, and the increased emphasis on the quality of telemetry data. Over the years, most of the major observability vendors have embraced OpenTelemetry tooling, both by supporting its data ingest format, OTLP, natively, and as by contributing to OpenTelemetry projects, making it the standard for instrumentation. “That tells me that it’s here to stay,” she says. “I can’t wait to see where the community takes it next.”

Reflections from the OpenTelemetry community on Villela’s impact

Comments from Adriana Villela’s fellow contributors on her engagement with the OTel community:

“Adriana is a tireless and omnipresent anchor of the OpenTelemetry community, whether it’s educating, building, or the organizing the end user SIG. My entry into the community was easy thanks to her efforts, and I’m sure countless others could say the same.”

— Josh Lee, Open Source Evangelist at Altinity

“I met Adriana at the very first OTel Unplugged in 2022, and together we developed the End User Working Group from a 3-person task force to the bigger and better End User SIG it is today (we’ve since added 2 more Maintainers!). I’m so stoked to be moving to the next stage of OTel community work alongside Adriana.”

— Reese Lee, Developer Relations Engineer and incoming OTel Community Manager, New Relic

“Adriana has a unique voice in the OpenTelemetry community, addressing current both end user and community challenges. Through her technical work and her public contributions in the open source observability ecosystem, Adriana connects with a wider audience: from developers to platform engineers or first time contributors. Her blog posts, conference talks and podcasts, help practitioners understand not just how OpenTelemetry works, but why it works the way it does, creating a trail of useful resources that anybody can use for onboarding or deepening specific topics.”

— Diana Todea, Developer Experience Engineer and OpenTelemetry contributor, Victoria Metrics

The post Adriana Villela of Dynatrace takes on OpenTelemetry community manager role appeared first on Dynatrace news.

In this blog post, we’ll walk you through how to observe your collectors using Dynatrace Dashboards and highlight the valuable insights you can gain from their internal telemetry. Next, we’ll show you how to set up Davis® Anomaly Detection to continuously track your collectors. You’ll learn how to configure alerts that notify you of any deviations from healthy behavior, ensuring proactive issue detection and faster response times.

Activate collector telemetry and access the dashboards

To try out the OpenTelemetry collector self-monitoring dashboards, first activate the collector’s internal telemetry. For details, follow the configuration example in our documentation.

Once your collector begins exporting its internal telemetry, go to Dynatrace Hub and install OpenTelemetry Dashboards.

Once the app is installed, the following dashboards are available in the Ready-made dashboards section:

• OTel Collector self-monitoring (all collectors): Shows an overview of all detected OpenTelemetry Collector instances.
• OTel Collector self-monitoring (single collector): Allows you to analyze a specific collector instance.

Collector insights in the dashboards

The Dynatrace Collector dashboards deliver comprehensive insights into the operational status and efficiency of your collector instances. Here are the key features:

• Collector uptime and CPU time since start: Monitor how long the process has been running and the CPU time consumed.
• Collector memory and heap usage: Track memory consumption and heap usage to ensure optimal performance.
• Receivers: View signals accepted and refused, categorized by data type.
• Processors: Monitor incoming and outgoing signals to understand data flow.
• Exporters: Track signals sent, failed to enqueue, and failed to send, categorized by data type. Additionally, monitor queue size and capacity.

HTTP/gRPC requests and responses: Count, duration, and size of requests and responses to analyze communication efficiency.

The dashboards rely on the internal telemetry exported by any standard OpenTelemetry Collector. They have been verified with Dynatrace OTel Collector and OTel Collector contrib. If your Collector distribution uses a different service.name value that is not in the dashboard drop-down list (top-left corner), you can extend the list with your custom value and then save a copy of the dashboard.

The value of internal telemetry

One of the most valuable aspects of collector self-monitoring is its ability to prevent data loss by helping you scale your infrastructure at the right time. When telemetry data is dropped due to resource constraints—for example, memory exhaustion or queue overflows—it can lead to blind spots in your observability pipeline, making it harder to troubleshoot issues or ensure compliance.

The Single Collector Dashboard plays a crucial role in avoiding this. It surfaces key performance indicators that highlight when your collectors are under pressure, enabling you to take proactive action before problems escalate. This ensures your system remains resilient, responsive, and capable of handling increasing workloads without sacrificing data integrity.

Let’s take a closer look at some of these indicators.

The Telemetry data passing through the collector section shows the number of requests that are accepted, sent, and refused for each signal. As a practical example, an increase in refused spans often indicates that the collector has exceeded its memory limit, and you should scale your collector instances to handle the increased load. Ignoring this can lead to continuous data loss at the collector’s ingestion stage.

The Memory and CPU time section displays your collector’s resource consumption, enabling effective capacity planning for future workloads and proper system scaling. These metrics help optimize infrastructure costs by preventing over-provisioning and revealing performance anomalies, like sudden spikes or drops, that prove crucial during troubleshooting and root cause analysis.

The Queue size metrics section provides insights into the exporter’s current queue size; it allows you to compare if you’re close to its maximum capacity. These metrics are vital for understanding the efficiency of data export processes. If the current queue size increases, it indicates that there are not enough workers available to send the data or that the backend receiving the data is too slow. This is another critical signal that scaling is necessary to maintain optimal performance.

Configure alerts using Davis anomaly detection

While dashboards offer a great overview for your collectors, they still rely on manual oversight. What if you could automate this process and get notified the moment something unusual happens? That’s where Davis Anomaly Detector comes in. With just a few simple steps, you can set up intelligent alerts that automatically notify you when collector behavior deviates from expected patterns, helping you catch issues early and respond proactively.

Step 1. Create a Davis Anomaly Detector

Start by creating a new Davis Anomaly Detector. In this example, we’ll monitor refused spans, which indicate that a Collector is overwhelmed and unable to ingest incoming telemetry.

Use the following metric expression to define your scope:

timeseries {sum(otelcol_receiver_refused_spans)}, by:{service.instance.id}

This configuration tracks refused spans per individual collector, identified by their unique service.instance.id.

Step 2. Define alert condition

Davis offers three powerful anomaly detection methods, each suited to different use cases:

• Static threshold – Triggers alerts when a metric crosses a fixed value.
• Auto-adaptive threshold – Learns from historical data to detect unusual behavior.
• Seasonal baseline – Accounts for recurring patterns (for example, daily or weekly cycles).

For our example, let’s use the Static Threshold method. Set the alert to trigger if refused spans exceed a certain limit—say, 100 per minute, to get an alert quite soon when the refusals start. This helps you catch overload situations before they lead to a significant data loss.

Use the Preview feature to visualize how the detector would behave with your current settings.

Step 3. Add alert details

Now, give your alert a clear and descriptive title, such as Refused Spans. Then, define the event message that will be triggered, for example:

Too many spans refused. Collector overloaded.

In the Advanced section, you can add more context or instructions for your team, making the alert even more actionable.

Once everything looks good, select Save. Your anomaly detector is now ready to go!

Configure alerts with auto-adaptive thresholds

In real-world environments, not all collectors behave the same. For example, memory usage can vary significantly between instances depending on their workload. A static threshold might catch extreme cases (maximum usage across all collectors), but it won’t detect sudden spikes in memory usage on a single collector—spikes that could signal performance degradation in an early phase.

This is where the Auto-Adaptive Analyzer becomes invaluable. Instead of applying a one-size-fits-all threshold, it dynamically creates a baseline for each individual collector, using historical data from the past seven days. These baselines are updated daily, allowing the system to adapt to evolving usage patterns and detect anomalies more precisely.

To set this up:

1. Create a new Davis anomaly detector with the following scope:

   
   timeseries {avg(otelcol_process_memory_rss)}, by:{service.instance.id}
   

   This configuration monitors the total physical memory (resident set size) per Collector instance.

2. Select the auto-adaptive threshold. This will automatically generate a tailored threshold for each collector.
3. Select Preview to visualize how the adaptive thresholds behave across all your collectors.

Monitor alerts and take action

Once your anomaly detectors are in place, head over to the Problems app in Dynatrace. Here, you can:

• Track triggered alerts in real time.
• Configure notifications to ensure the right teams are informed immediately—via email, Slack, or your preferred incident management tool.

This proactive alerting system helps you stay ahead of issues, reducing downtime and improving system reliability.

Try them out

Ready to explore the full potential of OpenTelemetry collector self-monitoring?

• Allow internal telemetry export from your collectors to Dynatrace.
• Install OpenTelemetry Dashboards, experiment with different metrics and anomaly detection strategies.
• Consult our documentation for step-by-step guidance, example configurations, and best practices.

We’d love to hear your feedback as you explore these features!

For more insights into OpenTelemetry Collector scaling and resiliency, check out our scaling guide.

The post Introducing OpenTelemetry collector self-monitoring dashboards appeared first on Dynatrace news.

In today’s rapidly evolving cloud-native landscape, managing observability efficiently can be a game-changer for any platform. During our recent tech talk at #KCDAustria, Observability as Code – DIY with Crossplane, we demonstrated how the Upjet project can streamline the creation of a custom Crossplane Dynatrace provider, unlocking robust observability capabilities in Kubernetes environments. This blog post covers what we did and how we did it.

In this blog post, we will deploy a monitoring dashboard with alerts and notifications as one simple Kubernetes resource. To help us achieve our goal, we have to set up a Kubernetes cluster and infrastructure components. But before that, we need to discuss some important concepts and patterns.

Configuration as Code

Managing vast amounts of configurations for organizational setups at scale is a hard problem to solve since they span over many tools and providers and have many different contributors.

However, there is a pattern for remedying many of these problems: treating these configurations as declarative code instead of applying changes manually.

Originally dubbed Infrastructure as Code, the pattern can be generalized and used for anything that provides a proper interface—simply put, Configuration as Code.

While many tools and their respective approaches exist to write and apply such configuration, one of the most interesting recent developments is extending Kubernetes and using its readily available REST API and reconciliation loops.

The operator pattern in Kubernetes

The mechanism that Kubernetes provides for interface extension is called the operator pattern. This is a powerful mechanism for automating the management of complex applications. It extends the Kubernetes API via custom resource definitions (CRDs), enabling new object types to be created. These are constantly watched by custom controllers, so-called operators.

Operators are designed with a “reconciliation loop,” meaning they continuously compare a resource’s real state against its desired state. When a resource deviates, the operator brings it back into alignment. This is the essence of Kubernetes automation and declarative infrastructure.

Crossplane and the operator pattern

Crossplane builds on the operator pattern and extends Kubernetes beyond managing containerized workloads. Through the Kubernetes API, it enables you to define and provision—among other things—cloud infrastructure resources such as databases, compute instances, and networking components.

Crossplane providers implement the operator pattern for external systems (for example, AWS, GCP, Azure). When you install one, it installs its external resources as Kubernetes native custom resource definitions (CRDs). The provider’s controller watches for changes in the desired state of objects—instanced from these CRDs, reconciling them to ensure they match your expectations.

Compositions

In Kubernetes, low-level resources are managed by high-level resources. Crossplane also allows you to build high-level resources using the Composition pattern.

An example would be an Application resource that abstracts away details like database, network, and compute needs. A Crossplane composition enables you to build something like this, giving you control over the new interface (also a Kubernetes CRD) and the implementation (which low-level resources are created and how they are created).

Using the Upjet project

Now that we have an overview of the concepts, let’s look at how we implement our demo. When building a custom Crossplane provider, we can take two approaches: build the provider from scratch or leverage an existing tool. We opted for the latter, by using the Upjet project, which automates the creation of Crossplane providers based on existing Terraform providers. Here’s why:

1. Speed and simplicity: Writing a provider from scratch requires a deep understanding of the external system API and how Crossplane manages resources. Upjet allows us to generate a provider much faster by transforming Terraform provider schemas into Crossplane CRDs, significantly reducing the development time.
2. Reuse of Terraform providers: Upjet allows us to tap into the vast ecosystem of Terraform providers. Since there are already many well-established Terraform providers for various cloud platforms and services, using Upjet means we don’t have to reinvent the wheel.

Generating a new Crossplane provider with Upjet

Creating a new Crossplane provider using Upjet is a streamlined process that allows you to extend Crossplane’s capabilities with minimal setup. Follow the official Upjet documentation to get started.

In our demonstration at the KCD Austria tech talk, we showcased how to build a Dynatrace provider. Below are the detailed steps we followed.

Step 1: Adjust the Makefile

The Makefile needs to reference the official Dynatrace Terraform module. This adjustment allows Upjet to use the correct source when generating the provider.

Here’s an example configuration:

export TERRAFORM_PROVIDER_SOURCE ?= dynatrace-oss/dynatrace
export TERRAFORM_PROVIDER_REPO ?= https://github.com/dynatrace-oss/terraform-provider-dynatrace
export TERRAFORM_PROVIDER_VERSION ?= 1.66.0
export TERRAFORM_PROVIDER_DOWNLOAD_NAME ?= terraform-provider-dynatrace
export TERRAFORM_PROVIDER_DOWNLOAD_URL_PREFIX ?= https://releases.hashicorp.com/$(TERRAFORM_PROVIDER_DOWNLOAD_NAME)/$(TERRAFORM_PROVIDER_VERSION)
export TERRAFORM_NATIVE_PROVIDER_BINARY ?= terraform-provider-dynatrace_v1.66.0
export TERRAFORM_DOCS_PATH ?= docs/resources

These settings specify the source, version, and download paths for the Dynatrace Terraform provider that Upjet will wrap as a Crossplane provider.

Step 2: Configure Provider Resources

Set Up the Provider Config

To configure the connection details, we need to modify internal/clients/dynatrace.go to reference the secret structure expected for the provider. In this case, define tenantURL and apiToken for Dynatrace connectivity:

const (
   tenantURL = "dt_env_url"
   apiToken  = "dt_api_token"
)

Then, reference these credentials in the TerraformSetupBuilder:

// TerraformSetupBuilder builds a Terraform setup function, returning provider configuration.
func TerraformSetupBuilder(version, providerSource, providerVersion string) terraform.SetupFn {
    return func(ctx context.Context, client client.Client, mg resource.Managed) (terraform.Setup, error) {
        ...
        // Set credentials in the provider configuration.
        ps.Configuration = map[string]any{}
        if v, ok := creds[tenantURL]; ok {
            ps.Configuration[tenantURL] = v
        }
        if v, ok := creds[apiToken]; ok {
            ps.Configuration[apiToken] = v
        }
    }
}

Define External Name Configurations

To identify external names for resources, update config/external_name.go by adding mappings for the Dynatrace resources:

// ExternalNameConfigs contains all external name configurations for this provider.
var ExternalNameConfigs = map[string]config.ExternalName{
    "dynatrace_alerting":           config.IdentifierFromProvider,
    "dynatrace_email_notification": config.IdentifierFromProvider,
    "dynatrace_json_dashboard":     config.IdentifierFromProvider,
    "dynatrace_metric_events":      config.IdentifierFromProvider,
}

This setup ensures that each resource is correctly identified using the provider’s unique identifier.

Add Custom Configurations for Resources

For each resource, create a corresponding config subfolder and add a config.go file with a Configure function. This function customizes the resource’s configuration and short group name as needed:

config/alerting/config.go

package alerting
import "github.com/crossplane/upjet/pkg/config"
// Configure customizes individual resources.
func Configure(p *config.Provider) {
  p.AddResourceConfigurator("dynatrace_alerting", func(r *config.Resource) {
    r.ShortGroup = "alerting"
  })
}

Repeat this for other resources, such as Dashboard, Event, and Notification.

config/dashboard/config.go

package dashboard
import "github.com/crossplane/upjet/pkg/config"
// Configure customizes individual resources.
func Configure(p *config.Provider) {
  p.AddResourceConfigurator("dynatrace_json_dashboard", func(r *config.Resource) {
    r.ShortGroup = "dashboard"
  })
}

config/event/config.go

package event
import "github.com/crossplane/upjet/pkg/config"
// Configure customizes individual resources.
func Configure(p *config.Provider) {
  p.AddResourceConfigurator("dynatrace_metric_events", func(r *config.Resource) {
    r.ShortGroup = "event"
  })
}

config/notification/config.go

package notification
import "github.com/crossplane/upjet/pkg/config"
// Configure customizes individual resources.
func Configure(p *config.Provider) {
  p.AddResourceConfigurator("dynatrace_email_notification", func(r *config.Resource) {
    r.ShortGroup = "notification"
  })
}

Register Custom Configurations

To ensure these custom configurations are applied, register each Configure function in config/provider.go:

import (
    "github.com/xoanmi/provider-dynatrace/config/alerting"
    "github.com/xoanmi/provider-dynatrace/config/event"
    "github.com/xoanmi/provider-dynatrace/config/dashboard"
    "github.com/xoanmi/provider-dynatrace/config/notification"
)
for _, configure := range []func(provider *ujconfig.Provider){
    alerting.Configure,
    event.Configure,
    notification.Configure,
    dashboard.Configure,
} {
    configure(pc)
}

This setup allows Upjet to apply each resource’s configuration during the provider generation process, customizing each resource group as specified.

Step 3: Generate the Code

Once all the necessary configurations are in place, you’re ready to generate the provider code by running the following command:

The make generate command will use the settings specified in the previous steps to:

• Generate the Crossplane provider code based on the Terraform provider configurations.
• Create the necessary Kubernetes Custom Resource Definitions (CRDs) for each resource, allowing Crossplane to manage them.

Running make generate will produce output similar to the following, showing the installation of required tools and the generation of the provider schema and resource CRDs:

➜ make generate
11:38:21 [ .. ] installing terraform darwin-arm64
…
11:38:22 [ OK ] installing terraform darwin-arm64
11:38:22 [ .. ] generating provider schema for dynatrace-oss/dynatrace 1.66.0
11:38:24 [ OK ] generating provider schema for dynatrace-oss/dynatrace 1.66.0
11:38:26 [ .. ] go generate linux_arm64

Generated 4 resources!
11:38:59 [ OK ] go generate linux_arm64
11:38:59 [ .. ] go mod tidy
11:39:00 [ OK ] go mod tidy

➜ tree package/crds
package/crds
├── alerting.crossplane.io_alertings.yaml
├── dashboard.crossplane.io_dashboards.yaml
├── dynatrace.crossplane.io_providerconfigs.yaml
├── dynatrace.crossplane.io_providerconfigusages.yaml
├── dynatrace.crossplane.io_storeconfigs.yaml
├── event.crossplane.io_events.yaml
└── notification.crossplane.io_notifications.yaml

With the generated code and CRDs in place, you can now deploy the provider and begin managing Dynatrace resources in your Kubernetes environment.

Step 4: Deploy and run

In the setup, we’re going to run the operator locally while applying and connecting to a Kubernetes cluster (where this cluster runs doesn’t matter, as long as it’s reachable).

First, we apply the CRDs make generate has created.

kubectl apply -f package/crds

Once this is done, you can run the operator itself.

make run

The last missing part is adding the credentials the operator needs to connect to the chosen Dynatrace tenant. The necessary token is in the Access management documentation.

Create the namespace and a secret containing your brand-new access token.

apiVersion: v1
kind: Secret
metadata:
  name: example-creds
  namespace: crossplane-system
type: Opaque
stringData:
  credentials: |
    {
     "dt_env_url": "https://my-tenant.com",
     "dt_api_token": "my-secret-token"
    }

kubectl create namespace crossplane-system
kubectl apply -f example-creds.yaml

Finally, the last puzzle piece, the ProviderConfig, can be created:

apiVersion: dynatrace.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
  namespace: crossplane-system
spec:
  credentials:
    source: Secret
    secretRef:
      name: example-creds
      namespace: crossplane-system
      key: credentials

All done! The previously generated CRDs are now available and their objects in your Kubernetes cluster will result in entities and changes in your Dynatrace tenant. Try it out with a dashboard resource!

apiVersion: dashboard.crossplane.io/v1alpha1
kind: Dashboard
metadata:
  name: example-dashboard
  namespace: crossplane-system
spec:
  forProvider:
    contents: |
      {
        "dashboardMetadata": {
          "name": "Our small example dashboard",
          "owner": "my@mail.com",
          "preset": true,
          "hasConsistentColors": true
      },
      "tiles": [
        {
          More config…
        }
     }

kubectl apply -f example-dashboard.yaml

FAQ

Several key questions were raised during the talk and in the following discussions. We’ve summarized the main points:

Q: Why use Crossplane when Terraform can do the same thing?

A: It’s not a matter of “should” versus “shouldn’t.” If you’re already invested in Kubernetes, Crossplane allows you to manage cloud resources while leveraging the same tooling you use to deploy, maintain, and monitor your applications. This makes Crossplane highly convenient for teams already embedded in the Kubernetes ecosystem.

Additionally, these tools don’t exclude each other. One is used to build platforms, and the other is a command-line tool. Their potential use cases differ quite a lot.

Q: How is state management handled?

A: With the Upjet approach, you’re essentially bridging two worlds. Kubernetes manages the state of each object through its etcd system. Simultaneously, the Crossplane operator runs Terraform in the background, continuously reconciling the state between the Kubernetes Custom Resource (CR) and the Terraform-managed infrastructure.

Q: Can I use the Upjet approach in production?

A: Yes, you can, but remember that the provider uses Terraform under the hood. This means that during each reconciliation loop, a terraform plan and terraform apply run. Due to the nature of these continuous operations, managing a large number of resources this way could demand significant resources.

While the Upject project is very good at translating the provider, some things need to be added manually. The concept of Kubernetes labels simply doesn’t exist in Terraform. If you want to utilize them, you need to implement them yourself.

Q: How does the mapping between Terraform objects and Kubernetes Custom Resources (CRs) work?

A: The mapping is defined in the `/config` folder when configuring the provider. Here, we specify the relationship between the Terraform object and the corresponding Kubernetes CR. Running the `make generate` command triggers the generation of all necessary code, including the API, client, provider, and CRD (Custom Resource Definition). This allows Kubernetes to manage the Terraform-defined resource seamlessly.

Q: I read about the proposal for Crossplane v2.0. Do you know if that will impact the described provider creation process?

A: Recently, the Crossplane developers created a draft for the next version of Crossplane. Here, they talk in-depth about how they want to change composite resources and their structure. This will impact provider creation since they reconcile aforementioned resources. This section discusses the proposed changes. The developers also plan on keeping things backward compatible. For now, we
have to wait and see what the final implementation looks like.

Get started

Crossplane enables a seamless cloud-native approach for managing any cloud resource by extending the Kubernetes API. By leveraging Kubernetes as a control plane and using Crossplane compositions, you can declaratively define and automate your entire observability stack.

It’s easy to get started, all you need to start is

• A terraform provider
• The Upjet project

If you’re interested in diving deeper, you can check out the following resources from our session:

• The recording
• GitHub Repo

We hope this talk inspired you to explore Crossplane for your infrastructure automation needs and provided valuable insights into building observability solutions using the power of Kubernetes and declarative infrastructure.

The post Observability as Code: DIY with Crossplane appeared first on Dynatrace news.

One of the things I love most about OpenTelemetry (OTel) is that it’s vendor-neutral, which means you can send the same OpenTelemetry data to different vendors. In fact, most of the major Observability vendors out there not only support ingesting OpenTelemetry data but also actively contribute to the project, including Dynatrace. Check out the 2023 OpenTelemetry Journey Report for more info.

Why does this matter? I used to work at another Observability vendor, and many of the OpenTelemetry examples that I played with and blogged about in the last 2 years or so featured sending OTel data to that backend.

Now that I work at Dynatrace, which, by the way, ingests OTLP natively, I wanted to educate myself on how to send OpenTelemetry data to Dynatrace. A great way to learn is to try to run my go-to examples using Dynatrace as the Observability backend. Luckily for me, since OTel is vendor-neutral, all I had to do was reconfigure my OTel Collector to point to Dynatrace to get my examples to work.

Want to learn how? Let’s do it together!

Note: If you’re evaluating multiple vendors, you can send the same data to different vendors at the same time (à la “vendor bake-off”) to help you determine which vendor best suits your organization’s needs.

Prerequisites for sending data to Dynatrace

To send OpenTelemetry data to Dynatrace, you need two pieces of information:

Dynatrace tenant: Each user (or, more likely, organization) is assigned a tenant. When sending OpenTelemetry data from your application to Dynatrace, you need to specify the Dynatrace OTLP endpoint (used by Dynatrace to receive data), which includes your tenant name.

Access token: The access token allows you to send OTel data to your Dynatrace instance. It also specifies what kind of data you’re allowed to send to Dynatrace. You can find more on Dynatrace access tokens here.

Before we get to any of that, you first need a Dynatrace account. If you already have a Dynatrace account, feel free to skip the following section.

Create a Dynatrace account

If you don’t have a Dynatrace account, you can create a free trial account, which is valid for 15 days.

1. Go here, and select the Free trial button.
   

   

2. Enter your email, select the Terms of Use checkbox, and then select Continue.

   

3. Enter the rest of the info and select Start free trial.
   

   

   You will receive an email once your account has been created. You will also see a page that looks like the one below. Select Launch Dynatrace to get started.

   

   This takes you to the Dynatrace login page.

   

Your Dynatrace tenant

To find your Dynatrace tenant, log into Dynatrace here, and select the Login button at the top right of the page.

This takes you to the sign-in page. Once you sign in, take note of the URL. It should look something like this:

https://<your-dynatrace-tenant>.apps.dynatrace.com

Take note of the value of <your-dynatrace-tenant>, because we’ll need that later.

Create a Dynatrace access token

After confirming that you’re logged into Dynatrace, press ctrl+k, and then type access token. Next, select Access Tokens from the top of the search results.

On the Access tokens page, select Generate new token.

On the Generate new token page, enter:

• Token name: be sure to give it a descriptive name
• Expiration date: this is optional
• Template: Kubernetes Data Ingest

Even if we’re not necessarily using Kubernetes, the Kubernetes Data Ingest template has the token scopes (permissions) that we need to send OpenTelemetry data to Dynatrace, namely:

• Ingest logs (ingest)
• Ingest metrics (ingest)
• Ingest OpenTelemetry traces (ingest)

Find more information on these and other Dynatrace token scopes here.

Once you’re done, select Generate token at the bottom of the page.

The next page shows your access token. Be sure to copy and store it somewhere for safekeeping (for example, a secrets manager such as HashiCorp Vault or your cloud provider’s secrets manager) before selecting Done, because after that, it’s gone forever. If you lose that token information, you should delete the old one (not necessary, but highly recommended), and create a new one.

Configure the OTel Collector for Dynatrace

Now that you have your tenant info and access token, you can plug this information into your OpenTelemetry Collector configuration.

Note: There are two ways to send OTel data to an Observability backend: (1) direct from the application, or (2) via the OTel Collector. There’s a time and place for each, and you can check out my blog post on OTel Collector Anti-patterns on the OTel Blog to learn more.

Your OTel Collector config YAML file should look something like this:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  cumulativetodelta:
  batch:

exporters:
  otlphttp:
    endpoint: "https://${DT_TENANT}.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token ${DT_API_TOKEN}"
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp,debug]
    metrics:
      receivers: [otlp]
      processors: [cumulativetodelta,batch]
      exporters: [otlphttp,debug]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp,debug]

Dynatrace accepts data in the native OpenTelemetry Protocol (OTLP) format via HTTP (gRPC is not yet supported). You need to specify the Dynatrace OTLP endpoint (used by Dynatrace to receive data), which includes your tenant name, ${DT_TENANT}:

https://${DT_TENANT}.live.dynatrace.com/api/v2/otlp

${DT_TENANT} is the value of your Dynatrace tenant name, which you hopefully jotted down and stored in a secrets manager for safekeeping.

Finally, “Dynatrace requires metrics data to be sent with delta temporality, not cumulative temporality”. This means that you’ll need to include the cumulativetodelta processor in:

• Your Collector configuration (cumulativetodelta)
• Your metrics pipeline (pipelines.metrics)

Never store your Dynatrace token and tenant name in plain text. Instead, store them in a secrets manager and pull them from the secrets manager at runtime.

Dynatrace and the OTel Operator

If you’re using the OpenTelemetry Operator to send OpenTelemetry data to Dynatrace, you’ll need to configure your OpenTelemetryCollector resource as follows:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otelcol
  namespace: opentelemetry
spec:
  mode: statefulset
  image: ghcr.io/dynatrace/dynatrace-otel-collector/dynatrace-otel-collector:0.7.0
  env:
    - name: DT_API_TOKEN
      valueFrom:
        secretKeyRef:
          key: DT_API_TOKEN
          name: otel-collector-secret
    - name: DT_TENANT
      valueFrom:
        secretKeyRef:
          key: DT_TENANT
          name: otel-collector-secret
  config:
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}

    processors:
      cumulativetodelta: {}
      batch: {}

    exporters:
      otlphttp:
        endpoint: "https://${DT_TENANT}.live.dynatrace.com/api/v2/otlp"
        headers:
          Authorization: "Api-Token ${DT_API_TOKEN}"
      debug:
        verbosity: detailed

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp,debug]
        metrics:
          receivers: [otlp]
          processors: [cumulativetodelta,batch]
          exporters: [otlphttp,debug]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp,debug]

Notice that the spec.config looks the same as what we defined in the otelcol-config.yaml file we saw earlier. The only added thing here is that DT_API_TOKEN and DT_TENANT are environment variables pulled from a Kubernetes Secret. The secret YAML definition looks like this:

apiVersion: v1
kind: Secret
metadata:
 name: otel-collector-secret
 namespace: 
data:
 DT_API_TOKEN: 
 DT_TENANT: 
type: "Opaque"

Both DT_API_TOKEN and DT_TENANT values must be base64 encoded before being added to the secrets YAML. To base64 encode a value and copy the encoded value to your buffer, use this action for easy copy/paste:

echo <value_to_encode> | base64

Remember that storing secrets in Kubernetes (or storing a secrets YAML in version control, for that matter), is not recommended because base64 does not encrypt your data. You should instead consider using sealed secrets or the Kubernetes Secrets Store CSI Driver + your favorite secrets provider. For more info on these better alternatives check out this article.

The Dynatrace OTel Collector Distribution

Many vendors have their own OTel Collector Distributions. These distributions are curated with Collector components that are specific to that vendor. They can be a combination of vendor-developed custom components and components from Collector Core and Contrib. Using vendor-specific distributions ensures you’re using just the Collector components you need, reducing overall bloat. You can learn more here.

Dynatrace also has its own Collector distribution. It features a set of Collector components for sending Observability data to Dynatrace from various sources. It stays up-to-date with upstream components of the opentelemetry-collector and opentelemetry-collector-contrib repositories.

In addition, the Dynatrace Collector distribution offers the following advantages:

• It is covered by Dynatrace support
• Collector components are verified by Dynatrace
• Security patches are independent of OpenTelemetry Collector releases

Try it out!

Want to try sending data to Dynatrace yourself? Then check out my example repo. I created this repo for a talk on the Target Allocator that I gave at KubeCon in March 2024. It has been updated to include instructions on configuring the OpenTelemetryCollector resource to send OTel data to Dynatrace.

OTel Data in Dynatrace

And if you’re curious to see what OTel data looks like in Dynatrace, here are some screenshots of the web UI.

I’m not going in-depth on how to navigate the Dynatrace UI because there are already some great videos on the Dynatrace YouTube channel. I encourage you to check them out for a more in-depth look.

Final thoughts

As someone with experience sending OpenTelemetry data to various backends, I found that getting OpenTelemetry data into Dynatrace was fairly straightforward. My only personal hiccup was in generating the application token, but I got that sorted out, and now I’ve passed on my knowledge and highly detailed screenshots along to you.

I have to say that it’s always fun to use a product with fresh eyes, a fresh perspective, and a newbie point of view. There’s nothing quite like it. And, having worked at another observability vendor before, it’s always fun to see the similarities and differences. It’s like learning a new programming language and comparing it to another one that you already know. What a blast!

One final point that I want to make. I don’t want to trivialize things and give you the impression that moving from one observability vendor to another is simply a matter of repointing your OTel Collector from one vendor backend to another. That is only one aspect of a vendor migration, no matter what vendor you’re moving to/from. You also must consider the fact that you’ll likely have a bunch of dashboards, alerts, and whatnot that you created with your original vendor. When you migrate to another vendor, there won’t be a 1:1 translation; so keep that in mind.

But that may be a sacrifice that you’re willing to make because OTel’s vendor neutrality means that all vendors supporting OpenTelemetry are ingesting the same data. What sets them apart is what they do with your data. And if one vendor does something with your data better than another, well, don’t you owe it to yourself to check that out?

What’s Next?

If you’re interested in learning first-hand what Dynatrace and OpenTelemetry can do together, then dive into Dynatrace! You can do so in one of two ways:

• Check out the Dynatrace Playground to explore Dynatrace using pre-populated OpenTelemetry data
• Get started with a free trial and ingest your own OpenTelemetry data today!

The post Let’s learn how to send OpenTelemetry data to Dynatrace together! appeared first on Dynatrace news.

This blog post was co-written with Reese Lee.

If you already have an application running in Kubernetes and are exploring using OpenTelemetry to gain insights into the health and performance of your app and cluster, you might be interested in an implementation of the Kubernetes Operator called the OpenTelemetry Operator.

As you’ll learn shortly, due to its range of capabilities, the Operator is your go-to for (almost) hassle-free OpenTelemetry management. But, as with any powerful tool, what happens when things go sideways?

In this blog post, you’ll learn about the OpenTelemetry Operator (hereafter referred to as “the Operator”), along with issues commonly encountered across installation, Collector deployment, and auto-instrumentation. You’ll also learn how to resolve these issues, and be better prepared for running the Operator.

Overview of the Operator

Let’s take a closer look at the Operator’s main capabilities.

Managing the Collector

The Operator automates the deployment of your Collector, and makes sure it’s correctly configured and running smoothly within your cluster. The Operator also manages configurations across a fleet of Collectors using Open Agent Management Protocol (OpAMP), which is a network protocol for remotely managing large fleets of data collection agents. Since the protocol is vendor-agnostic, this helps ensure consistent observability settings and simplifies management across agents from different vendors.

Managing Auto-Instrumentation in Pods

The Operator automatically injects and configures auto-instrumentation for your applications, which enables you to collect telemetry data without modifying your source code. If your application isn’t already instrumented with OpenTelemetry, this is a fantastic option to feed two birds with one scone, and start generating and collecting application telemetry.

Installing the Operator

This might seem obvious, but before installing the Operator, you must have a Kubernetes cluster you can install it into, running Kubernetes 1.23+. Check the compatibility matrix for specific version requirements. You can spin up a cluster on your machine using a local Kubernetes tool such as minikube, k0s, or KinD, or use a cluster running on a cloud provider service.

Next, and this is less obvious: You must have a component called cert-manager already installed in that cluster. This piece manages certificates for Kubernetes by making sure the certificates are valid and up to date. You can install both the cert-manager and the Operator via kubectl or a Helm chart.

Note that in either case, you have to wait for cert-manager to finish installing before you install the Operator; otherwise, the operator installation will fail.

Using kubectl

To install cert-manager, run the following command:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml

Next, install the Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Using Helm

To install cert-manager, first add the Helm repository:

helm repo add jetstack https://charts.jetstack.io --force-update

Next, install the cert-manager Helm chart:

helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.16.1 \
--set crds.enabled=true

Expect the preceding step to take up to a few minutes. You can verify your installation of cert-manager by following the steps in this link, or check the deployment status by running:

kubectl get pods -namespace cert-manager

To install the Operator, note that Helm 3.9+ is required. First, add the repo:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts 
helm repo update

Then, install the Operator:

helm install --namespace opentelemetry-operator-system \  
  --create namespace \  
  opentelemetry-operatoropen-telemetry/opentelemetry-operator 

Deploying the OpenTelemetry Collector

Once you have cert-manager and Operator set up in your cluster, you can deploy the Collector. The Collector is a versatile component that’s able to ingest telemetry from a variety of sources, transform the received telemetry in a number of ways based on its configuration, and then export that processed data to any backend that accepts the OpenTelemetry data format (also referred to as OTLP, which stands for OpenTelemetry Protocol).

The Collector can be deployed in several different ways, referred to as “patterns.” Which pattern or patterns you deploy is dependent on your telemetry needs and organizational resources. This topic is out of scope for this blog post, but you can read more about them via this link.

Collector Custom Resource

A custom resource (CR) represents a customization of a specific Kubernetes installation that isn’t necessarily available in a default Kubernetes installation; CRs help make Kubernetes more modular.

The Operator has a CR for managing the deployment of the Collector, called OpenTelemetryCollector. The following is a sample OpenTelemetryCollector resource:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otelcol
  namespace: opentelemetry
spec:
  mode: statefulset
  config:
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}
      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 10s
              static_configs:
              - targets: [ '0.0.0.0:8888' ]

    processors:
      batch: {}

    exporters:
      logging:
        verbosity: detailed

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging]
        metrics:
          receivers: [otlp, prometheus]
          processors: []
          exporters: [logging]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging]

There are many configuration options for the OpenTelemetryCollector resource, depending on how you plan on instantiating it; however, the basic configuration requires:

• mode, which should be one of the following: deployment, sidecar, daemonset, or statefulset. If you leave out mode, it defaults to deployment.
• config, which may look familiar, because it’s the Collector’s YAML config.

Common Collector deployment issues and troubleshooting tips

If you’re not seeing the data you expect, or you suspect something isn’t working right, try the following troubleshooting tips.

Check that the Collector resources deployed properly

When an OpenTelemetryCollector YAML is deployed, the following objects are created in Kubernetes:

1. OpenTelemetryCollector

2. Collector pod:

• If you specified non-sidecar mode, look for Deployment, StatefulSet, or DaemonSet resources named <collector_CR_name>-collector-<unique_identifier>).
• If you specified the mode as sidecar, a Collector sidecar container will be created in an app pod, named otc-container.

3. Target Allocator pod:

• If you enabled the Target Allocator, look for a resource named <collector_CR_name>-targetallocator-<unique_identifier>.

4. ConfigMap of Collector configurations:

• If you specified non-sidecar mode, look for Deployment, StatefulSet, or DaemonSet resources named <collector_CR_name>-collector-<unique_identifier>.
• If you specified the mode as sidecar, note that the Collector config is included as an environment variable.

Thus, when you deploy the OpenTelemetryCollector resource, make sure that the preceding objects are created.

First, confirm that the OpenTelemetryCollector resource was deployed:

kubectl get otelcol -n <namespace>

When you deploy the Collector using the OpenTelemetryCollector resource, it creates a ConfigMap containing the Collector’s configuration YAML. Confirm that the ConfigMap was created in the same namespace as the Collector, and that the configurations themselves are correct.

List your ConfigMaps:

kubectl get configmap -n <namespace> | grep <collector-cr-name>-collector

We also recommend checking your Collector pods by running the appropriate command based on the Collector’s mode:

• deployment, statefulset, daemonset modes:

kubectl get pods -n <namespace> | grep <collector_cr_name>-collector

• sidecar mode:

kubectl get pods <pod_name> -n opentelemetry -o jsonpath='{.spec.containers[*].name}'

This will list all the containers created in the pod, including the Collector sidecar container, which includes the Collector config as an environment variable.

Check the Collector CR version

Take a look at the OpenTelemetryCollector CR version you’re using. There are two versions available: v1alpha1:

apiVersion: opentelemetry.io/v1alpha1 
kind: OpenTelemetryCollector 
metadata: 
  name: otelcol 
  namespace: opentelemetry 
spec: 
  mode: statefulset 
  config: | 
    receivers: 
      otlp: 
        protocols: 
          grpc: 
          http: 
 
    processors: 
      batch: 
 
    exporters: 
      otlp: 
        endpoint: "<my_o11y_backend>" 
      logging: 
        verbosity: detailed 
 
    service: 
      pipelines: 
        traces: 
          receivers: [otlp] 
          processors: [batch] 
          exporters: [otlp/ls, logging] 
        metrics: 
          receivers: [otlp, prometheus] 
          processors: 
          exporters: [otlp/ls, logging] 
        logs: 
          receivers: [otlp] 
          processors: [batch] 
          exporters: [otlp/ls, logging] 

and v1beta1:

apiVersion: opentelemetry.io/v1beta1 
kind: OpenTelemetryCollector 
metadata: 
  name: otelcol 
  namespace: opentelemetry 
spec: 
  mode: statefulset 
  config: 
    receivers: 
      otlp: 
        protocols: 
          grpc: {} 
          http: {} 
 
    processors: 
      batch: {} 
 
    exporters: 
      otlp: 
        endpoint: "<my_o11y_backend>" 
      logging: 
        verbosity: detailed 
 
    service: 
      pipelines: 
        traces: 
          receivers: [otlp] 
          processors: [batch] 
          exporters: [otlp/ls, logging] 
        metrics: 
          receivers: [otlp, prometheus] 
          processors: [] 
          exporters: [otlp/ls, logging] 
        logs: 
          receivers: [otlp] 
          processors: [batch] 
          exporters: [otlp/ls, logging] 

There are two main differences between these two API versions:

1. The config sections are different; for v1beta1, the config values are key-value pairs that are part of the CR configuration, whereas for v1alpha1, the config value is one long text string. Keep in mind that the text string still needs to follow YAML formatting.

2. If you’re using v1beta1, you can’t leave the Collector config values empty. You must specify either empty curly braces ({}) for scalar values or empty brackets ([ ]) for arrays. This isn’t necessary if you’re using v1alpha1.

Check the Collector base image

By default, the OpenTelemetryCollector CR uses the core distribution of the Collector. The core distribution is a bare-bones distribution of the Collector for OpenTelemetry developers to develop and test. It contains a base set of components: extensions, connectors, receivers, processors, and exporters.

If you want access to more components than the ones offered by core, you can use the Collector’s Kubernetes distribution instead. This distribution is made specifically to be used in a Kubernetes cluster to monitor Kubernetes and services running in Kubernetes. It contains a subset of components from the core and contrib distributions. Alternatively, you can build your own Collector distribution.

You can set the Collector’s base image by specifying the image attribute in spec.image, as in the following example:

apiVersion: opentelemetry.io/v1beta1 
kind: OpenTelemetryCollector  
metadata: 
  name: otelcol 
  namespace: opentelemetry 
spec: 
  mode: statefulset  
  image: otel/opentelemetry-collector/contrib:0.102.1 
config:  
  receivers:  
    otlp: 
      protocols:  
      grpc: {} 
       http: {} 
  processors:  
    batch: {} 
  exporters:  
    otlp: 
      endpoint: "<olly_backend_endpoint>"

Check your backend vendor’s access requirements

If you’re using a backend vendor to ingest your telemetry data, you’ll likely need to configure an account license key or some kind of access token, which you’ll want to keep confidential.

To store it as a secret and prevent it from appearing as plain text, first create a Kubernetes secret, and Base64-encode it:

apiVersion: v1  
kind: Secret  
metadata: 
  name: otel-collector-secret  
  namespace: opentelemetry 
data: 
  ACCESS_TOKEN: <base64_encoded_token> 
type: "Opaque"

Check your exporter configuration

Confirm that you’ve configured the correct endpoint according to your region in your exporter configuration.

When all else fails…check Kubernetes events

Kubernetes events provide detailed and chronological information about what’s happening within various components of your cluster. To view events for a specific namespace, use:

kubectl get events -n <namespace>

Replace <namespace> with the actual namespace where your OpenTelemetry Operator and resources are deployed.

Instrumentation

Instrumentation is the process of adding code to software to generate telemetry signals–logs, metrics, and traces. You have several options for instrumenting your code with OpenTelemetry, the primary two being code-based and zero-code solutions.

Code-based solutions require you to manually instrument your code using the OpenTelemetry API. While it can take time and effort to implement, this option enables you to gain deep insights and further enhance your telemetry, as you have a high degree of control over what parts of your code are instrumented and how.

To instrument your code without modifying it (or if you’re unable to modify the source code), you can use zero-code solutions (or auto-instrumentation agents). This method uses shims or bytecode agents to intercept your code at runtime or at compile-time to add tracing and metrics instrumentation to the third-party libraries and frameworks you depend on. At the time of publication, auto-instrumentation is currently available for Java, Python, .NET, JavaScript, PHP, and Go. Learn more about zero-code instrumentation at this link.

You can also use both options simultaneously. Some end users opt to start with a zero-code agent and manually insert additional instrumentation, such as adding custom attributes or creating new spans. Alternatively, OpenTelemetry also provides options beyond code-based and zero-code solutions. Learn more at this link.

Zero-code Instrumentation with the Operator

The Operator has a CR called Instrumentation that can automatically inject and configure OpenTelemetry instrumentation into your Kubernetes pods, providing the benefit of zero-code instrumentation for your application. This is currently available for the following: Apache HTTPD, .NET, Go, Java, nginx, Node.js, and Python.

The following is a sample Instrumentation resource definition for a Python service:

apiVersion: opentelemetry.io/v1alpha1  
kind: Instrumentation  
metadata: 
  name: python-instrumentation  
  namespace: application 
spec: 
  env: 
    - name: OTEL_EXPORTER_OTLP_TIMEOUT 
      value: "20" 
    - name: OTEL_TRACES_SAMPLER 
      value: parentbased_traceidratio 
    - name: OTEL_TRACES_SAMPLER_ARG 
      value: "0.85" 
  exporter: 
    endpoint: http://localhost:4317 
  propagators: 
    - tracecontext 
    - baggage  
  sampler: 
    type: parentbased_traceidratio  
    value: "0.25" 
  python:  
    env: 
      - name: OTEL_METRICS_EXPORTER 
        value: otlp_proto_http 
      - name: OTEL_LOGS_EXPORTER 
        value: otlp_proto_http 
      - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED 
        value: "true" 
      - name: OTEL_EXPORTER_OTLP_ENDPOINT 
        value: http://localhost: 4318 

You can use a single auto-instrumentation YAML to serve multiple services written in different languages (provided they are supported for auto-instrumentation). List your global environment variables under spec.env, and list language-specific environment variables under spec.<language_name>.env. You can mix and match language-specific environment variable configurations in the same Instrumentation resource.

In order to use the Operator’s auto-instrumentation capability, deploying an Instrumentation resource alone isn’t enough. The auto-instrumentation configuration must be associated with the code being instrumented. This is done by adding an auto-instrumentation annotation in your application’s Deployment YAML, in the template definition section, such as in the following example:

apiVersion: apps/v1 
kind: Deployment  
metadata: 
  name: my-deployment-with-sidecar  
spec: 
  replicas: 1  
  selector: 
    matchLabels: 
      app: my-pod-with-sidecar 
  template: 
    metadata:  
      labels: 
        app: my-pod-with-sidecar 
      annotations: 
        sidecar.opentelemetry.io/inject: "true" 
        instrumentation.opentelemetry.io/inject-python: "true" 
spec: 
  containers: 
    - name: py-otel-server 
      image: otel-python-lab:0.1.0-py-otel-server ports: 
    - containerPort: 8082 
      name: py-server-port 

When the annotation called instrumentation.opentelemetry.io/inject-python is set to true, it tells the Operator to inject Python auto-instrumentation (in this case) into the containers running in this pod. For other languages, simply replace python with the appropriate language name (for example, instrumentation.opentelemetry.io/inject-javafor Java apps). You can disable instrumentation by setting this value to false.

If you have multiple Instrumentation resources, you need to specify which one to use, otherwise the Operator won’t know which one to pick. You can do this as follows:

• By name. Use this if the Instrumentation resource resides in the same namespaces as the Deployment. For example, opentelemetry.io/inject-java: my-instrumentation will look for an Instrumentation resource called my-instrumentation.
• By namespace and name. Use this if the Instrumentation resource resides in a different namespace. For example: opentelemetry.io/inject-java: my-namespace/my-instrumentation will look for an Instrumentation resource called my-instrumentation in the namespace my-namespace.

You must deploy the Instrumentation resource before the annotated application; otherwise, your code won’t be automatically instrumented. The Operator injects auto-instrumentation by adding an init container to the application’s pod when it starts up, which means that if the Instrumentation resource isn’t available by the time your service is deployed, the auto-instrumentation will fail.

Common instrumentation issues and troubleshooting tips

If your Collector doesn’t seem to be processing data or if you think the auto-instrumentation isn’t working, try the following steps to troubleshoot and resolve the problem.

Check that the instrumentation resource deployed properly

Run the following command to make sure the Instrumentation resource(s) was created in your Kubernetes cluster:

kubectl describe otelinst -n <namespace>

Confirm the resource deployment order

Double check that your Instrumentation CR is deployed before your Deployment. As we learned earlier, if you’re auto-instrumenting via the Operator, you must deploy the Instrumentation resource before deploying your service’s Deployment resource, because the Deployment will create an init-container for the auto-instrumentation. You should therefore see an auto-instrumentation init-container when you run the following command:

kubectl get pod  -n  \ 
  -o jsonpath='{.spec.initContainers[*].name}' 

Check your auto-instrumentation CR annotations

1- Confirm that there are no typos in the annotations.

2- Confirm that they are in the pod’s metadata definition (spec.template.metadata.annotation), not the deployment’s metadata definition (metadata.annotation), as in the following example:

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: py-otel-server 
  namespace: opentelemetry 
  labels: 
    app: my-app 
    app.kubernetes.io/name: py-otel-server 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: my-app 
      app.kubernetes.io/name: py-otel-server 
  template: 
    metadata: 
      labels: 
        app: my-app 
        app.kubernetes.io/name: py-otel-server 
      annotations: 
        instrumentation.opentelemetry.io/inject-python: "true" 
    spec: 
      containers: 
      - name: py-otel-server 
        image: otel-target-allocator-talk:0.1.0-py-otel-server 
        imagePullPolicy: IfNotPresent 
        ports: 
        - containerPort: 8082 
          name: py-server-port 
        env: 
          - name: OTEL_RESOURCE_ATTRIBUTES 
            value: service.name=py-otel-server,service.version=0.1.0 

Check your endpoint configurations

The endpoint, configured in the following example under spec.exporter.endpoint, refers to the destination for your telemetry within your Kubernetes cluster:

apiVersion: opentelemetry.io/v1alpha1 
kind: Instrumentation 
metadata: 
  name: python-instrumentation 
  namespace: opentelemetry 
spec: 
  exporter: 
    endpoint: http://otelcol-collector.opentelemetry.svc.cluster.local:4318 
  env: 
  propagators: 
    - tracecontext 
    - baggage 
  python: 
    env: 
      - name: OTEL_METRICS_EXPORTER 
        value: console,otlp_proto_http 
      - name: OTEL_LOGS_EXPORTER 
        value: otlp_proto_http 
      - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED 
        value: "true" 

The spec.exporter.endpoint configuration in the Instrumentation resource allows you to define the destination for your telemetry data. If you omit it, it defaults to http://localhost:4317.

If you’re sending out your telemetry to a Collector, the value of spec.exporter.endpoint must reference the name of your Collector Service.

Looking at the example above, otel-collector is the name of the OTel Collector Kubernetes Service.

In addition, if the Collector is running in a different namespace, you must append opentelemetry.svc.cluster.localto the Collector’s service name, where opentelemetry is the namespace in which my Collector happens to be deployed to. It can be any namespace of your choosing.

Finally, make sure that you are using the right Collector port. Normally, you can choose either 4317 (gRPC) or 4318(HTTP); however, for Python auto-instrumentation, you can only use 4318. Confirm whether there are similar caveats for the language(s) you’re using.

Note: If you’re deploying your Collector as a Sidecar, your endpoint needs to be  http://localhost:4317 or http://localhost:4318 (remember: it has to be 4318 for Python).

When all else fails…check the Operator logs

Run the following command to check the Operator logs for any occurrences of error in the log messages:

kubectl logs -l app.kubernetes.io/name=opentelemetry-operator \ 
  --container manager \ 
  -n opentelemetry-operator-system --follow 

Note that the above only applies if you have admin access to your Kubernetes cluster. If you don’t, you can still tell what’s going on by checking your Kubernetes event log, just like we did when troubleshooting issues with the OpenTelemetryCollector resource:

kubectl get events -n <namespace>

Summary

The OpenTelemetry Operator manages the deployment and configuration of one or more Collectors, and injects and configures zero-code instrumentation solutions into your Kubernetes pods. This enables you to get started with OpenTelemetry instrumentation, and you can further enhance your telemetry by adding manual instrumentation to your application.

In this blog post, you learned the ins and outs of the Operator, from common installation hurdles to resolving auto-instrumentation and Collector deployment issues. With detailed installation steps and troubleshooting tips, you’re now equipped to leverage the Operator effectively for the deployment, configuration, and management of your Collectors and auto-instrumentation of supported libraries.

This blog post is based on a talk that Adriana and Reese did at KubeCon North America’s 2024 co-located event, Observability Day. You can check out the recording of the talk here:

When Things Go Sideways: Troubleshooting the OTel Operator – Adriana Villela & Reese Lee

The post When things go sideways: Troubleshooting the OpenTelemetry Operator appeared first on Dynatrace news.

Imagine you’re using a lot of OpenTelemetry and Prometheus metrics on a crucial platform. You’re gathering a lot of data, but you can’t make sense of it. You need to visualize the distribution of your measurements to identify patterns, outliers, and trends. But there’s a problem: Your current tools don’t support histograms.

Incorporating histograms is not just a technical upgrade—it’s a necessity for any observability professional. By starting with histograms, you can unlock deeper insights and drive more informed decisions in your projects.

We’re excited to announce that Dynatrace has introduced support for OpenTelemetry histograms in connection with the new visualization options in Dashboards and Notebooks. The histograms are supported starting from Dynatrace version 1.301. OpenTelemetry histograms complement the Distributed Tracing app, which uses histograms as the default visualization tool for response times.

In this blog, we will focus on histograms and why to use them. We will cover their main value and possibilities in OpenTelemetry.

Histograms are commonly used to define and monitor service-level objectives (SLOs). They can help determine the percentage of requests that meet a specific response-time threshold, which is essential for maintaining service quality.

In practice, histograms are useful when the measurement distribution is relevant and the data sets are large. Teams can also change queries to get answers on already-collected data without needing to redefine metrics or wait for new data to accumulate.

Breaking down the benefits of OpenTelemetry histograms

OpenTelemetry instrumentation automatically generates histograms for HTTP client and server request durations. This feature, available by default for OTel-instrumented services, gives users a standard way to consistently measure and compare response times across different services.

Moreover, the OpenTelemetry Collector can measure service span durations, categorized by span names, span kinds, and status codes. The span metrics connector creates these measurements and presents them as histograms, which you can analyze in Dynatrace for deeper insights.

Histograms also enhance the self-monitoring capabilities of the Collector. It reports batch sizes and HTTP/RPC measurements of its own pipelines as histograms, providing valuable metrics for performance monitoring. This self-monitoring aspect is crucial for maintaining the health and efficiency of the Collector itself, ensuring that it can handle the demands of large-scale data collection and processing without degradation.

Additionally, the Collector supports converting Prometheus and StatsD histograms into the OpenTelemetry protocol (OTLP), making them compatible with Dynatrace. By exporting metrics from different sources into a single platform, teams can achieve a holistic view of their system’s performance, facilitating proactive issue resolution and faster decision-making.

Percentiles to simplify analysis

Percentiles are statistical measures that divide a data set into 100 equal parts, providing a way to interpret specific points within your histograms. For instance, the 90th percentile (p90) is the value below which 90% of the data falls.

In practical applications, percentiles are particularly useful for web performance analysis. By examining the p90, you can identify the maximum response time experienced by 90% of users. This insight is crucial for optimizing performance for the majority of users. However, it also highlights that the remaining 10% of users experience longer wait times, which could lead to dissatisfaction.

With the Dynatrace Grail data lakehouse, extracting percentiles from histograms is straightforward, especially when using Notebooks. You can seamlessly integrate percentile graphs into dashboards, providing clear and actionable insights.

Support for explicit and exponential histograms

The first metrics API/SDK release in the OpenTelemetry project introduced histograms with explicit bucket boundaries. These histograms are very popular and are also widely used by Prometheus. Dynatrace now fully supports them.

Later, OpenTelemetry introduced exponential histograms, with each consecutive bucket exponentially larger than the previous one. These histograms are more efficient in carrying a high dynamic range of different values and ensure that the relative error for every bucket remains stable. Dynatrace now supports exponential histograms by calculating histogram summaries (min, max, sum, count). But for now, percentile calculation and buckets are available only for explicit bucket histograms.

Try OpenTelemetry histograms

To experiment with OpenTelemetry histograms, you can deploy the OpenTelemetry Demo Application (Astronomy shop) with the span metrics connector. See this blog about exporting the data from the demo app to Dynatrace.

To learn more about the histograms in Dynatrace, see Histogram Visualization in Dynatrace docs.

For easy analysis of trace data with histograms, check out the new Distributed Tracing app. You can also check out this demo: Transform OpenTelemetry data into actionable insights.

As a leading contributor to the OpenTelemetry project, Dynatrace is committed to advancing its features and maximizing its value. By collaborating with the community and other vendors, Dynatrace ensures that OpenTelemetry remains cutting-edge, accessible, and user-friendly for everyone.

The post OpenTelemetry histograms reveal patterns, outliers, and trends appeared first on Dynatrace news.

With the move towards cloud-native computing, having a clear understanding of your Kubernetes platform’s performance and application behavior is a requirement. Having the ability to troubleshoot quickly and easily is especially important. Logging provides platform and application owners with an effective way to understand the state of their workloads by giving them a view into the inner workings of their systems. Log collection solutions, such as the Dynatrace Kubernetes Log Module or Fluent Bit, give organizations a much-needed solution for quickly gathering and processing log data to make it available in different backends for further analytics. This is especially necessary within complex cloud-native—and, more specifically, Kubernetes—environments.

Fluent Bit, an open source tool within the CNCF ecosystem, is a popular choice when choosing which log collection tool to use. At Dynatrace, we’re happy to make your life easier by providing a ready-to-use integration with Fluent Bit. The integration ensures all logs are enriched properly to enable context-rich troubleshooting and allows you to connect the streamed logs with distributed traces and the Kubernetes domain model.

Speed up your troubleshooting processes

Log analysis is typically the first step in the troubleshooting process. Therefore, it’s critical that when an issue arises, you have the right data and tools to quickly and easily understand the full scope of what’s happening within your applications.

Dynatrace integration with Fluent Bit allows platform and development teams to examine Kubernetes logs in the context of their workloads and pods. For containers, they get the full context of individual traces throughout the hybrid microservices application stack.

When your microservices have an issue, you can look into error logs within the Kubernetes app at the respective namespace or workload and search for erroneous log lines.

When using Dynatrace OneAgent® for automated distributed tracing of all your Kubernetes workloads, the logs are automatically enriched with tracing metadata. You get the complete picture of the logs of each workload or namespace.

From there, you can drill down to the distributed traces and inspect the spans of the full trace down to the code level that caused the suspicious log line in Kubernetes.

Get started with a simple installation

In addition to an easier troubleshooting process, you benefit from less friction when onboarding.

Dynatrace is tech agnostic, having been purpose-built with cloud-native architectures in mind. This gives you more freedom and flexibility in your workspaces, allowing you to work with whatever tools you already have. Specifically, the release of the Fluent Bit integration gives you two options to get started:

• The in-product onboarding experience lets you quickly roll out a ready-to-use and preconfigured Fluent Bit installation as a DaemonSet. This is the most convenient way to get started. Dynatrace ensures that all logs are properly enriched with metadata to enable context-rich analytics on the platform. You can restrict streaming logs to certain namespaces or pods during the in-product onboarding or customize the created Fluent Bit filters in the YAML file of the used helm chart.
• You can also integrate the required Dynatrace configuration into your existing Fluent Bit installations by adding the required sections. The Dynatrace-specific configurations can be found in the preconfigured YAML file and added to existing and customized Fluent Bit.

Get started on your improved log streaming journey

Troubleshooting is only the beginning of what you can do with a proper log streaming solution. Whether it’s seamlessly using logs for health checks to understand why pods are failing, faster innovation, using log data to understand the issue and context of problems detected by Davis® AI in your Kubernetes setup, or investigating security incidents, the options are endless.

Ready to experience the Fluent Bit integration for yourself?

Get started on your log streaming journey and see how the Fluent Bit integration with Dynatrace simplifies your troubleshooting process.

If you’re looking for a full-service, fully automated approach to monitoring Kubernetes, check out our Kubernetes Log Module to seamlessly manage your logs.

The post Full Kubernetes logging in context: Easily stream logs from Fluent Bit to Dynatrace appeared first on Dynatrace news.

Deploying software in Kubernetes is often viewed as a straightforward process—just use kubectl or a GitOps solution like ArgoCD to deploy a YAML file, and you’re all set, right? Unfortunately, Kubernetes deployments can be fraught with challenges beyond the surface level. Numerous hurdles can hinder successful deployments, from resource constraints to external dependencies and monitoring inadequacies. In this article, we’ll explore these challenges in detail and introduce Keptn, an open source project that addresses these issues, enhancing Kubernetes observability for smoother and more efficient deployments.

Understanding Kubernetes deployment challenges

Deploying software using Kubernetes is, on the surface, easy, but there is a lot more to ensure a healthy deployment. There are a lot of potential problems that can prevent the successful deployment of a Kubernetes application, such as:

Resource constraints

In Kubernetes, managing resources efficiently is crucial to prevent performance bottlenecks and application failures. Insufficient CPU and memory allocation to pods can lead to resource contention and stop Pods from being created.

External dependencies

Many applications rely on external services, such as databases, APIs, or third-party services. Ensuring seamless connectivity to these dependencies during deployment is essential for application stability. Consider a scenario where a web application depends on an external payment gateway. Any connectivity issues with the payment gateway during deployment could result in transaction failures and impact the application’s functionality.

Infrastructure health

The underlying infrastructure’s health directly impacts application availability and performance. Vulnerabilities or hardware failures can disrupt deployments and compromise application security. For instance, if a Kubernetes cluster experiences a hardware failure during deployment, it can lead to service disruptions and affect the user experience.

Monitoring inadequacies

Comprehensive monitoring is essential for detecting and resolving issues during deployment. Inadequate monitoring tools or configurations can result in operational blind spots, delaying issue detection and resolution.

Performance degradation

Deploying new versions of applications can sometimes lead to unexpected performance degradation. Changes in application code or configurations can impact performance metrics, affecting user experience and application functionality. For instance, deploying a new version of a web application that introduces inefficient database queries could lead to increased response times and decreased user satisfaction.

This is the problem domain on which Keptn can help Platform Engineers provide a solution and guard rails for teams to deploy software that works perfectly.

Addressing Kubernetes deployment challenges with Keptn

Using Keptn, combined with your standard deployment tooling or practices,

you can move from “I guess it’s okay” to “I know it’s okay”. Keptn allows you to wrap governance and automated checks around the deployment process to ensure that the end-to-end deployment process is healthy and that your application meets the SLOs you’ve defined. This is done by configuring Keptn to execute KeptnTasks and KeptnEvaluations before and after deploying a Kubernetes Workload.

With these, you can implement checks that ensure your infrastructure is in a healthy state before a new version of a deployment is rolled out. This way, you can avoid the problems mentioned above by:

• Verifying that your cluster has enough resources to run a workload.
• Checking the reachability of external services that your workloads might require.
• Ensuring that your monitoring solution monitors your cluster.
• Verifying that there are no open problems within your cluster.

After successfully deploying a workload, Keptn also provides the means for post-deployment checks. These can be crucial in ensuring that the performance of your newly deployed workload meets your expectations before promoting it to the next stage.

How to get started

Keptn can be installed through Helm with this simple set of commands:

helm repo add keptn https://charts.lifecycle.keptn.sh
helm repo update
helm upgrade --install keptn keptn/keptn Dynatrace® Software Intelligence Platform
   -n keptn-system --create-namespace –wait

For further information and configuration options, head to the Keptn installation documentation. Otherwise, if your application uses the recommended Kubernetes labels, you are ready to go. Keptn will detect your application and provide observability of your deployment out of the box.

Usually, Keptn does not act in isolation during a deployment. In the logical timeline of events, there will be tools acting before Keptn (such as GitOps tools like ArgoCD) and tools that act after Keptn (such as security scanning tools).

If the tools before Keptn begin generating OpenTelemetry data (for example, spans and traces), it would be beneficial to see Keptn’s portion of the work in the correct context as part of the same distributed trace. Similarly, anything that happens after Keptn but still in the same logical “deployment” operation should be included in the end-to-end trace view.

To achieve this, Keptn now accepts tools to pass W3C trace IDs into Keptn

and Keptn will pass such W3C trace IDs out of Keptn, creating true end-to-end visibility.

With this information, Keptn can track from “PR merged” all the way to “deployment running in production,” allowing DevOps to slice and dice observability data to understand how the application reached that state.

After installing Keptn, you can get started by going through the several guides listed below that will introduce you step by step to the features provided by Keptn.

• Integrate Keptn with your applications, which shows you how to configure your applications to be deployed with Keptn.
• OpenTelemetry observability, which shows how to configure Keptn to make use of its observability features, so you can have a holistic overview of your application deployments and any issues that arise during deployment.
• Multi-stage application delivery, which serves as an example of how you can integrate Keptn with ArgoCD and GitHub to take the previous concepts and apply them across multiple application environments, giving you full observability of your application deployment from development into production.

Conclusion

Keptn empowers DevOps teams to conquer the Kubernetes deployment challenge confidently, ensuring smoother and more efficient deployments. By addressing the complexities inherent in Kubernetes deployments and seamlessly integrating with existing tools, Keptn delivers unparalleled visibility and control. Start your journey with Keptn by downloading the first Release Candidate for v2 available now on GitHub.

The post Mastering Kubernetes deployments with Keptn: a comprehensive guide to enhanced visibility appeared first on Dynatrace news.

As organizations strive for observability and data democratization, OpenTelemetry emerges as a key technology to create and transfer observability data. OpenTelemetry is gaining popularity because it’s considered a standard, and that’s why it’s a common choice for creating future-proof solutions for years to come.

To answer the growing demand for OpenTelemetry, Dynatrace is proud to announce the release of the Dynatrace OTel Collector distribution. This collector, fully supported and maintained by Dynatrace, is entirely open source. Before we get into the specifics, let’s first recap the benefits OpenTelemetry offers and why using collectors is a best practice.

Understanding OpenTelemetry

OpenTelemetry is an open, vendor-neutral standard for creating, collecting, and transferring telemetry data, like traces, metrics, and logs. Developers and operators can gain insights into their applications and infrastructure without fear of vendor lock-in because OpenTelemetry is fully open source and owned by CNCF. The OpenTelemetry project is supported and maintained by representatives from Microsoft, Google, Amazon, and many others, including Dynatrace.

Why do I need an OpenTelemetry collector?

As the name suggests, an OpenTelemetry collector gathers data from multiple sources and sends it to observability backends, like Dynatrace, for analysis. A collector helps developers control their telemetry data streams for each signal. Different data streams can be directed to different backends or even multicast to multiple backends simultaneously. The configuration is highly flexible in solving various user needs.

A collector is also a powerful component for data processing. It removes the burden of managing retries, batching, and sampling from monitored applications, which can reduce the CPU and memory requirements of applications. A collector can also transform and enrich the data with additional context. For example, in a Kubernetes environment, a collector can automatically attach metadata about pods and namespaces to all observability data. This ensures that application telemetry is contextualized with the infrastructure, enabling the observability backend to link the application and infrastructure for enhanced insights and root cause analysis.

From a user perspective, a collector can also serve as an open source platform that can be extended with custom components. You can create internal collector components of your own, for example, to receive telemetry data in a special format or to process it in a certain way. Using a collector as a telemetry processing platform can be much easier than creating an entirely new application.

Why the Dynatrace OTel Collector

The OpenTelemetry community releases different distributions of the collector, many of which our customers use to send OpenTelemetry data to Dynatrace. So, why should you consider using the Dynatrace Otel Collector? Quite simply, support and stability.

We have seen many customers identify collectors as a potential solution for their needs. Still, they haven’t been able to deploy collectors in production due to the lack of support. Deploying an open source component without external support and the needed expertise is undoubtedly a risk. That’s why we provide Dynatrace customers with a Dynatrace-supported solution.

The Dynatrace Otel Collector comes with collector components that have been verified by Dynatrace for seamless operation. This removes the burden of manually validating each component and use case. To further help you with your collector journey, we publish configuration examples of typical Dynatrace use cases and best practices to provide a good starting point.

The Dynatrace Otel Collector includes components that we know run stably in production, which means we can offer full Dynatrace support. At the same time, innovations from the OpenTelemetry community can be added to the Dynatrace Otel Collector only after they are mature enough and have proven their stability.

Deployment and the typical use cases

The Dynatrace Otel Collector can be deployed on Kubernetes or Docker using a provided container image or directly on a host with the published binary. For more details, refer to our Dynatrace OTel Collector deployment guide.

In the initial release, the Dynatrace Otel Collector comes with components for:

• Receiving, processing, and exporting OpenTelemetry protocol (OTLP) for traces, metrics, and logs
• Integrating Jaeger, Prometheus, and different log protocols, such as syslog, Fluentd, and log files
• To learn more about ingesting log data over syslog with the new Dynatrace OTel Collector, see the blog post Bring syslog into Dynatrace over OpenTelemetry to get open source value with enterprise support.

Additionally, the Dynatrace Otel Collector includes a rich toolset for data processing to enrich, filter, transform, sample, and batch. The complete list of the components is available in our GitHub repository.

What’s next

After the initial release, we’ll continue enhancing the Dynatrace OTel Collector with new features and capabilities to make it even easier to integrate with Dynatrace. We also intend to offer more automated methods to deploy the collector with pre-configurations. So, stay tuned.

As a significant contributor to the OpenTelemetry project, Dynatrace remains committed to working with the community and other vendors to enrich its capabilities and make it user-friendly for everyone.

The post Enhance data collection with Dynatrace OTel Collector distribution appeared first on Dynatrace news.

We’ve all been there. An important fix is waiting to be delivered, and a manager asks you why it’s taking so long and says that the pipeline was much faster a few months ago. Although you disagree with the assessment, you can’t dispute it because you don’t have the data to show that the pipeline is not slower.

At Dynatrace, Jenkins is the most used Continuous Integration (CI) tool. It’s open source, allows for great scalability with the possibility to run workloads on the cloud and on-premises Kubernetes clusters, and is easily extensible via plugins.

To monitor these pipelines, OpenTelemetry can be used to trace every step of the build and send the traces to various backends. You’re probably already familiar with the view of a trace in Dynatrace, as shown below. If you’re also familiar with Jenkins, you’ll notice that the trace steps are the same as in your Jenkins builds—you get a one-to-one mapping.

In this trace, we see that the checkout took up a large portion of the build. The question is, what can be done with this data? How can you show things like the failure rate of a pipeline during the last two weeks with as little effort as possible?

What we measure at Dynatrace

It’s important to agree upon what should be measured; there is no right or wrong answer here. At Dynatrace, we measure the DORA metrics from Google.

These include:

• Deployment frequency: How often is code deployed?
• Lead time for changes: How long does it take to get committed code into production?
• Time to restore service: How long does it take to restore a service?
• Change failure rate: How often does the service fail following changes?

While tracking the time to restore a service and the change failure rate require an issue management system, the first two metrics can be acquired by looking at the pipelines.

Unified services: Easy, out-of-the-box service visualization

If you use GitHub or GitLab, you can use the Four Keys project (https://github.com/dora-team/fourkeys), which can be integrated into your project, or you can write your own service that takes the OpenTelemetry data and executes the necessary operations on it.

But if you want to get everything out of the box, you can use Dynatrace unified services. This service visualization experience for OpenTelemetry was introduced with  Dynatrace version 1.274. As soon as you have unified services enabled on your Dynatrace tenant, your OpenTelemetry API-ingested services will look like the image below.

There’s an endpoint for each branch that exists within Jenkins. You get the number of failures, the throughput, and the response time, which maps to the duration of the build.

Now, let’s jump to a real-world example. Let’s say that your developers believe that their pull requests sometimes take longer than usual, not a lot, but a noticeable amount. And sometimes, the pull requests aren’t triggered at all. In such a scenario, check the durations of all the pull requests. In this case, we can see a clustering of pull requests and some outliers. With this information, we can guess that the webhook to Jenkins isn’t correctly configured, and only when the pipeline is scanned on Jenkins are the PRs picked up.

Normally, you get one endpoint per pull request, but as you can see, we summarized all the pull requests into one chart. This can be achieved by adapting the query a bit.

builtin:service.request.response_time_chart:filter(and(eq("dt.entity.service",SERVICE-AE4EB9E449B856CA),eq("endpoint.name","EP/backstage/PR-156"))):splitBy("endpoint.name"):median

You can adapt the equals to a contains and remove the PR number from the endpoint name and you get data for all pull requests in one chart.

builtin:service.request.response_time_chart:filter(and(eq("dt.entity.service",SERVICE-AE4EB9E449B856CA),contains("endpoint.name","EP/backstage/PR-"))):splitBy("endpoint.name"):median

Now, let’s say we want to check the Backstage pipeline. The first DORA metric is rather easy; we simply look at the throughput of the main branch. You can view the throughput in the Dynatrace Data Explorer, which translates to the number of builds that ran. In the view below, you can see the number of builds in the last two hours.

To learn the amount of time it takes until a commit appears in production, you need the duration of pull requests and the duration of the main branch. The chart below shows you the duration of builds on the main branch.

The chart for the main branch works out of the box. For the pull requests, we must adapt the query a bit, because based on the built-in rules that come with the unified services, they create a new endpoint for every pull request, which allows for exact control, but this is not what we want in this case. So, we will leave the query as-is .

One last thing that you might be interested in is the failure rate of the main branch. As with the other metrics, simply search your main branch, view the data in the Dynatrace Data Explorer and be happy that your main branch only broke twice in the last seven days.

Sneak-peek time!

While Traces powered by Grail™ is still in development, we already use it for internal use cases and can give you a sneak peek. With a simple Dynatrace Query Language (DQL) query, we can filter for each and every step of a trace, or as is the case in this example, a Jenkins pipeline. As you can see, the allocation times look fine, but there is a huge outlier. This outlier points us to an overloaded cloud; too many workloads were running, and the cluster ran out of resources. Therefore, the pods had to wait a long time to be spawned. With this data in hand, we don’t need to rely on a gut feeling—you can now back up your report with data.

See the current state of your pipeline at a glance

With Dynatrace unified services, you can assess the state of your pipelines at a glance and can make confident statements based on data and not your gut feelings. You can now provide answers to tough questions like “Are the pipelines getting slower?” or “Are my builds breaking more often?” Another big advantage is the fact that once you have the services enabled, you have the data available all the time and no longer need to search when new questions arise. If you want a deeper look into this topic, watch Dynatrace Tips & Tricks #18 – Jenkins Pipeline Analytics with OpenTelemetry.

Try it yourself

Using Jenkins? The following links help you to export the traces from Jenkins and to ingest them into Dynatrace.

• Jenkins OpenTelemetry Plugin
• OpenTelemetry Collector
• Dynatrace Tips & Tricks #18 – Jenkins Pipeline Analytics with OpenTelemetry

The post Unified services: Easily keep track of your pipelines appeared first on Dynatrace news.

Because open source software (OSS) is taking over the world, optimizing open source contributions is becoming an essential competitive strategy. OSS is a faster, more collaborative, and more flexible way of driving software innovation than proprietary-only code. This flexibility appeals to developers and can help organizational leadership drive down costs while supporting digital transformation goals. The figures speak for themselves: 80% of organizations increased their OSS use in 2022. Especially those operating in critical infrastructure sectors such as oil and gas, telecommunications, and energy.

However, open source is not a panacea. Governance, security, and balancing between contributing to OSS development and preserving a commercial advantage pose challenges for many organizations. These considerations are important if developers want to maximize the impact of their contributions to open source projects.

Benefits of making open source contributions

There’s no one-size-fits-all approach with OSS. Projects could range from relatively small software components, such as general-purpose Java class libraries, to major systems, such as Kubernetes for container management or Apache’s HTTP server for modern operating systems. Organizations are more likely to adopt projects that receive regular contributions and updates from reputable developers. These projects provide a range of proven benefits.

1 Saves time and resources

Open source can save time and resources, as developers don’t have to expend their own energies to produce code. One study estimates that the top four OSS ecosystems recorded more than three trillion component requests last year. That utilization represents a great deal of time and effort developers are saving. That saving means teams can devote more time to developing proprietary functionality to boost revenue streams. The same study estimates that $1.1 billion in OSS investments in the EU generated around $100 billion.

2 Benefits from diverse contributions

OSS also encourages experts from across the globe—whether individual hobbyists or DevOps teams from multinational companies—to contribute their coding skills and industry knowledge. The idea is projects benefit from a diverse pool of developers, driving up the quality of the final product. In contributing to these projects, businesses and individuals can also stake a claim to the future direction of a particular product or field of technology. With such a stake, they can help shape the technology to advance their priorities. Organizations also benefit from being at the leading edge of any new discoveries and innovations. Fostering such involvement can help organizations gain an advantage over the competition by being first to market.

3 Drives a culture of innovation

Organizations that regularly contribute to OSS can also help to drive a culture of innovation. Alongside an organization’s track record on patents, a commitment to OSS projects can allow prospective new hires to contribute their own innovation, which can help attract the brightest and best talent.

Three ways to get the most out of open source contributions

To maximize the benefit of their contributions to the OSS community, DevOps leaders should ensure their organization has a clear, strategic approach. There are three key points to consider in these efforts:

1 Define the scope of the organization’s contribution

OSS is built on the expertise of a potentially wide range of individuals and organizations, many of whom are otherwise competitors. This “wisdom of the crowd” can ultimately help to create better-quality products more quickly. However, it can also raise difficult questions about how to keep proprietary secrets under wraps. Often, there is pressure from the community to share certain code bases or functionality that could benefit others. By defining what they want to keep private at the outset, contributors can draw a clear line between commercial advantage and community benefit.

2 Contribute to open standards

Open standards are the foundation on which OSS contributors can collaborate. By contributing to these initiatives, organizations have a fantastic opportunity to shape the future direction of OSS. Helping to solve common problems can enhance the value of their own commercial products. OpenTelemetry is one such success story. This collection of tools, APIs, and SDKs enables organizations to capture and export telemetry data from applications to make tracing more seamless across boundaries and systems. As a result, OpenTelemetry has become a de facto industry standard for how organizations capture and process observability data. OpenTelemetry brings organizations closer to achieving a unified view of hybrid technology stacks in a single platform.

3 Build robust security practices

Despite the benefits of OSS, there’s always a risk of vulnerabilities slipping into production if teams don’t detect and remedy them quickly and effectively in development environments. Three-quarters (75%) of chief information security officers (CISOs) worry the prevalence of team silos and point solutions throughout the software development lifecycle makes it easier for vulnerabilities to fly below the radar. Their concerns are valid. For example, according to one estimate, the average application development project contains 49 vulnerabilities. These risks will only grow as more organizations adopt ChatGPT-like tools to support software development by compiling code snippets from open source libraries.

Essential capabilities needed to support open source contributions

Given the dynamic, fast-changing nature of cloud-native environments and the increasing use of open source software, teams need a way to manage and monitor OSS at scale. To support safe and efficient OSS development and usage, teams should enlist a unified source of end-to-end observability. By combining comprehensive OSS observability data with trustworthy AI, teams can understand the full context behind what OSS they use and how. With this insight, organizations can unlock precise, real-time answers about OSS-related vulnerabilities in their environment. In turn, DevOps teams can implement security and quality gates throughout the delivery pipeline. These gates help teams automatically detect and resolve vulnerabilities and bugs before releasing software into production.

OSS is increasingly important to long-term success, even for commercially motivated organizations. Developing a strategy for effectively using and contributing to key OSS projects can help determine an organization’s success.

The post Three ways to optimize open source contributions appeared first on Dynatrace news.

Kubernetes has become the leading container orchestration platform for organizations adopting open source solutions to manage, scale, and automate application deployment. Adopting this powerful tool can provide strategic technological benefits to organizations — specifically DevOps teams. At the same time, it also introduces a large amount of complexity. This complexity has surfaced seven top Kubernetes challenges that strain engineering teams and ultimately slow the pace of innovation.

What is Kubernetes? And how does it benefit organizations?

Kubernetes is an open source container orchestration platform for managing, automating, and scaling containerized applications.

Containerized microservices have made it easier for organizations to create and deploy applications across multiple cloud environments without worrying about functional conflicts or software incompatibilities. This ease of deployment has led to mass adoption, with nearly 80% of organizations now using container technology for applications in production, according to the CNCF 2022 Annual Survey. However, as the use of containers has grown, so too has the need for more effective management of these highly distributed environments at scale.

To manage this complexity, teams have turned to container orchestration solutions such as Kubernetes. The platform aims to help DevOps teams optimize the allocation of compute resources across all containerized workloads in deployment.

While a handful of these solutions exist, Kubernetes has become the de facto industry standard, offering the following benefits:

• Container deployment. Container orchestration platforms automate daily operations for processes such as the re-creation of failed containers and rolling deployments. This helps to avoid downtime for end users.
• Automated scaling. Kubernetes enables efficient resource utilization by easily scaling applications and services based on demand.
• Self-healing. The platform will automatically restart, replace, or kill failed containers, as well as reschedule unhealthy pods and manage node failures. This key feature helps in maintaining availability and reduces the need for manual intervention.
• Extensibility and technology ecosystem. As an open source solution, Kubernetes has a large, active, and growing ecosystem of extensions, plug-ins, and technologies to enhance its capabilities. The ability to extend functionality across a wide range of use cases allows teams to tailor the platform to meet their organization’s specific requirements.

The top Kubernetes challenges and potential solutions

Despite its benefits, Kubernetes has some potential pitfalls that engineering leaders should consider when managing the complexity it introduces. The top seven Kubernetes challenges include the following:

1. Complexity. Kubernetes environments tend to be complex, multilayered, and dynamic. This creates limitations and blind spots when it comes to observability. Teams often need to know where to look to find issues and resolve them, but that can be time-consuming in large-scale deployments. Comprehensive platform solutions that provide full-stack monitoring with AI at their core can automatically detect anomalies and provide root-cause analysis to prevent future issues.

2. Networking. Large-scale, multicloud deployments can introduce challenges related to network visibility and interoperability. Traditional ways of operating networks using static IPs and ports simply don’t work in dynamic Kubernetes environments. Container Network Interface (CNI) provides a common way to seamlessly integrate various technologies with the underlying Kubernetes infrastructure. Additionally, service meshes — such as those offered by Istio, Linkerd, and Consul Connect — help to manage internetwork communication at the platform layer using purpose-built application programming interfaces.

3. Observability. While there are many observability and monitoring tools on the market today, most are specific in nature. A full-stack observability platform such as Dynatrace provides easy access to the three key types of monitoring signals — logs, traces, and metrics — in context. Automated data collection correlated with topology information, real-user experience, security events, and metadata makes it easier to identify issues and determine effective remediation paths. With the ability to monitor resource utilization metrics such as CPU and memory in real time, teams can optimize their operations, resulting in reduced cost and greater overall efficiency.

4. Cluster stability. Kubernetes containers are naturally short-lived and ephemeral, meaning they’re constantly being created, altered, and removed. This creates challenges in monitoring and debugging distributed applications at scale, often resulting in reliability issues. Effective monitoring, logging, and tracing mechanisms need to be in place to identify and resolve issues quickly. Additionally, ensuring cluster stability through monitoring critical components at the control plane is essential to preventing failures. To minimize negative effects, consider setting limits and alerts on CPU and memory resource requests.

5. Security. Kubernetes security incidents are primarily related to pod communications or misconfigurations that ultimately lead to delayed application deployment. Pods are not isolated, which leaves them vulnerable to malicious actors. Bad actors can use misconfigurations to gain access to sensitive data. However, Kubernetes configurations are complicated, which makes managing them at scale almost impossible. Network policies can restrict pod communications, and teams can use pod security policies to ensure pods are securely configured.

6. Logging. Logs provide critical visibility into the ongoing health of Kubernetes clusters. While Kubernetes makes it easier to generate logs from the various components and layers of a cluster, challenges remain in aggregation and analysis. Solutions that offer log management and analysis capabilities can help to streamline these efforts and provide actionable insights to teams maintaining the health of the workloads and underlying infrastructure.

7. Storage. Containers need to spin up and down easily. Therefore, they are built to be non-persistent by design. However, applications require persistent data to run successfully in production. Traditional storage solutions were not created to address these requirements, which are common among modern deployments. To ease the pain of managing storage at scale within a Kubernetes environment, Kubernetes has released features such as Container Storage Interface (CSI), StatefulSets, Persistent Volume (PV), and Persistent Volume Claim (PVC).

How a cloud-native observability and security platform can overcome Kubernetes challenges

To adequately address the complexity introduced by Kubernetes and other cloud-native technologies, organizations need more complete solutions that provide end-to-end, full-stack visibility, advanced performance and security analytics, and automated workflow capabilities all in a single, comprehensive platform.

Dynatrace integrates extensive Kubernetes observability with continuous runtime application security. This combination helps organizations more effectively meet business goals and minimize risk. The Dynatrace platform offers the following benefits:

• Automated observability at scale. As organizations deploy Kubernetes clusters across on-premises, cloud, and edge environments, end-to-end observability becomes mandatory. Dynatrace OneAgent and open ingest deliver the deepest and broadest observability on the market, with hundreds of out-of-the-box integrations covering the complete Kubernetes ecosystem. Dynatrace Grail unifies observability, security, and business data at a limitless scale for any analysis at any time.
• AI-powered analytics. Dynatrace provides precise and explainable answers in real time, identifying performance issues and anomalies across the full Kubernetes technology stack. Davis, the Dynatrace AI engine, helps organizations understand and optimize Kubernetes platform health and application performance, enabling IT teams to proactively pinpoint and rectify performance issues.
• Platform and application security. Optimized for Kubernetes, Dynatrace Application Security automatically and continuously detects vulnerabilities and protects against injection attacks that exploit critical vulnerabilities, such as Log4Shell. These capabilities remove blind spots, ensuring development teams aren’t wasting time chasing false positives and providing business leaders with confidence in the security of their organizations’ applications.
• Acceleration of innovation. Dynatrace enables IT teams to shift their effort from code maintenance and troubleshooting to innovation. Leveraging Dynatrace’s analytics and workflow automation enhances software quality, minimizes the mean time to resolve anomalies, and detects security vulnerabilities in near-real time.