In practice, most organizations attempt to solve this with one of two compromises: lock everything down centrally and become a bottleneck for every pipeline change, or copy rules across pipelines and hope nothing drifts. Neither approach scales. And as data requirements shift faster than ever to address new services, new regulatory mandates, and new object types to observe, the cost of such compromise keeps growing.

Pipeline Groups change that dynamic. Generally available now in Dynatrace SaaS version 1.332, Pipeline Groups let platform engineering teams, the teams with admin permissions over the observability platform’s configuration, mandate and standardize ingest behavior across many pipelines, while safely delegating day-to-day configuration of individual pipelines to the teams best suited to own them.

The problem: Pipeline governance at scale

In large enterprises, dozens of teams may each operate their own custom pipelines. Common requirements, security context enrichment, cost allocation tagging, and storage bucket assignment must be applied consistently across the board. However, without a structured mechanism to enforce such standards, platform teams face a choice that gets harder with every new team, every new data source, and every new regulation:

• Lock everything down and become the gatekeeper for every pipeline change. Every adjustment, no matter how small, becomes a ticket, a review, a delay. Innovation stalls. Teams wait days for changes that should take minutes.
• Copy rules everywhere and accept the risk of configuration drift. What starts as a manageable set of shared rules gradually fragments—one team forgets to apply the latest cost allocation tag, another skips the security enrichment step, a third routes data to the wrong bucket. The inconsistencies compound silently until they surface as compliance gaps or billing surprises.

Customers consistently tell us that this approach doesn’t scale. What they need is a way to separate what must always happen from what teams should be free to decide, and to encode that separation directly into the platform, not just into process documents that team members forget to follow.

What are Pipeline Groups?

A Pipeline Group is a first-class configuration object that separates a global pipeline list into distinct sets; the pipelines in these sets are then members of the group. A Pipeline Group orchestrates a special type of reusable pipeline to define what happens before or after the stages of the member pipelines.

Pipeline Groups allow you to:

• Organize pipelines into groups with clearly defined membership.
• Define execution order so processing happens in a predictable, layered sequence.
• Control stage execution for the member pipelines so that teams can turn on or turn off specific pipeline stages.
• Enforce mandatory global processing that no team can bypass or override.
• Allow team-level customization within boundaries defined by the platform team.

Pipeline Groups determine how data flows through your pipelines. This ensures that governance isn’t an afterthought bolted onto a pipeline configuration, but rather it’s built into the execution model itself.

How Pipeline Group ownership works

The design behind Pipeline Groups draws a deliberate line between group-level configuration and member-pipeline configuration. How organizations map this division to team ownership is up to them. Pipeline Groups provide the mechanism, not a prescriptive policy. The pattern we see most often is straightforward: platform or SRE teams own the pipeline groups, and application teams own their member pipelines within those groups.

To make this more concrete, consider a platform team that’s responsible for observability across multiple business units. They create a Pipeline Group that adds a business segment field to every record for organizational attribution, applies cost allocation tags for accurate chargeback, sets security context so sensitive data is handled consistently, and assigns data to the correct storage bucket based on retention and compliance needs. These are the rules that must always apply, and because they operate at the group level, no member pipeline can bypass or override them.

Member pipeline ownership can be more granular than a simple admin-vs-team split. While a group is always admin-owned, the individual pipelines that make up the Pipeline Group can each have different owners. For example, a pipeline that enriches every record with organizational metadata or classifies data sensitivity might be globally relevant and owned by the platform team, or by a specialist team like the security team. A pipeline that handles domain-specific compliance logic, such as PCI field masking for a financial services division, might be owned by that compliance team because they have the expertise that the platform team lacks. This arrangement reflects how responsibility within that organization is distributed.

Within that group, application teams each get their own member pipeline. One team configures parsing for a specific log format. Another sets up filtering to reduce noise. A third extracts metrics tailored to their services. Each team works independently within its own pipeline without touching the platform-level rules above and without needing to coordinate with other teams or the platform team for routine changes.

The value of this separation becomes clear when things change, as they do frequently in large enterprises. Say that one of those business units onboards a new microservice that generates a completely new log format. Under a centralized-only model, the team would file a request, wait for the platform team to update the pipeline, validate, and iterate. With Pipeline Groups, the team simply adds their parsing and filtering rules to their own member pipeline. The mandatory enrichment, cost tagging, and routing are already guaranteed by the group. The new service is observable in hours, not weeks, and the platform team doesn’t need to be involved at all.

Or picture the reverse direction: a new regulatory requirement lands, say, a mandate that all log data from EU-based services must be assigned to region-specific storage. The platform team updates the group configuration once. Every member pipeline inherits the change immediately.

Why the flexibility of Pipeline Groups matters

The pace at which enterprise data requirements evolve has fundamentally changed. New services are spun up in days, not months. Regulatory landscapes shift across jurisdictions. The types of software entities that organizations need to observe, from traditional infrastructure to AI model outputs, edge devices, and third-party SaaS telemetry, keep expanding. In this environment, a rigid, centralized-only pipeline configuration becomes a constraint rather than a benefit.

Pipeline Groups are built for this reality. They give enterprises a mechanism that:

1. Creates room for innovation. Application teams can iterate on their pipeline configurations independently, experimenting with new parsing rules, extracting new metrics, and adapting to new data shapes without waiting for central approval on every change.
2. Eliminates bottlenecks. Platform teams define the rules once and let the system enforce them. Teams are freed from serving as gatekeepers for routine changes and can focus on architecture, standards, and strategy.
3. Ensures that ever-changing guardrails are applied. As compliance requirements, security policies, or cost structures evolve, platform teams can update them all in one place. The changes propagate automatically to every pipeline in the Pipeline Group.

Pipeline Groups give platform teams the governance controls and flexibility they need

Pipeline Groups give enterprise platform teams the governance controls they’ve been asking for, mandatory processing, centralized ownership of sensitive stages, and structured delegation, all without taking away the flexibility that makes Dynatrace OpenPipeline® valuable to individual teams in the first place.

Ready to get started with OpenPipeline and Pipeline Groups?

If your organization manages observability across multiple teams and struggles with consistency, Pipeline Groups are now generally available.

Learn how to define global guardrails once and empower your application teams to build their own pipelines confidently within standardized boundaries.

The post Pipeline Groups in Dynatrace OpenPipeline: Enterprise-grade governance explained appeared first on Dynatrace news.

The February Dynatrace® SaaS releases continue a clear trend: extending powerful platform capabilities to help practitioners move faster, stay in context, and scale with less friction. This edition of Release Radar focuses on five enhancements that stand out in enabling practitioners to get their data in context autonomously while helping them to better understand their environment and troubleshoot easily.

If you want to see these enhancements in action, head on over to our release radar launchpad on our playground environment.

Bring topology into the natural flow of troubleshooting

Smartscape® is a powerful tool for understanding live dependencies across modern environments. With version 1.333, Dynatrace makes that context easier to access by adding View Topology directly to entity pages in Clouds, Infrastructure & Operations, and Kubernetes.

This is a meaningful refinement because effective troubleshooting depends on staying in flow. When practitioners spot an issue on an entity page, they can now move directly into real-time topology and explore surrounding dependencies without detouring into a separate workflow.

Scale governance without slowing teams down

OpenPipeline® gives teams a flexible foundation for ingest and processing. With version 1.333, pipeline groups are now generally available, giving central teams a way to enforce shared policies across multiple pipelines at once, including security context, cost allocation, and sensitive data scanning, while still leaving parsing and extraction logic in the hands of individual teams.

This is where Dynatrace extends flexibility with cleaner operational guardrails – unlocking a way for safe scaling. Platform teams can standardize the controls that matter most, while application teams keep the autonomy they need to move quickly. For larger environments, especially, such a balance is essential.

Add additional dimensions for OTLP metrics

One enhancement for OpenTelemetry Protocol users is the expansion of OpenTelemetry Protocol (OTLP) metrics ingest so that all OTLP resource attributes are now available as dimensions in Metrics powered by Grail®. Dynatrace also added controls to limit which attributes are carried forward, including a deny list for OTLP sources and an allow list for Metrics Classic.

This update adds richer metric dimensions and stronger controls, making OTLP-based metrics easier to analyze, govern, and own while extending the core strength of Dynatrace Grail and preserving context at scale.

Get sharper answers from synthetic analysis

Synthetic monitoring helps teams catch digital experience issues early. Now, Dynatrace makes that workflow more actionable with the new browser monitor execution analysis page, which helps isolate the specific resources in specific executions that are affecting performance.

Once a run is flagged as slow, the next question is always the same: What exactly made this run degrade? This enhancement gives practitioners a more direct path from symptom to likely cause.

Make Primary Grail tags more useful across the platform

Across the February releases, Primary Grail tags have appeared in more places. In 1.332, Dynatrace added them to Segments, where they now show up in suggestions and autocomplete. In 1.333, they were also added to Synthetic monitor configurations.

This is a quiet but important improvement. The more consistently teams can use the same organizational language across telemetry, filtering, and configuration, the easier it becomes to move between workflows without losing context. Dynatrace already does a strong job of preserving meaning across data; this enhancement makes that consistency even more useful in practice.

Why these five Dynatrace enhancements matter

Taken together, these updates show Dynatrace continuing to strengthen the workflows practitioners already trust. Smartscape is easier to reach during troubleshooting. OpenPipeline becomes easier to govern at scale. OTLP metrics on Grail carry richer, more useful context. Synthetic analysis gets more precise. And Primary Grail tags continue to unify how teams organize and work with data across the platform.

Check out the updates in action on our release radar launchpad.

The post Dynatrace Release Radar 02.26 appeared first on Dynatrace news.

Protecting Personally Identifiable Information (PII) data in logs is crucial for supporting privacy and compliance. While Dynatrace OneAgent can mask data at capture/source, sensitive information like social security numbers (SSNs), payment details, IP addresses, or email addresses might still get through — especially when logs are ingested via other mechanisms such as directly using an API, via Fluent Bit, Cribl, or using the OpenTelemetry collector.

In this blog, practitioners will walk away with an in-depth understanding of how to create your own parsing rules. We’ll focus on masking/obfuscating email addresses in log events during the ingest process, leveraging Dynatrace OpenPipeline before logs are retained. Outside of this blog’s scope is the attribute-based access capability and mask on read.

Best Practice: Before we start masking data at the time of ingest, we’ll validate our configuration within a Dynatrace Notebook to prevent unwanted data loss caused by inaccurate patterns. We’ll walk through:

• Identifying patterns
• Creating a masking pipeline in OpenPipeline
• Validating the results

Step 1: Simulate logs with sensitive data

To demonstrate, we’ll work with logs containing email addresses and IP addresses. These logs use example domains and IPs (e.g., example.com and 203.0.113.x) for documentation purposes.

Here are the eight sample logs sent from various demo applications to the Dynatrace Playground:

• 2025-04-22 11:46:38 [ERROR|[203.0.113.13]
  |K9p8Q3xJwZrStUv4XyZ7AbC2n5M6h8T1v0L2r4 
  |com.example.exmplstore.security.examplePersistentTokenRepository] 
  Can't find credentials for series marie_curie@example.com

• 2025-04-22 13:50:43 [INFO 
  |[203.0.113.112] |A1b2C3d4EfGhIjK5LmN6oPq7RsT8uVw9XyZ0 
  |class com.example.exmplfacades.order.exampleCheckoutLogger Checkout ABC] 
  Receive API Request:method=placePayOrder, cartCode=302132310, 
  customerID=freddie@example.com
  |com.example.exmplstore.checkout.request.PayPlaceOrderRequestDto@3d3a53d5|]

• 2025-04-22 13:58:24 [INFO |||com.example.exmpl.BusinessProcessLoggingAspect] 
  Finish Action:[ PerformSubscriptionAction ], BusinessProcessCode: 
  [ customerRegistrationProcess-martin_luther@example.com-1745294290737],
  OrderCode:[ n/a ]

• 2025-04-29 10:16:13 [INFO ||
  |com.example.exmplmarketing.action.exampleSendCustomerNotificationAction] 
  Successfully sent email forgottenPassword message for process 
  forgottenPasswordProcess-jane_austen@example.com-1745885767984

• 2025-04-29 10:24:26 [INFO |[203.0.113.14] 
  |M3n4P5q6RsTuVwXyZ7aBc8DeF9gHi0JkL2|com.example.exmpl.UserDeleteInterceptor] 
  Deleting userId: cleopatra@example.com, actioned by userId: anonymous

• 2025-04-29 10:24:36 [INFO |[203.0.113.19] 
  |T2u3V4w5XyZaBc6DeF7gHi8JkL9mNo0PqR1|com.example.exmpl.AdvantageApiClient] 
  Loyalty: Check loyalty account exist for email: leonardo_davinci@example.com , 
  wodCorrelationId 2fee137d-915e-46fb-b390-c69aae3f150

• 2025-04-29 10:22:47 [INFO ||
  |com.example.exmplbusproc.aop.BusinessProcessLoggingAspect] 
  Begin Action: [ exampleAdvantageLinkEmailAction ], BusinessProcessCode: 
  [ advantageLinkEmailProcess-albert_einstein@example.com-1745886166709], 
  OrderCode:[ n/a ]

• 10.96.152.11 - - [29/Apr/2025:10:33:43 +1000] 
  "GET /reminder/shakespeare@example.com/1 HTTP/1.1" 200 90 
  "https://www.example.com/my-account/order-details/306399901" 
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 
  (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36" 203.0.113.145 - 
  T2u3V4w5XyZaBc6DeF7gHi8JkL9mNo0PqR1 100 

As you can see, all of these logs contain email addresses, which must be masked before storing in Dynatrace Grail™.

Step 2: Identify patterns in logs

We can use the Notebooks app to query stored logs and search for log lines that contain email addresses. To do so, we create a new notebook, add a DQL section, and execute the following command:

fetch logs
| filter matchesPattern(content, "LD '@' [A-Za-z0-9'_']* '.'
[A-Za-z]* (LD | EOS)") 

The second part of this command executes a pattern matching command, which makes use of the Dynatrace Pattern Language (DPL) to define the pattern and reduces the result to show only those entries that contain an email address.

Let’s have a closer look at the DPL pattern itself:

LD '@' [A-Za-z0-9'-']* '.' [A-Za-z]* LD EOS 

and break down the elements:

• LD: Matches any characters (letters, digits, or others) until the next non-optional matcher (in this case, ‘@’) within the scope of a line.
• '@': Matches the actual @ symbol.
• [A-Za-z0-9'-']*: Matches zero or more letters, digits, or hyphens, forming the domain part of the email address.
• '.': Matches the literal “.” (e.g. in .com).
• [A-Za-z]*: Matches zero or more letters, representing the top-level domain. Note: If you wish to match double-dotted TLD’s like .co.uk use this pattern instead: [A-Za-z’.’]*
• (LD | EOS): The pattern matches either a sequence of characters (letters, digits, or others) via LD to the end of the line after the email address. However, if the last word in the log line is the email address, then optionally the EOS will match the end of string.

Step 3: Analyze logs for email address patterns

After confirming email addresses exist, we examine the log lines to identify patterns preceding them. Here are the literals just before the email addresses, found in our returned sample logs:

• order confirmation email sent to
• find credentials for series
• customerID=
• customerRegistrationProcess-
• forgottenPasswordProcess-
• Deleting userId: 
• Check loyalty account exist for email: 
• advantageLinkEmailProcess-
• GET /reminder/ 

These literals will help us target the email addresses for masking.

Step 4: Check the pattern to test the process of masking the email addresses

We will now parse the email addresses and mask the local part (before the @ symbol) using Dynatrace Query Language (DQL). This is parsing at query time to test whether the pattern is valid or not. It does not change or update the unmasked data that is already stored in Grail. This step ensures that the pattern we’ll be using in the following steps with OpenPipeline are valid. To achieve this, we must open the DPL architect by selecting the 3 dots on the right hand of a row-element in the content column and select “Extract fields.”

Parse the email field

We will use the following parsing pattern with DQL to extract email addresses based on the literals identified:

Pattern:

 
LD  
( 
'order confirmation email sent to' |  
'find credentials for series' | 
'customerID=' |  
'customerRegistrationProcess-' |  
'forgottenPasswordProcess-' |  
'Deleting userId: ' |  
'Check loyalty account exist for email: ' | 'advantageLinkEmailProcess-' |  
'GET /reminder/') LD:email 
 

Breakdown: 

• LD 
  Matches any non-whitespace characters before the literal.
• ('Can't find credentials for series' | ...) 
  Matches any of the specified literals.
• LD:email 
  Captures the email address into a field named email.

Validate the parsing rule using DPL architect in Dynatrace Notebooks by pasting the parse command (starting with LD):

Make sure you select the ‘Add to preview’ action button to ensure there are no “Unmatched records” and all the matched records based on the previous filter can be truly passed:

Test the process of masking the email

In the screenshot below, you can notice that the logs do contain unmasked email addresses:

Test replacing and obfuscating the email address in the log content with a masked value (e.g., xyz):
Add below your existing query the fieldsAdd operation:

| fieldsAdd content = replacePattern(content,  
 
" 
<< 
( 
 'order confirmation email sent to' | 
'find credentials for series ' |  
 'customerID=' |  
 'customerRegistrationProcess-' |  
 'forgottenPasswordProcess-' |  
 'Deleting userId: ' |  
 'Check loyalty account exist for email: ' |  
 'advantageLinkEmailProcess-' |  
 'GET /reminder/' 
) LD:email  
>> 
'@' 
",  
 
“xyz”) 
 

This replaces the email address (stored in the email field) with xyz in the content field.

What it does:

This DQL command modifies the content field in a Dynatrace log record by replacing specific patterns defined – email addresses in this case – with the defined string “xyz”.
Let’s have a closer look at the technical details:

1. replacePattern Function:
   The replacePattern function searches the content field for matches of a specified pattern and replaces them with a given string (here, “xyz”).
2. DPL Modifier – Lookaround:
   Positive Look Behind Modifier <<
   Pattern:

<<
(
'for series' |
'customerID=' | 
'order confirmation email sent to' |
'customerRegistrationProcess-' |
' process forgottenPasswordProcess-' |
'userId: ' |
'Check OnePass account exist for email: ' |
'MobileApp customerRef=' |
'advantageLinkEmailProcess-' |
'GET /reminder/'
)

1. • Explanation: The << modifier looks up to 64 bytes before the current position in the log to check for any of the specified phrases. If one is found, the pattern continues. Our pattern includes <<('customerID=' 'GET /reminder/'). If the log line contains customerID=marie_currie@example.com, the modifier confirms that customerID= appears within 64 bytes before the email address, so the match succeeds. If that phrase isn’t found in the preceding 64 bytes, the match fails. Read more about DPL modifiers in our product documentation
2. Positive Look Ahead >>
   Pattern:
   LD:email >> ‘@' Explanation:  The >> modifier performs a look-ahead check. It ensures the pattern only matches if a specific condition appears after the current position. In this case, after capturing the local part of the email with LD:email, the pattern verifies that an @ symbol follows. This confirms the captured text is indeed part of an email address. For example, take the logline “Can't find credentials for series marie_currie@example.com” The pattern first uses LD:email to capture marie_currie as the local part. The >> '@' check then looks ahead and confirms that @ immediately follows. Because the condition is met, the match succeeds. But if the @ symbol were missing (e.g., Can't find credentials for series marie_curieexample.com), the match would fail.
3. Replacement:
   The matched pattern (e.g., marie_currie) is replaced with “xyz”. This masks the local part of the email address while leaving the rest of the log entry intact. For instance:
   • Before: customerID=marie_currie@example.com
   • After: xyz@example.com
4. fieldsAdd content = …: 
   The modified content (with the email local part masked) is stored back into the content field, overwriting the original value.

Additional guidance:

Follow these steps if you wish to mask the entire email address, including the domain:
If you wish to mask or extract the entire email address, you could use a pattern like below:

<<
( 
'order confirmation email sent to ' | 
'find credentials for series ' |  
'customerID=' |  
'customerRegistrationProcess-' |  
'forgottenPasswordProcess-' |  
'Deleting userId: ' |  
'Check loyalty account exist for email: ' |  
'advantageLinkEmailProcess-' |  
'GET /reminder/' 
) 
(LD '@'[A-Za-z0-9'-'']* '.' [A-Za-z]*):email 

In addition to the modifiers and pattern matching literals that were explained earlier, below is the explanation of the last line on how the breakdown of the DPL pattern language:

(LD '@'[A-Za-z0-9'-'']* '.' [A-Za-z]*):email 

1. LD: Matches line data.
2. '@‘: Matches the ‘@’ symbol.
3. [A-Za-z0-9'-'']*: Matches any sequence of alphanumeric characters, hyphens, and single quotes, occurring zero or more times.  Read more here.
4. '.': Matches the ‘.’ (dot) character.
5. [A-Za-z]*: Matches any sequence of alphabetic characters, occurring zero or more times.
6. :email: Captures the matched data and labels it as email. This is not needed if you don’t wish to capture the email address.

Step 5: Filter logs to avoid unintended masking

To ensure we only mask logs from specific sources and pattern containers, apply additional filters to isolate those logs only.

As a best practice, you should pre-filter your logs, by applying this filter directly underneath your fetch logs statement.

Filter by Container Names:

| filter k8s.container.name == "checkout"
OR k8s.container.name == "astroshop"
OR k8s.container.name == "aks-playground"

Filter by Content Patterns:

filter  
( 
matchesPhrase(content, "find credentials for series ") OR  
matchesPhrase(content, "order confirmation email sent to ") | 
matchesPhrase(content, "customerID=") OR  
matchesPhrase(content, "customerRegistrationProcess-") OR  
matchesPhrase(content, "forgottenPasswordProcess-") OR  
matchesPhrase(content, "Deleting userId: ") OR  
matchesPhrase(content, "Check loyalty account exist for email: ") OR 
matchesPhrase(content, "advantageLinkEmailProcess-") OR 
matchesPhrase(content, "GET /reminder/") 
) 

Complete DQL Query

Here’s the full DQL query combining all steps for your reference and to copy/paste:

fetch logs 
| filter  
( 
k8s.container.name == "checkout" OR  
k8s.container.name == "astroshop" OR  
k8s.container.name == "aks-playground" 
) 
AND 
// Match only logs that have specific phrases  
( 
matchesPhrase(content, "find credentials for series ") OR  
matchesPhrase(content, "order confirmation email sent to ") OR  
matchesPhrase(content, "customerID=") OR  
matchesPhrase(content, "customerRegistrationProcess-") OR  
matchesPhrase(content, "forgottenPasswordProcess-") OR  
matchesPhrase(content, "Deleting userId: ") OR  
matchesPhrase(content, "Check loyalty account exist for email: ") OR 
matchesPhrase(content, "advantageLinkEmailProcess-") OR 
matchesPhrase(content, "GET /reminder/") 
) 
| // extract the contents just after these literals that contains the localpart/username within the email address and replace the pattern that comes after these strings. 
fieldsAdd content = replacePattern(content,  
 
" 
<< 
( 
'order confirmation email sent to' | 
'find credentials for series ' |  
'customerID=' |  
'customerRegistrationProcess-' |  
'forgottenPasswordProcess-' |  
'Deleting userId: ' |  
'Check loyalty account exist for email: ' |    
'advantageLinkEmailProcess-' |  
 'GET /reminder/' 
) LD:email  
>> 
'@' 
",  
 
“xyz”) 

Store this query in a Dynatrace Notebook for future reference, and feel free to select the Run button to validate that the output matches your expectations.

All our earlier steps did not actually mask the incoming logs, but applied our masking at read/query, and secondly, just for us as the users of the Notebook.

The Notebook and its DQL snippet simply masked the data we visualized. Although this is useful for preventing users from displaying unmasked data when automated, it is even better if logs with PII are masked during ingest time.

Now that we have validated that our pattern works, we can use OpenPipeline to ensure that any log matching the DQL processor rule will undergo a series of processes that transform the log events during the ingest process.

Step 6: Set up OpenPipeline for real-time masking

Let’s configure OpenPipeline to mask email addresses at log ingestion.

1. Launch OpenPipeline in Settings app: Navigate to OpenPipeline using search or the CTRL+K shortcut. Then select Logs from the OpenPipeline menu.
2. Create a New Pipeline:
   • Go to Pipelines and create a new pipeline.
   • Provide a meaningful name (e.g., “Mask Email Addresses”).
   • In the Processing tab, add a DQL processor.
3. Configure the DQL Processor:
   • Name the processor (e.g., “mask email address in astroshop OR checkout OR aks-playground”).
   • Replace the default matching condition true with our patterns defined:

   • ( 
     k8s.container.name == "astroshop" OR  
     k8s.container.name == "checkout" OR  
     k8s.container.name == "aks-playground" 
     ) AND 
     // Match only logs that have specific phrases 
     ( 
     matchesPhrase(content, "order confirmation email sent to ") OR matchesPhrase(content, "find credentials for series ") OR  
     matchesPhrase(content, "customerID=") OR  
     matchesPhrase(content, "customerRegistrationProcess-") OR  
     matchesPhrase(content, "forgottenPasswordProcess-") OR  
     matchesPhrase(content, "Deleting userId: ") OR  
     matchesPhrase(content, "Check loyalty account exist for email: ") OR 
     matchesPhrase(content, "advantageLinkEmailProcess-") OR 
     matchesPhrase(content, "GET /reminder/") 
     )

   • Define what action should be applied when the condition matches, by defining the DQL processor:

     fieldsAdd content = replacePattern(content,  
      
     " 
     << 
     ( 
      'find credentials for series ' |  
      'order confirmation email sent to ' |  
      'customerID=' |  
      'customerRegistrationProcess-' |  
      'forgottenPasswordProcess-' |  
      'Deleting userId: ' |  
      'Check loyalty account exist for email: ' |  
      'advantageLinkEmailProcess-' |  
      'GET /reminder/' 
     ) LD:email  
     >> 
     '@' 
     ",  
      
     “xyz”)

While it is suggested to test your new masking with sample data, you can alternatively input ‘example’ as text and select to save the processor and pipeline, as we have validated them earlier in our Notebook.

4. Set up and define Dynamic Routing:

• Select the Dynamic routing tab
• Create a new Dynamic route named “Email ID masking for specific container names.”
• Use the same condition as the matching condition before.
• Connect the route to the pipeline you’ve just created.

Step 7: Validate the masking

To confirm the email addresses are masked:

1. Re-run the DQL query in your Dynatrace Notebook, but this time without the parsing.
2. Check the content field in the logs to ensure email addresses are replaced with xyz.

For example, marie_currie@example.com should now appear as xyz@example.com.
We can verify this by looking for these logs, and we can observe that the email addresses have been fully masked without any parsing at query time. Additionally, you can notice that the logs have the dt.openpipeline.pipelines attribute attached with the pipeline that was used during ingest time. This also confirms that the log was processed in that pipeline.

Conclusion

In this blog, we used email masking at ingest to demonstrate how to effectively obfuscate various types of sensitive data in logs using OpenPipeline, protecting sensitive data in real-time during the ingest process. By leveraging OpenPipeline’s powerful DQL-based processing and dynamic routing capabilities, you can precisely target specific Kubernetes containers or just any log source with patterns to enhance data privacy and security.

Take the next step

Explore OpenPipeline’s advanced features to:

• Mask sensitive data like SSNs, IDs, and more
• Streamline log management workflows with dynamic routing
• Build custom pipelines in Dynatrace to control your log data

Streamline Compliance

For comprehensive data-subject rights management, consider the Sensitive Data Center. app. Efficiently manage end-user personal data requests in Grail, with support for regulations like GDPR and CCPA.

…

Addendum: Exploring this approach in the Dynatrace Playground tenant:

A sample dataset, preloaded with tailored pattern matching and masking for the following steps and explanations, is available in the Dynatrace Playground tenant within this Dynatrace Notebook for hands-on exploration.

The individual steps can also be observed in the playground tenant with the logs ingested from the astroshop demo, while unfortunately, you lack the permissions to configure new pipelines and processors in Playground tenant.

If you look to test this within your personal tenant or a free trial tenant without actual logs containing PII, or running Astroshop yourself, you can download a copy of this Notebook or test with the pattern below, using the data command to generate sample data in a Dynatrace Notebook.

data  
record(content = "2025-04-22 11:46:38 [ERROR|[203.0.113.13] |K9p8Q3xJwZrStUv4XyZ7AbC2n5M6h8T1v0L2r4 |com.example.exmplstore.security.examplePersistentTokenRepository] Can't find credentials for series marie_curie@example.com", log.source = "bash-script"), 
record(content = "2025-04-22 13:50:43 [INFO |[203.0.113.112] |A1b2C3d4EfGhIjK5LmN6oPq7RsT8uVw9XyZ0 |class com.example.exmplfacades.order.exampleCheckoutLogger Checkout ABC] Receive API Request:method=placePayOrder, cartCode=302132310, customerID=freddie@example.com|com.example.exmplstore.checkout.request.PayPlaceOrderRequestDto@3d3a53d5|", log.source = "bash-script"), 
record(content = "2025-04-22 13:58:24 [INFO |||com.example.exmpl.BusinessProcessLoggingAspect] Finish Action: [ PerformSubscriptionAction ], BusinessProcessCode: [ customerRegistrationProcess-martin_luther@example.com-1745294290737], OrderCode:[ n/a ]", log.source = "bash-script"), 
record(content = "2025-04-29 10:16:13 [INFO |||com.example.exmplmarketing.action.exampleSendCustomerNotificationAction] Successfully sent email forgottenPassword message for process forgottenPasswordProcess-jane_austen@example.com-1745885767984", log.source = "bash-script"), 
record(content = "2025-04-29 10:24:26 [INFO |[203.0.113.14] |M3n4P5q6RsTuVwXyZ7aBc8DeF9gHi0JkL2|com.example.exmpl.UserDeleteInterceptor] Deleting userId: cleopatra@example.com, actioned by userId: anonymous", log.source = "bash-script"), 
record(content = "2025-04-29 10:24:36 [INFO |[203.0.113.19] |T2u3V4w5XyZaBc6DeF7gHi8JkL9mNo0PqR1|com.example.exmpl.AdvantageApiClient] Loyalty: Check loyalty account exist for email: leonardo_davinci@example.com , wodCorrelationId 2fee137d-915e-46fb-b390-c69aae3f150", log.source = "bash-script"), 
record(content = "2025-04-29 10:22:47 [INFO |||com.example.exmplbusproc.aop.BusinessProcessLoggingAspect] Begin Action: [ exampleAdvantageLinkEmailAction ], BusinessProcessCode: [ advantageLinkEmailProcess-albert_einstein@example.com-1745886166709], OrderCode:[ n/a ]", log.source = "bash-script"), 
record(content = "10.96.152.11 - - [29/Apr/2025:10:33:43 +1000] \"GET /reminder/shakespeare@example.com/1 HTTP/1.1\" 200 90 \"https://www.example.com/my-account/order-details/306399901\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36\" 203.0.113.145 - T2u3V4w5XyZaBc6DeF7gHi8JkL9mNo0PqR1 100", log.source = "bash-script") 
 
| filter matchesPattern(content, "LD '@' [A-Za-z0-9'_']* '.' [A-Za-z]* (LD | EOS)")

The post How to mask PII like email addresses appearing in logs with Dynatrace: An advanced use case appeared first on Dynatrace news.

Log forwarding includes metrics, spans, and events

Logs are a core pillar of observability, and in many organizations, logs serve at least a dual purpose. They drive day-to-day troubleshooting, root cause analysis, security investigations, and many other use cases.

Logs also need to be available for compliance audits, Business Intelligence (BI) analysis, and traditional Security Information Event Management (SIEM) solutions. The challenge is doing this without relying on bolted-on forwarders or rigid, costly integrations.

With the introduction of the new Dynatrace OpenPipeline capability, we’re providing you with the freedom to precisely define your organization’s forwarding, paired with flexible and unified log collection and processing.

_{Figure 1. OpenPipeline provides rich configuration and customization options for each forwarding setup.}

Update – June 2026

Data egress will be in General Availability starting June 17, 2026.

OpenPipeline now supports forwarding for all data types ingested into Dynatrace, including metrics, spans, and events, giving you a single, unified pipeline for your entire observability dataset. The same principles described below apply across the board: forward before or after processing, export in open formats, and route to the storage or third-party destination of your choice, regardless of signal type.

Read on for the full details of how OpenPipeline forwarding works.

Centralized log collection and processing

OneAgent, Cribl, OpenTelemetry, AWS Firehose, journalD, Logstash, and Fluent Bit … Have you heard of any of these tools? Maybe you’re using all of these at the same time.

With Dynatrace, you have the choice of using one or all of these tools simultaneously. Dynatrace OpenPipeline allows you to process and transform them in the same way, regardless of their source, and to move seamlessly from one shipping method to another. Just as our customer, United Wholesale Mortgages, successfully consolidated their tools while moving logs off Splunk.

Forwarding logs un/processed

Depending on your industry or geography, you might be required to retain logs in an unprocessed state on third-party-managed storage, just as you might have used tape backups in the old days. With Dynatrace OpenPipeline, you can configure a forward-before-process to retain log events in an unaltered state.

Alternatively, you can first send log events to your defined processing pipeline, extract or convert the logs into metrics, and even drop unnecessary details before determining what to forward-after-processing to your object storage.

The last step in the pipeline is to decide whether to route log events to a Dynatrace Grail® bucket for retention or drop them.

Overcome vendor lock-in with precise, open forwarding

While other log management solutions provide proprietary log archive formats, you can overcome these challenges with Dynatrace and log forwarding for data archiving.

The Dynatrace OpenPipeline log forwarding capability delivers log events in a standardized and compressed NDJSON (Newline Delimited JSON) (.JSON.GZ) archive format, for which you can define the details, such as segmentation and filename prefixes.

Residing on destinations like AWS S3 or Azure Blob storage, other 3^rd party solutions such as Microsoft Sentinel, Snowflake, or Tibco Spotfire can easily leverage these archives for use cases not covered by your observability platform.

Cost-effective long-term retention for logs

Dynatrace Grail offers cost-effective log retention for up to 10 years, accessible at any time with the same speed as on day one of ingestion, supporting a daily log ingest volume of 1 PB per tenant.

Many of our customers already retain years’ worth of logs today, always accessible without the headaches of re-ingestion, re-indexing, or storage tiering and scaling concepts.

Still, there are use cases where aged logs become low-value, low-touch data.

You may want to keep certain log events available in Grail for the first 60 or 90 days for troubleshooting purposes, and then store a copy in an archive for years.

NDJSON-exported log archives are an ideal file format for further increasing cost-effectiveness and are ready for handover to other teams or parties for investigation, thanks to the standardized JSON.GZ format. These are especially valuable for use cases like:

• Reviewing access logs during a post-mortem security incident
• Providing transaction logs to fulfill a court order
• File access audit-trial review

Tool consolidation with OpenPipeline and log forwarding

Log archiving presents an opportunity and an integration strategy for solutions that don’t yet integrate seamlessly with Dynatrace.

More importantly, OpenPipeline provides you with processing and transformation capabilities, volume control, and a reliable cadence for transmitting your log events.

You no longer have to rely on custom-defined scripts for the splitting and shipping of your logs. And there are no worries about truncating large log events; Dynatrace supports up to 10 MB.

Forget about all the retransmission headaches with custom log shippers when a connection breaks; all this is baked natively into Dynatrace components such as OneAgent, ActiveGate, and others.

Start leveraging OpenPipeline and log forwarding today and retire your third-party forwarding tool stack.

For complete details, go to Log Forwarding in Dynatrace Documentation.

The post Flexible vendor-agnostic log forwarding with OpenPipeline appeared first on Dynatrace news.

Empowering real-time insights through petabyte-scale data processing of all your data

Modern cloud native systems generate massive volumes of telemetry data, spanning logs, metrics, events, and traces. Extracting real-time, actionable insights and enabling meaningful alerting and root cause analysis requires more than just centralized data storage; it requires a scalable data pipeline. Without such a pipeline, even the most advanced analytics tools are constrained by data flow and preparation bottlenecks. To address these challenges, we launched OpenPipeline, our unified data ingest solution for the Dynatrace® platform. OpenPipeline is a customizable, scalable data pipeline for ingesting, transforming, and harmonizing data for reliable analysis and correlation.

With its latest enhancements, OpenPipeline supports real-time data processing even at petabyte scale, by reliably handling high volumes of logs, traces, and other telemetry. This level of scalable pipeline processing is essential for delivering timely insights, enabling Dynatrace to:

• Detect anomalies across distributed systems in real time
• Perform proactive security analytics and real-time vulnerability detection
• Offer real-time insights into your business processes

Scale is not only about the sheer amount of data signals. It’s also represented in the size of individual signals that can be handled. An increasing number of use cases, such as parsing JSON/XML request bodies, capturing detailed audit logs, exploring transactional data dumps, or analyzing input vectors from ML model logs, require the ability to efficiently process large log files. With the release of Dynatrace version 1.311, OpenPipeline now supports log records up to 10 MB in size, empowering you to use Dynatrace for high-volume use cases. For more information, have a look at this Dynatrace support for large log records blog post.

Break silos with unified ingestion across all data types

Alongside scale, OpenPipeline provides unified ingestion across all data types, removing silos, simplifying integration, and ensuring that all data, regardless of format or source, is enriched, contextualized, and processed in the same tool, ready for advanced analytics.

Recent enhancements to data type processing include:

• Spans: You can now configure how spans are processed, including dropping specific fields or entire records, and assign a security context for fine-grained, record-level access control. Spans can also extract metrics and route them into defined target buckets. Full ingest functionality for spans is coming soon.
• RUM data: Ingesting Real User Monitoring (RUM) data, including user events, sessions, and associated metrics, is currently in preview and will soon be made generally available for all Dynatrace SaaS customers.
• Events: OpenPipeline now supports custom processing rules for a broad range of event types, including security events, software development lifecycle (SDLC) events, business events, and Dynatrace internal system-level events.

Simplified pipeline management from setup to optimization

Historically, setting up pipelines for data ingestion involved time-consuming configuration of parsing, field mapping, and transformation logic for each data source. OpenPipeline now streamlines this process with ready-made processor bundles for widely used technologies and formats, accelerating data onboarding and standardizing data handling at scale for ingest sources such as:

• Hyperscaler support, including AWS and Azure
• Web servers like Apache, IIS, JBoss, HAProxy, or Nginx
• Programming languages like .NET, PHP, Java, Python, or NodeJS
• Databases and app frameworks like Elastic or Cassandra

With just a few clicks, you can apply a processor to a source, ensuring all fields are parsed correctly, attributes are renamed, and data structures are accurately standardized. Each bundle includes example records that can be tested interactively, with further customization available to meet specific requirements. This allows your teams to onboard new telemetry streams in minutes instead of hours. We’ll continue to expand our support to cover more technologies in the future. You can also create your own processors to handle any custom format.

Reveal hidden value with data transformation

Upon ingestion, modern observability, security, and business data can be structured in varying, often complex forms. OpenPipeline offers advanced transformation capabilities that allow for easy extraction of relevant data beyond the creation of simple events and metrics:

• Extract values from deeply nested JSON fields. For example, extract the threat identifier from a JSON-formatted security event and convert it into a security metric.
• Create unified metrics from different data types. For example, consolidate error codes from both logs and spans into a single metric for cross-service comparison.

Stay informed: Real-time visibility and alerting for your pipelines

To optimize pipelines, teams need visibility into their performance. OpenPipeline now exposes detailed metrics at key processing stages: ingest, routing, and output, as well as not-stored-records. With these metrics, you can instantly verify any pipeline configuration and detect anomalies early.

These metrics power:

• Real-time microcharts within the OpenPipeline user interface, showing data trends and ratios over the last 30 minutes.
• A ready-made dashboard used to explore daily or weekly summaries, historic volume trends, and detailed routing stats by pipeline or source.
• Smart alerting. Traffic fluctuations on ingest are common, so you can leverage AI-powered dynamic baselining to raise custom alerts and detect anomalies before an issue occurs.
• Automated operations that trigger notifications or perform autonomous remediation steps once an ingest anomaly or traffic volume deviation is detected.

Easy transition to OpenPipeline for existing customers

For Dynatrace SaaS customers using classic pipelines, we’ve simplified the transition to OpenPipeline. Start utilizing OpenPipeline for new data sources while keeping your existing log processing configurations fully operational and uninterrupted. This side-by-side, risk-free approach supports gradual adoption and allows you to modernize your data processing without disrupting ongoing workflows.

Get started today

OpenPipeline is now available to all customers running the latest version of Dynatrace SaaS. Already today, you can:

• Benefit from processing a broad range of data types, including logs, traces, events, and RUM data (currently in preview).
• Take advantage of processing the full payload of large log records, parse embedded XML or JSON structures, parse events or metrics from complex log records, or use them for deep search analytics and forensic use cases—all without splitting individual log lines into separate events. Leverage the built-in processor bundles to get started with popular formats.
• Explore the ready-made OpenPipeline dashboard on the Dynatrace Playground.
• Use real-time pipeline metrics to tune and optimize your configurations.

The post Latest OpenPipeline upgrades simplify high-volume, real-time data processing appeared first on Dynatrace news.

In this blog post, we guide you through configuring a project to visualize real-time CI/CD build and release data for your Azure DevOps pipelines. The resulting visibility enhances collaboration with transparent data, increases productivity by automating monitoring tasks, and enables teams to detect issues proactively. With insights into your Azure DevOps pipelines, you can continuously improve processes, ensuring smoother and more reliable deployments over time.

A notebook version of the blog is also available on the Dynatrace GitHub repository. Download the AzureDevOps - Dynatrace Integration.json file and upload it to your Notebooks app in Dynatrace.

Prerequisites

Before we begin, ensure you have the following:

• Access to your Dynatrace Tenant and permission to create tokens.
• Your tenant ID, which can be found in your environment URL: https://<YOUR_TENANT_ID>.live.dynatrace.com/.
• A token with Ingest Logs v2 scope.
• A token with Write Settings scope.

Create webhooks in Azure DevOps

First, we need to create two service hooks subscriptions in Azure DevOps: one for Builds Completed and one for Release Deployment Completed.

1. Navigate to https://{orgName}/{project_name}/_settings/serviceHooks.
2. During the configuration, do not apply any filters.
3. In the settings page of the subscription, fill in the following fields:
   • URL: https://<YOUR_TENANT_ID>.live.dynatrace.com/api/v2/logs/ingest
   • HTTP Headers: Authorization: Api-token <YOUR_LOG_INGEST_TOKEN>
   • Ensure the text above is copied exactly, replacing only the token.
   • Change “Messages to send” and “Detailed Messages to send” to Text.

Add a new Grail logs bucket

Follow these steps to create a new logs bucket in Dynatrace. Note: This step is optional. You’re welcome to use an existing Grail logs bucket.

1. Open the Storage Management app in your tenant: Select CTRL/CMD + K and enter Storage.
2. Create a new bucket by selecting + in the top right corner.
3. Name the bucket azure_devops_logs.
4. Set the retention time as desired.
5. Set the bucket type to logs.

Configure OpenPipeline with log processing rules

Using OpenPipeline™, you can easily define a rule that routes all relevant log lines you created in the previous step into the bucket. This process also performs the necessary processing steps, such as renaming fields or transforming log data into dedicated events.

1. Open the OpenPipeline app and select Logs in the left pane.
2. Select the Pipelines tab and create a new one by selecting + Pipeline.
3. Name the new pipeline AzureDevOps.
4. Go to Dynamic Routing and create the following new rule:

matchesPhrase(eventType,"ms.vss-release.deployment-completed-event") OR matchesPhrase(eventType,"build.complete")

5. From the Pipeline dropdown, select “AzureDevOps”.
6. Return to your pipelines and open “AzureDevOps”.
7. Under the Storage section, add a new processor > Bucket assignment, set a name and select the azure_devops_logs bucket from the dropdown.

8. Next, open the Processing tab and define a couple of rules.
   1. First, create a new Rename Fields rule and call it “Rename Build Fields” where the left value is the new field name and the right value is the existing field name.

1. resource.buildNumber: buildNumber
2. resource.result: result

1. 2. Second, create another Rename Fields rule called “Rename Release Fields” where the left value is the new field name and the right value is the existing field name.

1. resource.stageName: stageName  
2. resource.project.name: projectName 
3. resource.deployment.release.name: releaseName 
4. releaseStatus:resource.environment.status: releaseStatus

Use the following sample data to verify your rules are working as expected.

Release event sample data:

{ 
       "timestamp": "2024-11-11T15:14:51.104000000-05:00", 
       "loglevel": "NONE", 
       "status": "NONE", 
       "createdDate": "2024-11-11T20:14:50.6300269Z", 
       "detailedMessage.text": "Deployment of release Release-946 on stage Staging succeeded. Time to deploy: 00:14:14.", 
       "dt.auth.origin": "dt0c01.YFMJ6LUO43SFFDW2SF7EW5YZ", 
       "eventType": "ms.vss-release.deployment-completed-event", 
       "id": "1862ab11-c0d4-451a-8b9b-0dfe7f517297", 
       "message.text": "Deployment of release Release-946 on stage Staging succeeded.", 
       "resource.environment.status": "succeeded", 
       "resource.project.name": "devlove-alpha", 
       "resource.stageName": "Staging", 
       "resource.deployment.release.name": "Release-946" 
    }

Create an Azure DevOps dashboard and visualize log data

Now that we have ingested the logs coming from AzureDevOps, let’s visualize the data to get better and quicker insights into our CICD pipelines.

1. Go to the AzureDevOps Git Repository and download the AzureDevOps Dashboard (on Logs).json file.
2. Within Dynatrace, open the Dashboards app and select Upload at the top left corner.
3. Upload the JSON file to start visualizing your Azure DevOps data.

Extract “release” and “build” events

Taking this a step further, we can convert the ingested log data to SDLC (Software Development Lifecycle) events in case we detect a new release or build and discard the related log line afterward.

This helps reduce the number of stored log data and supports further platform engineering use cases, such as calculating DORA metrics, automating development processes, or observing the health of your engineering pipeline.

Disclaimer: The following instructions extract Business events from log data. Once supported by OpenPipeline we propose to extract Software Development Lifecycle Events (SDLC events), which are the preferred way of storing the extracted information.

Note: You may need to request a Log Content Length (MaxContentLength_Bytes) increase depending on how many steps your Release Events have. The integration generates ingest costs for logs and metrics (according to your rate card), depending on how many build/release events you ingest.

1. In Dynatrace, open OpenPipeline:
   • Go to OpenPipeline > Logs > Pipelines > AzureDevOps.
   • Navigate to the Data Extraction section.
2. Next, create a Business Event Processor rule, for all build events, using the following parameters:
   • Name: Build Result
   • Matching condition: matchesPhrase(eventType,"build.complete")
   • Event type: field name eventType
   • Event provider: Change to Static String: AzureDevOps
   • Fields to extract: result, buildNumber, resource.reason

1. Finally, we use another Business Event Processor rule, to extract release events:
   • Name: Release Result
   • Matching condition: matchesPhrase(eventType,"ms.vss-release.deployment-completed-event")
   • Event type: field name eventType
   • Event provider: Change to Static String: AzureDevOps
   • Fields to extract: stageName, projectName, releaseName, releaseStatus, resource.deplyoment.startedOn, resource.deployment.completedOn

Extract Davis® AI events (optional)

Besides transforming the log line into SDLC events, you can also extract Davis events in case something goes wrong. These events can be used to create an alert or trigger a (remediation) workflow.

1. First, add an event if a build has failed. Add a new “Davis event” processor using the following data:
   • Name: Build Complete Failed
   • Matching condition: matchesPhrase(eventType,"build.complete”) AND result== “build”
   • Event description: Unable to generate build {buildNumber}

Note: You can change the event.type in case you want to increase the severity level.

2. Next, extract a Davis event if the deployment is rejected:
   • Name: Release deployment rejected
   • Matching condition: matchesPhrase(eventType,"ms.vss-release.deployment-completed-event”) AND releaseStatus== “rejected”
   • Event description: Unable to deploy release {releaseName}

Discard logs (optional)

Now that we have successfully converted the log event into a SDLC (Business) and a Davis event, you can disable the storage assignment rule. This will reduce the amount of data stored in Grail and help you saving some money (and consequently speeding up your log queries). Go to OpenPipeline > Logs > Pipelines > AzureDevOps.

1. Open the OpenPipeline app and select the pipeline we created before.
2. Select the tab Storage and change the matching condition to false.

Analyzing data

After deleting the logs events we need to adapt our dashboard to consume business instead of log data. To speed up things you can upload a ready-to-use dashboard into your environment:

1. Go to the AzureDevOps Git Repository and download the AzureDevOps Dashboard (on BizEvents).json file.
2. Within Dynatrace, open the Dashboards, select Upload at the top left corner, and select the JSON file.

Additionally, we recommend that you create a segment to filter all of your monitored entities across different apps.

1. Go to the Segments app and create a new segment by selecting + in the top right corner.
2. Rename the segment to AzureDevOps.
3. Select + Business events and add the following filter: event.provider = AzureDevOps
4. Select + Logs and add the following filter: dt.system.bucket = azure_devops_logs
5. Select Preview to validate the filters and Save once you’re done.

What’s next

By following these steps, you’ll be able to seamlessly integrate Azure DevOps with Dynatrace, enabling efficient log management and insightful data visualization.

Happy monitoring! 🚀

The post Insights into your Azure DevOps pipelines appeared first on Dynatrace news.

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. To unlock business value, the data must be:

• Accessible from anywhere. Data has value only when you can access it, no matter where it lies.
• Easy to access. Simplicity accelerates time-to-value and reduces implementation and maintenance costs.
• Real time. Agile business decisions rely on fresh data.
• Precise. Accuracy provides the confidence needed for business automation.
• Contextualized. Metadata enrichment improves collaboration and increases analytic value.

The Dynatrace® platform continues to increase the value of your data—broadening and simplifying real-time access, enriching context, and delivering insightful, AI-augmented analytics. Our Business Observability solution is a prominent beneficiary of this commitment.

Business events: Delivering the best data

It’s been two years since we introduced business events, a special class of events designed to support even the most demanding business use cases. Since then, many of our customers have embraced the opportunity to explore and adopt new business observability use cases. Most of these leverage the unique capability of Dynatrace OneAgent® to extract business data from in-flight application payloads—without writing any code. Other data sources, including APIs and log files—are used to expand access, often to external or proprietary systems. Enhancing access to business data from log files is an important priority, and OpenPipeline makes this a reality.

Dynatrace OpenPipeline is a new stream processing technology that ingests and contextualizes data from any source. You can now use OpenPipeline to extract business events from log files, complementing OneAgent as a primary source of business data. This is especially important when legacy or proprietary systems don’t meet OneAgent’s environmental prerequisites; in these cases, log files are usually the preferred source of business data.

In fact, it’s likely that some of your critical business systems already write business data to log files. For years, logs have been the dominant approach many observability vendors have taken to report business metrics on dashboards. You might still be using one of these tools despite the drawbacks, which include a lack of IT context, constraints on data privacy and data retention, and, as the only convenient source of business data, a limiting lack of breadth. OpenPipeline makes migrating these use cases to Dynatrace easy, helping you overcome these limitations to realize greater value.

Use case categories

It’s helpful to categorize the nearly unlimited use cases covered by Business Observability. Two of the most important categories are:

• Business reporting, analytics, and automation. Track business metrics, key performance indicators (KPIs), and service level objectives (SLOs)—automatically and in context with IT infrastructure and services—to promote collaboration between business and IT teams.
• Business process monitoring and optimization. Monitor and optimize business processes with real-time visibility into process KPIs and detailed analytics for each step to improve customer satisfaction, increase operational efficiency, and reduce cost.

Most of the use cases in these two broad categories benefit from the flexibility that comes from multiple available sources of business data. For example:

• An airline’s reporting and analytics dashboard includes data showing flights, passengers, available seats, passenger load, revenue per passenger, flight crew staffing, arrival delays, and customer satisfaction metrics. The data may come from a mix of systems, including a departure control system (DCS), an airline reservation system (ARS), a legacy inventory control system, and a SaaS-based Voice of the Customer (VoC) solution.
• A financial institution’s loan origination business process includes a series of process milestones, including loan application, credit scoring, application review, contract generation, CRM, and loan funding. All of these steps are critical components of the process, likely to be implemented using different systems. Furthermore, to understand process health, individual steps must be viewed in the context of the entire process. For this, we use Business Flow to track, analyze, and optimize complex business processes, treating the process as an observable IT and business asset.

Use OpenPipeline to extract business events from logs

Using OpenPipeline, you can ingest log data into Dynatrace from a wide range of sources, including OneAgent, Extensions, the Log ingest API, and OpenTelemetry. The pipeline includes a configurable business event processor as part of the data extraction stage; this processor is used to extract and convert relevant business data into business events.

There are many benefits to extracting business events from log files. These include:

• Improved data privacy. Sensitive business data is separated from IT observability data.
• Improved data management. Fine-grained permission and retention policies can be tailored to individual business use cases.
• Reduced storage and query overhead for business use cases. Business events are a small, often negligible subset of log data.
• Simplified and enhanced analytics efficiency. Business events from log files are unified with business events from OneAgent, RUM, and API.

Extracting business events from logs: Configuring OpenPipeline

The OpenPipeline app guides you through the three configuration steps required to extract business events from log files:

1. Identify the source of the log file.
2. Define OpenPipeline’s dynamic routing matching criteria used to route the log lines of interest to a second pipeline.
   
3. Create the target pipeline. Use the Data extraction tab to define the matching condition that creates the business event. You need to include static or dynamic Event type and Event provider fields.
   

Within the target pipeline, you can also define processing rules, extract metrics, set the security context, and define retention periods. Log data is then processed accordingly, stored in Dynatrace Grail™ causational data lakehouse, and available for your Business Observability use cases.

The value is in the data

OpenPipeline broadens Dynatrace’s unified access to the best data to deliver the best value. Now’s the time to see how it can benefit your organization.

For more details about OpenPipeline read this Dynatrace OpenPipeline blog post.

The post OpenPipeline: Simplify access to critical business data appeared first on Dynatrace news.

Unrealized optimization potential of business processes due to monitoring gaps

Imagine a retail company facing gaps in its business process monitoring due to disparate data sources. Due to separated systems that handle different parts of the process, the view of the process is fragmented. On top of that, the data sources are inconsistent. While some data comes from modern systems with APIs, other data stems from older systems that generate log files, and some data originates from external vendors. This inconsistency complicates uniform gathering and data analysis, resulting in incomplete or inaccurate insights. This scenario is shared among large organizations that rely on multiple internal and external systems for data collection.

Get business process observability across data silos with Dynatrace OpenPipeline

Traditional observability platforms often focus on technical metrics and logs, which are essential for troubleshooting, but they don’t take business value into consideration. Dynatrace OpenPipeline goes beyond this by offering the possibility of extracting business-relevant data from logs and using them to monitor end-to-end business observability.

In our retail company example, older systems are involved in shipping the order. They only produce logs that have not yet been monitored from a business perspective, although they contain valuable business information.

By leveraging Dynatrace OpenPipeline, the retail company can integrate data from all sources across the whole process, including Dynatrace OneAgent^®, logs, and external business tools. This approach ensures that the company can easily track the entire journey from online order to its successful delivery.

Benefits of capturing business events

Logs often contain valuable insights into your business; however, this information can be difficult to process, particularly as you probably only need data from some specific log lines.

Dynatrace OpenPipeline extracts this business information from logs and stores it as a separate data type, so-called business events. This has several advantages:

• Separation of concerns: Logs are often used for technical troubleshooting. Extracting business events allows for a clear separation between technical and business-relevant data.
• Access control: Different teams can access different types of data. For example, support teams might access logs for troubleshooting, while business teams access business events for analytics.
• Ease of access: Having business events in a uniform format simplifies querying and visualization, making it easier to analyze and derive insights.

How to find valuable business information in logs

In our example of a retail company, we need to extract business information from shipping logs to know if our order has already been shipped. See a typical log file with shipping details below.

From this log file, we can identify and capture the following business information:

1. Correlation ID – The Correlation ID is part of the log payload and identifies this event in the context of a business process.

2. Message – The message is also part of the log payload and contains information on which part of the business process it represents.

3. Context – The context contains information on which IT system handled this part of the business process.

4. Status—The status information tells us if something was successful or if we hit an issue.

Use OpenPipeline to identify relevant log events

To turn this information into a business event for analytics, we utilize the capabilities of Dynatrace OpenPipeline to identify relevant log events, parse the data, and create a business event out of it. The approach is the following:

1. Go to the OpenPipeline app (available by default in your Dynatrace tenant)

1. Create a route that specifies which logs should be processed. For example, you can only process log events coming from a specific ingest source or containing a specific phrase in its log message.
2. Define a pipeline that will process these logs.
3. Configure the data extraction rules within the pipeline. This involves:

1. • Extracting correlation IDs: Identify and extract correlation IDs from the log content. Rename these IDs to make them uniform with other events (for example, rename “order ID”).
   • Extracting the message: The relevant message will be extracted from the log content and used as part of the business event.
   • Parse out contextual information such as the context (which is the related host) or status information in our case.
   • Defining event types: Specify the event type using the extracted message and provider information. For example, you might name the provider retail.logs for consistency.

The extracted information is re-ingested into the pipeline as a new business event. The event is then stored in Grail™ datalake house and can be used for further analysis or process visualization.

Do you want to dig deeper into extracting the data? Watch the dedicated Dynatrace Lab episode with Andreas Grabner and Alistair Emslie for a step-by-step guide on how this is done:

Turn your business data into tangible value

The extracted business data enables you to optimize your processes and gain valuable new insights. You can easily create a dashboard like the one below: it provides the example retail company with real-time data and KPIs, such as the success rate of shipping orders. Unlike traditional dashboards focusing on specific applications or technical aspects, this dashboard tracks the performance of the complete business journey with business KPIs and metrics, utilizing the information extracted from the log files.

Of course, you can also leverage the full power of the Dynatrace platform, like AI-powered monitoring of your business KPIs through Davis^® AI. You can also visualize the end-to-end process flow in our Business Flow app, which enables you to identify delays, errors, and exceptions, providing insights into IT and business-related issues.

Learn more about the capabilities of Dynatrace OpenPipeline

If you’re struggling with gaps in your business process monitoring caused by disparate data sources and outdated systems, consider how Dynatrace can transform your approach.

The post Leverage logs for an end-to-end view of your business processes via Dynatrace OpenPipeline appeared first on Dynatrace news.

Dynatrace integrates with AWS Elastic Container Registry (ECR) to enable visibility, orchestration, and prioritization of cross-container-registry vulnerability findings. This integration provides a single pane of glass for container image scans of your containerized applications and is part of a larger effort to enrich vulnerability findings with runtime context.

In complex multicloud environments, vulnerability findings are often siloed between build-time and run-time tooling. Thus, getting a holistic view of security risks is challenging.

Dynatrace addresses this issue by providing unified ingest and analysis of container vulnerability findings across cloud and container registries. This ensures that SecDevOps has a continuous and comprehensive understanding of its security posture.

In addition, security findings detected during the build phase and in your artifact registries, such as Amazon ECR, might not be relevant to your production-critical applications. By enriching runtime context from the monitored entities, Dynatrace helps filter out the noise, prioritize critical findings, and focus your remediation efforts on what truly matters for your production environment.

Key Steps in the Integration Process

Container image scanning

Amazon ECR scans container images for vulnerabilities. You can choose between basic and enhanced scanning.

Data ingestion

The vulnerability findings are pushed into the Dynatrace platform through AWS Event Bridge via the dedicated security ingest endpoint powered by OpenPipeline^TM. You can set it up using an AWS CloudFormation template provided by Dynatrace. For instructions, see the documentation.

Data mapping

The ingested data is mapped according to the Dynatrace Semantic Dictionary, ensuring a unified format for analysis.

Analysis and automation

Once the findings are ingested, you can visualize, analyze, and automate in Dynatrace with Dashboards, Notebooks, and Workflows.

Use cases

Once security findings and scan events are ingested into Dynatrace Grail™, you can analyze them and perform automation tasks, leveraging the uniform data format.

Amazon ECR ingested data can be consumed as follows:

• Dashboards: Use the provided sample dashboards or create custom visualizations of the security findings and scan events.
• Notebooks and Security Investigator: Use vulnerability findings as an additional dimension for threat hunting and forensic investigations.
• Workflows: Automate the orchestration of critical vulnerability findings by creating alerts and tickets.

Explore individual use cases in Dynatrace Documentation:

• Visualize and contextualize security findings
• Automate and orchestrate security findings
• Discover coverage gaps in security scans
• Visit the Dynatrace Hub and leverage our Amazon ECR app

Get started

Visit Dynatrace Documentation and get started setting up your Amazon ECR data integration.

The post Enrich Amazon ECR vulnerability findings with runtime context appeared first on Dynatrace news.

You’re in the middle of threat-hunting activities using Dynatrace and discover that some of your assets are trying to resolve a suspicious DNS name to extract data via covert channels like DNS tunneling. You now want to detect such events automatically by creating a custom Dynatrace security event. This blog post explains how to achieve this goal.

Following the Route53 log use case in Dynatrace threat-hunting documentation, a DNS query record is structured in the following way:

{
    "version": "1.100000",
    "account_id": "684399999995",
    "region": "us-east-1",
    "vpc_id": "vpc-026e4d09999995",
    "query_timestamp": "2024-08-17T07:24:57Z",
    "query_name": "250607.56755a335668636d52634969497343694167496e4a6c59584e766269493649.434a4762334a696157526b5a5734694c416f6749434a6b5a5852686157787a.tiitha-maliciousdomain.com.",
    "query_type": "A",
    "query_class": "IN",
    "rcode": "NOERROR",
    "answers": [
        {
            "Rdata": "34.667.67.208",
            "Type": "A",
            "Class": "IN"
        }
    ],
    "srcaddr": "172.31.1.6",
    "srcport": "42241",
    "transport": "UDP",
    "srcids": {
        "instance": "i-0b56150a99995e6d",
        "resolver_endpoint": "rslvr-out-c8e7c49999999fb5a"
    }
}

Let’s say that the meaningful fields you need for the security event are:

• timestamp when the DNS query was executed (query_timestamp);
• source IP where the query was performed (srcaddr);
• DNS query requested (query_name).

The JSON matcher, with its ability to parse out selected members, allows you to use the following simple DQL expression to extract only the fields you need:

json{
  timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp,
  string:query_name,
  ipaddr:srcaddr
}(flat=true)

As a result, you now have three additional fields with their respective types and values.

To add more accuracy to the query you’re about to automate, you need to extract the domain portion from the query_name field. This can be achieved with the following DQL expression:

LD* '.'? ( (LD '.' LD):domain '.' EOS)

That DQL pattern translates to something like this:

“Please extract two line-data (LD) portions from the content separated by a dot LD '.' LD) that is immediately followed by a single dot ('.') and then reaches the end of the string (EOS). A single dot and some line data might precede it.”

So, the pattern will match the highlighted DNS names below:

Adding a filter for the correct domain and limiting the query to only the last 5 minutes gives you the following DQL query to automate:

fetch logs, from: -5m
| filter aws.log_group == "/aws/route53/unguard-secla-demo/resolver-logs"
| parse content, "json{timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp, string:query_name, ipaddr:srcaddr}(flat=true)"
| parse query_name, "ld* '.'? ( (ld '.' ld):domain '.' eos) "
| filter domain == "tiitha-maliciousdomain.com"
| fields query_timestamp, srcaddr, query_name

Automate the query

Dynatrace AutomationEngine provides the ability to automate such periodic query executions. When you’re satisfied with your DQL query and are ready to automate it, select the Open with button in the lower right-hand corner of the query window.

As a result, a modal dialog will display, providing you with different possibilities about where to open the DQL query. In the current use case, you should open the query in Workflows.

Modify the default workflow

To execute the workflow automatically every 5 minutes, change the default on-demand trigger to either a cron or time-interval trigger. You can easily achieve this by selecting the trigger box in Workflows and following the instructions in the right-hand pane. You can read more about workflow triggers in Workflow schedule trigger documentation

Ingest query results as security events

The simplest way to do this is to use Dynatrace OpenPipeline. OpenPipeline allows you to create custom endpoints for data ingestion and process the events in the pipeline (for example, adding custom pipe-dependent fields to simplify data analysis in a later phase).

Set up a custom pipeline

The best way to set up a security event ingestion to Dynatrace is via Dynatrace OpenPipeline. You can create your own ingest endpoint that will then perform custom processing activities, create additional fields, store these events in separate buckets, and much more. For the current use case, we can create a custom security event pipeline to which we can send all the automated security events we have created from the Security Investigator, add respective fields for future analysis, and store them in the default security events bucket.

Now, create a general ingest endpoint for this use case and any automated detections we want to initiate from the Security Investigator in the future.

1. Go to OpenPipeline.
2. Choose Events > Security Events.
3. On the Ingest Sources tab, select + Source.
4. Give your pipeline a name, for example, Automated discovery from Security Investigator.
5. Choose a path to the new ingest source, for example, Signals.
6. Since we currently don’t want to create any dynamic rules to our pipeline, leave Routing set to static and choose the Custom events pipeline as the route.
7. Go to the Pre-processing tab and add a new processor with the type Add Fields.
8. Give the processor a name, for example, My custom processor.
9. Add the fields that you want to be automatically added when an event passes the pipeline. In our case, we will add the following fields:
   – kind == “SECURITY_EVENT”
   – provider == “Automated Detection Script”
   – category == “AUTOMATED_DETECTION”
   – provider_product == “Security Investigator App”
10. To figure out which fields to add for custom security events, refer to the semantic dictionary.
    Following the overall convention here will simplify analyzing this data later, together with other security events created by other sources.
11. You can test the processor with sample data to see what kind of event will be created based on the processor, and if you’re happy with the result, select Save.

For the current demo, this is enough, however, there are lots of things that you could configure additionally. Take a look at the Ingest custom security events via API documentation to learn more about sending security events to OpenPipeline.

Create and secure the token for event ingest

To create the token needed to ingest events, follow the steps in How to ingest data (events) documentation. When you have created the token, we recommend that you protect it in the Credential Vault instead of just using it in plain text in the automation engine.

1. Open Credential Vault.
2. Select Add new credential.
3. Select token as the credential type and give it a meaningful name.
4. Select App Engine as the credential scope.
5. Paste your ingest token into the token field and select Save.

After saving the credentials, an ID is presented to you in the form of CREDENTIALS_VAULT-12345. Write this down for later usage.

Ingest records through the OpenPipeline custom endpoint

Let’s return to our workflow and add a follow-up task for the DQL query. Choose the JavaScript task type and start writing the code to ingest security events.

Consider the following code for your function:

import { execution } from '@dynatrace-sdk/automation-utils';
import { credentialVaultClient } from '@dynatrace-sdk/client-classic-environment-v2';

export default async function ({ execution_id }) {

  const ex = await execution(execution_id);

  // load the token from the credential vault by its' ID
  const token = await credentialVaultClient.getCredentialsDetails({
    id: "CREDENTIALS_VAULT-F12345",
  }).then((credentials)=> credentials.token);

  // loop through the DQL query results and ingest them one by one
  var result = await ex.result('execute_dql_query_1');
  result.records.forEach (record => {
    ingest_event(token, record);
  });
}

async function ingest_event(token, payload) {

  payload["event.description"] = "Suspicious DNS query from AWS DNS Logs";

  fetch('https://<dynatrace_env>.dynatrace.com/platform/ingest/custom/events.security/signals', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json; charset=utf-8",
      "Authorization": `Api-Token ${token}`
    },   
    body: JSON.stringify(payload)
  }).then(function(response) {
    console.log(response);
  })
}

The code loads the token from the Credential Vault that will be used to ingest the events to OpenPipeline. Then, we iterate through the result set we got from the execute_dql_query_1 task and pass each event along with the ingest token to the ingest function, which will POST the event to the OpenPipeline ingest endpoint that we created earlier. The response to the request is shown in the console for debugging purposes.

We’re also adding a custom field called event.description to the event. You can add custom fields to your event from the OpenPipeline processing as well, but since we want to keep the ingest source as general as possible, it makes more sense to add the event-specific fields near the event, not the pipeline.

Now, give it a distinct name and test if it runs perfectly. Ingesting events to Grail via OpenPipeline should result in a status: 202 that is visible in the run_javascript_1 log console.

Fetch the security events from Grail

To view the newly created security events, follow the same principles as with fetching any other security data from Grail. Using the following DQL query, you can see all the events you have started to ingest:

fetch events
| filter event.kind == "SECURITY_EVENT"
| filter event.category == "AUTOMATED_DETECTION"

To feel the power of the semantic dictionary and to illustrate what you gain from using it, consider the following query. If you want to visualize all the security events ingested to Grail over the last 12 hours by event.category, it is quite simple with the following query thanks to the unified field names:

fetch events, from: -12h
| filter event.kind == "SECURITY_EVENT"
| makeTimeseries count(), by: { event.category }, interval:1h

The result could be a line chart like this:

Of course, you can follow your own naming convention, but this would make it much harder to view all the security events (both custom and Dynatrace-created events) in conjunction.

Operationalize the results

You can create additional workflows for your security events in Grail. For example, you can create a workflow that notifies you in Slack when a new event is created or integrate your workflows with Jira to create a ticket every time such events arise.

You can create a custom integration to your own SIEM system (similarly as we did for the OpenPipeline) for events that can enrich your security data in other environments. Consider your use case and work processes to determine the best way to approach security events available in Grail.

The post Generate security events from Dynatrace Security Investigator via OpenPipeline appeared first on Dynatrace news.

Organizations choose data-driven approaches to maximize the value of their data, achieve better business outcomes, and realize cost savings by improving their products, services, and processes. However, there are many obstacles and limitations along the way to becoming a data-driven organization. The following are some of the most pressing challenges:

• Managing cost and scale of large data. The exponential growth of data volume—including observability, security, software lifecycle, and business data—forces organizations to deal with cost increases while providing flexible, robust, and scalable ingest.
• Understanding the context. With siloed data sources, heterogeneous data types—including metrics, traces, logs, user behavior, business events, vulnerabilities, threats, lifecycle events, and more—and increasing tool sprawl, it’s next to impossible to offer users real-time access to data in a unified, contextualized view.
• Addressing security requirements. Organizations need to ensure their solutions meet security and privacy requirements through certified high-performance filtering, masking, routing, and encryption technologies while remaining easy to configure and operate.

Dynatrace is addressing these challenges with a single, built-in data ingest functionality: Dynatrace OpenPipeline™, the ultimate addition for data-driven organizations.

Introducing Dynatrace OpenPipeline

OpenPipeline is a stream-processing technology that transforms how the Dynatrace platform ingests data from any source, at any scale, and in any format. With OpenPipeline, you can easily collect data from Dynatrace OneAgent®, open source collectors such as OpenTelemetry, or other third-party tools. OpenPipeline then filters and preprocesses that data to manage and reduce costs.

OpenPipeline also includes data contextualization technology, which enriches data with metadata and links it to other relevant data sources. By putting data in context, OpenPipeline enables the Dynatrace platform to deliver AI-driven insights, analytics, and automation for customers across observability, security, software lifecycle, and business domains.

Furthermore, OpenPipeline is a data security and privacy technology that ensures data is collected and processed securely and compliantly, with high-performance filtering, masking, routing, and encryption capabilities that are easy to configure and operate.

OpenPipeline works seamlessly with Dynatrace Grail™ to handle data ingest at an unparalleled scale. During the data ingestion process, OpenPipeline enriches data and data signals with their context and simultaneously interconnects them with Smartscape®, a real-time interactive map that reflects the topology and dependencies of all data signals. This “data in context” feeds Davis® AI, the Dynatrace hypermodal AI, and enables schema-less and index-free analytics. Unlike other pipelining solutions that serve to consolidate and move data from one place to another, OpenPipeline enhances the Dynatrace platform and allows users to extract even more value from their data.

Scale beyond petabytes

Dynatrace can collect data from the full application stack without configuration—including metrics, traces, logs, user sessions, security events, business events, and more. OpenPipeline unifies the ingestion of data sent to Dynatrace from any source in any format. During transport, data is prioritized, compressed and encrypted, ensuring data integrity and protection. Data is then dynamically routed into pipelines for further processing.

Designed to reach beyond petabyte (PB) scale, OpenPipeline will—at its launch—quintuple the ingest throughput from 100 TB a day per tenant to 500 TB a day per tenant. Further scaling, to and beyond 1 PB per day, will be announced in the near future.

This massive increase in data-ingestion throughput is possible thanks to several patent-pending high-performance stream-processing technologies, including a breakthrough in the simultaneous processing of thousands of data-processing rules. OpenPipeline rule processing outperforms most rule-processing algorithms by magnitudes (by a factor of 6-10 compared to prior art), with lower memory consumption and no degradation in data throughput.

Manage the cost of data with ease

With the exponential growth of data, and the need to retain data longer for business and security reasons, it’s important to preprocess and manage data appropriately to maximize value and minimize cost. OpenPipeline high-performance filtering and preprocessing provides full ingest and storage control for the Dynatrace platform. As a result, dedicated data pipeline tools are unnecessary for preprocessing data before ingest.

Filtering data is crucial for privacy and compliance to minimize the exposure of sensitive data. Additionally, it helps to reduce data volume and keep the cost of storing and querying data under control by eliminating duplicates, redundancies, and dropping unnecessary data fields. As an example, in early preview usage, AWS GuardDuty events were reduced by 84% by filtering out security-irrelevant events, which reduced cost and alert noise at the same time.

Transformations preprocess data and reduce data volume further, especially in situations where raw data is not required. OpenPipeline extracts data with context and transforms it into more efficient formats, for example, logs to metrics. Such transformations can reduce storage costs by 99%.

Routing of data to specific Grail buckets of varying retention durations lets you decide which data to keep and for how long. It also separates data organizationally for improved access control and focused query scope.

Protect your sensitive data

• Privacy by design. One essential aspect of OpenPipeline is the ability to mask data at capture using OneAgent® and automatic, rule-based processing. This approach suppresses sensitive data capture entirely before the data leaves the process, service, or customer environment. This ensures compliance and protects sensitive information from the start.
• Commitment to privacy. Dynatrace Trust Center demonstrates the Dynatrace commitment to privacy (and security) by design. Masking personal and sensitive data is of vital importance. With OpenPipeline, Dynatrace users can configure filtering and masking to their specific needs.

Bring data into context for improved analytics, automation, and AI

As OpenPipeline processes data streams in real time and it retains context during data normalization. At the same time, it performs contextual enrichment to ensure high-fidelity analytics, automation, and AI. With OpenPipeline processing power, you can:

• Enrich data and improve its value and quality by adding supplemental attributes (such as IP address geolocation or the related trace ID to a log line).
• Normalize data without losing context, as it detects known data structures automatically and provides optional rules for custom data structures.
• Transform contents into well-defined fields, convert raw data to time series, calculate metrics, or create business events from log lines.
• Converge heterogeneous data sources with ease, as data normalization and contextual enrichment can also happen on read, thanks to the schemaless and indexless approach of Grail.
• Contextualize and map data and get a unified view by identifying and connecting data points to the topology and dependencies within a software environment using the Dynatrace Semantic Dictionary. This contextualization provides Grail with additional semantic information in real time.
• Prioritize business data for a desired quality of service (QoS). Dynatrace provides unmatched accuracy by treating relevant business data (for example, real-time consumption or revenue dashboarding and analytics) with a higher priority and ensuring that data is not only in context but also not dropped or sampled.

Such contextually enriched data is the key to unlocking the full potential of your data for improved analytics, automation, and AI. By adding metadata and linking data to other relevant data sources, you enhance the quality, accuracy, and value of your data.

Davis—the unique Dynatrace hypermodal AI—builds on more than a decade of AI expertise in predictive and causal AI, and as announced recently, generative AI technologies. Automation and business observability require precise results. Contextually enriched data enables unrivaled predictive and causal AI power that provides real-time risk and root-cause analysis, enabling immediate insights and automated, AI-powered IT operations.

OpenPipeline in action

Let’s take a look at a concrete example—a logline that contains the following information:

2024-01-31 15:08:12 INFO [user-service] User john.doe@example.com logged in from 192.168.0.1

OpenPipeline can parse this logline and extract the following fields:

• Timestamp: 2024-01-31 15:08:12
• Log level: INFO
• Service name: user-service
• User email: john.doe@example.com
• User IP: 192.168.0.1

From here, OpenPipeline can convert this log entry into a time series metric that counts the number of logins per service, or create a business event that triggers an alert or notification when a user logs in.

With no schema or indexing, Grail even handles such data normalization and contextual enrichment “on read” as well, which makes it even easier to converge heterogeneous data sources. This additional power is enabled by the Dynatrace Semantic Dictionary, which provides Grail additional semantic information in real-time for the mapping of topology and dependencies within a software environment. Let’s look at another example, where a data source contains the following information:

{

"user_id": "123456789",

"user_email": "john.doe@example.com",

"user_name": "John Doe",

"user_role": "admin",

"user_location": "Linz, Austria"

}

Grail can use the Dynatrace Semantic Dictionary to map the user_email field to the User email field extracted above by OpenPipeline and enrich the data with additional context, such as user_id, user_name, user_role, and user_location. This way, Grail can provide a holistic view of the user and their activities across different data sources.

OpenPipeline is part of the unified Settings experience, inside the built-in Settings app 

OpenPipeline offers an easy way to create and configure routes and pipelines at scale, comfortably situated within the Settings app. This app simplifies the configuration process by utilizing Dynatrace query language (DQL) for matching and processing routes. Of course, configuration-as-code using an application programming interface (API) is also available.

Visual guidance within the user interface is available to administrators for managing rules, setting up ingestion configurations, dynamic routing into individual pipelines, controlling data enrichment and transformation policies, and sending data from a source to its destination (such as specific Grail buckets).

You can read more about setting up and managing pipelines in our documentation. 

Get more value from your data with OpenPipeline

OpenPipeline complements Grail with high-performance stream processing to maximize security and ease the management of large heterogeneous data, while minimizing cost. It manages parallel pipelines to ingest, transport, mask, filter, enrich, normalize, transform, contextualize, route, and persist data. OpenPipeline is available to Dynatrace customers at no additional cost.

• OpenPipeline enables cost-effective and scalable data ingest, with up to 500 TB per day per tenant, with plans to go beyond the petabyte-per-day level in the near future. It also offers built-in data transformation features that reduce storage costs by up to 99%.
• OpenPipeline contextualizes data in real time with enrichment, discovered topology, and tags. Using its patent-pending stream-processing technologies, OpenPipeline optimizes data for Dynatrace analytics and AI.
• OpenPipeline ensures data security and privacy with source-side masking and encryption at the source, additional filtering, and masking at ingest. In combination with Grail, this ensures data privacy and compliance at the storage and query level. Such a centralized data management approach has substantial potential to reduce security and audit efforts.

The first release of Dynatrace OpenPipeline, supporting logs, business events, and generic events, will be released within 90 days. Once available, Dynatrace SaaS customers running on AWS or Azure, and using the latest version of Dynatrace, can start using OpenPipeline without installing anything: the OpenPipeline Configuration app will be pre-installed on all eligible tenants. Existing processing rules for logs and business events will be automatically migrated to new pipelines, so existing customers will benefit from the new OpenPipeline functionality from day one.

The post Dynatrace OpenPipeline: Stream processing data ingestion converges observability, security, and business data at massive scale for analytics and automation in context appeared first on Dynatrace news.