The business observability gap in cloud-based transactions

Business observability, at its best, delivers real-time insights into business health, reporting, and interpretation of the metrics and KPIs that business and IT leaders need to make decisive and effective decisions.

But how can you access this data when cloud-based solutions deliver your critical services? One of the more common examples is payment gateways, or payment service providers (PSPs)—hosted services that verify account details, check for available funds, and transfer money into your business merchant account.

There’s a wealth of business data in your systems’ transaction requests to these PSPs and other SaaS solutions. Without visibility into this data, you’re left with an incomplete and delayed understanding of transaction success, sales data, and business health. This is particularly problematic since the transaction success or failure status is often embedded within the response body.

Here are some of the challenges our customers have identified:

• Delays in resolving issues. Failed transactions are only discovered in scheduled reports or when customers complain.
• Reconciliation discrepancies. Inconsistencies between internal records and PSP reports lead to frustrating, time-consuming, and error-prone manual work.
• Inability to understand short-term trends. Identifying failure patterns for specific providers or transaction types is nearly impossible without timely and detailed transaction data.
• Compliance risks. Lack of granular detail makes auditing and demonstrating compliance with payment regulations more challenging.

Business events from outgoing web requests

Dynatrace business events are in a special class designed to support even the most demanding business use cases. Whether derived from OneAgent, log files, RUM sessions, or APIs, business events deliver real-time lossless access to critical business data—wherever it lies. In particular, OneAgent’s unique capability to extract business data from in-flight application payloads—without writing any code—has proven invaluable to hundreds of our customers.

OneAgent now supports capturing business events from outgoing web requests that are specifically designed to illuminate these blind spots. You can now transform critical business information within your outgoing web requests and their corresponding responses into structured business events, which can then be analyzed, correlated, and acted upon within Dynatrace.

Use case: Business events and payment gateways

Here’s how it works, and what it means for our payment gateway use case.

Granular configuration for targeted capture

• Define endpoint-specific rules to capture data from requests to specific URLs or IP addresses (for example, api.stripe.com/payments or paypal.com/checkout).
• Define method-specific rules to capture data from relevant HTTP methods like POST or PUT that are used to initiate transactions.

Deep insight into request and response payloads

Support for business-event capture from outgoing web requests adds compelling new value and expands on current OneAgent capabilities for extracting business data from in-flight application payloads.

• Extract data from the request body. Imagine you’re sending payment requests that include customer account numbers, transaction amounts, and transaction identifiers. You can now define rules to extract these key pieces of information directly from your JSON or XML request bodies. This is crucial for linking your internal transaction IDs to the business data sent to the provider.
• Extract data from the response body. Potentially greater value lies in the response body. Often, PSPs return transaction status messages (for example, transactionStatus:success, authorizationResult:approved, or responseCode:1954) within the response body. Extracting these status messages allows you to report, alert, and act on failures or anomalies as they occur.

Transform raw data into meaningful business insights

Business data increases in value when enriched with business and IT attributes. For example, you could map extracted fields to meaningful attributes of a “Payment Transaction” business event. These attributes might include payment_provider_id, transaction_amount, and customer_id, extracted from the request body, and payment_status extracted from the response body.

These events are automatically enriched with Dynatrace observability data, such as the request’s originating host, service, user journey, or geographical source, facilitating business and IT collaboration.

Real-time visibility and proactive alerts

• Real-time business transaction monitoring. Know each PSP transaction’s success, failure, and value as it happens, not hours later.
• Custom dashboards. Dashboards make it easy to visualize different PSPs’ success rates and revenue, exposing bottlenecks and highlighting business and failure trends.
• Proactive alerting. Set up alerts based on real-time payment status messages. Trigger alerts on failed transactions over a certain amount, or when the success rate SLA degrades.

Enhanced traceability and root cause analysis

• End-to-end transaction tracing. Trace the entire lifecycle of a payment, from the user initiating the purchase through your internal services to the outgoing request to the PSP and the response status indicating success or failure.
• Accelerated root cause analysis. Quickly determine the source of payment transaction failures, bottlenecks, and anomalies to speed resolution and improve business outcomes.

Payment gateway example: From opaque to transparent

Here’s what this could mean for your organization:

1. A customer attempts a major purchase. Your e-commerce system sends the request to your PSP, but the transaction is declined.
2. OneAgent captures relevant business data from the outgoing request, including customer ID and transaction amount attributes. Dynatrace also captures the relevant response fields, including the status:declined message.
3. A “Payment Failure” business event is automatically generated in Dynatrace. This event triggers an alert to notify your operations and customer support teams.
4. Your customer support team can now proactively contact the customer while your operations team works to resolve the issue, knowing exactly why the transaction failed and which provider was involved.

This is possible without any code changes. Extracting business data from outgoing requests and responses turns opaque customer interactions into transparent, actionable business insights, empowering you to improve customer experience, optimize payment workflows, and reduce operational risk.

Initial support for outgoing web request capture is available for Java; see Supported Technologies for details. Additional technology support will be available soon.

The post Extend business observability: Extract business events from outgoing web requests appeared first on Dynatrace news.

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. To unlock business value, the data must be:

• Accessible from anywhere. Data has value only when you can access it, no matter where it lies.
• Easy to access. Simplicity accelerates time-to-value and reduces implementation and maintenance costs.
• Real time. Agile business decisions rely on fresh data.
• Precise. Accuracy provides the confidence needed for business automation.
• Contextualized. Metadata enrichment improves collaboration and increases analytic value.

The Dynatrace® platform continues to increase the value of your data—broadening and simplifying real-time access, enriching context, and delivering insightful, AI-augmented analytics. Our Business Observability solution is a prominent beneficiary of this commitment.

Business events: Delivering the best data

It’s been two years since we introduced business events, a special class of events designed to support even the most demanding business use cases. Since then, many of our customers have embraced the opportunity to explore and adopt new business observability use cases. Most of these leverage the unique capability of Dynatrace OneAgent® to extract business data from in-flight application payloads—without writing any code. Other data sources, including APIs and log files—are used to expand access, often to external or proprietary systems. Enhancing access to business data from log files is an important priority, and OpenPipeline makes this a reality.

Dynatrace OpenPipeline is a new stream processing technology that ingests and contextualizes data from any source. You can now use OpenPipeline to extract business events from log files, complementing OneAgent as a primary source of business data. This is especially important when legacy or proprietary systems don’t meet OneAgent’s environmental prerequisites; in these cases, log files are usually the preferred source of business data.

In fact, it’s likely that some of your critical business systems already write business data to log files. For years, logs have been the dominant approach many observability vendors have taken to report business metrics on dashboards. You might still be using one of these tools despite the drawbacks, which include a lack of IT context, constraints on data privacy and data retention, and, as the only convenient source of business data, a limiting lack of breadth. OpenPipeline makes migrating these use cases to Dynatrace easy, helping you overcome these limitations to realize greater value.

Use case categories

It’s helpful to categorize the nearly unlimited use cases covered by Business Observability. Two of the most important categories are:

• Business reporting, analytics, and automation. Track business metrics, key performance indicators (KPIs), and service level objectives (SLOs)—automatically and in context with IT infrastructure and services—to promote collaboration between business and IT teams.
• Business process monitoring and optimization. Monitor and optimize business processes with real-time visibility into process KPIs and detailed analytics for each step to improve customer satisfaction, increase operational efficiency, and reduce cost.

Most of the use cases in these two broad categories benefit from the flexibility that comes from multiple available sources of business data. For example:

• An airline’s reporting and analytics dashboard includes data showing flights, passengers, available seats, passenger load, revenue per passenger, flight crew staffing, arrival delays, and customer satisfaction metrics. The data may come from a mix of systems, including a departure control system (DCS), an airline reservation system (ARS), a legacy inventory control system, and a SaaS-based Voice of the Customer (VoC) solution.
• A financial institution’s loan origination business process includes a series of process milestones, including loan application, credit scoring, application review, contract generation, CRM, and loan funding. All of these steps are critical components of the process, likely to be implemented using different systems. Furthermore, to understand process health, individual steps must be viewed in the context of the entire process. For this, we use Business Flow to track, analyze, and optimize complex business processes, treating the process as an observable IT and business asset.

Use OpenPipeline to extract business events from logs

Using OpenPipeline, you can ingest log data into Dynatrace from a wide range of sources, including OneAgent, Extensions, the Log ingest API, and OpenTelemetry. The pipeline includes a configurable business event processor as part of the data extraction stage; this processor is used to extract and convert relevant business data into business events.

There are many benefits to extracting business events from log files. These include:

• Improved data privacy. Sensitive business data is separated from IT observability data.
• Improved data management. Fine-grained permission and retention policies can be tailored to individual business use cases.
• Reduced storage and query overhead for business use cases. Business events are a small, often negligible subset of log data.
• Simplified and enhanced analytics efficiency. Business events from log files are unified with business events from OneAgent, RUM, and API.

Extracting business events from logs: Configuring OpenPipeline

The OpenPipeline app guides you through the three configuration steps required to extract business events from log files:

1. Identify the source of the log file.
2. Define OpenPipeline’s dynamic routing matching criteria used to route the log lines of interest to a second pipeline.
   
3. Create the target pipeline. Use the Data extraction tab to define the matching condition that creates the business event. You need to include static or dynamic Event type and Event provider fields.
   

Within the target pipeline, you can also define processing rules, extract metrics, set the security context, and define retention periods. Log data is then processed accordingly, stored in Dynatrace Grail™ causational data lakehouse, and available for your Business Observability use cases.

The value is in the data

OpenPipeline broadens Dynatrace’s unified access to the best data to deliver the best value. Now’s the time to see how it can benefit your organization.

For more details about OpenPipeline read this Dynatrace OpenPipeline blog post.

The post OpenPipeline: Simplify access to critical business data appeared first on Dynatrace news.

Should business data be part of your observability solution?

Technology and business leaders express increasing interest in integrating business data into their IT observability strategies, citing the value of effective collaboration between business and IT. Exploding volumes of business data promise great potential; real-time business insights and exploratory analytics can support agile investment decisions and automation driven by a shared view of measurable business goals.

But are observability platforms—born from the collision between the demands of cloud computing and the limitations of APM and infrastructure monitoring—the best solution for managing business analytics?

Observability fault lines

The monitoring of complex and dynamic IT systems includes real-time analysis of baselines, trends, and anomalies. This is achieved, in part, by establishing actionable statistical accuracy—not necessarily precise accuracy—through practical levels of metric sampling, aggregation, and extrapolation. Statistical accuracy is typically not precise enough for most business use cases, and business actions can’t be automated based on anything less than 100% precise data.

Similarly, many performance analytics goals are served by capturing method calls and arguments to give developers the information they need to diagnose and correct anomalies. Traditional observability solutions don’t capture or analyze application payloads. Such analysis is intentionally excluded from most observability solutions because payload details are unnecessary for DevOps purposes, problematic for agent overhead, and risky for data privacy. Therefore, to add the “Biz” to “DevOps,” observability solutions typically rely on application log files, which incurs significant development and maintenance overhead.

To close these critical gaps, Dynatrace has defined a new class of events called business events. Dynatrace OneAgent^® prioritizes business events over observability metrics to ensure the lossless precision you need to support demanding business use cases. At the same time, deep payload inspection makes it easy to extract important business data locked in application payloads—without writing any code. When combined with Dynatrace Grail™, business events gain long-term, cost-effective storage, unaggregated granularity, instant indexless queries, and automatic Smartscape^® topology context.

Introducing metric extraction from business events

Beginning with Dynatrace SaaS version 1.257, you can extract metrics from ingested business events. Metric extraction is a convenient way to create your business metrics, delivering fast, flexible, and cost-effective analytics. Since these metrics are derived from business events, they inherit key characteristics, including lossless precision and deep access to application payloads. Extracting metrics from business events allows you to use metric expressions, view baselines and trends, and create custom alerts when issues require further evaluation by Dynatrace Davis®, our causational AI engine.

From business events to metrics: the ingest pipeline

All incoming business events must pass through the business event ingest pipeline where transformational processing occurs. It’s at this point—immediately after processing—where you can extract metrics from your events by defining extraction rules.

Extraction rules include the following four fields:

• Key: Always starts with the prefix bizevents.
• Matcher: A matcher-specific DQL query that selects specific incoming business events
• Measure: Choose an event-attribute value or an event-count value
• Dimension: Define up to 50 dimensions

In this example, we use a DQL matcher query to extract the trading_volume attribute value from all incoming business events that have an event.type of com.easytrade.buy-assets. We assign the metric key of bizevents.EasyTrade.TradingDollarVolume, and include the dt.entity.host dimension, which will allow us to split or filter the results by host.

Once defined, custom business-event metrics will be extracted from all future ingested business events that match this rule.

View and analyze

You can use the interactive Data explorer to view, query, transform, and visualize your metrics. Once you’ve refined the visualization of a metric, you can pin it to a dashboard.

Through filtering, aggregation, or arithmetic operations, Data explorer also supports metric expressions that combine one or more existing metrics to create new, on-demand metrics. For example, you could divide the TradingDollarVolume metric by a TradeCount metric to calculate a new metric, AverageTradeValue. The new metric can then be pinned to a dashboard. The Data explorer supports many types of visualizations, making it easy to understand trends or to view the relationship between business and IT metrics.

Metric events for alerting

Metric event configuration can be used to automatically detect anomalies in metric timeseries using thresholds or baselines. The resulting alerts can be based on attribute values or specific event occurrences. The extraction of metrics from business events automatically preserves the Smartscape context of each event, allowing Davis AI to automatically analyze anomalies and identify any relevant impact and root cause.

Conclusion: Metric extraction adds new value to business events

Lossless precision and easy, code-free access to business data distinguish business events from traditional IT metrics. Powered by Grail, business events gain long-term cost-effective storage, unaggregated granularity, instant indexless queries, and automatic Smartscape context.

Extracting metrics from business events delivers additional value, including comprehensive metric calculations, a wide range of visualizations, threshold-based alerting, and AI-driven causation analytics.

What’s next?

Metric extraction from business events gives you new options for monitoring, analyzing, and automating your business use cases. For additional technical insights on extracting metrics from business events, watch this Business Events Observability Clinic.

The post Extract metrics from business events to increase the value of Business Analytics appeared first on Dynatrace news.