The business observability gap in cloud-based transactions

Business observability, at its best, delivers real-time insights into business health, reporting, and interpretation of the metrics and KPIs that business and IT leaders need to make decisive and effective decisions.

But how can you access this data when cloud-based solutions deliver your critical services? One of the more common examples is payment gateways, or payment service providers (PSPs)—hosted services that verify account details, check for available funds, and transfer money into your business merchant account.

There’s a wealth of business data in your systems’ transaction requests to these PSPs and other SaaS solutions. Without visibility into this data, you’re left with an incomplete and delayed understanding of transaction success, sales data, and business health. This is particularly problematic since the transaction success or failure status is often embedded within the response body.

Here are some of the challenges our customers have identified:

• Delays in resolving issues. Failed transactions are only discovered in scheduled reports or when customers complain.
• Reconciliation discrepancies. Inconsistencies between internal records and PSP reports lead to frustrating, time-consuming, and error-prone manual work.
• Inability to understand short-term trends. Identifying failure patterns for specific providers or transaction types is nearly impossible without timely and detailed transaction data.
• Compliance risks. Lack of granular detail makes auditing and demonstrating compliance with payment regulations more challenging.

Business events from outgoing web requests

Dynatrace business events are in a special class designed to support even the most demanding business use cases. Whether derived from OneAgent, log files, RUM sessions, or APIs, business events deliver real-time lossless access to critical business data—wherever it lies. In particular, OneAgent’s unique capability to extract business data from in-flight application payloads—without writing any code—has proven invaluable to hundreds of our customers.

OneAgent now supports capturing business events from outgoing web requests that are specifically designed to illuminate these blind spots. You can now transform critical business information within your outgoing web requests and their corresponding responses into structured business events, which can then be analyzed, correlated, and acted upon within Dynatrace.

Use case: Business events and payment gateways

Here’s how it works, and what it means for our payment gateway use case.

Granular configuration for targeted capture

• Define endpoint-specific rules to capture data from requests to specific URLs or IP addresses (for example, api.stripe.com/payments or paypal.com/checkout).
• Define method-specific rules to capture data from relevant HTTP methods like POST or PUT that are used to initiate transactions.

Deep insight into request and response payloads

Support for business-event capture from outgoing web requests adds compelling new value and expands on current OneAgent capabilities for extracting business data from in-flight application payloads.

• Extract data from the request body. Imagine you’re sending payment requests that include customer account numbers, transaction amounts, and transaction identifiers. You can now define rules to extract these key pieces of information directly from your JSON or XML request bodies. This is crucial for linking your internal transaction IDs to the business data sent to the provider.
• Extract data from the response body. Potentially greater value lies in the response body. Often, PSPs return transaction status messages (for example, transactionStatus:success, authorizationResult:approved, or responseCode:1954) within the response body. Extracting these status messages allows you to report, alert, and act on failures or anomalies as they occur.

Transform raw data into meaningful business insights

Business data increases in value when enriched with business and IT attributes. For example, you could map extracted fields to meaningful attributes of a “Payment Transaction” business event. These attributes might include payment_provider_id, transaction_amount, and customer_id, extracted from the request body, and payment_status extracted from the response body.

These events are automatically enriched with Dynatrace observability data, such as the request’s originating host, service, user journey, or geographical source, facilitating business and IT collaboration.

Real-time visibility and proactive alerts

• Real-time business transaction monitoring. Know each PSP transaction’s success, failure, and value as it happens, not hours later.
• Custom dashboards. Dashboards make it easy to visualize different PSPs’ success rates and revenue, exposing bottlenecks and highlighting business and failure trends.
• Proactive alerting. Set up alerts based on real-time payment status messages. Trigger alerts on failed transactions over a certain amount, or when the success rate SLA degrades.

Enhanced traceability and root cause analysis

• End-to-end transaction tracing. Trace the entire lifecycle of a payment, from the user initiating the purchase through your internal services to the outgoing request to the PSP and the response status indicating success or failure.
• Accelerated root cause analysis. Quickly determine the source of payment transaction failures, bottlenecks, and anomalies to speed resolution and improve business outcomes.

Payment gateway example: From opaque to transparent

Here’s what this could mean for your organization:

1. A customer attempts a major purchase. Your e-commerce system sends the request to your PSP, but the transaction is declined.
2. OneAgent captures relevant business data from the outgoing request, including customer ID and transaction amount attributes. Dynatrace also captures the relevant response fields, including the status:declined message.
3. A “Payment Failure” business event is automatically generated in Dynatrace. This event triggers an alert to notify your operations and customer support teams.
4. Your customer support team can now proactively contact the customer while your operations team works to resolve the issue, knowing exactly why the transaction failed and which provider was involved.

This is possible without any code changes. Extracting business data from outgoing requests and responses turns opaque customer interactions into transparent, actionable business insights, empowering you to improve customer experience, optimize payment workflows, and reduce operational risk.

Initial support for outgoing web request capture is available for Java; see Supported Technologies for details. Additional technology support will be available soon.

The post Extend business observability: Extract business events from outgoing web requests appeared first on Dynatrace news.

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. To unlock business value, the data must be:

• Accessible from anywhere. Data has value only when you can access it, no matter where it lies.
• Easy to access. Simplicity accelerates time-to-value and reduces implementation and maintenance costs.
• Real time. Agile business decisions rely on fresh data.
• Precise. Accuracy provides the confidence needed for business automation.
• Contextualized. Metadata enrichment improves collaboration and increases analytic value.

The Dynatrace® platform continues to increase the value of your data—broadening and simplifying real-time access, enriching context, and delivering insightful, AI-augmented analytics. Our Business Observability solution is a prominent beneficiary of this commitment.

Business events: Delivering the best data

It’s been two years since we introduced business events, a special class of events designed to support even the most demanding business use cases. Since then, many of our customers have embraced the opportunity to explore and adopt new business observability use cases. Most of these leverage the unique capability of Dynatrace OneAgent® to extract business data from in-flight application payloads—without writing any code. Other data sources, including APIs and log files—are used to expand access, often to external or proprietary systems. Enhancing access to business data from log files is an important priority, and OpenPipeline makes this a reality.

Dynatrace OpenPipeline is a new stream processing technology that ingests and contextualizes data from any source. You can now use OpenPipeline to extract business events from log files, complementing OneAgent as a primary source of business data. This is especially important when legacy or proprietary systems don’t meet OneAgent’s environmental prerequisites; in these cases, log files are usually the preferred source of business data.

In fact, it’s likely that some of your critical business systems already write business data to log files. For years, logs have been the dominant approach many observability vendors have taken to report business metrics on dashboards. You might still be using one of these tools despite the drawbacks, which include a lack of IT context, constraints on data privacy and data retention, and, as the only convenient source of business data, a limiting lack of breadth. OpenPipeline makes migrating these use cases to Dynatrace easy, helping you overcome these limitations to realize greater value.

Use case categories

It’s helpful to categorize the nearly unlimited use cases covered by Business Observability. Two of the most important categories are:

• Business reporting, analytics, and automation. Track business metrics, key performance indicators (KPIs), and service level objectives (SLOs)—automatically and in context with IT infrastructure and services—to promote collaboration between business and IT teams.
• Business process monitoring and optimization. Monitor and optimize business processes with real-time visibility into process KPIs and detailed analytics for each step to improve customer satisfaction, increase operational efficiency, and reduce cost.

Most of the use cases in these two broad categories benefit from the flexibility that comes from multiple available sources of business data. For example:

• An airline’s reporting and analytics dashboard includes data showing flights, passengers, available seats, passenger load, revenue per passenger, flight crew staffing, arrival delays, and customer satisfaction metrics. The data may come from a mix of systems, including a departure control system (DCS), an airline reservation system (ARS), a legacy inventory control system, and a SaaS-based Voice of the Customer (VoC) solution.
• A financial institution’s loan origination business process includes a series of process milestones, including loan application, credit scoring, application review, contract generation, CRM, and loan funding. All of these steps are critical components of the process, likely to be implemented using different systems. Furthermore, to understand process health, individual steps must be viewed in the context of the entire process. For this, we use Business Flow to track, analyze, and optimize complex business processes, treating the process as an observable IT and business asset.

Use OpenPipeline to extract business events from logs

Using OpenPipeline, you can ingest log data into Dynatrace from a wide range of sources, including OneAgent, Extensions, the Log ingest API, and OpenTelemetry. The pipeline includes a configurable business event processor as part of the data extraction stage; this processor is used to extract and convert relevant business data into business events.

There are many benefits to extracting business events from log files. These include:

• Improved data privacy. Sensitive business data is separated from IT observability data.
• Improved data management. Fine-grained permission and retention policies can be tailored to individual business use cases.
• Reduced storage and query overhead for business use cases. Business events are a small, often negligible subset of log data.
• Simplified and enhanced analytics efficiency. Business events from log files are unified with business events from OneAgent, RUM, and API.

Extracting business events from logs: Configuring OpenPipeline

The OpenPipeline app guides you through the three configuration steps required to extract business events from log files:

1. Identify the source of the log file.
2. Define OpenPipeline’s dynamic routing matching criteria used to route the log lines of interest to a second pipeline.
   
3. Create the target pipeline. Use the Data extraction tab to define the matching condition that creates the business event. You need to include static or dynamic Event type and Event provider fields.
   

Within the target pipeline, you can also define processing rules, extract metrics, set the security context, and define retention periods. Log data is then processed accordingly, stored in Dynatrace Grail™ causational data lakehouse, and available for your Business Observability use cases.

The value is in the data

OpenPipeline broadens Dynatrace’s unified access to the best data to deliver the best value. Now’s the time to see how it can benefit your organization.

For more details about OpenPipeline read this Dynatrace OpenPipeline blog post.

The post OpenPipeline: Simplify access to critical business data appeared first on Dynatrace news.

Unrealized optimization potential of business processes due to monitoring gaps

Imagine a retail company facing gaps in its business process monitoring due to disparate data sources. Due to separated systems that handle different parts of the process, the view of the process is fragmented. On top of that, the data sources are inconsistent. While some data comes from modern systems with APIs, other data stems from older systems that generate log files, and some data originates from external vendors. This inconsistency complicates uniform gathering and data analysis, resulting in incomplete or inaccurate insights. This scenario is shared among large organizations that rely on multiple internal and external systems for data collection.

Get business process observability across data silos with Dynatrace OpenPipeline

Traditional observability platforms often focus on technical metrics and logs, which are essential for troubleshooting, but they don’t take business value into consideration. Dynatrace OpenPipeline goes beyond this by offering the possibility of extracting business-relevant data from logs and using them to monitor end-to-end business observability.

In our retail company example, older systems are involved in shipping the order. They only produce logs that have not yet been monitored from a business perspective, although they contain valuable business information.

By leveraging Dynatrace OpenPipeline, the retail company can integrate data from all sources across the whole process, including Dynatrace OneAgent^®, logs, and external business tools. This approach ensures that the company can easily track the entire journey from online order to its successful delivery.

Benefits of capturing business events

Logs often contain valuable insights into your business; however, this information can be difficult to process, particularly as you probably only need data from some specific log lines.

Dynatrace OpenPipeline extracts this business information from logs and stores it as a separate data type, so-called business events. This has several advantages:

• Separation of concerns: Logs are often used for technical troubleshooting. Extracting business events allows for a clear separation between technical and business-relevant data.
• Access control: Different teams can access different types of data. For example, support teams might access logs for troubleshooting, while business teams access business events for analytics.
• Ease of access: Having business events in a uniform format simplifies querying and visualization, making it easier to analyze and derive insights.

How to find valuable business information in logs

In our example of a retail company, we need to extract business information from shipping logs to know if our order has already been shipped. See a typical log file with shipping details below.

From this log file, we can identify and capture the following business information:

1. Correlation ID – The Correlation ID is part of the log payload and identifies this event in the context of a business process.

2. Message – The message is also part of the log payload and contains information on which part of the business process it represents.

3. Context – The context contains information on which IT system handled this part of the business process.

4. Status—The status information tells us if something was successful or if we hit an issue.

Use OpenPipeline to identify relevant log events

To turn this information into a business event for analytics, we utilize the capabilities of Dynatrace OpenPipeline to identify relevant log events, parse the data, and create a business event out of it. The approach is the following:

1. Go to the OpenPipeline app (available by default in your Dynatrace tenant)

1. Create a route that specifies which logs should be processed. For example, you can only process log events coming from a specific ingest source or containing a specific phrase in its log message.
2. Define a pipeline that will process these logs.
3. Configure the data extraction rules within the pipeline. This involves:

1. • Extracting correlation IDs: Identify and extract correlation IDs from the log content. Rename these IDs to make them uniform with other events (for example, rename “order ID”).
   • Extracting the message: The relevant message will be extracted from the log content and used as part of the business event.
   • Parse out contextual information such as the context (which is the related host) or status information in our case.
   • Defining event types: Specify the event type using the extracted message and provider information. For example, you might name the provider retail.logs for consistency.

The extracted information is re-ingested into the pipeline as a new business event. The event is then stored in Grail™ datalake house and can be used for further analysis or process visualization.

Do you want to dig deeper into extracting the data? Watch the dedicated Dynatrace Lab episode with Andreas Grabner and Alistair Emslie for a step-by-step guide on how this is done:

Turn your business data into tangible value

The extracted business data enables you to optimize your processes and gain valuable new insights. You can easily create a dashboard like the one below: it provides the example retail company with real-time data and KPIs, such as the success rate of shipping orders. Unlike traditional dashboards focusing on specific applications or technical aspects, this dashboard tracks the performance of the complete business journey with business KPIs and metrics, utilizing the information extracted from the log files.

Of course, you can also leverage the full power of the Dynatrace platform, like AI-powered monitoring of your business KPIs through Davis^® AI. You can also visualize the end-to-end process flow in our Business Flow app, which enables you to identify delays, errors, and exceptions, providing insights into IT and business-related issues.

Learn more about the capabilities of Dynatrace OpenPipeline

If you’re struggling with gaps in your business process monitoring caused by disparate data sources and outdated systems, consider how Dynatrace can transform your approach.

The post Leverage logs for an end-to-end view of your business processes via Dynatrace OpenPipeline appeared first on Dynatrace news.

In part 1 of this blog series, we explored the concept of business observability, its significance, and how real-time visibility aids in making informed decisions. In part 2, we’ll show you how to retrieve business data from a database, analyze that data using dashboards and ad hoc queries, and then use a Davis analyzer to predict metric behavior and detect behavioral anomalies.

Dataflow overview

Dynatrace ActiveGate extensions allow you to extend Dynatrace monitoring to any remote technology that exposes an interface. Dynatrace users typically use extensions to pull technical monitoring data, such as device metrics, into Dynatrace.

However, as we highlighted previously, business data can be significantly more complex than simple metrics. To accommodate this complexity, we created a new Dynatrace extension.

Create an extension to query complex business data

Creating an ActiveGate extension with the Dynatrace extension framework is easy; there’s a tutorial on using the ActiveGate Extension SDK that guides you through making an extension to monitor a demo application bundled with the SDK.

Similar to the tutorial extension, we created an extension that performs queries against databases. Notably, the SQL query is not limited to specific columns or data with specific metric values (int or float). Instead, the data can be of any type, including string, Boolean, timestamp, or duration.

There are three high-level steps to set up the database business-event stream.

1. Create and upload the extension that connects to the database and extracts business data in any form.
2. Configure the extension with the appropriate database credentials, query names, Dynatrace endpoint, and tokens necessary to send the business data to Grail.
3. Once the data is received in Grail, you can explore, manipulate, and analyze the data, utilizing advanced techniques such as filtering, grouping and aggregation, calculations and transformations, time windowing, and much more. Further, you can set alerts based on predefined or auto-adaptive thresholds.

Step-by-step: Set up a custom MySQL database extension

Now we’ll show you step-by-step how to create a custom MySQL database extension for querying and pushing business data to the Dynatrace business events endpoint.

Create and upload the extension

1. Download the extension ZIP file. Don’t rename the file. This is a sample extension that connects to a MySQL database and pushes business events to Dynatrace.
2. Unzip the ZIP file to the plugin deployment directory of your ActiveGate host (found at /opt/dynatrace/remotepluginmodule/plugindeployment/).
3. In the Dynatrace menu, go to Settings > Monitored technologies > Custom extensions and select Upload Extension.
4. Upload the ZIP file.
5. Once uploaded, extract the ZIP file at the same location.
6. Configure the information needed to query business observability data from the target database.
   There are three configuration sections, as shown below in the Dynatrace web UI.

Configuration details

Database configuration

• Endpoint name: Any label to identify this connection. This is used for identification purposes.
• SQL IP/Hostname: The database IP or hostname.
• SQL Username: Username of the user who has permission to login on the SQL server remotely and access the database.
• SQL Password: Password for the username.
• SQL DB: The database name.

Bizevents API and token configuration

• Endpoint to Push Bizevents: Bizevents API that will receive the business data.
  • Replace tenantid with your tenant ID
• Client ID to generate token: Client ID used to generate OAuth token. To generate client-id, refer to our OAuth documentation.
• Client secret to generate token: Client secret for token generation.

Define your SQL Queries

• Queryname 01: Unique name to identify the query to ensure data identification and retention within Dynatrace.
• Query 01: SQL query to retrieve data.
• Interval 01: Frequency in minutes for executing the configured query.
• Add multiple queries (depending on the requirement) with the above config for each query.

Define the retention period with matcher DQL and bucket assignment

Data stored in Grail can be preserved for extended periods, up to 10 years. To achieve this, we’ll create a Grail bucket specifically designed to retain data for a duration of 10 years (3,657 days).

Here is a JSON response from an API that successfully created a bucket capable of storing data for a period of up to 10 years.

After obtaining a bucket with a suitable retention period, it’s time to create a DQL matching rule that effectively filters events and directs them to the appropriate Grail bucket. This ensures that the data is retained for the correct duration while restricting access to users who are authorized for that specific bucket.

Analyze the data in real-time using Dashboards or collaborate with colleagues using Notebooks

In the screen recording provided below, we begin by examining the business data ingested into Grail using a notebook. This initial overview provides a broad perspective of the ingested data. However, real insights emerge when we delve deeper and analyze specific events over time. As you follow along in the video, you’ll notice the ability to determine the day of the week for each transaction and visualize the data in a user-friendly bar chart.

The video below showcases a business dashboard that effectively visualizes important events, including pending withdrawals and deposits from the past hour, transaction amounts throughout the week, transaction queue status from the previous hour, and the overall transaction status.

Enhance data insights with real-time ad hoc queries

While predefined dashboards can offer comprehensive overviews, they don’t always anticipate and meet the needs of business analysts. Dynatrace Query Language (DQL) is a powerful tool for exploring your data and discovering patterns, identifying anomalies and outliers, creating statistical modeling, and more based on data stored in Dynatrace Grail. Now we’ll use a Dynatrace Notebook to execute our DQL queries.

In the below query, we’re specifically searching for pending deposit transactions greater than $8,000 that occurred between 10:00:04 AM and 12:00:00 AM on August 21, 2023. The query for pending deposit transactions within a specific time frame is useful for real-time analysis, issue investigation, performance assessment, impact assessment, and compliance/auditing purposes.

Proactive alerting for accumulating business transactions: Mitigating business impact

To ensure timely action and address potential bottlenecks, we can set up alerts that notify you when pending transactions accumulate within a short period. These alerts serve as early business warnings, allowing you to take necessary measures to prevent disruptions and minimize delays in transaction processing.

In the above recording, we demonstrate an alert specifically designed to notify when there is a significant increase in pending transactions. This alert serves as a valuable tool in maintaining operational efficiency, ensuring business continuity, and delivering optimal customer experiences.

Forecast business data Using a Davis analyzer

In the context of monitoring business-related data such as sales, orders, payments, withdrawals, deposits, and pending transactions, Dynatrace Davis analyzers offer valuable forecast analysis capabilities. Davis analyzers offer a broad range of general-purpose artificial intelligence and machine learning (AI/ML) functionality, such as learning and predicting time series, detecting anomalies, or identifying metric behavior changes within time series.

By utilizing a Davis analyzer, organizations can predict future trends and patterns in their payment and transaction data. This forecast analysis helps businesses anticipate customer behavior, plan for fluctuations in transaction volumes, and optimize their operations accordingly.

For example, by applying forecast analysis to payment data, businesses can identify potential cash flow issues or predict periods of high transaction activity. This type of insight enables you to proactively manage liquidity, ensure sufficient funds are available, and make informed decisions about resource allocation.

Conclusion

By combining proactive alerts and leveraging AI-powered insights, we can effectively manage pending transactions, optimize processes, and ensure smooth operations.

To address the business need for extracting business data from databases, we demonstrated using a custom database extension to bring the data into Dynatrace. This integration allows seamless connectivity to a variety of databases, enabling the real-time retrieval and storage of business data.

By leveraging the powerful combination of business, security, and observability, organizations gain immediate access to their critical business data without any delays or data staleness. The real-time nature of the data extraction ensures that decision-makers have up-to-date information at their fingertips, empowering them to make timely and informed decisions.

Furthermore, we showcased the flexibility and versatility of the Dynatrace platform in exploring and analyzing the extracted data. By seamlessly integrating the data into Notebooks and Dashboards, organizations can gain comprehensive insights into trends, patterns, and key performance indicators relevant to their business. This empowers data analysts and business users to delve deep into the data, uncover valuable insights, and derive actionable intelligence.

Additionally, we demonstrated the power of custom alerts in Dynatrace. By defining specific thresholds for key business KPIs, the platform can proactively monitor data and generate alerts whenever a breach or potential issue is detected. This proactive alerting capability ensures that stakeholders are promptly notified of any anomalies or deviations, enabling them to take immediate corrective actions and mitigate risks. More advanced use cases integrate with automation workflows to automate recovery actions.

Through seamless database connectivity, real-time data retrieval, exploratory capabilities, proactive alerting, and automation, organizations can enhance their overall operational efficiency, customer satisfaction, and business performance. The integration of the Dynatrace observability platform with the custom database extension provides organizations with a solution to extract, analyze, and act upon their at-rest business data, driving success in a rapidly evolving business landscape.

The post Extend business observability: Extract business events from online databases (Part 2) appeared first on Dynatrace news.

Business leaders benefit from in-the-moment business insights. They frequently articulate the need for real-time visibility into business data to support agile business decisions. But existing business intelligence (BI) tools often lack the broad context, ease of data access, and real-time insights needed to understand and improve customer experience and complex business processes.

The key challenges include:

• Business data is often difficult to access, resulting in fragile data pipelines.
• Data is not delivered in real time; it’s often delayed by weeks or longer.
• Business data often lacks IT context, which prevents effective BizOps collaboration.

Dynatrace business events address these systemic problems, delivering real-time business observability to business and IT teams with the precision and context required to support data-driven decisions and improve business outcomes.

Dynatrace business events provide precise, real-time business metrics that support fine-grained business decisions and auditable business reporting. They offer lossless access to hard-to-reach business data embedded in in-flight application payloads, ensuring that valuable information is not missed. Additionally, Dynatrace business events enable organizations to explore and analyze large, long-term data sets without pre-indexing, which allows for flexible and comprehensive data analysis.

Extend business observability to data at rest

In our past blog post about business agility, we looked at a retail sales use case example to investigate potential causes of underperforming store locations. We also looked at a pizza chain example, connecting each customer order to the fulfillment process milestones that followed, including the handoff to the delivery agent.

In both examples, we used Dynatrace OneAgent® deep payload inspection to capture business data in motion. There are also many cases where business data—transactional, inventory, or financial—is at rest or in use, stored in a database. For comprehensive business observability, you need access to this data in real time. This can be accomplished using Dynatrace extensions. Dynatrace extensions can easily query data from various databases and store the results in Grail™, the Dynatrace data lakehouse. Once the data is in Grail, it can be transformed, queried, reported to dashboards, and more.

Business data is more than metrics

Dynatrace Extensions enable the expansion of Dynatrace monitoring to encompass any technology that provides an interface. For instance, the SQL datasource facilitates universal database queries across commonly used databases, subsequently transmitting the results to Dynatrace in the form of metrics or logs.

However, in the real world, business-related data isn’t limited to metrics. Business data should be viewed through a different lens, storing it separately while preserving the unique characteristics that enable business observability:

• Certain business data, such as product names, customer details, sentiments, order dates, payment methods, and more, are not simple metrics. Instead, they can consist of various data types: strings, integers, float, timestamps, and combinations of values.
• Such business observability can’t reside in traditional databases or data warehouses and thus needs to be in a data lakehouse that can unify and contextually analyze observability, security, and business.
• Metrics lack the contextual information to automatically trigger actions such as targeted outreach to impacted customers or automations to remediate process anomalies. Business events, however, capture specific occurrences or actions, allowing organizations to understand triggers, respond promptly, and foster collaboration among teams for improved customer experiences and business outcomes.

To get past the basic metric limitations, we created a custom extension to extract business data from existing databases and store it in Grail. Here’s a peek at the approach:

Business observability

Business observability refers to gaining insights into a business’s operation, performance, and behavior in real time. It involves collecting and analyzing data from various sources within an organization, such as IT systems, applications, customer interactions, and business processes, to gain a comprehensive view of how the business is functioning. An effective business observability solution should make it easy to ingest business data from any source, including databases.

Similar to the concept of observability in IT systems and applications, business observability focuses on capturing data at different layers of the business and making it easily accessible and understandable for analysis and decision-making. It goes beyond traditional business intelligence by providing real-time, granular, and contextual data that enables organizations to identify patterns, trends, anomalies, and correlations across different business dimensions.

Illustrating the value of business observability

Business observability helps you understand and evaluate the performance and effectiveness of systems in achieving their intended business goals. While observing individual requests is essential for performance engineering purposes, taking a business lens perspective provides deeper insights into the actual value delivered by the underlying system.

For example, consider an e-commerce website aiming to maximize sales. By implementing business observability, you can analyze conversion rates, sales patterns, and order fulfillment times. This enables you to identify bottlenecks, optimize user experiences, and make data-driven decisions to improve sales performance.

Similarly, in the case of a ride-sharing app, business observability allows you to monitor metrics like ride acceptance rates, driver and rider satisfaction, and average wait times. By analyzing these business-oriented indicators, you can optimize an app’s algorithms, allocate resources effectively, and enhance the overall experience for both riders and drivers.

For an insurance provider, business observability provides insights into key metrics such as policy sign-ups, claim processing times, and customer satisfaction levels. By closely monitoring these business-focused metrics, you can identify areas for improvement, streamline processes, and deliver better service to your customers.

Business observability not only ensures that systems perform well technically, it also ensures that systems are aligned with their intended business objectives. By gaining visibility into the business value delivered by these systems, you can make informed decisions, optimize performance, and ultimately achieve your business goals more effectively.

The post Extend business observability: Extract business events from online databases (Part 1) appeared first on Dynatrace news.

Digital experience is often considered the most important customer-facing aspect of digital commerce. This is typically the first thing that comes to mind for IT professionals working in the retail industry when evaluating holiday readiness. While digital experience has many facets, transaction speed usually ranks among the most important. Almost two decades ago, a Google experiment showed that fast-loading transactions are more important to customers than content quality—even small increases in transaction delay result in substantially more abandoned sessions. That lesson remains important. (Though the three-second rule for page load time is often misinterpreted).

CEOs of hybrid retailers prioritize e-commerce growth over in-store shopping, investing heavily in their online storefronts. IT teams spend months preparing for the peak traffic they anticipate will arrive with holiday shopping. However, this is a dynamic target; shopping behaviors are increasingly unpredictable, customer expectations continue to rise, and fierce competition makes cultivating loyalty more challenging than ever. These challenges can be summarized by this quote, paraphrased here from Adobe’s 2021 Digital Trends report: “Your customers are digital, unpredictable, and easy to lose.”

From first to lasting impressions

But there’s more to digital experience than speed. Digital experience, measured by fast, frictionless user journeys, paints an incomplete picture, tracking just the beginning of the customer relationship. What happens after the conversion creates a lasting impression with a larger impact on loyalty and your business.

Let’s shift our focus to the backend systems and business processes, the behind-the-scenes heroes of end-to-end customer experience. These retail-business processes must work together efficiently to orchestrate customer satisfaction:

• Inventory management ensures you can anticipate and meet dynamic customer demand.
• Order processing workflow is triggered by customer orders.
• Order fulfillment is the packaging and delivery of orders to customers.

From a customer perspective, the nuances of these business processes are uninteresting as long as they work. Increasingly, however, order fulfillment is a differentiating customer-facing aspect of the end-to-end customer journey, often with digital touchpoints woven into the experience. The fulfillment clock starts ticking the moment a customer purchases your product. Yet fulfillment is often an area over which retailers have little visibility or control.

Customers value real-time visibility into order status and delivery tracking. However, these fulfillment processes are often strained under the pressure of increased online shopping, next-day delivery expectations, and environment-friendly choices. Flexible delivery options, including “buy online, pick up in store” (BOPIS), curbside pickup, self-service lockers, and gig economy delivery require even greater real-time coordination to commit to competitive and narrowing delivery windows. Decentralized last-mile delivery strategies such as micro-fulfillment centers complicate inventory management and order fulfillment oversight.

Technology to the rescue?

Solutions such as inventory management, order management, and delivery optimization can introduce new challenges:

System integration. To effectively leverage multiple systems to manage orders, inventory, and logistics, retailers must invest in often complex integrations. Unsynchronized and siloed data prevents real-time decision-making and business automation.

Multi-channel logistics. Most retailers work with multiple carriers to handle deliveries, resulting in disparate tracking systems. Aggregating tracking information and presenting it to customers in a uniform way can be a challenge.

Real-time updates: Customers expect real-time visibility into fulfillment milestones beyond order confirmation, including packing, shipping, and delivery notifications. Self-service tracking information, preferred by most customers, becomes especially difficult when there are delays or disruptions.

Embracing business observability

Successful retailers benefit from real-time insights into business processes across all milestones. While each system and service provider might adhere to SLOs, the end-to-end health of the process is greater than the sum of its parts. How can you discover optimization opportunities, patterns behind recurring disruptions, or the root cause of an anomaly? The answer lies in the context—connecting business process KPIs to system performance becomes the starting point for real-time business/IT collaboration and automated remediation. The resulting agility supports targeted responses to process disruptions, anomalies, and bottlenecks as they happen, not when daily or weekly reports are produced, not when your call center is inundated, not when your Net Promoter Score (NPS) plummets. To accomplish this transformation, IT teams need to expand their observability scope to include business KPIs.

How Dynatrace can help

Recent platform innovations have made monitoring end-to-end business processes such as order fulfillment easier. Consider these requirements for effective business observability.

• Business data must be accurate to instill the confidence to make business decisions.
• Business data can come from many sources, including OneAgent, RUM, external business systems, and log files.
• Business data must be easy to access without modifying code to reduce the burden on development and maintenance resources.
• Business data must remain granular over long retention periods to support long-running business processes and “needle in the haystack” queries.
• Business data must be unified, regardless of the source or data type.
• Business data must be easily queried to answer unanticipated questions without upfront indexing.

Business events deliver real-time business observability to business and IT teams with the precision and context to support data-driven decisions and improve business outcomes. Business events extract critical business data from your IT systems with lossless precision and can illuminate dark data quickly and easily, wherever that data exists.

Order fulfillment process example

Retail order fulfillment is a good example of business process monitoring, a use case enabled by these innovations. Fulfillment processes vary between retailers, with subprocesses that might introduce branches and loops. It’s a good practice to identify process milestones as a starting point; these should be relatively consistent. For example:

1. Purchase confirmation
2. Order picked from the warehouse
3. Shipping label created
4. Order accepted by the delivery agent
5. Delivery confirmation
6. Survey completed

Once you’ve defined the list of milestones, identify where to capture the data.

• Purchase confirmation: E-commerce platform (via OneAgent)
• Order picked: Warehouse management system (via OneAgent)
• Shipping label created: Warehouse management system (via OneAgent)
• Order scanned by delivery agent: Agent logistics system (via API)
• Delivery confirmation: Agent logistics system (via API)
• Survey: VoC solution (via API or database query)

The Business Flow app, developed using Dynatrace® AppEngine, makes it easy to configure business process milestones for an end-to-end view of process throughput, delays, and anomalies.

The post Customer expectations for retail: Beyond digital experience appeared first on Dynatrace news.

Should business data be part of your observability solution?

Technology and business leaders express increasing interest in integrating business data into their IT observability strategies, citing the value of effective collaboration between business and IT. Exploding volumes of business data promise great potential; real-time business insights and exploratory analytics can support agile investment decisions and automation driven by a shared view of measurable business goals.

But are observability platforms—born from the collision between the demands of cloud computing and the limitations of APM and infrastructure monitoring—the best solution for managing business analytics?

Observability fault lines

The monitoring of complex and dynamic IT systems includes real-time analysis of baselines, trends, and anomalies. This is achieved, in part, by establishing actionable statistical accuracy—not necessarily precise accuracy—through practical levels of metric sampling, aggregation, and extrapolation. Statistical accuracy is typically not precise enough for most business use cases, and business actions can’t be automated based on anything less than 100% precise data.

Similarly, many performance analytics goals are served by capturing method calls and arguments to give developers the information they need to diagnose and correct anomalies. Traditional observability solutions don’t capture or analyze application payloads. Such analysis is intentionally excluded from most observability solutions because payload details are unnecessary for DevOps purposes, problematic for agent overhead, and risky for data privacy. Therefore, to add the “Biz” to “DevOps,” observability solutions typically rely on application log files, which incurs significant development and maintenance overhead.

To close these critical gaps, Dynatrace has defined a new class of events called business events. Dynatrace OneAgent^® prioritizes business events over observability metrics to ensure the lossless precision you need to support demanding business use cases. At the same time, deep payload inspection makes it easy to extract important business data locked in application payloads—without writing any code. When combined with Dynatrace Grail™, business events gain long-term, cost-effective storage, unaggregated granularity, instant indexless queries, and automatic Smartscape^® topology context.

Introducing metric extraction from business events

Beginning with Dynatrace SaaS version 1.257, you can extract metrics from ingested business events. Metric extraction is a convenient way to create your business metrics, delivering fast, flexible, and cost-effective analytics. Since these metrics are derived from business events, they inherit key characteristics, including lossless precision and deep access to application payloads. Extracting metrics from business events allows you to use metric expressions, view baselines and trends, and create custom alerts when issues require further evaluation by Dynatrace Davis®, our causational AI engine.

From business events to metrics: the ingest pipeline

All incoming business events must pass through the business event ingest pipeline where transformational processing occurs. It’s at this point—immediately after processing—where you can extract metrics from your events by defining extraction rules.

Extraction rules include the following four fields:

• Key: Always starts with the prefix bizevents.
• Matcher: A matcher-specific DQL query that selects specific incoming business events
• Measure: Choose an event-attribute value or an event-count value
• Dimension: Define up to 50 dimensions

In this example, we use a DQL matcher query to extract the trading_volume attribute value from all incoming business events that have an event.type of com.easytrade.buy-assets. We assign the metric key of bizevents.EasyTrade.TradingDollarVolume, and include the dt.entity.host dimension, which will allow us to split or filter the results by host.

Once defined, custom business-event metrics will be extracted from all future ingested business events that match this rule.

View and analyze

You can use the interactive Data explorer to view, query, transform, and visualize your metrics. Once you’ve refined the visualization of a metric, you can pin it to a dashboard.

Through filtering, aggregation, or arithmetic operations, Data explorer also supports metric expressions that combine one or more existing metrics to create new, on-demand metrics. For example, you could divide the TradingDollarVolume metric by a TradeCount metric to calculate a new metric, AverageTradeValue. The new metric can then be pinned to a dashboard. The Data explorer supports many types of visualizations, making it easy to understand trends or to view the relationship between business and IT metrics.

Metric events for alerting

Metric event configuration can be used to automatically detect anomalies in metric timeseries using thresholds or baselines. The resulting alerts can be based on attribute values or specific event occurrences. The extraction of metrics from business events automatically preserves the Smartscape context of each event, allowing Davis AI to automatically analyze anomalies and identify any relevant impact and root cause.

Conclusion: Metric extraction adds new value to business events

Lossless precision and easy, code-free access to business data distinguish business events from traditional IT metrics. Powered by Grail, business events gain long-term cost-effective storage, unaggregated granularity, instant indexless queries, and automatic Smartscape context.

Extracting metrics from business events delivers additional value, including comprehensive metric calculations, a wide range of visualizations, threshold-based alerting, and AI-driven causation analytics.

What’s next?

Metric extraction from business events gives you new options for monitoring, analyzing, and automating your business use cases. For additional technical insights on extracting metrics from business events, watch this Business Events Observability Clinic.

The post Extract metrics from business events to increase the value of Business Analytics appeared first on Dynatrace news.

Business events are a special class of events, new to Business Observability; together with Grail, our data lakehouse, they provide the precision and advanced analytics capabilities required by your most important business use cases. Business events deliver real-time business data from anywhere and empower business and IT teams to analyze, discover, and automate to improve business outcomes.

Let’s look at the unique enabling characteristics of business events through two use case examples, then see how easy it’s to get started.

What are business events?

Business events are prioritized over metric events and observability data to deliver lossless precision. It’s an important distinction; many business use cases won’t tolerate sampling or other resource-conservation approaches common to operational monitoring. Inventory counts, revenue reports, and service activation flows are just a few examples that demand precision.

Business events capture business data from anywhere; they can come from OneAgent, from real-user monitoring (RUM) sessions, from log files, or from external tools and data sources. Dynatrace automatically enriches business events with topology metadata before storing them in Grail. From there, you use the Dynatrace platform to analyze, discover, and automate—without worrying about where the data is or where it comes from.

This diagram shows a few examples of business events. Note that most of these have a clear business—and not IT—focus, but that’s an arbitrary distinction; business events can include any data important to your use case.

Analytics without boundaries

There are other characteristics important for business use cases, delivered through Grail and Dynatrace Query Language (DQL). These include the following highlights:

• Long-term cost-effective storage to support seasonal trending and forecasting
• Unaggregated granularity to support “needle in a haystack” queries
• Instant indexless storage to support unanticipated retrospective questions
• Smartscape context to support drill-down analytics and promote collaboration

Let’s take a closer look at two use cases that are uniquely enabled by business events and Grail.

Use case 1: real-time business intelligence

Business intelligence tools have earned a reputation for being inflexible, lacking the context and real-time insights needed to understand and improve business processes and customer experience. They are generally backward-looking, relying on data stored in accounting and other business systems for a historical perspective. How can business teams gain the same real-time observability and analytics that IT teams enjoy?

Business events are the answer; with real-time precision and easy access to in-flight business data, Business Observability offers a compelling and agile complement to business intelligence tools. Consider the following business revenue use case:

A retail chain wants to boost the average checkout transaction value by encouraging customers to add additional items to their purchase. To track this goal, regional sales managers need real-time visibility into the number of items per transaction and the transaction value, segmented by store. But the company’s point-of-sale system limits daily reports to sales volume, with additional detail available only in monthly reports.

With a few simple configuration steps (outlined below), OneAgent begins capturing key sales data as business events, sending them to Grail in real time. Immediately, the sales managers can track transaction data for each store, focusing on the stores with the lowest items to transaction ratio.

Once they’ve identified which stores are at risk, the managers want additional granularity to analyze a store’s performance by duty manager and sales associate.

These questions—and others like them—can easily be answered retrospectively through ad hoc queries, with no need for additional configuration.

Use case 2: from digital experience to customer experience

Your digital teams focus on conversion funnels, creating web experiences that encourage them to become customers. We categorize these user journeys as digital experience, and our customers enjoy best-in-class digital experience monitoring.

But the conversion—from visitor to customer—is just the start of the end-to-end customer experience. What happens after the conversion? You can deliver a perfect digital experience and still disappoint your customer. Whether you’re providing gated access to a video, activating a service, or delivering food to your customer’s doorstep, there are critical business processes that must work together to fulfill the expectations you’ve set.

A pizza chain wants to improve customer satisfaction by tracking delivery times from each shop. They already monitor digital experience using RUM for web and mobile users, but lose visibility into the order fulfillment process once the order is received. Using the order ID as the unique identifier, they can easily connect each conversion to the process milestones that follow, including the handoff to the delivery agent; dashboards summarize each shop’s performance. They also decide to add a follow-up survey after each delivery; since the survey responses are also tagged with the order ID, they can learn how delivery delays impact survey results through ad hoc queries.

Deliveries with survey responses ≤ 3.0

Get started with OneAgent for business events

OneAgent is a great starting point for using business events. Deep payload inspection extracts up to 64KB of business data from in-flight application requests and responses, and configuration is simple and doesn’t require code changes.

The Business Analytics documentation covers configuration steps in detail; here we’ll provide an overview.

Configuration overview

Dynatrace business events loosely follow a subset of the Cloud Events specification for describing event data; if you’re familiar with the specification, you’ll recognize the similarity as you configure your business events. For OneAgent, there are just a few mandatory fields to complete your capture rule and start using business events. Refining the rule with specific data fields and transforming the data via processing rules will add to the value you get from the business data you’re capturing.

Configuration example

Consider the pizza shop example above, with this snippet from the JSON file. We want OneAgent to capture the data as a business event for reporting and analysis.

Step 1: Create a capture rule with trigger conditions, event metadata, and data fields.

• The trigger defines the condition that, when met, will cause OneAgent to capture the business event. We’ll use the request path that contains the value /easypizza/purchase.
• The event metadata fields (provider, type, and category) add context, and can be a fixed value or extracted directly from the event itself. We’ll use easypizza.com as the event source, and com.easypizza.purchase as the event type. The event category field is optional.
• Event data fields are used to extract attribute/value pairs from the event. We’ll capture orderId, quantity, and unitPrice.

Step 2: Add processing rules with matcher DQL, data fields, and processor definition.

Note that there is no total amount field in the sample JSON – just a unit price and quantity. We can use a processing rule to add this new field.

• The matcher uses DQL to filter event data for processing. We’ll use matchesValue(action, “purchase”)
• The data fields will be used in the processor definition; we’ll need unitPrice and quantity to derive totalAmount.
• Processing commands define how to transform the matching data; we’ll use FIELDS_ADD(totalAmount: quantity*unitPrice)

The processing rule will add the totalAmount field to the business event stored in Grail.

Step 3: Define the retention period with matcher DQL and bucket assignment.

Business events are assigned to buckets that have pre-defined retention periods.

• The matcher uses DQL to filter business events for bucket assignment. We’ll use matchesValue(event.provider, "www.easypizza.com"). All business events from this event provider will be assigned to the selected bucket.
• Choose from three bucket options: 35 days, 1 year, or 3 years.

Automatic enrichment

Dynatrace automatically enriches business events with important topology and application information. For OneAgent, these include host, process group, and trace, and provide the context necessary to analyze anomalies and foster collaboration. Intermittent order update failures? Drill down to see the health of the host or process group or open the distributed trace to discover bottlenecks or downstream failures.

The post The value of business events: How IT can increase business agility appeared first on Dynatrace news.

Following the launch of Dynatrace® Grail for Log Management and Analytics, Dynatrace is excited to announce a major update to our Business Analytics solution. Business events powered by our new Grail™ data lakehouse and by other Dynatrace platform technologies ensures the real-time precision that business and IT teams need to make data-driven decisions and improve business outcomes. Business events deliver the industry’s broadest, deepest, and easiest access to your critical business data. Powered by Grail, you can now unify, store, and instantly analyze massive volumes of business data from anywhere, automatically enriched with the IT context needed to unlock precise AI-powered answers and automation.

The need for real-time business observability

Business leaders benefit from in-the-moment business insights, frequently articulating the need for real-time visibility into business data to support agile business decisions. In fact, business teams are often envious of the real-time IT observability that operations teams enjoy. That’s because their existing business intelligence tools often lack the broad context, ease of data access, and real-time insights that are needed to understand and improve customer experience and complex business processes. Business intelligence tools are generally backward-looking, relying on accounting, CRM, ERP, and other business systems to surface anomalies, report results and trends, or mine historical data. To highlight the problem, a recent report from Dimensional Research indicates that 86% of the data used by business analysts is out of date, often by two months or more. While last week’s or last month’s data might be acceptable for historical reporting, it doesn’t help teams respond to dynamic business conditions or react to process anomalies.

The business observability gap

Over the past few years, observability vendors have sought to address some of these analytics challenges, opportunistically introducing real-time reporting for a limited set of business metrics. Using existing APM agent and log monitoring capabilities made it reasonably easy to access certain business metrics and metadata to add to IT dashboards. But these solutions have largely fallen short of business and IT users’ needs. Since they rely on capabilities designed for IT monitoring, they inherit a series of architectural design constraints that limit their usefulness. Top among these are sampling-induced imprecision, limited access to critical business data embedded in application payloads, and lack of context.

Business events, Grail, and OneAgent

Business events, Grail, and OneAgent work together to overcome these limitations, prioritizing business data separately from observability data to deliver precise, real-time business insights with deep visibility into application payloads. This is an entirely new approach to monitoring, investigating, and acting on business metrics. It represents a significant enhancement to our previous Business Analytics capabilities, which emphasized the value and simplicity of business data captured from real user sessions. Business events can be captured by Dynatrace OneAgent, sent from web or mobile real user monitoring RUM, from external sources via public API, or (coming soon) extracted from log files. Business events make it easy for teams to spot business anomalies, long-term trends, and process optimization opportunities, answering even the most challenging business questions with accuracy and clarity. Since business events are automatically enriched with IT context, business and IT teams can collaborate effectively to achieve shared business goals.

Unlike business analytics offerings from other observability vendors, Business Observability easily supports a wide range of important business use cases that demand auditable accuracy. From order fulfillment and bill payments to service activation flows and customer onboarding milestones, business events deliver on the promise of real-time business observability. Business and IT teams can monitor and optimize complex business processes that might span weeks or months, uncovering anomalies before they affect business outcomes. Deep access to in-flight application payloads makes it simple to capture critical business data without code changes. Grail unifies business and IT data at a massive scale for powerful exploratory analytics to answer even unforeseen questions as they arise.

Here’s a quick look at how business events and Grail work together:

1. Business event sources include OneAgent, web and mobile RUM sessions, external business tools via a public API, and (coming soon) log files.
2. Business events are processed through the ingest pipeline using the Dynatrace Query Language (DQL) to transform and enrich the data and set retention periods.
3. Grail unifies business and IT observability data from all sources, leveraging the Smartscape® dependency model to automatically build causational context.
4. Dynatrace Query Language provides exploratory analytics through ad hoc queries without the need to index. Explore interactively with the Log and event explorer, pin your queries to dashboard tiles, or execute queries via the API.

Unlock the value of your business data

With business events from Dynatrace, business and IT teams enhance collaboration and improve agility by leveraging real-time insights and operational analytics for informed business decisions.

Grail, DQL, and business events are available now in select AWS regions and will be available on Azure in early 2024. To learn more or to get started, contact your Dynatrace representative.

What’s next for Business Observability?

There’s much more in the works, using Grail as a game-changing enabling technology. Next to come will be business events from logs, complementing existing OneAgent, RUM, and external sources.

The post Dynatrace enhances Business Analytics with business events powered by Grail appeared first on Dynatrace news.