Testing for mishaps you can predict is essential. But with the complexity that comes with digital transformation and cloud-native architecture, teams need a way to make sure applications can withstand the “chaos” of production. Chaos engineering answers this need so organizations can deliver robust, resilient cloud-native applications that can stand up under any conditions.

What is chaos engineering?

Chaos engineering is a method of testing distributed software that deliberately introduces failure and faulty scenarios to verify its resilience in the face of random disruptions. These disruptions can cause applications to respond unpredictably and break under pressure. Chaos engineers ask why.

Practitioners subject software to a controlled, simulated crisis to test for unstable behavior. The crisis could be technical, natural, or malicious events, for example, an earthquake affecting data center availability, or a cyberattack infecting applications and websites. As software performance degrades or fails, the chaos engineers’ findings enable developers to add resiliency to the code, so the application remains intact in an emergency.

As chaos engineers grow confident in their testing, they change more variables and broaden the scope of the disaster. The many disaster scenarios and outcomes allow chaos engineers to better model what happens to applications and microservices, which gives them increasing intelligence to share with developers to perfect software and cloud-native infrastructure.

The history of chaos engineering

Netflix pioneered chaos engineering out of necessity. In 2009, the purveyor of online videos migrated to AWS cloud infrastructure to deliver its entertainment to a growing audience. But the cloud brought new complexities, such as increasing connections and dependencies. It created more uncertainty than the load-balancing issues the entertainment firm saw in its data centers. If any touchpoint in the cloud failed, the quality of the viewers’ experience could degrade. So, the organization sought to reduce complexity and raise production quality.

In 2010, Netflix introduced a technology to switch production software instances off randomly—like setting a monkey loose in a server room—to test how the cloud handled its services. Thus, the tool Chaos Monkey was born.

Chaos engineering matured at organizations such as Netflix, and gave rise to technologies such as Gremlin (2016), becoming more targeted and knowledge-based. The science has spawned specialized chaos engineers who dedicate themselves to disrupting cloud software and the on-prem systems they interact with to make them resilient. Now, chaos engineering is an established profession, stirring up managed trouble to stabilize cloud software.

How does chaos engineering work?

Chaos engineering starts with understanding the software’s expected behavior.

1. Hypothesis. Engineers ask themselves what should happen if they change a variable. If they terminate services randomly, they assume the service will continue uninterrupted. The question and the assumption form a hypothesis.
2. Testing. To test the hypothesis, chaos engineers orchestrate simulated uncertainty combined with load testing and watch for signs of upheaval in the services, infrastructure, networks, and devices that deliver the application. Any failures in the stack break the hypothesis.
3. Blast radius. By isolating and studying failures, engineers can understand what happens under unstable cloud conditions. Any damage or influence caused by the test is known as the ‘blast radius.’ Chaos engineers can manage the blast radius by controlling the tests.
4. Insights. The discoveries form inputs into the software development and delivery process, so new software and microservices will better stand up to unforeseeable events.

To mitigate damage to production environments, chaos engineers start in a non-production environment, then slowly extend to production in a controlled way. Once established, chaos engineering becomes an effective way to fine-tune service-level indicators and objectives, improve alerting, and build more efficient dashboards, so you know you are collecting all the data you need to accurately observe and analyze your environment.

Who uses chaos engineering?

Chaos engineering generally originates from small teams within DevOps, often involving applications running in both pre-production and production environments. Because it can touch many systems, chaos engineering can have broad implications, affecting groups and stakeholders across the organization.

A disruption spanning hardware, networks, and cloud infrastructure can require input and participation from network and infrastructure architects, risk experts, security teams, and even procurement officers. That’s a good thing. The greater the scope of the test, the more useful chaos engineering becomes.

Although a small team generally owns and manages the chaos engineering effort, it’s a practice that often requires input from—and provides benefits to—the village.

The benefits of chaos testing

The insights you can gain by testing the limits of your applications deliver a lot of benefits for your development teams and your overall business. Here are just a few benefits of a healthy, well-managed chaos engineering practice.

• Increases resiliency and reliability. Chaos testing enriches the organization’s intelligence about how software performs under stress and how to make it more resilient.
• Accelerates innovation. Intelligence from chaos testing funnels back to developers who can implement design changes that make software more durable and improve production quality.
• Advances collaboration. Developers aren’t the only group to see advantages. The insights chaos engineers glean from their experiments elevate the expertise of the technical group, leading to response times and better collaboration.
• Speeds incident response. By learning what failure scenarios are possible, these teams can speed up troubleshooting, repairs, and incident response.
• Improves customer satisfaction. Increased resilience and faster response times mean less downtime. Greater innovation and collaboration from development and SRE teams means better software that meets new customer demands quickly with efficiency and high performance.
• Boosts business outcomes. Chaos testing can extend an organization’s competitive advantage through faster time-to-value, saving time, money, and resources, and producing a better bottom line.

The more resilient an organization’s software is, the more consumers and business customers can enjoy its services without distraction or disappointment.

The challenges and pitfalls of chaos engineering

Although the benefits of chaos testing are clear, it is a practice that should be undertaken with deliberation. Here are the top concerns and challenges.

• Unnecessary damage. The major concern with chaos testing is the potential for unnecessary damage. Chaos engineering can lead to a real-world loss that exceeds the allowances of justifiable testing. To limit the cost of uncovering application vulnerabilities, organizations should avoid tests that overrun the designated blast radius. The goal is to control the blast radius so you can pinpoint the cause of failure without unnecessarily introducing new points of failure.
• Lack of observability. Establishing that control can be easier said than done. Lack of end-to-end observability and monitoring of all systems a blast radius might affect is a common problem. Without comprehensive observability, it can be difficult to understand critical dependencies vs non-critical dependencies, or to have adequate context to understand the true business impact of a failure or degradation in order to prioritize fixes. Lack of visibility can also make it difficult for teams to determine the exact root cause of an issue, which can complicate remediation plans.
• Unclear starting system state. Another issue is having a clear picture of the starting state of the system before the test is run. Without this clarity, teams can have difficulty understanding the true effects of the test. This can diminish the effectiveness of chaos testing and can put downstream systems at greater risk, which makes it harder to control the blast radius.

How to get started with chaos engineering

As with any scientific experiment, getting started with chaos engineering requires a little preparation, organization, and the ability to monitor and measure results.

1. Know the starting state of your environment. To plan a well-controlled chaos test, you should understand the applications, microservices, and architectural design of your environment so you can recognize the effects of the test. Having a baseline you can compare to the end state creates a blueprint for monitoring during the testing and analyzing results after.
2. Ask what could go wrong and establish a hypothesis. With a clear idea of the system’s starting state, ask what could go wrong. Understand service-level indicators and service-level objectives and use them as a basis for establishing an assumption for how the system should work under pressure.
3. Introduce chaos one variable at a time. To keep control of the blast radius, introduce only a little chaos at a time, so you can appreciate the results. Be prepared to abort the experiment under certain conditions, so no harm comes to production software, and also have a roll-back plan if something goes wrong. During the test, try to disprove the hypothesis to discover areas to focus on to improve system resilience.
4. Monitor and record the results. Monitor the experiment to record any nuances in application behavior. Analyze the results to see how the application responds and whether testing achieved the teams’ expectations. Use investigation tools to understand the precise root causes of slow-downs and failures.

Controlling the chaos

Solutions like Gremlin provide crucial management tools to plan and execute chaos engineering experiments. It makes experiments repeatable and scalable so teams can apply them to future experiments of the same or larger stacks.

Automatic and intelligent observability from Dynatrace delivers insights into the effects of chaos testing, so engineers can steer chaos experiments with care. To monitor the blast radius, Dynatrace observes the systems undergoing chaos experiments. With visibility across the full software stack, Dynatrace provides crucial contextual analysis to isolate the root cause of failures exposed by chaos testing.

Effective monitoring from Dynatrace offers an essential panoramic lens for the engineers driving chaos testing, helping them understand dependencies and predict how outages will affect the system at large. Should the chaos reach further than intended, insights from Dynatrace help teams quickly remediate any actual harm to the application’s functionality.

Organizations can achieve application resiliency in any stage of digital transformation, and chaos engineering is a great tool. However, before playing with fire, it’s critical to have the right measures in place to predict and cope with the multitude of failure scenarios this approach can bring.

The post What is chaos engineering? appeared first on Dynatrace news.

Shift left and shift right are core testing concepts of the agile DevOps methodology. This methodology speeds up application development by releasing small builds frequently as code evolves. As part of the continuous cycle of progressive delivery, DevOps teams are also adopting shift left and shift right principles to ensure software quality in these dynamic environments.

All this shifting may sound abstract, but I’ll explain how this software verification approach benefits DevOps methods and outcomes—and makes software more reliable.

In DevOps, what is shift left? And what is shift right?

To understand shift left and shift right, consider the software development cycle as a continuum, or infinity loop, from left to right.

On the left side of the loop, teams plan, develop, and test software in pre-production. The main concern in pre-production on the left side of the loop is building software that meets design criteria.

When teams release software into production on the right side of the loop, they make it available to users. The concern in production is maintaining software that meets business goals and reliability criteria.

What does shift left mean in DevOps?

Shift left is the practice of moving testing, quality, and performance evaluation early in the development process, often before any code is written.

Shift-left testing helps teams anticipate changes affecting performance or other delivery processes that arise during the development process. With shift-left testing, teams verify APIs, container configurations, interactions between microservices, and container configurations.

Why is shift left important?

In addition to testing for functionality, shift left testing also checks that software meets customer requirements. This enables developers and stakeholders to identify improvements that could enhance customer experience and functionality. Making these changes early in the development process reduces the cost of making them after the code is released.

As a result, shift left testing has become increasingly important as teams face pressure to deliver software faster and more frequently with higher quality. Shift left speeds up development efficiency and reduces costs by detecting and addressing software defects earlier in the development cycle before they get to production.

The benefits of shift left

The concept of shift left has been gaining traction in the world of software development and testing. This approach, which introduces testing and quality assurance early in the development lifecycle, offers several considerable benefits. 

1. Early Bug Detection: By testing early in the development process, teams can identify and address bugs when they are easier and less costly to fix. This not only improves the quality of the final product but also reduces the time and resources spent on debugging and rework.
2. Faster Time to Market: With shift left, testing is integrated into every stage of development. This continuous testing allows for faster feedback and quicker iterations, which can significantly reduce the overall time to market.
3. Improved Customer Satisfaction: By ensuring that quality is a priority from the outset, shift left helps to deliver a product that meets or exceeds customer expectations. This can lead to higher customer satisfaction and loyalty.
4. Reduced Costs: While implementing shift left may require an initial investment, the cost savings from reduced rework and faster time to market can quickly offset this. In addition, by catching and fixing issues early, teams can avoid the potentially high costs associated with late-stage or post-release fixes.
5. Enhanced Collaboration: Shift left encourages collaboration between testers, developers, and other stakeholders from the beginning of the project. This collaborative approach can lead to better communication, shared understanding of the product, and a better end result.

What does shift right mean in DevOps?

Shift right is the practice of performing testing, quality, and performance evaluation in production under real-world conditions. Shift-right methods ensure that applications running in production can withstand real user load while ensuring the same high levels of quality.

With shift right, DevOps teams test a built application to ensure performance, resilience, and software reliability. The goal is to detect and remediate issues that would be difficult to anticipate in development environments.

Why is shift right important?

With shift right, teams can test code in an environment that mimics real-world production conditions that they can’t simulate in development. This practice enables teams to catch runtime issues before customers do. To automate part of the process, teams can use application programming interface (API) calls. Organizations can also apply shift-right testing to code configured or monitored in the field. The result is more comprehensive testing coverage that better addresses user experience concerns.

Like shift left testing, the objective of shift right testing is to fail small and fail fast. The assumption is that problems caught early in the pre-deployment environment are easier to solve than issues caught by customers in live production.

Shift right testing is especially useful for organizations practicing progressive delivery, wherein developers release new software features incrementally to minimize the impact of unforeseen issues. Testing in a production-ready environment is a crucial final phase before declaring features ready for prime-time.

The benefits of shift right

Shift right is a concept that extends testing into the production environment. It involves monitoring and testing systems in production to gain insights and improve quality, offering several significant advantages.

1. Realistic User Testing: Shift right testing involves testing with real users. By including them in the process, software professionals gather valuable feedback about how users interact with the software. This helps make informed decisions about future improvements.
2. Continuous Feedback Loop: Shift right testing promotes continuous monitoring and improvement throughout all phases of development. This iterative feedback loop ensures that issues are addressed promptly, leading to better product quality.
3. Test Coverage: By extending testing to real-world scenarios, shift right testing provides broader test coverage. It helps identify issues that might not appear during earlier development stages, resulting in more comprehensive testing.
4. Enhanced Monitoring: Monitoring software behavior in production environments allows for better visibility into its performance, stability, and user experience. This helps teams proactively address any anomalies or performance bottlenecks.
5. Customer-Centric Approach: Shift right testing prioritizes users’ needs and experiences. By involving real users, software teams gain insights that lead to improvements aligned with customer satisfaction.

Why shift right and shift left are critical for microservices architecture

Both shift left and shift right testing have become more important as software is increasingly built from microservices. Instead of applications being built and run as a monolith, microservices are loosely coupled functions called at runtime. How microservices-based applications perform depends on individual service responsiveness, which complicates testing in a simulated environment.

Shift left testing allows teams to observe how services interact in a simulated environment and detect potential performance impacts before deploying the code.

Shift right testing at or near the deployment point allows teams to observe real-world forces and measure their impact. Shift-right tests typically cover functionality, performance, failure tolerance, and user experience. Teams often automate such production-environment testing and translate feedback into technical specifications for developers.

With shift-left and shift-right testing, testers can isolate issues so teams can fix and tackle improvements in parallel. As an application becomes more stable, teams can start testing and optimizing performance. Notably, shifting left and right have become an important practice in Agile software development. Together, they’re part of the continuous feedback loop that characterizes DevOps.

Types of shift right tests

A shift right approach may enlist various types of test suites. The following are tests your team might find useful:

• A/B testing. Web designers commonly use this method. Users see two versions of a page, and testers measure which generates a greater response. Teams often conduct this type of test in a production environment to gather real-world feedback.
• Synthetic monitoring. Synthetic monitoring is a form of shift-right testing that uses software tools to emulate the paths users might take when engaging with an application. With synthetic monitoring, you can automatically track application uptime and how your application responds to typical user behavior. It uses scripts to generate simulated user behavior for various scenarios, geographic locations, device types, and other variables.
• Chaos engineering. With chaos engineering, also called chaos testing, developers intentionally “break” the application by introducing errors to determine how well it recovers from disruption. DevOps and IT teams set up monitoring tools to see precisely how the application responds to different types of stresses. Teams perform these tests in a controlled production environment to minimize the impact on mission-critical systems.
• Canary releases. This strategy is named for the canaries that miners use to lower into coal mines to detect toxic gases. Technology has thankfully rendered this inhumane tactic obsolete. However, the term survives to describe slowly rolling out changes to a small subset of instances for testing before teams apply them to the full infrastructure.
• Blue-green deployment. With a blue-green deployment, an organization runs two nearly identical production environments, shifting users (real or synthetic) between the two as they make small changes to one or the other. This practice is important to shift-right methodology as it can minimize downtime. It also provides a mechanism for rapid rollback should something go wrong with the latest version.

Shift right tools

Shift right tools focus on the examination of code quality after it has reached production. Such tools are important for determining and analyzing how software behaves in real-word environments, offering teams insights into its actual performance. Examples of shift right tools include: 

• Application Performance Monitoring (APM) tools: APM tools offer metrics, logs, and traces relating to application performance, giving teams insights into how to remediate issues faster and further optimize their applications 

• Digital Experience Monitoring (DEM) tools: DEM tools provide insights into the user experience (UX) of an application. These tools may offer: 

• Real-user monitoring (RUM) for analyzing the behavior and experience of real-world users. 

• Synthetic monitoring that performs artificially simulated user journeys to test for any UX bottlenecks or deficiencies 

• Session replay that provides video-like documentation of real user journeys 

• Security monitoring and incident response tools: These tools help detect, manage, and communicate security incidents and facilitate the response process. They also assist teams in prioritization processes to ensure that the most business-impacting vulnerabilities are addressed first. 

• Chaos engineering tools: These tools introduce simulated failures to test the resiliency of code and the efficacy of response processes to such failures. This helps inform teams of any code deficiencies and opens room for improvement 

• Feature flagging tools: These tools provide capabilities to toggle certain features of a deployment on and off without deploying any additional code. Such tools offer a greater level of control to developers, as they allow for more experimentation and finetuning of the user experience. Feature flagging provides increased reliability in feature rollouts and helps to ensure that a deployment provides the best user experience possible. 

Types of shift left tests

The four basic types of shift left testing are the following:

• Traditional shift left testing. The traditional approach focuses on unit and integration testing, typically involving APIs and cross-browser compatibility. This approach focuses more on unit testing small pieces of code to deliver information early and often rather than more systems-level operational and acceptance testing.
• Incremental shift left testing. This is a popular approach for teams migrating from a waterfall development approach to one that breaks complex projects into smaller pieces. In this scenario, teams conduct systems-level operational and acceptance testing on a smaller, more incremental scale. Incremental testing is popular for validating large and complex systems.
• Agile/DevOps testing. This is a series of short, continuous sprints during the development stage. This approach does not intend to address operational performance but rather validate adherence to basic requirements architecture.
• Model-based shift left. Model-based testing aims to reduce errors introduced during the requirements definition, architecture, and design phases. It tests for executable requirements, architecture, and design. The advantage of model-based testing is that it can begin almost immediately instead of after completing all other test cycles.

Shift left tools

1. Security Scanning Tools:

These tools streamline the integration of security with DevOps processes by identifying vulnerabilities early in the development cycle. They help ensure that code is secure as it is being developed, rather than waiting until the testing phase. 

Examples of security scanning tools include: 

• Static Application Security Testing (SAST): SAST tools analyze the source code without executing it. They identify potential security issues, coding standards violations, and vulnerabilities. Linters (e.g., ESLint for JavaScript, pylint for Python) fall into this category. 

• Dynamic Application Security Testing (DAST): DAST tools test the application during runtime by simulating attacks. They identify vulnerabilities that can be exploited by attackers. Examples include OWASP ZAP and Burp Suite. 

• Dependency Scanning: These tools check for vulnerabilities in third-party libraries and dependencies used in the application. 

• Secrets Detection: Tools like GitGuardian scan code repositories for exposed secrets (e.g., API keys, passwords) and prevent accidental leaks. 

• Integrated Application Security Testing (IAST): IAST tools combine static and dynamic analysis to provide comprehensive security coverage. 

2. Quality control tools

These tools help ensure and validate that code in the DevOps pipeline is not only secure but also functional and effective. Implementing such tools early in the development process helps guarantee that errors or deficiencies are caught and corrected before reaching production. 

• Static Code Analysis tools: Similar to SAST tools, Static Code Analysis tools assess the source code without executing it to determine whether the code is compliant, bug-free, and generally of good quality. 

• Code review tools: Code review tools help automate the code review process by organizing and displaying code changes, facilitating communication between developers regarding changes, and examining the efficacy of the code review process.

• Continuous integration (CI) tools: CI tools enable developer collaboration by automatically integrating code from multiple developers into a single repository, such as GitHub, GitLab, and Bitbucket.

The application security dividend of shift right and shift left

An important benefit of shifting right is improved application security. “Scanning a static image, either in a repository or in a development environment, can’t give you the same rich insights you can get if you observe the application running in production,” a Dynatrace report on security evolution in the cloud notes. “For example, you don’t get to see what libraries are actually called, how they are used, whether a process is exposed to the Internet, or whether a process interacts with sensitive corporate data.”

The growing use of software containers has complicated aspects of cybersecurity. Containers can obscure the processes running in them, and attackers even containerize exploits. Production testing exposes the behavior of container-based software, even if the contents of containers are obscured. Organizations can also use shift-right testing to test for the presence of novel zero-day exploits, such as Log4Shell.

From a shift left perspective, security testing during development helps identify vulnerabilities as early in the life cycle as possible, when they are easiest to remediate.

Shift left, shift right, or both?

As organizations modernize their application stacks around cloud-native constructs like microservices and containers, a best practice is to adopt both shift left and shift right strategies. Shift left testing reduces software defects and speeds up time to market. Shift right better ensures reliability in production by testing under real-world conditions to affirm that software is doing its job where it matters most: serving the user. The combination of the two moves the organization closer to the ultimate goal of continuous integration and continuous delivery using automation and interdisciplinary communication. The result is a faster and more agile organization.

An essential capability for comprehensive shift left and shift right testing is automated full-stack observability and monitoring. With the capacity for end-to-end analysis at all phases of development, DevOps teams can monitor all requests and processes. Automatic analysis of all services across a sprawling complex of multicloud applications from a single interface enables teams to automate. Testers can push deployment information and metadata to the monitoring environment using scripts to track builds, revisions, and configuration changes.

The better an automated DevOps platform understands the full context of an issue, the better it can detect root causes, flag with the proper parties, and even implement self-healing measures.

Whether your organization has shifted testing left to the development phase or right in production — or simply wants to monitor performance in the field—an AI-driven, full-stack observability solution can take your software development to the next level.

The post Shift left vs shift right: A DevOps mystery solved appeared first on Dynatrace news.

This guest blog is authored by Raphael Pionke, DevOps Engineer at T-Systems MMS. Credits on content go to him and the work he has been doing around performance & resiliency testing automation. I, Andreas Grabner, helped to bring the blog to life!

Our Application Performance Management (APM) and load test team at T-Systems MMS helps our customers reduce the risk of failed releases. To do so we have successfully established AI-based White box load and resiliency testing with JMeter and Dynatrace, helping identify and resolve major performance and scalability problems in recent projects before deploying to production.

In recent years, customer projects have moved towards complex cloud architectures, including dozens of microservices and different technology stacks which are challenging to develop, maintain, and optimize for resiliency. Therefore, we implemented Dynatrace in almost all major applications for our customers to gain visibility from end-user to code-level and to reduce time with problem fixing and pro-active scalability optimizations by using Dynatrace’s AI-based root cause analysis.

Our customers usually involve us 2-4 weeks before the production release. While this seems very late in the development cycle, our white-box release readiness validation approach allows us to point out concrete recommendations for fixing scalability bottlenecks as well as performance and resiliency improvements.

Black box load testing, where load gets generated against a system you don’t get any insights from while putting it under load, is unimaginable nowadays. Any testing that’s done without getting insights with an observability platform or monitoring solution ends up consuming too much time to analyze problems. It also lacks automation capabilities due to missing data and therefore doesn’t scale. Our solution to modernize this legacy approach is an approach we call white box testing. In white-box testing, we combine open-source load testing tools such as JMeter with Dynatrace’s observability and analytics capabilities. This allows us to provide our services to customers with a focus on these three key pillars:

• Scalability: Our solution uses scalable cloud infrastructure.
• Automation: Single load test executions can be repeated and tracked. Each step is automated from provisioning infrastructure to problem analysis.
• Transparency: We rely on deep insights Dynatrace provides, including distributed tracing of single requests.

Now let’s jump into more technical details on how our solution works and how we connect JMeter with Dynatrace to enable automated root cause detection and resiliency validation:

Step 1: Integrating JMeter with Dynatrace

The Dynatrace documentation and video tutorials give a good overview of integrating any HTTP-based testing tool with Dynatrace. Here’s how we have implemented and extended those practices:

Test context through web request header

We first add the “x-dynatrace-test” header to each request, which contains the current test step and thread group/use case name:

This header is parsed by Dynatrace and can be used during analysis to differentiate between load test requests and other requests (e.g., from other test tools or real users).

Leveraging the W3C Trace context header

While the test step, use case, and thread group name context are helpful we also want to speed up the analysis of individual requests that fail. When JMeter reports an error, it should only be a single click to analyze the captured distributed trace (a Dynatrace PurePath).

A decent solution is the W3C Trace context standard, created by Dynatrace, Google, Microsoft, and others. This request header consists of different components to uniquely identify each individual section of an end-to-end distributed trace. In our case, the parent id and the trace ID were the most important ones.

We generate a unique trace ID for every single request across all threads and JMeter agents. On one side we add this value as an additional HTTP header that is then picked up by Dynatrace. On the other side we save this trace ID together with the failed request and other meta information (screenshot, timestamp, etc.) to form part of our test results:

{
  "TimeStamp": "2021-11-24 14:11:00",
  "VUID": "VU-4",
  "Sample": "0010_Startseite_Labelkauf_RT",
  "ThreadGroup": "Labelkauf - Basic load",
  "APM-TracingId": "00-19c0383f590caeebffbaa09b819f50bd-1637762145174fff-01",
  "SamplerClass": "org.apache.jmeter.protocol.java.sampler.JSR223Sampler",
  "CurrentUrl": "https://mycompany.com/send-parcel/gw/restweb/nepal/portal.html?type=ShipmentEditorProductSelection",
  "failedOnLabeledWebElement": "Section fuer MyCompany Application",
  "Screenshot": "Labelkauf/0010_Startseite_Labelkauf_RT-20211124-141101-VU4-10.175.18.82.png",
  "responseCode": "500",
  "responseMessage": "javax.script.ScriptException: org.openqa.selenium.TimeoutException: Expected condition failed: waiting for visibility of element located by By.cssSelector: section.mm_section.mm_section-primary.position-relative (tried for 20 second(s) with 500 milliseconds interval)",
  "responseData": ""
}

This tracing ID can then be used in the Distributed Traces view in Dynatrace to look up the correlating Dynatrace PurePath for that W3C Trace Context ID. With another click, Dynatrace shows us the full distributed trace (=Dynatrace PurePath) making it easy to identify why a request was slow or failing and which backend services or databases were involved:

The error log of the JMeter load test contains a message about a missing element, which can be traced down to an exception in the application code with the help of the W3C Trace Context Id.

Dynatrace back-end listener for JMeter:

Dynatrace engineering open sourced a backend listener for JMeter that they use internally. It automatically sends JMeter metrics to the Dynatrace cluster via the Metrics Ingest API. These metrics can be used to validate the load test plan or target load and to correlate between different application metrics. The screenshot below shows a Dynatrace dashboard showing JMeter metrics alongside metrics natively captured by Dynatrace:

As you can see here, we have all metrics on one single dashboard and can share them at once together with all team members. In this example, the application has reached its limit near the 90 percent load.

From such kind of dashboard, we can easily read out the tagged web requests for each use case with response time and count. In the given example, the goal was to find the maximum of concurrent requests per hour and to investigate the response time behavior over the test run time. From the graphs can be read that the response times have increased significantly at the end of the test.

The dashboard combines the load test metrics from JMeter with CPU/Memory/JVM metrics for the JMeter infrastructure (Agents and Controller) and the application and infrastructure metrics captured by Dynatrace. The application under test uses an Azure Application Gateway and a load balancer in front of the worker nodes, which are also part of this dashboard as Dynatrace provides automated support for pulling in metrics from cloud services such as Azure.

The white box load testing project setup

After explaining how we integrate JMeter with Dynatrace and what we can do with the data once it is in Dynatrace let’s switch over to explain how we automate performance test execution and analysis as part of our load testing projects. The following illustration gives an overview of our architecture and setup:

Each load test project is defined in a separate git repository with all necessary metainformation:

• Property files for workloads,
• Test data,
• Test plan and
• a Jenkinsfile.

Our Jenkins instance has a seeder job, which creates the jobs by reading all git repositories automatically.

Terraform and Ansible to provision infrastructure and configure Dynatrace

The test infrastructure and the Dynatrace configuration are provisioned by terraform. The “azurerm” terraform provider handles all infrastructure-related tasks. The new Dynatrace terraform provider (https://github.com/dynatrace-oss/terraform-provider-dynatrace) is used to apply all necessary Dynatrace configuration steps:

locals {
  apps = [
    { name : "ABC", zone : "1809_ABC" },
    { name : "DEF", zone : "3328_DEF" }
  ]
}

resource "dynatrace_calculated_service_metric" "teststep-metric" {
  count            = length(local.apps)
  name             = "Teststep avg. response time (${local.apps[count.index].name})"
  enabled          = true
…
 }

resource "dynatrace_calculated_service_metric" "usecase-metric" {
  count            = length(local.apps)
  name             = "UseCase avg. response time (${local.apps[count.index].name})"
  enabled          = true
…
  }

resource "dynatrace_request_attribute" "x-dynatrace-test_NA" {
  name                       = "x-dynatrace-test NA"
  enabled                    = true
…
}

resource "dynatrace_request_attribute" "x-dynatrace-test_PC" {
  name                       = "x-dynatrace-test PC"
  enabled                    = true
…
}

resource "dynatrace_dashboard" "golden_signals" {
  dashboard_metadata {
    name   = "${local.apps[count.index].name} – Jmeter loadtest results"
    owner  = "Dynatrace"
    shared = true
    filter {
      timeframe = "-1h to now"
      management_zone {
        name = local.apps[count.index].zone
      }
    }
  }
}

This file shows how to implement a starter package for each new project.

After creating the test infrastructure, we use Ansible as a configuration management tool to install docker, prepare the Jenkins Agents and the Dynatrace OneAgent.

Test Result Analysis

Each test run is documented in our self-developed test management tool to track configuration changes between different test runs:

The test management tool we developed can be used to call different user interfaces of our test automation, i.e. Jenkins, Git, JMeter, the Dynatrace dashboard together with the test timeframe and our self-developed ReportEngine test report:

The result overview currently shows data from JMeter split by Use Case by parsing the JLT raw data.

Next Steps: Collaboration on Keptn

We always strive for more and easier-to-manage automation. Therefore, future projects will use a dedicated Jenkins pipeline library to reduce the manageable code in the Jenkinsfiles and to iterate faster.

Some already established and running load test projects are currently migrating to continuous load testing with pipeline integration rather than involving the load test team for manually triggering tests only when publishing major releases.

Our team is also collaborating with Dynatrace to standardize some of this functionality through the open-source project Keptn. At the core, Keptn uses an SLO (Service Level Objective) scoring approach to automate the analysis across multiple metrics (SLIs). This is functionality we intend to better integrate into Jenkins and our reporting capabilities. Furthermore, Keptn solves the interoperability challenges between multiple tools such as configuration management, infrastructure, deployment, testing, and observability tools. As switching or integrating new tools in an automation process is always a lot of work, we expect Keptn to help us lower the automation efforts in our current Jenkins pipelines. As Dynatrace also offers Keptn orchestration capabilities to their enterprise customers as part of their Cloud Automation solution the entry-level barrier for many organizations is very small!

I, Andreas Grabner, want to say Thank you to Raphael Pionke for his work and contribution to the performance engineering community. It was great collaborating on this blog post and I am looking forward to working closer with you on integrating your best practices into Keptn.

The post Release readiness through AI-based white box resiliency testing with JMeter and Dynatrace appeared first on Dynatrace news.

The challenge with ASP.NET Web Service Observability, Diagnostics, and Performance Optimization is that most monitoring & diagnostic tools only provide global Service Level Indicators metrics (SLIs) such as response time, failure rate, and throughput, but not metrics for each individual service endpoint. The reason for this is because a classical ASP.NET SOAP Web Service (ASMX) exposes its individual endpoints through the same URL and using the SoapAction HTTP Header to determine which method to execute and therefore most tools only report on the global service name.

The same default behavior is also true for Dynatrace when installing the OneAgent on your servers that host your ASP.NET services. Once installed, every single web service request processed by ASP.NET will be traced across all tiers and captured as a PurePath (Distributed Trace). The screenshot below shows a PurePath that was shared with me by our partner triscon from Vienna, which specializes in Load and Performance testing for large enterprise applications. During a recent project, they immediately found long-running transactions that made several web services calls where one took more than 4 seconds. While this is great information the PurePath left some questions open:

What triscon really needed was not just PurePaths showing many calls to the same service instance. To monitor SLIs and optimize Service Level Objectives (SLOs) down to individual web service endpoints, the additional context from the soapaction header was required.

Additional context without code change through request attributes

In order to get the required additional context to detect service endpoints triscon leveraged Dynatrace’s Request Attributes, which enables users to extract additional context, such as service method name, for each distributed trace without the need for actual code modification. The data can be extracted from HTTP request or response headers, method arguments, return values, and more. Having this additional context as part of the PurePath speeds up the analysis and diagnostics work for performance engineers, developers, or architects. Below are before and after screenshots of the same PurePath shown above.

While this additional context information is great for engineers when analyzing individual traces, it can be used for multi-dimensional analysis across all captured PurePaths during a test or even in production to answer much bigger questions.

Diagnosing architectural issues in (micro)service architectures

As I said, the ability to analyze individual PurePaths down to code level is a great capability. However – it’s not the approach that allows us to quickly identify performance or architectural hotspots in your distributed application. Why? Because nobody has the time to analyze billions of distributed traces that get collected in a performance or production environment. This is where the strength of Dynatrace comes in, as it automates the analysis of all captured PurePaths and allows us to phrase questions that really matter – such as: “Do we have service methods that make excessive database calls? And if they exist – how we can optimize this behavior?” – or – “My critical SLIs are response time, failure rate and throughput per service endpoint: can you give me that data?”

These are some of the questions triscon asked for all of their projects and the answer comes through Dynatrace’s Multidimensional Analysis View:

From this view it’s just a single click to more details such as database hotspots, method hotspots, PurePaths or Service Flow:

The multi-dimensional analysis view is also your starting point to define your important SLIs, e.g. Response time, failure rate or throughput split by service endpoint name. If you want to learn more check out the blog from Michael Kopp on Analyze microservice-based metrics without code changes or watch my YouTube Tutorial on Calculated Service Metrics.

If you have your own web services and aren’t happy with your current approach of observability, diagnostics, and performance optimization then follow the next steps.

How to enable this for your web service?

It all starts with installing the Dynatrace OneAgent on your environment. If you don’t yet have Dynatrace – simply get download a Dynatrace Trial. Once you have Dynatrace capturing data from your apps and services simple follow these best practices shared by our friends from triscon so you can get even more out of Dynatrace.

Step 1: Identify the context data

In the case of ASP.NET Web Services (ASMX) the information about the actual service method is passed as part of the soapaction HTTP Header. Dynatrace allows you to capture any HTTP Header as well as other context information such as method arguments, return values, cookies, messaging information and more.

Triscon’s first attempt was to capture the full soapaction header to learn about the format of the value. Once configured as a Request Attribute this value will show up on each captured PurePath like shown in the following screenshot:

Step 2: Define Request Attributes

Looking at the soapaction value it’s apparent that the important piece is the last part of the URL. Luckily Dynatrace Request Attributes allow you to specify a regular expression to parse out the piece of information you really want Dynatrace to extract and make available for diagnostics. Here is a screenshot that shows the final Request Attribute configuration:

In addition to the service method name, triscon also suggests defining a second request attribute for the name of the entire web service which is part of the URL. Here’s the configuration for that request attribute:

Request attributes can be used for multi-dimensional analysis as shown earlier in the blog. But there is more use you see in the next step.

Step 3: Request naming rules

Do you remember the before & after PurePath picture from the beginning? It showed the proper ASMX Service Name and the ASMX Service Method Name as part of the PurePath names. This is possible because Dynatrace allows us to define custom naming rules for PurePaths. The following shows the request naming rule configuration which we should add to our best practices for ASP.NET Web Service monitoring:

Once configured these names are the official PurePath request names like shown in the top requests view when analyzing an individual service. Here is another before/after comparison that clearly shows how valuable this additional context is:

Get started with your own web service monitoring

While Dynatrace gives you a lot of out of the box data and context we can easily extend it with more context needed for better monitoring, diagnostics, and performance optimizations on frameworks such as ASP.NET Web Services.

Thanks again to our partner triscon who is not holding these best practices back but sharing it with the larger community. I am sure I will also invite them soon to one of my performance clinics and other events where these practices should be shared.

The post A guide to ASP.NET web service observability, diagnostics and performance optimization with Dynatrace appeared first on Dynatrace news.