Shift left and shift right are core testing concepts of the agile DevOps methodology. This methodology speeds up application development by releasing small builds frequently as code evolves. As part of the continuous cycle of progressive delivery, DevOps teams are also adopting shift left and shift right principles to ensure software quality in these dynamic environments.

All this shifting may sound abstract, but I’ll explain how this software verification approach benefits DevOps methods and outcomes—and makes software more reliable.

In DevOps, what is shift left? And what is shift right?

To understand shift left and shift right, consider the software development cycle as a continuum, or infinity loop, from left to right.

On the left side of the loop, teams plan, develop, and test software in pre-production. The main concern in pre-production on the left side of the loop is building software that meets design criteria.

When teams release software into production on the right side of the loop, they make it available to users. The concern in production is maintaining software that meets business goals and reliability criteria.

What does shift left mean in DevOps?

Shift left is the practice of moving testing, quality, and performance evaluation early in the development process, often before any code is written.

Shift-left testing helps teams anticipate changes affecting performance or other delivery processes that arise during the development process. With shift-left testing, teams verify APIs, container configurations, interactions between microservices, and container configurations.

Why is shift left important?

In addition to testing for functionality, shift left testing also checks that software meets customer requirements. This enables developers and stakeholders to identify improvements that could enhance customer experience and functionality. Making these changes early in the development process reduces the cost of making them after the code is released.

As a result, shift left testing has become increasingly important as teams face pressure to deliver software faster and more frequently with higher quality. Shift left speeds up development efficiency and reduces costs by detecting and addressing software defects earlier in the development cycle before they get to production.

The benefits of shift left

The concept of shift left has been gaining traction in the world of software development and testing. This approach, which introduces testing and quality assurance early in the development lifecycle, offers several considerable benefits. 

1. Early Bug Detection: By testing early in the development process, teams can identify and address bugs when they are easier and less costly to fix. This not only improves the quality of the final product but also reduces the time and resources spent on debugging and rework.
2. Faster Time to Market: With shift left, testing is integrated into every stage of development. This continuous testing allows for faster feedback and quicker iterations, which can significantly reduce the overall time to market.
3. Improved Customer Satisfaction: By ensuring that quality is a priority from the outset, shift left helps to deliver a product that meets or exceeds customer expectations. This can lead to higher customer satisfaction and loyalty.
4. Reduced Costs: While implementing shift left may require an initial investment, the cost savings from reduced rework and faster time to market can quickly offset this. In addition, by catching and fixing issues early, teams can avoid the potentially high costs associated with late-stage or post-release fixes.
5. Enhanced Collaboration: Shift left encourages collaboration between testers, developers, and other stakeholders from the beginning of the project. This collaborative approach can lead to better communication, shared understanding of the product, and a better end result.

What does shift right mean in DevOps?

Shift right is the practice of performing testing, quality, and performance evaluation in production under real-world conditions. Shift-right methods ensure that applications running in production can withstand real user load while ensuring the same high levels of quality.

With shift right, DevOps teams test a built application to ensure performance, resilience, and software reliability. The goal is to detect and remediate issues that would be difficult to anticipate in development environments.

Why is shift right important?

With shift right, teams can test code in an environment that mimics real-world production conditions that they can’t simulate in development. This practice enables teams to catch runtime issues before customers do. To automate part of the process, teams can use application programming interface (API) calls. Organizations can also apply shift-right testing to code configured or monitored in the field. The result is more comprehensive testing coverage that better addresses user experience concerns.

Like shift left testing, the objective of shift right testing is to fail small and fail fast. The assumption is that problems caught early in the pre-deployment environment are easier to solve than issues caught by customers in live production.

Shift right testing is especially useful for organizations practicing progressive delivery, wherein developers release new software features incrementally to minimize the impact of unforeseen issues. Testing in a production-ready environment is a crucial final phase before declaring features ready for prime-time.

The benefits of shift right

Shift right is a concept that extends testing into the production environment. It involves monitoring and testing systems in production to gain insights and improve quality, offering several significant advantages.

1. Realistic User Testing: Shift right testing involves testing with real users. By including them in the process, software professionals gather valuable feedback about how users interact with the software. This helps make informed decisions about future improvements.
2. Continuous Feedback Loop: Shift right testing promotes continuous monitoring and improvement throughout all phases of development. This iterative feedback loop ensures that issues are addressed promptly, leading to better product quality.
3. Test Coverage: By extending testing to real-world scenarios, shift right testing provides broader test coverage. It helps identify issues that might not appear during earlier development stages, resulting in more comprehensive testing.
4. Enhanced Monitoring: Monitoring software behavior in production environments allows for better visibility into its performance, stability, and user experience. This helps teams proactively address any anomalies or performance bottlenecks.
5. Customer-Centric Approach: Shift right testing prioritizes users’ needs and experiences. By involving real users, software teams gain insights that lead to improvements aligned with customer satisfaction.

Why shift right and shift left are critical for microservices architecture

Both shift left and shift right testing have become more important as software is increasingly built from microservices. Instead of applications being built and run as a monolith, microservices are loosely coupled functions called at runtime. How microservices-based applications perform depends on individual service responsiveness, which complicates testing in a simulated environment.

Shift left testing allows teams to observe how services interact in a simulated environment and detect potential performance impacts before deploying the code.

Shift right testing at or near the deployment point allows teams to observe real-world forces and measure their impact. Shift-right tests typically cover functionality, performance, failure tolerance, and user experience. Teams often automate such production-environment testing and translate feedback into technical specifications for developers.

With shift-left and shift-right testing, testers can isolate issues so teams can fix and tackle improvements in parallel. As an application becomes more stable, teams can start testing and optimizing performance. Notably, shifting left and right have become an important practice in Agile software development. Together, they’re part of the continuous feedback loop that characterizes DevOps.

Types of shift right tests

A shift right approach may enlist various types of test suites. The following are tests your team might find useful:

• A/B testing. Web designers commonly use this method. Users see two versions of a page, and testers measure which generates a greater response. Teams often conduct this type of test in a production environment to gather real-world feedback.
• Synthetic monitoring. Synthetic monitoring is a form of shift-right testing that uses software tools to emulate the paths users might take when engaging with an application. With synthetic monitoring, you can automatically track application uptime and how your application responds to typical user behavior. It uses scripts to generate simulated user behavior for various scenarios, geographic locations, device types, and other variables.
• Chaos engineering. With chaos engineering, also called chaos testing, developers intentionally “break” the application by introducing errors to determine how well it recovers from disruption. DevOps and IT teams set up monitoring tools to see precisely how the application responds to different types of stresses. Teams perform these tests in a controlled production environment to minimize the impact on mission-critical systems.
• Canary releases. This strategy is named for the canaries that miners use to lower into coal mines to detect toxic gases. Technology has thankfully rendered this inhumane tactic obsolete. However, the term survives to describe slowly rolling out changes to a small subset of instances for testing before teams apply them to the full infrastructure.
• Blue-green deployment. With a blue-green deployment, an organization runs two nearly identical production environments, shifting users (real or synthetic) between the two as they make small changes to one or the other. This practice is important to shift-right methodology as it can minimize downtime. It also provides a mechanism for rapid rollback should something go wrong with the latest version.

Shift right tools

Shift right tools focus on the examination of code quality after it has reached production. Such tools are important for determining and analyzing how software behaves in real-word environments, offering teams insights into its actual performance. Examples of shift right tools include: 

• Application Performance Monitoring (APM) tools: APM tools offer metrics, logs, and traces relating to application performance, giving teams insights into how to remediate issues faster and further optimize their applications 

• Digital Experience Monitoring (DEM) tools: DEM tools provide insights into the user experience (UX) of an application. These tools may offer: 

• Real-user monitoring (RUM) for analyzing the behavior and experience of real-world users. 

• Synthetic monitoring that performs artificially simulated user journeys to test for any UX bottlenecks or deficiencies 

• Session replay that provides video-like documentation of real user journeys 

• Security monitoring and incident response tools: These tools help detect, manage, and communicate security incidents and facilitate the response process. They also assist teams in prioritization processes to ensure that the most business-impacting vulnerabilities are addressed first. 

• Chaos engineering tools: These tools introduce simulated failures to test the resiliency of code and the efficacy of response processes to such failures. This helps inform teams of any code deficiencies and opens room for improvement 

• Feature flagging tools: These tools provide capabilities to toggle certain features of a deployment on and off without deploying any additional code. Such tools offer a greater level of control to developers, as they allow for more experimentation and finetuning of the user experience. Feature flagging provides increased reliability in feature rollouts and helps to ensure that a deployment provides the best user experience possible. 

Types of shift left tests

The four basic types of shift left testing are the following:

• Traditional shift left testing. The traditional approach focuses on unit and integration testing, typically involving APIs and cross-browser compatibility. This approach focuses more on unit testing small pieces of code to deliver information early and often rather than more systems-level operational and acceptance testing.
• Incremental shift left testing. This is a popular approach for teams migrating from a waterfall development approach to one that breaks complex projects into smaller pieces. In this scenario, teams conduct systems-level operational and acceptance testing on a smaller, more incremental scale. Incremental testing is popular for validating large and complex systems.
• Agile/DevOps testing. This is a series of short, continuous sprints during the development stage. This approach does not intend to address operational performance but rather validate adherence to basic requirements architecture.
• Model-based shift left. Model-based testing aims to reduce errors introduced during the requirements definition, architecture, and design phases. It tests for executable requirements, architecture, and design. The advantage of model-based testing is that it can begin almost immediately instead of after completing all other test cycles.

Shift left tools

1. Security Scanning Tools:

These tools streamline the integration of security with DevOps processes by identifying vulnerabilities early in the development cycle. They help ensure that code is secure as it is being developed, rather than waiting until the testing phase. 

Examples of security scanning tools include: 

• Static Application Security Testing (SAST): SAST tools analyze the source code without executing it. They identify potential security issues, coding standards violations, and vulnerabilities. Linters (e.g., ESLint for JavaScript, pylint for Python) fall into this category. 

• Dynamic Application Security Testing (DAST): DAST tools test the application during runtime by simulating attacks. They identify vulnerabilities that can be exploited by attackers. Examples include OWASP ZAP and Burp Suite. 

• Dependency Scanning: These tools check for vulnerabilities in third-party libraries and dependencies used in the application. 

• Secrets Detection: Tools like GitGuardian scan code repositories for exposed secrets (e.g., API keys, passwords) and prevent accidental leaks. 

• Integrated Application Security Testing (IAST): IAST tools combine static and dynamic analysis to provide comprehensive security coverage. 

2. Quality control tools

These tools help ensure and validate that code in the DevOps pipeline is not only secure but also functional and effective. Implementing such tools early in the development process helps guarantee that errors or deficiencies are caught and corrected before reaching production. 

• Static Code Analysis tools: Similar to SAST tools, Static Code Analysis tools assess the source code without executing it to determine whether the code is compliant, bug-free, and generally of good quality. 

• Code review tools: Code review tools help automate the code review process by organizing and displaying code changes, facilitating communication between developers regarding changes, and examining the efficacy of the code review process.

• Continuous integration (CI) tools: CI tools enable developer collaboration by automatically integrating code from multiple developers into a single repository, such as GitHub, GitLab, and Bitbucket.

The application security dividend of shift right and shift left

An important benefit of shifting right is improved application security. “Scanning a static image, either in a repository or in a development environment, can’t give you the same rich insights you can get if you observe the application running in production,” a Dynatrace report on security evolution in the cloud notes. “For example, you don’t get to see what libraries are actually called, how they are used, whether a process is exposed to the Internet, or whether a process interacts with sensitive corporate data.”

The growing use of software containers has complicated aspects of cybersecurity. Containers can obscure the processes running in them, and attackers even containerize exploits. Production testing exposes the behavior of container-based software, even if the contents of containers are obscured. Organizations can also use shift-right testing to test for the presence of novel zero-day exploits, such as Log4Shell.

From a shift left perspective, security testing during development helps identify vulnerabilities as early in the life cycle as possible, when they are easiest to remediate.

Shift left, shift right, or both?

As organizations modernize their application stacks around cloud-native constructs like microservices and containers, a best practice is to adopt both shift left and shift right strategies. Shift left testing reduces software defects and speeds up time to market. Shift right better ensures reliability in production by testing under real-world conditions to affirm that software is doing its job where it matters most: serving the user. The combination of the two moves the organization closer to the ultimate goal of continuous integration and continuous delivery using automation and interdisciplinary communication. The result is a faster and more agile organization.

An essential capability for comprehensive shift left and shift right testing is automated full-stack observability and monitoring. With the capacity for end-to-end analysis at all phases of development, DevOps teams can monitor all requests and processes. Automatic analysis of all services across a sprawling complex of multicloud applications from a single interface enables teams to automate. Testers can push deployment information and metadata to the monitoring environment using scripts to track builds, revisions, and configuration changes.

The better an automated DevOps platform understands the full context of an issue, the better it can detect root causes, flag with the proper parties, and even implement self-healing measures.

Whether your organization has shifted testing left to the development phase or right in production — or simply wants to monitor performance in the field—an AI-driven, full-stack observability solution can take your software development to the next level.

The post Shift left vs shift right: A DevOps mystery solved appeared first on Dynatrace news.

In this blog, I will be going through a step-by-step guide on how to automate SRE-driven performance engineering. You may have seen over the past few months we have been extensively promoting Service Level Indicators (SLIs) and Service Level Objectives (SLOs) as part of our Open Source project Keptn. Keptn uses SLO definitions to automatically configure Dynatrace or Prometheus alerting rules. If an alert is triggered, Keptn automates operational tasks such as triggering remediation actions. Keptn also uses the same SLOs to validate that these operational actions had the desired impact.

While Google’s SRE Handbook mostly focuses on the production use case for SLIs/SLOs, Keptn is “Shifting-Left” this approach and using SLIs/SLOs to enforce Quality Gates as part of your progressive delivery process. After a new build gets deployed and automated tests executed, SLIs are evaluated against their SLOs and, depending on that result, a build is considered good (promoted) or bad (rolled back).

“The earlier in the lifecycle we ensure SLIs meet their SLOs the better it will to ensure these service levels in production”.

Now, while we’ve been pushing these concepts in Keptn we haven’t explained well enough how to level-up your existing load testing scripts for better SLI monitoring and how to integrate them with Dynatrace in order to reap all the benefits of SLI-based Performance Analysis. This is what this blog is all about. Its a detailed tutorial blog that explains how you can make the first steps towards SLI/SLO-based Quality Gates. Let me walk you through how I have built my Dynatrace Performance Insights Dashboard showing SLIs split by Test Name as well as SLIs for the specific technology and infrastructure:

While I’ve included the automation script that creates the above dashboard, request attributes and calculated service metrics in Dynatrace as part of my Keptn Tutorial titled “Progressive Delivery with Keptn using Dynatrace” I didn’t explain how I got to this point so that anybody can replicate this in their own environment, with their own test scripts on their own applications.

So, let’s walk through the individual steps.

Step-by-step guide: SRE-driven performance analysis with Dynatrace

For this blog I will be using the same Node.js based sample application as in some of my Keptn Tutorials. If you want, you can use the same sample app and follow my steps – or – just apply the same concepts to your own environment! The app description and supporting files such as load testing scripts are on the Keptn Example GitHub

If you’re short on time, or want to see all of this in action instead then, please check out my Performance Clinic: Automated SRE-driven Performance Engineering with Dynatrace”

Step #1: Monitoring your test environment

Firstly, start by installing the Dynatrace OneAgent on the hosts where you’ll be running your application test. If you have a distributed environment with multiple servers hosting your webservers, app servers, and database, I suggest you install the OneAgent on all these servers to get full end-to-end visibility. If your apps are deployed in a PaaS Platform, e.g: Kubernetes, OpenShift, Cloud Foundry or Azure Web Apps then install the OneAgent by following the OneAgent PaaS installation options.

In my example, I’m going to run my sample node.js based application on an EC2 Linux machine using Docker. For that I simply followed the OneAgent installation instructions for Linux. Once installed I get full visibility into that host:

Step 1 accomplished! Now our environment where we’ll be deploying our application under test is now automatically monitored by Dynatrace!

Step #2: Deploy our application

Now let’s deploy our application. If you use your own application and it was already running before you installed the OneAgent, please restart your application and web servers. This will enable deep monitoring of those Java, .NET, Node, processes as well as your web servers. Please note – there is no need to restart your database servers.

In my case, I’m going to deploy my simple node application. This could either be done by deploying it on a k8s cluster or runnning it as a simple docker container. In this circumstance, I’m going to opt for a simple docker on my EC2 machine by executing the following command: docker run -p 80:8080 grabnerandi/simplenodeservice:1.0.0

After it’s deployed, verify that the application is up and running by accessing it through its public IP. Once Dynatrace sees the incoming traffic it will also show up in Dynatrace, under Transaction & Services. From here, you can explore the first service details:

Step 2 accomplished: We have deployed an application that is fully monitored with Dynatrace!

Step #3: Tagging of services

A key concept in monitoring is proper tagging. Tagging allows you to better organize your monitoring data. In Dynatrace, tagging also allows you to control access rights (via Management Zones), filter data on dashboards or via the API as well as allowing you to control calculation of custom service metrics or extraction of request attributes.

There are multiple ways to define tags and metadata in Dynatrace:

• Manual Tagging: Put them on service and the process group via the UI or the Dynatrace API
• Automated Tagging: Define a rule that tag entities based on existing metadata
• Cloud & Platform Tags: Extract tags & annotations from k8s, OpenShift or Cloud Foundry
• Environment Tags: specify tags via the DT_TAGS environment variable

In my example below, you’ll see I’ve tagged the service and processes my services run on with a tag called “AppUnderTest”. While I could have done this manually I have decided to automate this tagging by passing DT_TAGS=AppUnderTest to my container startup parameters. Here is my adapted docker run command: docker run -p 80:8080 –env DT_TAGS=AppUnderTest grabnerandi/simplenodeservice:1.0.0

Step 3 accomplished: Our application under test is now automatically tagged. These tags will allow us to create dashboards, request attributes or calculate service metrics specifically for our application under test. At a later stage, these tags allow us to automate data analysis through tools such as Keptn which provides an SLI/SLO based approach of performance analysis. Tags play a critical role there as well as it allows us to define which entities to have automatically analyzed.

Step #4: Tag your Load Testing steps

Dynatrace by default captures and traces every single incoming request on our instrumented applications and services. This allows us to analyze metrics (SLIs) for each individual endpoint URL. While this is great, the URL is not necessarily meaningful when we want to analyze details for a step from your load testing script, e.g: Login, Add to Cart, Checkout.

For my sample application, I’ve prepared a simple JMeter based test script which contains four test steps: homepage, invoke, echo and version.

When analyzing test results, I want to see the same test step names in Dynatrace so it’s easier for me to focus my diagnostics on these four steps instead of using URLs. To pass information such as Test Step Name (TSN), Load Test Name (LTN), Load Script Name (LSN), Virtual User Id (VU)  and others we can follow the load testing integration best practice as documented in Dynatrace and load testing tools integration. The best practices describes how testing tool can add an additional HTTP Header called x-dynatrace-test to each simulated request. The value for x-dynatrace-test is a set of semicolon separated name=value pairs including information such as TSN, LTN, LSN, VU … – the following screenshot shows my JMeter script and the corresponding x-dynatrace-test header for two of my four test steps:

Adding this custom header will be specific to the testing tool you use. If you’re using JMeter feel free to look at my sample load.jmx JMeter script that you can find in the example GitHub repo. For other tools either check out our documentation for Neoload, LoadRunner or JMeter. For any other tool I’m sure there’s a documented feature that allows you to add a custom HTTP Header with the documented format to your test requests. Neoload stands out as they have built an integration that automates many of the Dynatrace configuration steps.

Step 4 accomplished: When executing our test script, every request contains additional context information about the test script, load test name and test step as part of the x-dynatrace-test http header. This information can now be used in Dynatrace for Request Attributes and Calculated Service Metrics.

Step #5: Create Request Attributes in Dynatrace

Now that every simulated HTTP Request from our testing tool contains additional context information we can configure Requests Attributes which will instruct Dynatrace to pull out these name/value pairs and add it as context data on every single PurePath. Once this is done we can use this data for filtering and focused diagnostics, e.g: only show me requests that came in for TSN=homepage and LTN=TestMay18.

Request Attributes are rules that get applied to a certain set of Dynatrace monitored service entities. They allow you to extract meta data from a data source, e.g: HTTP Headers, URL, and then allow you to specify whether you want the full value or just a part of it. To make yourself familiar with Request Attributes I suggest you check out Basic Concepts – Request Attributes (doc).

In our case we want to create 4 rules that extract the individual name value pair (TSN, LTN, LSN, VU) from the x-dynatrace-test header. The following shows the screenshot of a rule for TSN. Be aware that it is a best practice to additionally limit this rule to only apply to certain technologies or services. Also be aware that the extraction where it says “between TSN= and ;”:

Not only can we create these rules through the UI. We can also automate them through the Dynatrace Configuration API. As part of my example GitHub tutorial repo you find a script that is called createTestRequestAttributes.sh. Feel free to use that script to create these rules automatically!

Step 5 accomplished: Whenever a test gets executed Dynatrace will capture the test context and applies it to the PurePaths. This data can then be used for targeted multi-dimensional analysis, filtering, calculated service metrics – more on that in the next steps.

Step #6: Multidimensional analysis based on test context

If we run load test script now, Dynatrace will give us the option to analyze data based on those Request Attributes we just specified. So, after running my simple JMeter script against my sample app I can open the Multidimensional Analysis view from my service and define queries like: 95^th Percentile Response Time by Test Step Name:

This analysis view is the perfect starting point for digging into response time hotspots, service flow analysis, failure analysis …

The multidimensional analysis view allows us to analyze all sorts of advanced metrics such as “number of database calls”, “number of backend service calls”, “time spent on CPU”. This opens up new analytics use case to e.g: detect those test cases that cause high database or CPU overhead. To learn more check out Michael Kopp’s blog post on Multidimensional Analysis 2.0.

Step 6 accomplished: After executing our tests we can leverage the multidimensional diagnostics option splitting or filtering by context data from our tests, e.g: test name, test step. We can also save that view on our service which makes it more convenient for future analysis exercises.

Step #7: Comparing results of two test runs

Analyzing one test run is great, but comparing two is better as we want to know what the difference between two test runs really is. In my example I deploy version 2.0.0 of my node.js sample application. If you read through my sample application readme on GitHub you will see that Build 2.0.0 has some built-in issues where requests to one of the api endpoints has a high failure rate.

So, I start with deploying Build 2.0.0 like this: docker run -p 80:8080 grabnerandi/simplenodeservice:2.0.0

Now I run the same JMeter load test again and bring up a new multidimensional analysis view. This time focusing on “Average Failure Rate split by Test Names” and from here start my comparison with the previous test:

Step 7 accomplished: While there are several options to compare test runs, the multi dimensional analysis view gives us a great starting point into the comparison views of Dynatrace. We can now do this for every test we run by comparing it with the test results of our last known good build.

Step #8: Create Calculated Service Metrics (SLIs)

We’ve seen how we can use the multidimensional analysis view to answer questions like “What’s my response time, failure rate, number of database statements, CPU or idle time per Test Step?” These are also questions you would phrase as SLOs when monitoring a production environment, e.g: Response Time of a Key Business Service or Key Business Transaction must be faster than 200ms at 95% of the time. Dynatrace however not just gives us the standard SLO metrics based on Google’s SRE handbook. We can also include SLOs that touch upon CPU Consumption, Number of Database Interactions, Number of calls to external Services.

Instead of getting these answers in the multi-dimensional analysis view, we can define Calculated Service Metrics to have these data points available as metrics (SLIs). So, for example, we can put them on a dashboard, define alerts or pull this data through an API which allows us to validate them against our objectives or – for load testing purposes – easily detect abnormal behavior within a test or compare them against previous tests.

As true for many features in Dynatrace there are multiple ways to get started with Calculated Service Metrics. The easiest though is to create them right out of the Multidimensional Analysis View:

As you will notice, not all aggregations are currently supported for calculated service metrics but support for more is coming. In case an aggregation is not supported, the Create metric dialog will tell you. If you click on “Advanced options” when creating such as calculated metric you will be taken to the settings screen where you’ll see all configuration options. If you want to learn more about this watch my Performance Clinic on Calculated Service Metrics and APIv2.

Just as with Request Attributes we can either manually create these rules or automate this creation through the Dynatrace API. As part of my example repository, I’ve prepared a script that creates seven calculated service metrics. As calculated service metrics are only applied to service entities that match a certain filter my script can be called with a tag filter option. In our case, it would be “ENVIRONMENT: AppUnderTest”. You can therefore run the script like this: ./createTestStepCalculatedMetrics.sh ENVIRONMENT AppUnderTest

If you want more sophisticated tag filters, you can either extend the script or modify the rules in Dynatrace. Just be aware that these metrics are only calculated for services that match the tagging rules. Also be aware that calculated service metrics will consume licenses. For more information please consult the Dynatrace documentation. The image below shows all the different rules my script is generating. Most of them are split by the TSN Request Attribute. One, however, splits by HttpStatusClass which allows us to later chart the number requests split by HttpStatusClass:

Step 8 accomplished: If we run our load test once more we can chart these calculated service metrics. This is the starting point for our final step which is creating a load testing dashboard that contains all our key SLIs (Service Level Indicators):

Step #9: Creating an SLI performance insights dashboard

Now we’re in the stage of putting all our key metrics and SLIs on a single dashboard. This can include the calculated service metrics we created, giving us insights by Test Step, overall service level metrics such as throughput and failure rate, and can also include metrics from the underlying processes or hosts.

The dashboard you’ve seen at the start of this blog is what you can build right now. The image below shows the different building blocks and which metrics I have put on. Depending on your application and environment you may put different aspects of it, but I hope this is a good starting point for you:

As there’s also a Dynatrace API to create dashboards, I created a script that creates the basic dashboard as shown above. It’s part of the example GitHub and is called createLoadTestingDashboard.sh. You can use this to get a head start and then customize the dashboard to your needs.

There are additional things I would suggest you do after building this dashboard:

#1: Links for drill-downs

I suggest you add a markdown tile at the top with links to other views such as the Multidimensional Analysis View, Response Time Hotspots, Service Flow. This will make it easier for you and others to start diagnostics in case the SLIs show an issue:

#2: Filter through Management Zones

Create Management Zones for your different “Application Under Test” environments. Once you have this set up you can use and reuse your dashboard for different test environments by applying the management zone as a filter:

Step 9 accomplished: We now have a dashboard we can use to analyze load testing data based on service level indicators split by test context data such as Test Steps. We can also use it to open diagnostics views to analyze the hotspots in our application under test or even compare the results with previous builds.

Next Step: Automate SLI validation through Keptn

While I hope I just made your lives as performance engineers easier with getting SLIs based on Test Steps and putting them on a dashboard next to other relevant service level and host health indicators, it’s still a manual task to look at these dashboards and comparing metrics across builds.

This is where Keptn comes in; automating the process by automatically pulling out SLIs for the period of the test and then automatically compares them against previous test timeframes (detects regressions) or against your SLOs:

If you want to read more on this, take a look at blog SLO Validation as a Self Service with Keptn Quality Gates or watch my YouTube Tutorial Leveling Up your Jenkins-based Delivery with Keptn. Also make sure to check out our Keptn Tutorials on https://keptn.sh

Dynatrace ACE Services for Quality Gates!

If you want some help with setting this up then I have got good news. At Dynatrace, we have a team of dedicated ACE (Autonomous Cloud Enablement) experts who are there for you to transform into an organization that truly embraces shift-left by adopting automated SLO driven quality gates. Our team will help you with:

• Implementing Keptn Quality Gates in your environment
• Educating on SLI/SLO best practices to ensure meaningful quality gates are used in the right phase of the CI/CD process
• Integrating the quality gates with your CI/CD platform of choice to achieve an automated quality gate

Reach out to ace@dynatrace.com or your account executive for more information

The post Tutorial: Guide to automated SRE-driven performance engineering appeared first on Dynatrace news.