When it comes to site reliability engineering (SRE) initiatives adopting DevOps practices, developers and operations teams frequently find themselves at odds with one another. Developers want to write high-quality code and deploy it quickly. Operations teams want to make sure the system doesn’t break.

Observability seeks to find a happy medium between the two. Observability is a set of practices and technologies that helps IT teams understand what’s happening across complex environments so that teams can detect and resolve issues quickly, without disruption to users.

Andreas Grabner, DevOps Activist at Dynatrace, took to the virtual stage at the recent Dynatrace Perform conference to describe how the open source Keptn project automates the configuration of observability tools, dashboards, and alerting based on service-level objectives (SLOs).

Keptn: A reference implementation of Google’s SRE principles

Keptn is an open source control plane that enables cloud-native continuous delivery and automated operations. The goal is to accelerate innovation by eliminating the need for custom automation scripts and point-to-point tool integrations.

Dynatrace developed and released Keptn to open source in 2020. Software engineer Taras Tsugrii of Meta (formerly Facebook) paid Keptn a high compliment, saying it feels like a reference implementation of Google’s SRE principles, which are the search giant’s techniques for ensuring the integrity of its sites and services. Keptn’s global user base is developing, as is adoption within companies including Citrix, Amasol, ERT, and Kitopi. One particular use case for Austrian banking software developer Raiffeisen involves using Keptn to automate the production release and readiness validation of all its products using scoring metrics.

One of Keptn’s key advantages is that it automates SLO-driven, multi-stage delivery of apps and services. Why is automated orchestration critical?

Too many SLOs create complexity for DevOps

DevOps and SRE engineers experience a lot of pressure to deliver applications faster and that adhere to standards like “the five nines” of availability, resulting in many new service level requirements. Developers also need to automate the release process to speed up deployment and reliability.

SLOs are a great way to define what software should do. But as teams adopt them more widely, they sometimes don’t break SLOs down into individual goals that make sense for development and release management.

By shifting SLOs left from operations back to development, teams can detect problems more quickly. SREs can then use SLOs for release quality checks, such as big bang, blue/green, and canary testing.

Bringing SLOs back into development also gives developers production-stage feedback on critical metrics that may later impact business SLOs. With instant feedback enabling teams to release clean software, developers can react faster and speed up the delivery of high-quality content.

With many pipelines to maintain, DevOps teams need automated orchestration. Standard automation techniques like scripting can be very powerful, but also quite complex.

Limits of scripting for DevOps and SRE

Classic automation has limits. Dieter Landenahuf, a senior ACE Engineer at Dynatrace, built Jenkins pipelines for new microservice architectures by creating templates and copying, pasting, and modifying them slightly. This created a classic “snowflake effect” because of the risk of code duplication: if something breaks, you need to fix it in multiple places. The process is error-prone, manual, and doesn’t scale.

Keptn reduces the complexity of pipelines while bringing automation into the delivery processes so that developers can focus on SREs.

Keptn presents a declarative way to define orchestration so that developers can create automation sequences based on what should happen for delivery, remediation, and testing. Developers can define any sequence with chains of preferred tools. Keptn retains the individual tool configurations under version control.

Optimizations are based on SLOs, meaning Keptn decides whether to move forward with an orchestration based on SLOs. Keptn doesn’t replace the tools developers prefer: it connects them. Since Keptn is an open-source project, it will be easier to eventually replace tools because they all adhere to the same event standard. Teams can plug in new tools for testing, deployment, or even monitoring without having to modify many existing automation tools.

Keptn lets developers keep their favorite tools

Developers can use their existing tools to build artifacts. If they need to add more tasks — such as enforcing SLOs, automating testing, or connecting an observability platform — Keptn handles that for them. Keptn orchestrates all tools so that developers don’t have to. It simply reaches out to monitoring platforms like Dynatrace to extract the necessary SLOs. There is no need to write or maintain any customizations, nor figure out when the job is done and how to parse the results.

Keptn includes best practices that help developers choose which sequences to use. Ultimately, Keptn reduces code automation by 90% and makes every component SLO-driven. It’s based on open standards and is fully declarative. Everything is code and version-controlled in GitOps.

Dynatrace has an enterprise version of Keptn as part of its cloud software-as-a-service (SaaS). Here are some of the advantages of using the enterprise version:

• Self-manages.
• Authenticates using Dynatrace single sign-on.
• Automatic upgrades.
• Enables developers to plug in additional tools.

Developers can stay in production sequences. Every time the system creates a new artifact, Keptn triggers a new delivery sequence using tools based on events that the developer has subscribed to. Keptn keeps all the configuration information for every state along with Helm scripts and SLOs. Developers can also create different sequences, such as deployment using Monaco.

SLO-based release validation made easy

Release validation centers around SLO-based evaluations. To check quality, developers can simply trigger a release that evaluates performance against SLOs. The dashboard provides a heat map and every service level indicator (SLI) metric, criteria, and scoring.

There are two configuration options. The first is to configure as code using YAML and upload the configuration to a Git repository. The second, for Dynatrace users, is the Dashboard Link, which defines a dashboard with all the relevant SLOs and technical metrics. The dashboard becomes the basis for automated validation — ­­with all SLIs and SLOs stored in Dynatrace — and problems detected by Dynatrace’s Davis automated dependency analysis.

Developers can also configure Slack and send notifications from Keptn cloud automation every time the system orchestrates a process. They can have multiple tools that subscribe to events, as well as a webhook service that lets them send events to an external tool.

Charting the course with Keptn

Developers who have been building many automation scripts for SRE should take a look at Keptn. The current version is 0.12, and the roadmap includes high-availability improvements, role-based access controls, and improved GitOps and auto-remediation. A Continuous Delivery Foundation special interest group is now standardizing events and the team is preparing a proposal for incubation in the Cloud Native Computing Foundation. Dynatrace’s hosted cloud version of Keptn, delivered through Cloud Automation, has planned support for a custom execution plane and custom integrations.

Check out Andreas’ presentation here along with other sessions from Perform 2022.

The post Scale DevOps and SRE with open source Keptn appeared first on Dynatrace news.

Shift left and shift right are core testing concepts of the agile DevOps methodology. This methodology speeds up application development by releasing small builds frequently as code evolves. As part of the continuous cycle of progressive delivery, DevOps teams are also adopting shift left and shift right principles to ensure software quality in these dynamic environments.

All this shifting may sound abstract, but I’ll explain how this software verification approach benefits DevOps methods and outcomes—and makes software more reliable.

In DevOps, what is shift left? And what is shift right?

To understand shift left and shift right, consider the software development cycle as a continuum, or infinity loop, from left to right.

On the left side of the loop, teams plan, develop, and test software in pre-production. The main concern in pre-production on the left side of the loop is building software that meets design criteria.

When teams release software into production on the right side of the loop, they make it available to users. The concern in production is maintaining software that meets business goals and reliability criteria.

What does shift left mean in DevOps?

Shift left is the practice of moving testing, quality, and performance evaluation early in the development process, often before any code is written.

Shift-left testing helps teams anticipate changes affecting performance or other delivery processes that arise during the development process. With shift-left testing, teams verify APIs, container configurations, interactions between microservices, and container configurations.

Why is shift left important?

In addition to testing for functionality, shift left testing also checks that software meets customer requirements. This enables developers and stakeholders to identify improvements that could enhance customer experience and functionality. Making these changes early in the development process reduces the cost of making them after the code is released.

As a result, shift left testing has become increasingly important as teams face pressure to deliver software faster and more frequently with higher quality. Shift left speeds up development efficiency and reduces costs by detecting and addressing software defects earlier in the development cycle before they get to production.

The benefits of shift left

The concept of shift left has been gaining traction in the world of software development and testing. This approach, which introduces testing and quality assurance early in the development lifecycle, offers several considerable benefits. 

1. Early Bug Detection: By testing early in the development process, teams can identify and address bugs when they are easier and less costly to fix. This not only improves the quality of the final product but also reduces the time and resources spent on debugging and rework.
2. Faster Time to Market: With shift left, testing is integrated into every stage of development. This continuous testing allows for faster feedback and quicker iterations, which can significantly reduce the overall time to market.
3. Improved Customer Satisfaction: By ensuring that quality is a priority from the outset, shift left helps to deliver a product that meets or exceeds customer expectations. This can lead to higher customer satisfaction and loyalty.
4. Reduced Costs: While implementing shift left may require an initial investment, the cost savings from reduced rework and faster time to market can quickly offset this. In addition, by catching and fixing issues early, teams can avoid the potentially high costs associated with late-stage or post-release fixes.
5. Enhanced Collaboration: Shift left encourages collaboration between testers, developers, and other stakeholders from the beginning of the project. This collaborative approach can lead to better communication, shared understanding of the product, and a better end result.

What does shift right mean in DevOps?

Shift right is the practice of performing testing, quality, and performance evaluation in production under real-world conditions. Shift-right methods ensure that applications running in production can withstand real user load while ensuring the same high levels of quality.

With shift right, DevOps teams test a built application to ensure performance, resilience, and software reliability. The goal is to detect and remediate issues that would be difficult to anticipate in development environments.

Why is shift right important?

With shift right, teams can test code in an environment that mimics real-world production conditions that they can’t simulate in development. This practice enables teams to catch runtime issues before customers do. To automate part of the process, teams can use application programming interface (API) calls. Organizations can also apply shift-right testing to code configured or monitored in the field. The result is more comprehensive testing coverage that better addresses user experience concerns.

Like shift left testing, the objective of shift right testing is to fail small and fail fast. The assumption is that problems caught early in the pre-deployment environment are easier to solve than issues caught by customers in live production.

Shift right testing is especially useful for organizations practicing progressive delivery, wherein developers release new software features incrementally to minimize the impact of unforeseen issues. Testing in a production-ready environment is a crucial final phase before declaring features ready for prime-time.

The benefits of shift right

Shift right is a concept that extends testing into the production environment. It involves monitoring and testing systems in production to gain insights and improve quality, offering several significant advantages.

1. Realistic User Testing: Shift right testing involves testing with real users. By including them in the process, software professionals gather valuable feedback about how users interact with the software. This helps make informed decisions about future improvements.
2. Continuous Feedback Loop: Shift right testing promotes continuous monitoring and improvement throughout all phases of development. This iterative feedback loop ensures that issues are addressed promptly, leading to better product quality.
3. Test Coverage: By extending testing to real-world scenarios, shift right testing provides broader test coverage. It helps identify issues that might not appear during earlier development stages, resulting in more comprehensive testing.
4. Enhanced Monitoring: Monitoring software behavior in production environments allows for better visibility into its performance, stability, and user experience. This helps teams proactively address any anomalies or performance bottlenecks.
5. Customer-Centric Approach: Shift right testing prioritizes users’ needs and experiences. By involving real users, software teams gain insights that lead to improvements aligned with customer satisfaction.

Why shift right and shift left are critical for microservices architecture

Both shift left and shift right testing have become more important as software is increasingly built from microservices. Instead of applications being built and run as a monolith, microservices are loosely coupled functions called at runtime. How microservices-based applications perform depends on individual service responsiveness, which complicates testing in a simulated environment.

Shift left testing allows teams to observe how services interact in a simulated environment and detect potential performance impacts before deploying the code.

Shift right testing at or near the deployment point allows teams to observe real-world forces and measure their impact. Shift-right tests typically cover functionality, performance, failure tolerance, and user experience. Teams often automate such production-environment testing and translate feedback into technical specifications for developers.

With shift-left and shift-right testing, testers can isolate issues so teams can fix and tackle improvements in parallel. As an application becomes more stable, teams can start testing and optimizing performance. Notably, shifting left and right have become an important practice in Agile software development. Together, they’re part of the continuous feedback loop that characterizes DevOps.

Types of shift right tests

A shift right approach may enlist various types of test suites. The following are tests your team might find useful:

• A/B testing. Web designers commonly use this method. Users see two versions of a page, and testers measure which generates a greater response. Teams often conduct this type of test in a production environment to gather real-world feedback.
• Synthetic monitoring. Synthetic monitoring is a form of shift-right testing that uses software tools to emulate the paths users might take when engaging with an application. With synthetic monitoring, you can automatically track application uptime and how your application responds to typical user behavior. It uses scripts to generate simulated user behavior for various scenarios, geographic locations, device types, and other variables.
• Chaos engineering. With chaos engineering, also called chaos testing, developers intentionally “break” the application by introducing errors to determine how well it recovers from disruption. DevOps and IT teams set up monitoring tools to see precisely how the application responds to different types of stresses. Teams perform these tests in a controlled production environment to minimize the impact on mission-critical systems.
• Canary releases. This strategy is named for the canaries that miners use to lower into coal mines to detect toxic gases. Technology has thankfully rendered this inhumane tactic obsolete. However, the term survives to describe slowly rolling out changes to a small subset of instances for testing before teams apply them to the full infrastructure.
• Blue-green deployment. With a blue-green deployment, an organization runs two nearly identical production environments, shifting users (real or synthetic) between the two as they make small changes to one or the other. This practice is important to shift-right methodology as it can minimize downtime. It also provides a mechanism for rapid rollback should something go wrong with the latest version.

Shift right tools

Shift right tools focus on the examination of code quality after it has reached production. Such tools are important for determining and analyzing how software behaves in real-word environments, offering teams insights into its actual performance. Examples of shift right tools include: 

• Application Performance Monitoring (APM) tools: APM tools offer metrics, logs, and traces relating to application performance, giving teams insights into how to remediate issues faster and further optimize their applications 

• Digital Experience Monitoring (DEM) tools: DEM tools provide insights into the user experience (UX) of an application. These tools may offer: 

• Real-user monitoring (RUM) for analyzing the behavior and experience of real-world users. 

• Synthetic monitoring that performs artificially simulated user journeys to test for any UX bottlenecks or deficiencies 

• Session replay that provides video-like documentation of real user journeys 

• Security monitoring and incident response tools: These tools help detect, manage, and communicate security incidents and facilitate the response process. They also assist teams in prioritization processes to ensure that the most business-impacting vulnerabilities are addressed first. 

• Chaos engineering tools: These tools introduce simulated failures to test the resiliency of code and the efficacy of response processes to such failures. This helps inform teams of any code deficiencies and opens room for improvement 

• Feature flagging tools: These tools provide capabilities to toggle certain features of a deployment on and off without deploying any additional code. Such tools offer a greater level of control to developers, as they allow for more experimentation and finetuning of the user experience. Feature flagging provides increased reliability in feature rollouts and helps to ensure that a deployment provides the best user experience possible. 

Types of shift left tests

The four basic types of shift left testing are the following:

• Traditional shift left testing. The traditional approach focuses on unit and integration testing, typically involving APIs and cross-browser compatibility. This approach focuses more on unit testing small pieces of code to deliver information early and often rather than more systems-level operational and acceptance testing.
• Incremental shift left testing. This is a popular approach for teams migrating from a waterfall development approach to one that breaks complex projects into smaller pieces. In this scenario, teams conduct systems-level operational and acceptance testing on a smaller, more incremental scale. Incremental testing is popular for validating large and complex systems.
• Agile/DevOps testing. This is a series of short, continuous sprints during the development stage. This approach does not intend to address operational performance but rather validate adherence to basic requirements architecture.
• Model-based shift left. Model-based testing aims to reduce errors introduced during the requirements definition, architecture, and design phases. It tests for executable requirements, architecture, and design. The advantage of model-based testing is that it can begin almost immediately instead of after completing all other test cycles.

Shift left tools

1. Security Scanning Tools:

These tools streamline the integration of security with DevOps processes by identifying vulnerabilities early in the development cycle. They help ensure that code is secure as it is being developed, rather than waiting until the testing phase. 

Examples of security scanning tools include: 

• Static Application Security Testing (SAST): SAST tools analyze the source code without executing it. They identify potential security issues, coding standards violations, and vulnerabilities. Linters (e.g., ESLint for JavaScript, pylint for Python) fall into this category. 

• Dynamic Application Security Testing (DAST): DAST tools test the application during runtime by simulating attacks. They identify vulnerabilities that can be exploited by attackers. Examples include OWASP ZAP and Burp Suite. 

• Dependency Scanning: These tools check for vulnerabilities in third-party libraries and dependencies used in the application. 

• Secrets Detection: Tools like GitGuardian scan code repositories for exposed secrets (e.g., API keys, passwords) and prevent accidental leaks. 

• Integrated Application Security Testing (IAST): IAST tools combine static and dynamic analysis to provide comprehensive security coverage. 

2. Quality control tools

These tools help ensure and validate that code in the DevOps pipeline is not only secure but also functional and effective. Implementing such tools early in the development process helps guarantee that errors or deficiencies are caught and corrected before reaching production. 

• Static Code Analysis tools: Similar to SAST tools, Static Code Analysis tools assess the source code without executing it to determine whether the code is compliant, bug-free, and generally of good quality. 

• Code review tools: Code review tools help automate the code review process by organizing and displaying code changes, facilitating communication between developers regarding changes, and examining the efficacy of the code review process.

• Continuous integration (CI) tools: CI tools enable developer collaboration by automatically integrating code from multiple developers into a single repository, such as GitHub, GitLab, and Bitbucket.

The application security dividend of shift right and shift left

An important benefit of shifting right is improved application security. “Scanning a static image, either in a repository or in a development environment, can’t give you the same rich insights you can get if you observe the application running in production,” a Dynatrace report on security evolution in the cloud notes. “For example, you don’t get to see what libraries are actually called, how they are used, whether a process is exposed to the Internet, or whether a process interacts with sensitive corporate data.”

The growing use of software containers has complicated aspects of cybersecurity. Containers can obscure the processes running in them, and attackers even containerize exploits. Production testing exposes the behavior of container-based software, even if the contents of containers are obscured. Organizations can also use shift-right testing to test for the presence of novel zero-day exploits, such as Log4Shell.

From a shift left perspective, security testing during development helps identify vulnerabilities as early in the life cycle as possible, when they are easiest to remediate.

Shift left, shift right, or both?

As organizations modernize their application stacks around cloud-native constructs like microservices and containers, a best practice is to adopt both shift left and shift right strategies. Shift left testing reduces software defects and speeds up time to market. Shift right better ensures reliability in production by testing under real-world conditions to affirm that software is doing its job where it matters most: serving the user. The combination of the two moves the organization closer to the ultimate goal of continuous integration and continuous delivery using automation and interdisciplinary communication. The result is a faster and more agile organization.

An essential capability for comprehensive shift left and shift right testing is automated full-stack observability and monitoring. With the capacity for end-to-end analysis at all phases of development, DevOps teams can monitor all requests and processes. Automatic analysis of all services across a sprawling complex of multicloud applications from a single interface enables teams to automate. Testers can push deployment information and metadata to the monitoring environment using scripts to track builds, revisions, and configuration changes.

The better an automated DevOps platform understands the full context of an issue, the better it can detect root causes, flag with the proper parties, and even implement self-healing measures.

Whether your organization has shifted testing left to the development phase or right in production — or simply wants to monitor performance in the field—an AI-driven, full-stack observability solution can take your software development to the next level.

The post Shift left vs shift right: A DevOps mystery solved appeared first on Dynatrace news.

I don’t think there is a correct or one-size-fits-all definition of Shift-Left, neither is there one for DevOps, Cloud-Native or any of these other heavily used terms. What I do know is that there are experts in our industry trying to make it easier for engineers to get faster self-service quality, performance and scalability feedback about their proposed code changes.

In a recent chat with one of our customers who works in the in the financial sector, I was told about how they transformed their core financial services leveraging a modern cloud native architecture running on AWS. That transformation also included a shift of their development practices, their delivery pipeline and how to provide feedback to engineering and business earlier in the development cycle!

To give you one example: Instead of executing performance tests at the end of the Kanban time-box or at the end of the release cycle, performance tests are run for every build deployment and daily scheduled runs for latest code changes of each microservice. This gives both the infrastructure and development teams more and faster feedback about the impact of code or configuration changes on performance, scalability and resource consumption.

In a remote desktop session, their performance engineering team lead showed me their Dynatrace environment. I took a couple of screenshots and notes and hope that this will inspire others to redefine performance engineering with the power of the cloud, containers and Dynatrace AI. As I am respecting their request of anonymity, I have blurred out sensitive information in the screenshots!

#1: Eight Feature Based Environments powered by Terraform, EC2, Mesos, Docker & OneAgent

Let’s start with explaining what type of environment the performance engineering team is providing to the engineering teams to get faster performance feedback. They run a Dynatrace Managed installation in their AWS VPC, running 10 different Dynatrace Tenants to separate the different pre-production environments they are monitoring (Feature1, Feature2. Feature n, Performance, etc). If you want a Dynatrace Managed instead of a SaaS instance on AWS check out the recently announced Dynatrace Managed Cluster on AWS Quick Start.

On the infrastructure side their environments leverage Terraform and Ansible to deploy EC2 instances on which they run Mesos orchestration to host their Spring Boot Java services in Docker containers!

Here are some of the benefits they get out of Dynatrace, Cloud & Container Monitoring capabilities:

Benefit #1: Dynatrace allows easy management of containers and images!

One challenge in very dynamic containers environments, is to keep track of all the currently deployed containers, where they run and how many resources they consume. Making sure that those containers that should be running have enough resources and can reach all the other depending services is another challenge in highly dynamic environments. Dynatrace provides that type of visibility out-of-the-box! We call this “No-Touch Container Monitoring” as you don’t have to change anything in your container to get this visibility. This works for Docker, ECS, Fargate or any other Container flavor:

Benefit #2: Dynatrace provides automatic monitoring of container resource consumption

For a Docker host, Dynatrace provides visibility into the actual containers deployed on that host and shows how many resources these containers consume right now and over time:

Benefit #3: Dynatrace leverages CloudWatch for Metrics and for Tags

Our customer has also setup the Dynatrace AWS CloudWatch integration which not only automatically pulls in key CloudWatch metrics of AWS Services such as S3, EBS, RDS, DynamoDB, Lambda, ELB … but also pulls in Tags from EC2 instances, which get automatically applied to the monitored hosts where a Dynatrace OneAgent is installed. This makes managing all performance data much easier because dashboards, filters and notifications can be setup using the same tags as already defined on AWS.

And here is the Dynatrace AWS Overview showing how many AWS services and resources the teams are using in their Performance-Dev Environment:

TIP: If you want to learn more about basic AWS monitoring, I suggest you walk through my 101 AWS Monitoring GitHub Tutorial or watch my 101 AWS Monitoring Performance Clinic on YouTube. There I explain the integration with CloudWatch, how to deploy OneAgents on EC2 and how to monitor applications deployed on Beanstalk, ECS or Lambda!

#2: Ensuring Healthy Cloud Infrastructure with help of Dynatrace AI

While having more monitoring data available is great, thanks to the Dynatrace OneAgent and the AWS CloudWatch integration, it doesn’t mean that the team can spend more time analyzing more data points. This is where the Dynatrace automatic baselining, anomaly and root cause detection helps.

I asked them if they have an example of how the Dynatrace AI helps their infrastructure team to help a stable and healthy environment for the services that the development teams run on top.

They opened the Dynatrace Problem View, clicked on the Infrastructure filter and then walked me through the following screenshots. They showed me how Dynatrace detected a network connectivity issue of several haproxy instances running in Docker containers orchestrated by Mesos across several EC2 instances.

Benefit #1: Dynatrace’s automatic anomaly detection across all monitored entities and metrics

The Dynatrace OneAgent monitors every single container and all processes running in these containers. OneAgent automatically detects technologies and services such as haproxy, message queues, web- or application servers, databases, …

The Dynatrace anomaly detection understands which metrics are important for each type of service and reports an anomaly if a metric shows problematic or unusual behavior. Thanks to this auto-detection capability DevOps team can react much quicker to infrastructure related problems before they start impacting the services that run on them:

Benefit #2: Dynatrace Automatic Dependency and Impact Detection

Thanks to Smartscape, the infrastructure team not only knows which infrastructure components or critical services are currently in an unhealthy state. Smartscape also shows all the depending services that haproxy is connecting. Based on that information, it is easier to understand the potential impact to higher-level services, applications or even end users. This also helps to prioritize remediation actions – whether executed manually or automated.

TIP: If you want to learn more about how Dynatrace helps your IT Service and Operations (ITSM) check out the information around our ServiceNow Integration or how you can integrate Dynatrace with any CMDB YouTube Tutorial. If you want to see other examples of detected infrastructure problems check out my recent blog on AI In Action: RabbitMQ, Cassandra and JVM Memory.

#3: Shift-Left Performance Feedback with the help of Dynatrace AI

What runs on top of this dynamic cloud & container infrastructure? I am sure you guessed the answer: Services that their development teams are trying to get performance feedback on. The primary type of services they implement using Spring Boot expose REST APIs for their B2B offerings. Some of these APIs have well defined SLAs, which is why their team decided to define several custom thresholds for the different REST endpoints.

When developers make code changes, those get automatically deployed with the next scheduled build and get automatically tested. LoadRunner and JMeter are used to generate the load against their various REST APIs. These tests typically run for little over one hour after which developers pro-actively reach out to the Dynatrace dashboards to analyze how their code was performing, where the hotspots are and where there might have scalability issues.

Benefit #1: Dynatrace AI reduces time for analyzing performance test results

While Dynatrace gives them access to all data through the dashboards, the team started to see the benefit in time savings when using the Dynatrace AI automatic detected problems and root causes. The time saved can be better spent on building new features instead of manually analyzing the same metrics, log files, stack traces or CPU samples every time a test executes.

The following screenshot shows an automatically detected problem that happened during one of the automated load test runs. There was a 47% slowdown of a specific REST API endpoint caused by a CPU spike on that EC2 Linux machine, where Mesos hosts the Tomcat process in a container:

Benefit #2: Dynatrace Pro-Active Alerting of Dev Teams

Thanks to the host, process and service tagging capabilities of Dynatrace, each service is tagged with the name of the team responsible. In case Dynatrace detects a problem the team automatically gets notified thanks to the Alerting Profile feature in Dynatrace. Alerting Profiles allow sending problem notifications ONLY to those teams of services where a problem was detected. The notification (email, JIRA, Slack, Mattermost, Flowdock) also gets sent out immediately when the problem is detected and not only at the end of the test. This also speeds up the feedback loop cycle time in case a code change has an obvious issue which can be detected by Dynatrace within minutes.

Benefit #3: Dynatrace speeds up fault domain isolation between Infrastructure and Application

The CPU spike on that host makes you probably wonder: is it a problem with the infrastructure, the container or the actual app. When clicking into that root cause box in the problem ticket we end up seeing all the captured data from that EC2 Linux machine. We clearly see that there are a lot of other processes and containers running on that same box – all competing for CPU, memory, disk and the network. Very interesting to learn that the same box also runs Swagger (on Node.js), Kafka (two Jetty’s) and Filebeat (Go) besides our Tomcat that hosts the service under test:

A click on “Consuming processes” gives us a detailed CPU breakdown of all processes & containers on that machine – clearly highlighting that the cause is our Apache Tomcat process:

Benefit #4: Dynatrace detects code level problems. In custom and 3^rd party code

Now that we know the problem lies within Tomcat it is easy to find out what caused it. Dynatrace provides several hotspot detection options such as the response time analysis. Code execution is the clear “winner” in this case followed by two SQL statements:

As the SLA for their REST API endpoints is 200ms it is interesting to learn why most of this time is consumed in code execution. More interesting is WHERE in the code the time is spent. Clicking on “Code execution” in the response time analysis infographic, brings up the method hotspots view with a detailed breakdown. We got a winner: Hibernate!

The above screenshot shows the hotspot across all the request that exceeded the 200ms SLA. This is very useful and makes it easier to analyze and fix hotspots that are impacting many transactions and not just individual requests.

Thanks to the Dynatrace PurePath technology every single transaction is also available for inspection. Seeing the PurePath allows engineers to better understand the sequence of code execution which is very useful in distributed or asynchronous transactions.

Tip: I get a lot of questions from users that integrate Dynatrace with their load testing tools. Make sure to check out my blog on Load Testing Redefined or watch my Load Testing YouTube tutorial. If you want to learn more about diagnostics option with Dynatrace then check out Basic Diagnostics or Advanced Diagnostics with Dynatrace.

What else can be done with Dynatrace?

While the performance engineering team at our customer already leverages Dynatrace to speed up continuous feedback cycles to engineering as well as the infrastructure team there is more that can be done. Here are some additional ideas:

#1: Use the Dynatrace REST API to automate “sanity checks”:

• How many containers are running vs. how many should run?
• Do we run too many containers on a single host?

#2: Use the Dynatrace REST API to automate “deployment optimizations”:

• Re-distribute top resource consuming containers to avoid resource constraints
• Co-locate containers that are “tightly coupled”. This prevents network contention issues

#3: Use the Dynatrace REST API to automate “remediation actions”:

• Check out the auto-remediation blogs on Ansible Tower and StackStorm
• Check out my thoughts on Self-Healing use cases with Dynatrace

#4: Integrate Dynatrace tighter into your CI/CD Pipeline

• Read up on my thoughts about Building the Unbreakable Delivery Pipeline
• Learn how Dynatrace can be integrated into Atlassian DevOps Tools

If you have any further questions let me know. Happy to give you more insights into how Shift-Left can be done with Dynatrace and happy to share your own implementation. Just let me know!

The post Shift-Left with Dynatrace: Automating Performance Feedback with AWS, Mesos and Containers appeared first on Dynatrace news.