The surge in demand for digital services since 2020 is a trend that is here to stay and only expected to accelerate. To compete, organizations have to achieve both speed and reliability when bringing new products and services to market. To meet this demand, organizations are adopting DevOps practices, such as continuous integration and continuous delivery, and the related practice of continuous deployment, referred to collectively as CI/CD.

CI/CD is a series of interconnected processes that empower developers to build quality software through well-aligned and automated development, testing, delivery, and deployment. Together, these practices ensure better collaboration and greater efficiency for DevOps teams throughout the software development life cycle.

Here’s what you need to know about these software development practices, how they relate to each other, and how they benefit DevOps teams as they optimize and automate more processes to achieve ever-faster time to value for customers.

Continuous integration streamlines development

Continuous integration (CI) is a software development practice that streamlines the internal process of creating software. With CI, multiple software developers can work on different features or modules of the same application and individually commit their updates to a shared code repository as they complete them, often many times a day. When they check in their code, the build management system automatically creates a build and tests it. If the test fails, the system notifies the team to fix the code. This practice helps software teams quickly detect and resolve any bugs that come up during the development process.

Continuous integration also prevents “merge hell,” which can happen when two or more developers inadvertently make conflicting changes that break the build when the lines are merged back into the master branch. Continuous integration also avoids teams having to reconcile substantial amounts of conflicting or redundant code, which can require code freezes or even a dedicated integration stage in the pipeline.

When software development teams continuously integrate incremental changes, each developer is free to make changes without worrying about throwing someone else’s work off track or about their own work being stepped on by another. Smooth, regular merging helps software development teams complete projects more quickly and efficiently. It also ensures there is always a testable, up-to-date build that will properly compile, which is critical for frequent and rigorous application testing.

This approach saves time and resources that would otherwise be spent fixing issues later in the software development life cycle, or worse, after release, when issues are much more difficult to address.

Continuous delivery ensures code is always ready to deploy

Continuous delivery (CD) is a process in which DevOps teams develop and deliver complete portions of software to a repository, such as GitHub or a container registry, in short, controlled cycles. Continuous delivery seeks to make releases regular and predictable events for DevOps staff, and seamless for end-users. Another goal of continuous delivery is to always keep code in a deployable state so updates can go live at a moment’s notice with little or no issues.

DevOps teams can automate CI/CD pipelines to move code through the appropriate environments with no human input, which accelerates the build, test, and deployment stages of software development — or any additional stages they have in place, depending on the processes they use.

For example, when a feature is ready for client demonstration, the DevOps team can have a CD tool automatically deploy it to a test server so the client can see how it works and provide feedback before it is released to the production server.

Continuous deployment ensures customers always have the latest

Continuous deployment (also CD) is an extension of continuous delivery in which builds that pass testing are automatically deployed directly to production environments on a daily, or even hourly, basis.

Continuous deployment hastens the feedback loop with your customers and eases the burdens on operations teams by automating the next stage in the pipeline.

Once an organization’s continuous delivery practices are stable and mature, they often adopt automated continuous deployment and testing in a tiered “blue-green” sequence. In this scenario, the new build (green) is deployed in parallel with the existing build (blue) to ensure it works before the blue build is retired. Teams can also use a phased “canary” deployment approach, where the new build is gradually phased in to replace the existing build.

Continuous delivery vs. continuous deployment: which is it in CI/CD?

Continuous delivery (CD) often gets confused with continuous deployment (CD). If CD stands for continuous delivery and continuous deployment, which is it when people refer to CI/CD practices?

In short, it’s either continuous delivery, or continuous delivery and continuous deployment. Although related, the two terms refer to automated processes that happen at different points in the delivery workflow.

You can think of adopting these practices as three steps in a continuum of DevOps automation:

1. Continuous integration to streamline development and automate building and testing. It’s a way for developers to work on the same code base at the same time.
2. Continuous delivery to automate the process of delivering completed code blocks into the main branch, where it can be deployed to a production environment by the operations team. It’s a way for the development part of a DevOps team to automate the process of testing and committing code.
3. Continuous deployment to automate the process of deploying completed, tested code into a production environment. It’s a way for the operations part of a DevOps team to automate the process of deploying new code to customers.

When an organization first adopts agile DevOps practices, they often start out with continuous integration, and mature quickly into continuous delivery, thus achieving CI/CD. Many organizations stop here, preferring to release production code manually. Others progress into continuous deployment so they can automate the entire software development, delivery, and deployment pipeline.

The benefits of CI/CD

The main benefit of continuous integration and continuous delivery is that it reduces the time required to develop applications and features. This gives companies a competitive advantage over businesses that still take a manual approach to software development.

Besides the competitive advantage, CI/CD enables agile development teams to constantly iterate and release new features. These practices ensure everyone is working with the same version of the code as they develop and refine features and functions. With the logistics of integrating and testing builds automated, engineers can focus on what they do best: coding.

If teams have implemented continuous deployment, code can spend less time waiting for testing and deployment and more time in production, which accelerates feedback from users and, ultimately, improves business outcomes.

Increased efficiency, reduced time to market, and quicker innovation make CI/CD an attractive proposition for organizations of all types.

The drive to optimize and automate more CI/CD processes

To meet competitive demands, organizations know they must release new products and services more quickly than ever without compromising quality or reliability. They know basic CI/CD practices are fundamental to DevOps and DevSecOps initiatives that look to develop applications with more effective collaboration and greater precision.

Teams that implement CI/CD practices successfully rely on many tools and methods to manage features, versions, testing, and builds. Automation happens at every stage of the pipeline, from building, packaging, and testing to pushing applications to different production environments.

To accelerate the development pipeline for ever-faster releases with less risk, teams need continuous automation and advanced AI-driven observability across all the tools in their DevOps toolchain so they can automate manual steps and identify quality issues earlier in the software lifecycle.

If your organization uses or is considering adopting CI/CD DevOps processes, a best practice is to implement a full-stack observability platform that can provide code-level visibility of all software builds, apps, and services in your environment, whether they’re in development or deployed to end-users.

The Dynatrace Software Intelligence Platform continuously and automatically monitors the performance of DevOps tools, and seamlessly integrates with CI/CD workflows. With improved collaboration on a single platform and a shared data model, you can ensure your entire team has continuous situational awareness across the lifecycle.

To take a deeper look at continuous delivery, see my next post, What is continuous delivery and what are best practices for implementing it?

The post Continuous integration and continuous delivery (CI/CD): How it enhances DevOps and continuous deployment appeared first on Dynatrace news.

Monitoring as Code is a concept to define what metrics (functional, performance, architecture) are relevant for applications, services, process, containers or infrastructure. One implementation of Monitoring as Code is what my colleague Thomas Steinmaurer and my friends Matthias Fichtner and Raphael Pionke, from T-Systems MMS, have coined “Performance Signature”. The Performance Signature is a list of metrics including important thresholds stored in a JSON or YAML file. This file gets checked in with your source code and gets automatically validated as your code moves through your Continuous Delivery Pipeline. The signature acts as an automated quality gate between your different stages (dev, test, prod) as well as allows you to validate your production deployments (blue/green, canaries, …).

Matthias and Raphael recently released their implementation of the Performance Signature for Dynatrace as a Jenkins Plugin (free for everyone to download in your Jenkins). I invited both to showcase their integration and the key use cases in one of my recent Performance Clinics. Watch it on YouTube, or simply hit the play button on the following video:

Benefits of Performance Signature

During the recording, I chimed into the conversation a couple of times, re-iterating the benefits of integrating monitoring into your delivery pipeline. At Dynatrace we have been doing it for years and we have seen this approach being implemented by many organizations that are re-platforming or re-architecting towards cloud native delivery pipelines. The key benefits from my perspective are:

#1: Monitoring data capturing de-coupled from analysis

Developers can write a simple JSON file that gets automatically validated without having to worry about how the data gets collected. This basically means “Monitoring as a Pipeline Service” or better “Monitoring as a Self-Service” through the concept of “Monitoring as Code”

#2: Performance Signature for all stages of the delivery pipeline

Developers can use it in the Dev Environment to get feedback on every code change. It can be integrated into the Pull Request Approval process before merging to master. It can act as Quality Gates between staging and production after executing your load tests. And finally, it can be used to get Real-Time Production Feedback on e.g: Blue/Green, Canary of Feature Flag releases!

#3: Versioning of approved performance signature per build/release

When a build is pushed through the pipeline the actual values captured while validating your Performance Signature can be committed back to your source code repo. This not only allows you to historically see the trend of your performance – but also allows you to push these accepted values as your new production baselines. In Dynatrace we provide REST APIs to define your thresholds for your service endpoints or even for resource consumption. We can make this part of the production deployment by applying the thresholds as defined in your source code repo to your production monitoring environment!

Implementation Guide for your Jenkins Performance Signature

While the performance clinic video, as well as the documentation on the Plugins GitHub page, should give you all the information you need to implement the Performance Signature for Jenkins yourself, I thought I would write a quick step-by-step guide to better understand the use cases & benefits.

Step 0: Understand the End-to-End Continuous Delivery Workflow, Integrations & Use Cases

I created the following animation based on what Raphael & Matthias showed in their tutorial. I additionally highlighted the individual workflow steps, integrations and when which use case comes into play:

While Raphael & Matthias used a Kubernetes Cluster in their demo this also works for any other platform such as OpenShift, CloudFoundry, ECS, Fargate, Docker and even your more “classical” Java & .NET stacks! Now let’s get through all of this step-by-step

Step 1a: Install Dynatrace OneAgents in your Environment

First step is to rollout the Dynatrace OneAgent in your environment. If you don’t yet have a Dynatrace installation, simply sign up for our SaaS-based Trial and follow the instructions to deploy the OneAgent on your Linux, Windows, Solaris or AIX machines. If you run on Kubernetes, OpenShift, CloudFoundry, Azure, AWS, GCP … simply follow the instructions in the Deploy Dynatrace OneAgent section. The OneAgent will automatically take care of FullStack Monitoring.

Recommendation: Dynatrace can automatically pick up tags & metadata from the underlying PaaS or SaaS platforms, e.g: Kubernetes Namespace, Container Names, CloudFoundry Organizations, … – If you need to pass more metadata or define your own tag rules check out my session on 101 Tagging Essentials with Dynatrace and the blogs from my colleague Michael Kopp on Tagging & Naming. In my personal environment I run k8s on GCP and I have configured automated tagging to create tag for environment (from the k8s namespace) and the app (from the container name). Following screenshot shows my payment app-service deployed in the jx-staging space – fully automatically tagged through the automated tagging rules:

Step 1b: Install the Jenkins Performance Signature Plugin

Install the Performance Signature: Dynatrace plugin it via the Jenkins Plugin Manager. In the global system configuration section in Jenkins configure the endpoint to your Dynatrace Tenant. This can either be a SaaS or Managed Tenant. All you need is your Dynatrace Tenant URL, e.g: https://YOURTENANT.live.dynatrace.com and a Dynatrace API Token which you can create in Dynatrace via Settings -> Integration -> Dynatrace API.

Step 2: Push Deployment Information to Dynatrace

In the Jenkinsfile find the stage where you deploy into a certain stage, e.g: Staging. Now simply wrap this deployment with a call to createDynatraceDeploymentEvent. This Jenkins pipeline step is provided by the Performance Signature plugin. It will measure the time it takes for the deployment and upon success will push a Dynatrace Deployment Event via the Dynatrace REST API. The event will be pushed to those entities that match the tagMatchRules. Therefore, it is important to have proper tagging setup as described in Step 1a. In the following example I deploy the carts app-service into the environment jx-staging. Here is the modified Jenkinsfile:

Once the pipeline runs, we see Dynatrace not only picking up Monitoring of the deployed services, we also see the deployment events on those service entities that are tagged with app:carts & environment:jx-staging:

I believe this is the first game changer. Why? Because every time somebody runs the pipeline, either on the master or a dev branch, Dynatrace and Jenkins are automatically linked. When analyzing data in Dynatrace you immediately know which Jenkins job deployed at which time, what the Git Commit was, how long the deployment lasted, and which services were impacted by that deployment.

Step 3: Execute your Integration, API, Load or Performance Tests

This step now is straight forward. Simply execute your Selenium, JMeter, Neotys, Gatling, … tests that you normally execute in your pipeline. We do recommend that you tag your HTTP-based tests through the x-dynatrace-test HTTP Header as explained in How to I integrate Dynatrace into my load testing processes? This will make it easier to analyze, diagnose and compare individual test runs.

The following shows my modified Jenkinsfile for the Performance Testing stage. Just as we wrapped the deployment of the application in the previous step we are now wrapping the execution of the performance test into a call to recordDynatraceSession – another pipeline step that comes with the Performance Signature Jenkins Plugin. It will push a load testing event to those entities in Dynatrace once the test is complete. It also allows us to define the testCase name which is how data is grouped later on in the Jenkins Result view. You could run different tests, e.g: Load Test, Functional Test, API Test, … – and use that wrapping to pull in different metrics for different types of tests you execute.

recordDynatraceSession goes hand in hand with perfSigDynatraceReports. This is where the “real magic” behind the Performance Signature evaluation happens. Let me show you all the details about this in Step 4!

Step 4a: Understand and define your Performance Signature

Let’s have a quick look at the Performance Signature – the JSON file that we will define as an engineer for automated evaluated in our pipeline. The file is an array of timeseries definitions:

• a timeseriesId, e.g: response time, CPU, memory, failure rate …
• an aggregation: avg, min, max, count, percentile
• tags: pull metrics from those entities with these tags
• entityId: instead of tags you can also specify a specific EntityId
• upperLimit & lowerLimit: both must be specified and actually define a SEVERE and WARNING threshold. Depending on whether upperLimit > lowerLimit or lowerLimit > upperLimit the limits are validated against the upper or lower value range. See below screenshot for an example

The next screenshot shows a Performance Signature that contains two metrics: service.responsetime and service.server_side_requests (=throughput). Please not the different boundaries:

• Response Time validates against upper boundaries: We want to fail the build if the 98^th percentile of response time exceeds 1.8s and mark the build unstable if we are higher than 1.5s
• Throughput validates against lower boundaries: We want to fail the build if we have less than 100 requests / min and mark as unstable the build if we have less than 250

I hope this explains how to use upperLimit and lowerLimit. We might want to rename these though in the future to something like warningLimit and severeLimit as it makes a bit more sense.

Now let’s have a look how this gets evaluated and where we actually see the data.

Step 4a: Validating Performance Signature

In Step 3 we already saw the call to perfSigDynatraceReports. It was executed right after the performance test was executed. As a parameter it takes the references to the Performance Signature specfile:

Now – here is what happens in Dynatrace when the evaluation happens – we see the event on those entities referenced in the recordDynatraceSession step:

More important to this information are the actual performance signature results in Jenkins. We see them build over build in the Job overview:

For each individual build we see every metrics for every entity that matched and all data points for the timeframe of the test run. We also see if there were any violations to the defined thresholds (upperLimit, lowerLimit). This either creates a severe or warning incident as shown in the screen below with details about the violation: actual value vs thresholds.In case of a violation the build status would also change to either failed or unstable. Last bonus item is that the view also shows a comparison of the current to the previous build:

I believe this is a game change. WHY? Because it provides Monitoring Feedback as a Service automated into my continuous integration pipeline. As a developer I only have to define the list of metrics I want in a JSON file and everything else happens “auto-magically”. I get notified about performance, resource consumption or architectural regressions (e.g: too many unwanted dependencies) and can then easily drill down to the root cause. Which is what we are going to see next! Yeah – you heard me right – we can also extend this to architectural validations as we can also pull in Smartscape metrics such as incoming or outgoing dependencies!

Step 5: Comparing two builds and find the difference

Once we have a failed build due to the performance signature violation, we want to know what the root cause is. For HTTP-based applications & services I favor the following approach:

#1: Open the Top Web Requests View

I start in the Top Web Requests View and filter it down to those requests that came in from my load testing script. This works if you are following the Load Testing Integration for Dynatrace best practice and have request attributes configured to filter on your load tests executed from Jenkins:

In the screenshot above also see that I have filtered on the request attributed called LTN (=Load Test Name). I am only looking at those requests that come in from my specific Load Test. Now I can select one of the tests that was executed between 1:51 and 2:06 and click on Comparison of one of the tested endpoints!

In the following Comparison View I select the timeframe of the previous or next build as compare timeframe: 2:26 – 2:41. Then I open the Response Time Hotspot comparison:

In case of a code execution issue I can go directly into Method Hotspots to see which methods consumed all this wait time that wasn’t there in the previous build:

Depending on what the hotspot analysis tells us we can go off different routes of diagnosing the issue. For more details on diagnostics make sure to check out my Basic Diagnostics and Advanced Diagnostics with Dynatrace Online Tutorial.

Give us feedback on Performance Signature for Jenkins

I want to start to give feedback to Matthias and Raphael by saying: THANK YOU for implementing Monitoring as Code in such an elegant way and making it available through the Performance Signature for Jenkins plugin.

In discussion with them we already came up with a list of future improvements, e.g:

• Thresholds based on deviation of previous build vs hard coded upper & lower bounds
• Heatmap Visualization in Jenkins Build to Build Comparison Table
• Allow validating ANY percentile and not just the 98^th which is default
• Architectural validation using Smartscape data
• Promote a good performance signature as baseline for production alerting

If you have more feedback for them either leave a comment on the blog, contribute or post on the Performance Signature GitHub project or reach out via email.

The post Shift-Left in Jenkins: How to implement Performance Signature with Dynatrace appeared first on Dynatrace news.

I don’t think there is a correct or one-size-fits-all definition of Shift-Left, neither is there one for DevOps, Cloud-Native or any of these other heavily used terms. What I do know is that there are experts in our industry trying to make it easier for engineers to get faster self-service quality, performance and scalability feedback about their proposed code changes.

In a recent chat with one of our customers who works in the in the financial sector, I was told about how they transformed their core financial services leveraging a modern cloud native architecture running on AWS. That transformation also included a shift of their development practices, their delivery pipeline and how to provide feedback to engineering and business earlier in the development cycle!

To give you one example: Instead of executing performance tests at the end of the Kanban time-box or at the end of the release cycle, performance tests are run for every build deployment and daily scheduled runs for latest code changes of each microservice. This gives both the infrastructure and development teams more and faster feedback about the impact of code or configuration changes on performance, scalability and resource consumption.

In a remote desktop session, their performance engineering team lead showed me their Dynatrace environment. I took a couple of screenshots and notes and hope that this will inspire others to redefine performance engineering with the power of the cloud, containers and Dynatrace AI. As I am respecting their request of anonymity, I have blurred out sensitive information in the screenshots!

#1: Eight Feature Based Environments powered by Terraform, EC2, Mesos, Docker & OneAgent

Let’s start with explaining what type of environment the performance engineering team is providing to the engineering teams to get faster performance feedback. They run a Dynatrace Managed installation in their AWS VPC, running 10 different Dynatrace Tenants to separate the different pre-production environments they are monitoring (Feature1, Feature2. Feature n, Performance, etc). If you want a Dynatrace Managed instead of a SaaS instance on AWS check out the recently announced Dynatrace Managed Cluster on AWS Quick Start.

On the infrastructure side their environments leverage Terraform and Ansible to deploy EC2 instances on which they run Mesos orchestration to host their Spring Boot Java services in Docker containers!

Here are some of the benefits they get out of Dynatrace, Cloud & Container Monitoring capabilities:

Benefit #1: Dynatrace allows easy management of containers and images!

One challenge in very dynamic containers environments, is to keep track of all the currently deployed containers, where they run and how many resources they consume. Making sure that those containers that should be running have enough resources and can reach all the other depending services is another challenge in highly dynamic environments. Dynatrace provides that type of visibility out-of-the-box! We call this “No-Touch Container Monitoring” as you don’t have to change anything in your container to get this visibility. This works for Docker, ECS, Fargate or any other Container flavor:

Benefit #2: Dynatrace provides automatic monitoring of container resource consumption

For a Docker host, Dynatrace provides visibility into the actual containers deployed on that host and shows how many resources these containers consume right now and over time:

Benefit #3: Dynatrace leverages CloudWatch for Metrics and for Tags

Our customer has also setup the Dynatrace AWS CloudWatch integration which not only automatically pulls in key CloudWatch metrics of AWS Services such as S3, EBS, RDS, DynamoDB, Lambda, ELB … but also pulls in Tags from EC2 instances, which get automatically applied to the monitored hosts where a Dynatrace OneAgent is installed. This makes managing all performance data much easier because dashboards, filters and notifications can be setup using the same tags as already defined on AWS.

And here is the Dynatrace AWS Overview showing how many AWS services and resources the teams are using in their Performance-Dev Environment:

TIP: If you want to learn more about basic AWS monitoring, I suggest you walk through my 101 AWS Monitoring GitHub Tutorial or watch my 101 AWS Monitoring Performance Clinic on YouTube. There I explain the integration with CloudWatch, how to deploy OneAgents on EC2 and how to monitor applications deployed on Beanstalk, ECS or Lambda!

#2: Ensuring Healthy Cloud Infrastructure with help of Dynatrace AI

While having more monitoring data available is great, thanks to the Dynatrace OneAgent and the AWS CloudWatch integration, it doesn’t mean that the team can spend more time analyzing more data points. This is where the Dynatrace automatic baselining, anomaly and root cause detection helps.

I asked them if they have an example of how the Dynatrace AI helps their infrastructure team to help a stable and healthy environment for the services that the development teams run on top.

They opened the Dynatrace Problem View, clicked on the Infrastructure filter and then walked me through the following screenshots. They showed me how Dynatrace detected a network connectivity issue of several haproxy instances running in Docker containers orchestrated by Mesos across several EC2 instances.

Benefit #1: Dynatrace’s automatic anomaly detection across all monitored entities and metrics

The Dynatrace OneAgent monitors every single container and all processes running in these containers. OneAgent automatically detects technologies and services such as haproxy, message queues, web- or application servers, databases, …

The Dynatrace anomaly detection understands which metrics are important for each type of service and reports an anomaly if a metric shows problematic or unusual behavior. Thanks to this auto-detection capability DevOps team can react much quicker to infrastructure related problems before they start impacting the services that run on them:

Benefit #2: Dynatrace Automatic Dependency and Impact Detection

Thanks to Smartscape, the infrastructure team not only knows which infrastructure components or critical services are currently in an unhealthy state. Smartscape also shows all the depending services that haproxy is connecting. Based on that information, it is easier to understand the potential impact to higher-level services, applications or even end users. This also helps to prioritize remediation actions – whether executed manually or automated.

TIP: If you want to learn more about how Dynatrace helps your IT Service and Operations (ITSM) check out the information around our ServiceNow Integration or how you can integrate Dynatrace with any CMDB YouTube Tutorial. If you want to see other examples of detected infrastructure problems check out my recent blog on AI In Action: RabbitMQ, Cassandra and JVM Memory.

#3: Shift-Left Performance Feedback with the help of Dynatrace AI

What runs on top of this dynamic cloud & container infrastructure? I am sure you guessed the answer: Services that their development teams are trying to get performance feedback on. The primary type of services they implement using Spring Boot expose REST APIs for their B2B offerings. Some of these APIs have well defined SLAs, which is why their team decided to define several custom thresholds for the different REST endpoints.

When developers make code changes, those get automatically deployed with the next scheduled build and get automatically tested. LoadRunner and JMeter are used to generate the load against their various REST APIs. These tests typically run for little over one hour after which developers pro-actively reach out to the Dynatrace dashboards to analyze how their code was performing, where the hotspots are and where there might have scalability issues.

Benefit #1: Dynatrace AI reduces time for analyzing performance test results

While Dynatrace gives them access to all data through the dashboards, the team started to see the benefit in time savings when using the Dynatrace AI automatic detected problems and root causes. The time saved can be better spent on building new features instead of manually analyzing the same metrics, log files, stack traces or CPU samples every time a test executes.

The following screenshot shows an automatically detected problem that happened during one of the automated load test runs. There was a 47% slowdown of a specific REST API endpoint caused by a CPU spike on that EC2 Linux machine, where Mesos hosts the Tomcat process in a container:

Benefit #2: Dynatrace Pro-Active Alerting of Dev Teams

Thanks to the host, process and service tagging capabilities of Dynatrace, each service is tagged with the name of the team responsible. In case Dynatrace detects a problem the team automatically gets notified thanks to the Alerting Profile feature in Dynatrace. Alerting Profiles allow sending problem notifications ONLY to those teams of services where a problem was detected. The notification (email, JIRA, Slack, Mattermost, Flowdock) also gets sent out immediately when the problem is detected and not only at the end of the test. This also speeds up the feedback loop cycle time in case a code change has an obvious issue which can be detected by Dynatrace within minutes.

Benefit #3: Dynatrace speeds up fault domain isolation between Infrastructure and Application

The CPU spike on that host makes you probably wonder: is it a problem with the infrastructure, the container or the actual app. When clicking into that root cause box in the problem ticket we end up seeing all the captured data from that EC2 Linux machine. We clearly see that there are a lot of other processes and containers running on that same box – all competing for CPU, memory, disk and the network. Very interesting to learn that the same box also runs Swagger (on Node.js), Kafka (two Jetty’s) and Filebeat (Go) besides our Tomcat that hosts the service under test:

A click on “Consuming processes” gives us a detailed CPU breakdown of all processes & containers on that machine – clearly highlighting that the cause is our Apache Tomcat process:

Benefit #4: Dynatrace detects code level problems. In custom and 3^rd party code

Now that we know the problem lies within Tomcat it is easy to find out what caused it. Dynatrace provides several hotspot detection options such as the response time analysis. Code execution is the clear “winner” in this case followed by two SQL statements:

As the SLA for their REST API endpoints is 200ms it is interesting to learn why most of this time is consumed in code execution. More interesting is WHERE in the code the time is spent. Clicking on “Code execution” in the response time analysis infographic, brings up the method hotspots view with a detailed breakdown. We got a winner: Hibernate!

The above screenshot shows the hotspot across all the request that exceeded the 200ms SLA. This is very useful and makes it easier to analyze and fix hotspots that are impacting many transactions and not just individual requests.

Thanks to the Dynatrace PurePath technology every single transaction is also available for inspection. Seeing the PurePath allows engineers to better understand the sequence of code execution which is very useful in distributed or asynchronous transactions.

Tip: I get a lot of questions from users that integrate Dynatrace with their load testing tools. Make sure to check out my blog on Load Testing Redefined or watch my Load Testing YouTube tutorial. If you want to learn more about diagnostics option with Dynatrace then check out Basic Diagnostics or Advanced Diagnostics with Dynatrace.

What else can be done with Dynatrace?

While the performance engineering team at our customer already leverages Dynatrace to speed up continuous feedback cycles to engineering as well as the infrastructure team there is more that can be done. Here are some additional ideas:

#1: Use the Dynatrace REST API to automate “sanity checks”:

• How many containers are running vs. how many should run?
• Do we run too many containers on a single host?

#2: Use the Dynatrace REST API to automate “deployment optimizations”:

• Re-distribute top resource consuming containers to avoid resource constraints
• Co-locate containers that are “tightly coupled”. This prevents network contention issues

#3: Use the Dynatrace REST API to automate “remediation actions”:

• Check out the auto-remediation blogs on Ansible Tower and StackStorm
• Check out my thoughts on Self-Healing use cases with Dynatrace

#4: Integrate Dynatrace tighter into your CI/CD Pipeline

• Read up on my thoughts about Building the Unbreakable Delivery Pipeline
• Learn how Dynatrace can be integrated into Atlassian DevOps Tools

If you have any further questions let me know. Happy to give you more insights into how Shift-Left can be done with Dynatrace and happy to share your own implementation. Just let me know!

The post Shift-Left with Dynatrace: Automating Performance Feedback with AWS, Mesos and Containers appeared first on Dynatrace news.

In the recent years, I have blogged about how Dynatrace Labs approaches certain aspects of software engineering, continuous integration, continuous delivery or as we call it CDF (Continuous Delivery & Feedback).

This current blog post was triggered by a discussion I had with Dietmar Strasser, Acceptance Test Team Lead, in our lab in Linz. Our Acceptance Test team runs tests against the latest builds of Dynatrace in different test environments (dev, staging …). They primarily leverage Selenium as Functional Test tool monitor Dynatrace, the Application Under Test (AUT), with Dynatrace. So – it is Dynatrace on Dynatrace so to say. 😊

To speed up functional test result analysis Dietmar’s team found a smart and creative way to integrate Selenium with Dynatrace RUM (Real User Monitoring). In case a functional test fails or is slower as previous runs, testers and developers get a customized link in the test result report that opens up a Dynatrace RUM dashboard, showing all data of that specific test run. This data includes every single click including waterfalls, JavaScript exception details and soon a video replay through our upcoming Session Replay feature. This link eliminates the manual step of searching for the relevant RUM data, which saves time during triage and also reduces the need to educate new hires on how to navigate and search through the captured data in order to find what they are looking for!

The following screenshot shows how this looks like in the test result report the Acceptance Team generates: It’s really just a single click!

This integration is part of their fully automated Acceptance Test Environment and is therefore available for every test executed. The following diagram shows which tools they are using, how they are integrated and where Dynatrace comes into play it comes to monitoring:

While other teams within Dynatrace engineering have moved to Jenkins, the Acceptance Testing Team still uses QuickBuild, which was historically used more broadly within Dynatrace. The team has built several integrations over the years that enable the level of automation they have. In this moment there would be no real benefit for them to move to a different CI platform.

Based on the diagram above, you can see that Dynatrace OneAgent’s not only monitors the different Test Environments (Dev, Sprint, Sprint Managed). Dynatrace is also used to monitor the VMs that run the Selenium tests to detect any potential issues with the VMs. This allows them to eliminate bad test runs caused by an unhealthy test environment. I personally consider it as a best practice to monitor all your tools and systems, as you don’t want to troubleshoot issues based on a flaky test framework!

Alright! Are you ready to learn more? Ready to try it yourself?

If you are familiar with Dynatrace, the OneAgent and Dynatrace RUM, you can skip the next section where I give a quick overview of how Dynatrace works and how the Acceptance Test Team instruments their test environment. After that, we discuss integration options with Selenium – or in that respect – with ANY other functional testing tool that drives a browser.

Step #1: Enabling Dynatrace in the Acceptance Test Environment

If you do not yet have Dynatrace in your environment, simply sign up for a Dynatrace SaaS Trial or reach out to our teams and ask for a Managed (On Premises) trial. Once you have access to your Dynatrace web interface, simply follow these steps to install a OneAgent on those machines where your Application Under Test (AUT) will be running. Feel free to watch my “What is Dynatrace and How to get Started” YouTube Tutorial or any other tutorials we have on our YouTube channel that show you how to monitor applications running on physical, virtual, cloud hosts, containers, Kubernetes, OpenShift, CloudFoundry or in any other PaaS or even SaaS platform. Once installed, Dynatrace OneAgent will automatically instrument your application and automatically enables Real User Monitoring without having to modify and code or configuration files!

Validate data gets captured correctly!

I always recommend validating that Dynatrace captures RUM and backend data in the way you expect it. That can either be done by running parts of your functional test suite or by simply opening a browser and manually hitting a couple of pages of your application you want to test.

After that is done navigate to the User sessions dashboard, pick your just simulated user (it may take up to 1 minute until your user session shows up) and explore the data captured:

There are several configuration options available that allow you to optimize how and what type of data Dynatrace captures, e.g: if your web application uses certain JavaScript frameworks (Angular, jQuery, …) you can enable support for those. Have a look at the Real User Monitoring doc pages or watch my YouTube video on Getting Started with Dynatrace RUM.

Configure User Tag Capturing

If your application has a login form, you can instruct Dynatrace to capture that username and assign it to the captured user session. This makes it easier to identify a specific session, e.g: identify your test user from other users on the system. You can configure this in the application configuration settings by either specifying a CSS Selector, JavaScript variable or the name of an HTML Meta tag to be evaluated by the Dynatrace JavaScript agent. More details on how to configure user tag capturing in the documentation pages or in my YouTube video on Advanced Real User Monitoring.

From now on every test (manual or automated) that gets executed against your application under test will automatically be captured by Dynatrace as a visit and you will be able to easily identify a specific run through username.

Step #2: Integrating Dynatrace with Selenium

Having Dynatrace installed in the test environment where you deploy your AUTs automatically gives you full visibility into every single manual or automated test execution. On top of that we can enable a couple of additional integrations that will make your life even easier:

1. Pass test context (test or script names) to the captured Dynatrace user session
2. Create links to Dynatrace dashboards for easier data lookup
3. Pull data from Dynatrace into your functional test results

Let’s tackle it one by one:

1. Passing test context to a Dynatrace User Session

If you have configured User Tag capturing as explained earlier, you already have additional context information when analyzing captured data, e.g: the test user ID you use when logging in to your site during the test. If you re-use the same user ids for every test run – which I assume is very likely – but you want to do an automatic lookup of test user ABC from test run Sept 30^th on Build 256 you can do this by passing a customized user tag to the Dynatrace Visitor.

This is made possible through the Dynatrace JavaScript SDK that is automatically available for every instrumented application. Tools like Selenium allow you to execute JavaScript that runs within the browser context which allows passing this type of context from your test script to Dynatrace.

Here is a helper function I wrote using dtrum.identifyUser. Feel free to use this function for your own scripting or modify it to your needs:

/**

 * Passes the userNameTag to the Dynatrace captured Visitor via dtrum.identifyUser

 * @param browser Selenium browser object

 * @param testUserName tag value for Dynatrace Visitor Tag

 */

function setDynatraceUserTag(browser, testUserName) {

    browser.execute(function (tagValue) {

        if(dtrum) dtrum.identifyUser(tagValue);

    }, testUserName);

}

In the Dynatrace Acceptance Test Team, they generate a unique user tag for each test they run. I am sure you have noticed it earlier when looking at the screenshots, e.g: daily09_08_2018. That session identifies the daily regression suite that got executed on Aug 9^th 2018! This convention makes it very easy to open the details of the whole test run or even the waterfall of a single test step and analyze it:

At a recent implementation workshop with a customer, their automated testing team decided to pass the Jenkins Build ID to their Selenium script and add this to the user tag, e.g: LoginTest_JenkinsBuild_23. I am sure you will pick the best approach for your testing processes!

2. Create Links to Dynatrace Dashboards

When Dietmar showed me his nicely rendered test results page in QuickBuild I was intrigued to learn more about “the magic” behind the “Investigate User Session” button. Turns out, it is not that hard to do if you know what type of link you need to generate. Dietmar was gracious enough to share the function they use internally to generate that link. It gets called from their Selenium script at the end of each test case execution.

Instead of sharing their code I went ahead and created my own version which I hope serves you well as a template for your scripts. My function tries to get the rxVisitor cookie which is present in case your navigated to a page that is instrumented with Dynatrace. If that is the case, we use that value and pass it via “id=-<rxCookie>” (be aware of the leading minus symbol) to the web dashboard. If that cookie is not present we have a fallback option to search by testUserName.

Start and end time custom timeframe filters can additionally be passed. This option allows generating a general link to all visitors that were captured for your complete functional test suite.

/**

 * Generates a link to the Dynatrace Visitor Overview Dashboard for one specific user (if rxVisitor cookie is present) or filtered by username.

 * Optionally enables a timeframe filter

 * @param tenantURL Dynatrace Tenant Base URL, e.g: https://<LIVEID>.live.dynatrace.com or https://<yourdtserverurl>/e/<yourenvironmentid>

 * @param startTime If > 0 will be used as start time for custom timerange filter

 * @param endTime If > 0 will be used as end time for custom timerange filter

 * @param testUserName If rxVisitor cookie is not available the dashboard link will search for this username

 * @param browser Selenium browser object

 */

function generateDynatraceVisitorLink(tenantURL, startTime, endTime, testUserName, browser) {

    var rxVisitor = browser.getCookie("rxVisitor");

    var dashboardLink = tenantURL + "#usersearchdetailuser";

    if (rxVisitor != null && rxVisitor != "") {

        dashboardLink += ";id=-" + rsVisitor;

    } else {

        dashboardLink += ";filtrFilterSessTags=s" + testUserName;

    }

    if((startTime > 0) && (endTime > 0))

        dashboardLink += ";gtf_c" + startTime + "_" + endTime

    return dashboardLink;

}

At a recent implementation with a customer, we generated the link as described above and logged it to stdout which eventually ended up in the Jenkins output log. That made it easy for developers to scan the Jenkins console output and navigate to the dashboard that shows exactly the captured user of that test run.

The Dynatrace Acceptance Test team went a step further and produced a nice HTML formatting so that the Dynatrace and some other links, e.g: to Jira show up as buttons in the Quick Build result view. If you want to learn more about their formatting let me know and I get you in touch with them:

3. Pull Data from Dynatrace into Test Results

One of Dynatrace’s strengths lies within its automatic baselining, the anomaly detection and the fact that problem tickets get opened whenever there is an impact to your end users, availability or overall system health. This works extremely well in production or systems that see constant load where there is enough consistent data for baselining.

For environments with more “sporadic” load – such as a functional test environment – I propose to leverage the Dynatrace REST APIs at the end of your test suite execution and pull key data points from RUM via the USQL (User Session Query Language) API as well Timeseries and Smartscape data for your applications and services.

In my recent workshop with a customer we came up with a couple of interesting use cases:

#1: Add Dynatrace Real User Error and Performance metrics to test result

At the end of the test, query metrics such as JavaScript error count, failed request count, size and response time information for exactly the executed test via USQL. If you have passed a unique test user name as user tag you can filter on that value in the USQL where clause. Now we have more out-of-the-box diagnostics metrics captured by Dynatrace in your Selenium Test Result!

#2: Validate Environment Health before starting test execution

Too often test results are invalid because the application was not in a healthy state to begin with. As a pre-test check you can either query USQL or the Timeseries API for current failure rate on the application level or on individual visitor sessions. If you already see a high failure rate in the application of activity of other tests that get executed, then why even bother executing more tests? You can make this decision by simply looking at the Dynatrace RUM metrics!

#3: Query user sessions with similar errors after test failure!

A common question after a test fails is: is it just this test or are other tests failing as well for similar reasons? You can either do this manually after you see that tests are failing by looking up all visitor data captured while your tests were running and looking for similar errors – OR – you use USQL and simply query for all visitors that e.g: also saw JavaScript errors on that particular User Action. In our recent workshop one of the attendees wanted to log out all other visitors that had similar issues than the one the test ran into. Knowing how other users ended up in the same problem makes it easier to fix that issue for all the different click paths!

I am sure there are more use cases we can come up with but these 3 are a good start!

Thanks, Dynatrace Acceptance Team

I want to say Thank You to Dietmar and the rest of the Acceptance Test Team for sharing some of the internals on how they contribute to our CDF (Continuous Delivery and Feedback) philosophy. The key aspect here is that important metrics are easily accessible for every developer in the tools they are using anyway, e.g: Jenkins.

For everyone out there that wants to give this a try simply follow the steps laid out in this blog. It starts with getting your own Dynatrace Trial and installing the Dynatrace OneAgent in your test environment. If you have any questions don’t hesitate to leave a comment or ask our experts through the in-product chat channel or our online discussion forum.

The post Accelerating Functional Test Automation Analysis with Selenium Monitoring appeared first on Dynatrace news.

In this blog, I share the struggle I’ve seen with companies that buy a performance management product in the belief it will make them more proactive. In fact, it requires much more than tools to make the transition from reactive to proactive.

The Status-Quo: Faster in being Reactive – but not yet Proactive

Companies that I deal with have to solve the same problems and questions as described in the intro when it comes to performance management. Let’s have a look at the typical process when pushing a new release into production. Each team has a different way of reacting to problems as fast as possible in case anything goes wrong:

• The Operations Team may use as many as fifteen different tools to watch every aspect of the application. As soon as something is seen amiss everyone springs into action to try and identify the problem. Once the issue is isolated the deployment change is backed out. A feedback loop is started as teams try to determine the root cause.
• The Development Team gets data from Ops but soon demands more as standard log output is not deep enough and they are typically not allowed to get direct access to these machines for debugging. Additional tracing code is hastily written into the application to get more information out to logs. These logs are combed over to try and understand where the problem is really coming from. Once the root cause is isolated new code is written to address it.
• In haste the Testing Team attempts to use load generators to check and see if the problem is really solved. That only works if the problem is able to be recreated in a controlled environment. When a level of confidence is reached, the application is deployed and the cycle begins anew.

This process at its core is the definition of being reactive in practice. It is clear that – even with the best tools and additional resources one can only get faster in being reactive but this won’t help proactively prevent problems.

What does proactive mean to Ops, Test, Dev?

Proactive means different things depending on which team is asked:

• Operations teams feel that getting proactive means having better alerts that are “smarter”. Baselines that quickly adjust for new norms without having to fly blind when new builds are introduced. The problem with this is that it does not make a team more proactive at all. It just makes that team quicker at being reactive.
• Testing teams would like to catch issues earlier but the tools used make it hard to handle the changes coming from the development teams. They ask for a solution to help them manage things from release to release while adapting to new code coming through with very little setup time.
• Development teams are trying to be more agile in their process, moving to a more task-oriented process with more automation in the testing and building of these applications. Visibility into the performance of each build is not the main priority.

Lack of communication and collaboration prevents being proactive

To address the need to be more proactive we have to address the way IT organizations look at managing the performance of applications. Current processes make this hard for one main reason: The lack of communication and collaboration during the lifecycle makes identifying and addressing issues very hard.

Each team has very different drivers and goals that only compound this challenge. Let’s think about the scenario above. The tool sets used by each one of these teams are usually by a different vendor. These tools provide different types of information that are poorly shared with the other teams:

• Operation Teams have an arsenal of tools ranging from infrastructure level tools to home grown solutions in an effort to tease out the most obscure data. All are used in the hopes that performance problems can be identified.
• Testing teams have another set that allows for some level of performance but it is typically for response time analysis of an application under load. With more web 2.0 technology these tools are having a harder time delivering information around edge complexities like 3rd party APIs and toolkits.
• Development Teams, while understanding performance, is primarily testing for functionality and data integrity. The tools used are more to automate the build process and for functional testing and recursion. Unit tests are written to derive that the right response was returned versus how long it took for that response to return.

Each tool provides performance but none of them allow for that data to be shared between teams which leads to problems when communicating and collaborating on performance topics across the lifecycle.

Becoming Proactive requires Process Changes

The Definition of Proactive: creating or controlling a situation by causing something to happen rather than responding to it after it has happened.

Having smarter alerts just make a team better at not being proactive according to this definition. The only way to become proactive is to address the problem at its source: the Application Delivery Chain. To truly become proactive with your applications you must be in control of them. Teams must come together to cause the application to do what they expect. A change in the process is imperative to make a transition from reactive teams to a proactive ecosystem. The phases of the application cannot be treated as independent isolated steps at moving code from concept to a working product that is generating revenue. It must be looked at as part of a whole. Each team dependent on another working together to bring a system from design to production ready in a timely manner. Take the previously discussed scenario. Had the team’s process been utilizing a unified platform for performance, the outcome would look very different.

So how does an organization begin to address the delivery chain and start to become proactive? IT must put performance as a key metric for delivering quality products for doing business. In order to address this, an IT organization must have a process and platform implemented that facilitates some key features. Those being communication, providing rich understanding of performance details and a single source of truth. This will deliver actionable data to the appropriate teams and empower decisions to be made with that data. Using a single platform gives performance data across all the teams involved while moving applications from the drawing board to the end user. Operations now feeds data back to the development teams for rapid feedback. Once Development addresses performance issues thorough testing can occur. Testing teams can now use real test cases extracted from user transactions to test against the performance fixes. QA now becomes empowered to give the final approval for any release going into production. Now, instead of looking for the smallest hiccup, operations expects normal performance. There comes a level of repeatability and expectation.

Proactive comes when the decision is made to attack the problems at the root before they are in the wild causing consternation for your teams and your customers. This sounds hard to do and in fact, it can be if everyone is not bought into what the overall vision and goal is. Each team needs to understand that they are not an independent group. They are now a part of the overall ecosystem and if one part fails the whole ecosystem fails. Over the next few months we will explore what this process and platform should look like at each phase and ultimately look at the need for a Center of Excellence around performance and a template that can be used to start replicating this process in an IT organization.

The post Proactive vs Reactive: How to prevent problems instead of fixing them faster appeared first on Dynatrace news.