After organizations struggled to recover from the CrowdStrike update outage on their Windows hosts in July, teams are now looking for ways to bolster the resiliency of their software update and continuous delivery practices.

The widespread impact of the CrowdStrike issue demonstrated how critical it is to avoid outages and failures during software deployments to maintain user trust and satisfaction. Traditional deployment techniques that roll out updates or patches directly into full production can present significant risks and lead to potential downtime.

Modern deployment techniques using progressive delivery—such as rolling, blue/green, canary, and feature flagging—offer a more controlled and gradual approach to releasing software. By implementing these strategies, organizations can minimize the impact of potential failures and ensure a smoother transition for users.

Eliminating the potential for outages may not be possible in all situations. That said, a unified observability and security platform such as Dynatrace can enhance modern deployment practices and enable teams to proactively monitor performance, validate changes, and best protect their downstream customers and end users from disruptions.

Bolster continuous delivery with software development lifecycle integrations

Software testing is critical, yet issues can still make it into production that negatively impact the customer experience. Dynatrace strives to protect its own customers through extensive automated test validation, complemented by staged rollouts to customer segments, which include quick feedback loops to determine whether the software is safe to continue deployments. Customers can adopt many modern and safe continuous delivery practices with Dynatrace in combination with their existing software pipelines.

By embedding Dynatrace observability into their own CI/CD pipeline, customers can prevent issues from ever leaving pre-production environments. This integration enhances the pre-release phase and plays a crucial role in the quick feedback cycle after deployment, allowing teams to identify issues immediately. Furthermore, the Dynatrace Site Reliability Guardian ensures service-level objectives (SLOs) and further validation thresholds are not violated both before and after deployment.

Progressive delivery methods to release higher-quality software, faster

The following common progressive delivery methods enable organizations to release software in a more controlled manner. 

Rolling deployments

In a rolling deployment, software is gradually rolled out, replacing previous software in a serial one-by-one fashion or in batch sets. Dynatrace can monitor production environments for performance degradations and outage events that may cause customers to lose access. The Dynatrace AutomationEngine, in conjunction with the existing deployment platforms, enables immediate and automatic software rollbacks to previous versions, limiting the blast radius for customers.

Blue/green deployments

A blue/green deployment strategy involves selecting a “blue” group to run the new software while the “green” group continues to run the previous version. The Davis AI engine immediately recognizes any anomalies, performance issues, or outages between the two groups. In case there is a degradation of service, Dynatrace picks it up and, embedded by the pipeline, can ensure all customers are rooted back to the proven, stable deployment.

Canary deployments

The Canary deployment strategy releases software to customers in incremental phases, gradually increasing the production load on the new deployment. Dynatrace enables customers to set quality measures or SLO targets for performance, outages, or other usage metrics to mitigate risk. By integrating AutomationEngine into the pipeline, customers can safely increase or decrease the load on canary deployments.

Feature flagging

Feature flagging is another popular technique used in progressive delivery that allows teams to roll out new features incrementally and with minimal risk. It supports A/B testing, canary releases, and quick rollbacks, ensuring smoother transitions and more controlled feature releases. The result is testing features in production with specific user segments, gathering feedback and making data-driven decisions on a broader rollout. Embedding feature flags in the codebase can make it easier to control the visibility and behavior of features in real time without redeploying or disrupting an application. This leads to enhanced agility, improves quality, and accelerates time-to-market, all while maintaining a seamless user experience.

As organizations implement progressive delivery strategies to enhance their release processes, integrating automated validation tools such as the Dynatrace Site Reliability Guardian (SRG) becomes essential for further optimizing these practices and ensuring robust performance monitoring

Site Reliability Guardian and AutomationEngine bolster continuous delivery tools and technologies

Dynatrace Site Reliability Guardian (SRG) integrates with continuous deployment practices and platforms to provide the following capabilities:

1. Automated change impact analysis: SRG automates analyzing the impact of changes on service availability, performance, and capacity objectives across various systems before a deployment goes live or during any of the deployment strategies.
2. Integration with CI/CD pipelines: Teams can integrate SRG into existing delivery pipelines including Jenkins, Github, GitLab, AWS, or Azure pipelines. This integration enables teams to validate releases automatically as part of their software development lifecycle before they are released to customers.
3. Workflow automation: AutomationEngine and workflows automate the execution of guardians or problem remediation. This can be tied to specific events such as deployments or configuration changes, allowing for automated validation and response to changes.
4. Service-level objectives (SLOs): SLOs enable site reliability engineers (SREs) to manage and track thresholds for critical services. The Davis AI engine proactively monitors these SLOs for degradations and acts before an SLO violation happens.
5. Automated release validation: The platform supports automated release validation for security and quality gates to ensure that only high-quality code progresses through the delivery pipeline. This integration reduces the risk of deploying faulty code to production.

Embed Dynatrace into your release process and gain even more control

The Dynatrace platform is an essential tool for customers deploying software. It provides critical data that enables rapid, fact-based decisions about continuing or rolling back deployments. By offering real-time performance metrics and insights into application health, the Dynatrace platform empowers teams to assess the impact of changes quickly. This capability minimizes downtime and ensures potential issues are addressed before affecting end users, allowing organizations to navigate software deployment complexities and enhance reliability confidently.

Enhance continuous delivery quality and manage the risk of another CrowdStrike update outage

Incidents like the CrowdStrike update outage illustrate the importance of adopting modern progressive delivery strategies to enhance software reliability and customer satisfaction. By adopting approaches like rolling, blue/green, and canary deployments, teams can mitigate risks associated with large-scale releases and ensure a smoother user experience. Integrating Dynatrace into these processes provides invaluable insights and automated monitoring capabilities, allowing DevOps teams to detect issues early and respond swiftly.

As organizations continue to navigate the complexities of software delivery, prioritizing modern deployment techniques and leveraging robust monitoring solutions will be key to avoiding outages and failures. Embracing these practices will ultimately lead to more resilient applications, happier users, and a stronger competitive edge in the market.

Contact us to learn how you can bolster the resiliency of your progressive delivery techniques with AI-driven automation that can help you avoid the effects of an outage like CrowdStrike.

The post CrowdStrike update outage: Managing continuous delivery and deployment risk with Dynatrace appeared first on Dynatrace news.

The surge in demand for digital services since 2020 is a trend that is here to stay and only expected to accelerate. To compete, organizations have to achieve both speed and reliability when bringing new products and services to market. To meet this demand, organizations are adopting DevOps practices, such as continuous integration and continuous delivery, and the related practice of continuous deployment, referred to collectively as CI/CD.

CI/CD is a series of interconnected processes that empower developers to build quality software through well-aligned and automated development, testing, delivery, and deployment. Together, these practices ensure better collaboration and greater efficiency for DevOps teams throughout the software development life cycle.

Here’s what you need to know about these software development practices, how they relate to each other, and how they benefit DevOps teams as they optimize and automate more processes to achieve ever-faster time to value for customers.

Continuous integration streamlines development

Continuous integration (CI) is a software development practice that streamlines the internal process of creating software. With CI, multiple software developers can work on different features or modules of the same application and individually commit their updates to a shared code repository as they complete them, often many times a day. When they check in their code, the build management system automatically creates a build and tests it. If the test fails, the system notifies the team to fix the code. This practice helps software teams quickly detect and resolve any bugs that come up during the development process.

Continuous integration also prevents “merge hell,” which can happen when two or more developers inadvertently make conflicting changes that break the build when the lines are merged back into the master branch. Continuous integration also avoids teams having to reconcile substantial amounts of conflicting or redundant code, which can require code freezes or even a dedicated integration stage in the pipeline.

When software development teams continuously integrate incremental changes, each developer is free to make changes without worrying about throwing someone else’s work off track or about their own work being stepped on by another. Smooth, regular merging helps software development teams complete projects more quickly and efficiently. It also ensures there is always a testable, up-to-date build that will properly compile, which is critical for frequent and rigorous application testing.

This approach saves time and resources that would otherwise be spent fixing issues later in the software development life cycle, or worse, after release, when issues are much more difficult to address.

Continuous delivery ensures code is always ready to deploy

Continuous delivery (CD) is a process in which DevOps teams develop and deliver complete portions of software to a repository, such as GitHub or a container registry, in short, controlled cycles. Continuous delivery seeks to make releases regular and predictable events for DevOps staff, and seamless for end-users. Another goal of continuous delivery is to always keep code in a deployable state so updates can go live at a moment’s notice with little or no issues.

DevOps teams can automate CI/CD pipelines to move code through the appropriate environments with no human input, which accelerates the build, test, and deployment stages of software development — or any additional stages they have in place, depending on the processes they use.

For example, when a feature is ready for client demonstration, the DevOps team can have a CD tool automatically deploy it to a test server so the client can see how it works and provide feedback before it is released to the production server.

Continuous deployment ensures customers always have the latest

Continuous deployment (also CD) is an extension of continuous delivery in which builds that pass testing are automatically deployed directly to production environments on a daily, or even hourly, basis.

Continuous deployment hastens the feedback loop with your customers and eases the burdens on operations teams by automating the next stage in the pipeline.

Once an organization’s continuous delivery practices are stable and mature, they often adopt automated continuous deployment and testing in a tiered “blue-green” sequence. In this scenario, the new build (green) is deployed in parallel with the existing build (blue) to ensure it works before the blue build is retired. Teams can also use a phased “canary” deployment approach, where the new build is gradually phased in to replace the existing build.

Continuous delivery vs. continuous deployment: which is it in CI/CD?

Continuous delivery (CD) often gets confused with continuous deployment (CD). If CD stands for continuous delivery and continuous deployment, which is it when people refer to CI/CD practices?

In short, it’s either continuous delivery, or continuous delivery and continuous deployment. Although related, the two terms refer to automated processes that happen at different points in the delivery workflow.

You can think of adopting these practices as three steps in a continuum of DevOps automation:

1. Continuous integration to streamline development and automate building and testing. It’s a way for developers to work on the same code base at the same time.
2. Continuous delivery to automate the process of delivering completed code blocks into the main branch, where it can be deployed to a production environment by the operations team. It’s a way for the development part of a DevOps team to automate the process of testing and committing code.
3. Continuous deployment to automate the process of deploying completed, tested code into a production environment. It’s a way for the operations part of a DevOps team to automate the process of deploying new code to customers.

When an organization first adopts agile DevOps practices, they often start out with continuous integration, and mature quickly into continuous delivery, thus achieving CI/CD. Many organizations stop here, preferring to release production code manually. Others progress into continuous deployment so they can automate the entire software development, delivery, and deployment pipeline.

The benefits of CI/CD

The main benefit of continuous integration and continuous delivery is that it reduces the time required to develop applications and features. This gives companies a competitive advantage over businesses that still take a manual approach to software development.

Besides the competitive advantage, CI/CD enables agile development teams to constantly iterate and release new features. These practices ensure everyone is working with the same version of the code as they develop and refine features and functions. With the logistics of integrating and testing builds automated, engineers can focus on what they do best: coding.

If teams have implemented continuous deployment, code can spend less time waiting for testing and deployment and more time in production, which accelerates feedback from users and, ultimately, improves business outcomes.

Increased efficiency, reduced time to market, and quicker innovation make CI/CD an attractive proposition for organizations of all types.

The drive to optimize and automate more CI/CD processes

To meet competitive demands, organizations know they must release new products and services more quickly than ever without compromising quality or reliability. They know basic CI/CD practices are fundamental to DevOps and DevSecOps initiatives that look to develop applications with more effective collaboration and greater precision.

Teams that implement CI/CD practices successfully rely on many tools and methods to manage features, versions, testing, and builds. Automation happens at every stage of the pipeline, from building, packaging, and testing to pushing applications to different production environments.

To accelerate the development pipeline for ever-faster releases with less risk, teams need continuous automation and advanced AI-driven observability across all the tools in their DevOps toolchain so they can automate manual steps and identify quality issues earlier in the software lifecycle.

If your organization uses or is considering adopting CI/CD DevOps processes, a best practice is to implement a full-stack observability platform that can provide code-level visibility of all software builds, apps, and services in your environment, whether they’re in development or deployed to end-users.

The Dynatrace Software Intelligence Platform continuously and automatically monitors the performance of DevOps tools, and seamlessly integrates with CI/CD workflows. With improved collaboration on a single platform and a shared data model, you can ensure your entire team has continuous situational awareness across the lifecycle.

To take a deeper look at continuous delivery, see my next post, What is continuous delivery and what are best practices for implementing it?

The post Continuous integration and continuous delivery (CI/CD): How it enhances DevOps and continuous deployment appeared first on Dynatrace news.

Every other week Dynatrace ships a new version of our Software Intelligence Platform into production. It’s done through what I would refer to as “Progressive Delivery at Cloud Scale”. What does that mean? The new version that comes out of our hardening stage gets rolled out to a small initial set of Dynatrace clusters. Through automated self-monitoring, we observe whether the update succeeded within a certain time window and whether the cluster runs within our defined Service Level Objectives (SLOs). If that’s the case, the update process continues to the next set of clusters and that process continues until all clusters are updated to the new version.

An individual Dynatrace cluster update is done through a rolling update of each individual cluster node. As you may know – a Dynatrace cluster always has at least 3 nodes – this is true for our Dynatrace Managed installations as well as our Dynatrace clusters that power our SaaS offering.

During such a rolling update, one node after the other is brought down, updated, and restarted. When a node comes back up, several initialization steps happen for each Dynatrace Tenant / Environment that runs on that Dynatrace Cluster. Normally, initializing such a Tenant / Environment takes about 1-2 seconds and is not a big deal, even if we have hundreds or more tenants on one node!

During one of our recent upgrades, our teams observed that a node in one of our Dynatrace SaaS clusters took much longer than normal to restart. While all other nodes took about 10 minutes to initialize ~1200 tenants, one of the nodes took about 18 minutes to initialize one specific tenant. That raised some questions: Was this an isolated issue just for one tenant or could it also come up when rolling out the version to all our other clusters that were still waiting for the new version?

As all our Dynatrace clusters – whether deployed in dev, staging, or production – are automatically self-monitored using Dynatrace, it was easy to answer this question! Our Cluster Performance Engineering Team in collaboration with our Autonomous Cloud Enablement (ACE) and development teams quickly identified the root cause and fixed the problem in no time! All thanks to the data Davis® already had ready for them when analyzing this slow startup problem.

This is a great example of how valuable Dynatrace is for diagnosing performance or scalability issues, and a great testimony that we at Dynatrace use our own product and its various capabilities across our globally distributed systems. And the code-level root cause information is what makes troubleshooting easy for developers.

Best of all – all this work was done with zero customer impact 😊

Now, let me walk you through the actual steps the team took so you can see for yourself how not only we at Dynatrace but any engineering organization can benefit from Dynatrace in any stage of the software delivery cycle:

Step 1: Dynatrace thread analysis

Dynatrace thread analysis is a great starting point to identify CPU hogging or hanging threads in your application. For our teams, the analysis immediately pointed out that the “main” thread spiked with 100% CPU utilization during the problematic startup period. You can see this yourself in the following screenshot where the “Estimated CPU time” of 59,8s / min clearly tell us that this Java main thread is fully utilizing one of our cores:

From here it’s a single click to the CPU Method Hotspot analysis!

Step 2: CPU hotspot analysis

As Dynatrace automatically captures stack traces for all threads at all time the CPU Hotspot analysis makes it easy to identify which code is consuming all that CPU in that particular thread. In our case this could be attributed to the service that initializes tenant-specific Geo-Information as you can see from the following screenshot:

The method “hotspot view” provides many additional capabilities like decoding or downloading the byte code right here on the spot.

Step 3: Identifying root-cause in code

With Dynatrace already pointing to the problematic line of code, and the additional captured contextual information including the problematic tenant that caused this spike, it was easy to find the root cause.

It turned out that the method in question was recently added and was new in the latest rolled. The method was reading custom geographical locations for Dynatrace’s RUM geo-mapping capability and then validated against the Geographical Database Service. By default, checking these configurations is not a performance problem as a standard tenant only has a handful of these custom locations configured. The problematic tenant on that Dynatrace cluster node, however, had well over 15,000 custom locations configured, which caused the spike in CPU and long execution time as each location was checked individually against the database!

Step 4: Fixing the issue

To solve this problem several strategies were discussed. One of them being a small cache that would have brought the initial startup time down by about 95%. Another one was to rethink the whole requirement for this check during startup.

In the end, the team was able to provide a fix to this issue solving the startup time issue for this and for future updates to Dynatrace clusters.

Conclusion: Dynatrace for Developers

I’m honoured to be able to share these stories on how we use Dynatrace internally in our everyday development, performance engineering, and progressive delivery processes. It shows how Dynatrace makes life so much easier for performance engineers and developers that need to ensure our code is highly optimized for speed and resource consumption.

Special thanks to all the teams involved in bringing us this story: Cluster Performance Engineering (CPE), ACE, Development and of course our product team who is not only building great features for our customers but also features that we truly need in our own day two-day work!

The post Progressive delivery at cloud scale: Optimizing CPU intensive code with Dynatrace appeared first on Dynatrace news.

“This is a mouthful of buzzwords” is how I started my recent presentations at the Online Kubernetes Meetup as well as the DevOps Fusion 2020 Online Conference when explaining the three big challenges we are trying to solve with Keptn – our CNCF Open Source project:

• Automate build validation through SLI/SLO-based Quality Gates
• Breaking monolithic pipelines into event-driven Delivery Choreography
• Embrace event-driven auto-remediation with an SLO-based safety net

You can watch the recording of my Online Kubernetes Meetup on YouTube (embedded below), watch the recording of DevOps Fusion on their website as well as access the slides on my Slideshare:

Since the talk, I made a few adjustments based on feedback and plan on giving an updated version at upcoming events including our own DynatraceGo! in October 2020 where I’ll specifically focus on how Keptn integrates and automates on top of Dynatrace Davis®. It’s a free virtual event so I hope you join me.

In this blog, I want to dig a bit into a core capability of Keptn which is used across all use cases: SLI/SLO-based Evaluations for Quality Gates as well as Auto-Remediation.

Shifting-left SRE to automate Quality Gates

If you’re not familiar with Site Reliability Engineering (SRE) and the concepts of Service Level Indicators (SLIs), Service Level Objectives (SLOs) and Service Level Agreements (SLAs) I recommend watching the YouTube Video from Google Engineers called SLIs, SLOs, SLAs, oh my! (class SRE implements DevOps)!

While an SLI is just a metric, an SLO just a threshold you expect your SLI to be in and SLA is just the business contract on top of an SLO. It’s great that this discipline and ideas have got a lot of attention lately – most likely because of their catchy names, good marketing, and some really great success stories. This is also the reason why Keptn leverages the same ideas and terminology. But instead of just for production reporting and alerting, Keptn is using this concept for automated build validation. It’s a shift-left of SLIs & SLOs to automate the evaluation of each build based on indicators that will be very important later in production. The following animation shows this “Shifting-Left SRE” approach that Keptn implements:

While this concept of metrics-based quality gates is not new (think about the Unbreakable Pipeline or what we did 10 years ago with AppMon’s Test Automation feature), Keptn implements SLIs/SLOs as a core capability and powers its core use cases such as automated quality gates in delivery, performance engineering as a self-service or auto-remediation validation.

Thanks to its event-driven architecture, Keptn can pull SLIs (=metrics) from different data sources and validate them against the SLOs. Currently supported data sources are Prometheus, Dynatrace, Neoload with others in the works, e.g. Wavefront.

As I personally helped drive the latest innovation of the Dynatrace SLI Provider, let me show you how easy we made it for Dynatrace users to define SLIs & SLOs:

Visual SLIs & SLOs through Dynatrace dashboards

First, let me take a moment to thank Klaus Enzenhofer, Technical Product Manager at Dynatrace, who challenged me to make SLI & SLO definition simpler so that it is more appealing to a non-deep technical user base.

In our recent Performance Clinic Automate Business Level Objective Monitoring with Dynatrace & Keptn I was able to live demo the latest version of the Dynatrace SLI Provider for Keptn which now accepts a Dynatrace dashboard as input as you can see in the following example!

As you can see – it’s as easy as creating a dashboard in Dynatrace, putting the SLIs that are important for you on it, define the thresholds via the title description and that’s it. Next time Keptn must evaluate SLIs/SLOs for either quality gates, performance analysis after a performance test, or during auto-remediation, it will pull the metrics from that dashboard.

This approach allows every team to easily define their own dashboard for their own SLIs/SLOs. You can also start creating dashboard templates to faster onboard applications of a certain type or technology stack.

For those of you who know Klaus, he also challenged me to extend this concept to also include business-relevant metrics such as conversion rates, end-user experience, page load times, and bounce rates. To make Klaus happy, the Dynatrace SLI Provider now also supports any real user metrics, calculated metrics, as well as USQL (User Session Query Language), queries:

If you want to learn more about this capability check out the readme on the Dynatrace SLI Service GitHub page.

Level up your existing Continuous Delivery

The benefit of Keptn providing these capabilities is that every other tool you use in your DevOps toolchain can leverage it by integrating Keptn through the Keptn API or by extending Keptn with a Keptn Service.

We have several examples of users using Keptn SLI/SLO-based Quality Gates in Jenkins, Azure DevOps, XebiaLabs (now digital.ai), GitLab, or other delivery tools. A common scenario is what Christian Heckelmann from ERT is doing with their GitLab pipeline. He reduced the complexity of doing build validation in his GitLab pipeline by simply letting Keptn do the job:

Christian is also working on leveraging Keptn for performance test orchestration, which will allow him to also remove those test steps out of the delivery pipeline. For that, Keptn already offers integrations with JMeter and Neoload, with other testing tool integrations in the works. After that, I think Christian is looking into Auto-Remediation with Keptn to ensure healthy environments all the way into production!

Let’s drive this vision forward, together!

There’s much more shown in the meet-up presentation, so make sure you watch it!

I also encourage you to join our Keptn community, star us on GitHub, join our Keptn Slack and test drive Keptn by walking through our Online Keptn Tutorials. Together we can drive even more innovation.

The post A guide to event-driven SRE-inspired DevOps appeared first on Dynatrace news.

Self-Service Progressive Delivery of Microservices, Automated SLI/SLO based Quality Gates, Continuous Feedback through ChatOps and Automatic Remediation of Production Issues are some of the capabilities you expect from a modern cloud-native software delivery platform.

In December Dynatrace announced Keptn, an open-source pluggable control plane enabling autonomous software delivery and operations for cloud-native applications. Mid-January, Keptn was included in the CNCF Landscape for Continuous Delivery and at the same time, Avodaq presented the How Keptn solves their challenges moving to Cloud Native & K8S (watch it on YouTube).

The recent improvements released in Keptn 0.6, a strong community with contributions from partners like Avodaq, Citrix or Neotys make it easier for everyone to build an autonomous cloud platform powered by Keptn. To get started – just follow the new Keptn Quickstart with a special gift from our friends at Google Cloud Platform!

Keptn Quickstart on GKE with $500 GCP credits

Run your Keptn installation for free on GKE! If you sign up for a Google Cloud account, Google gives you an initial $300 credit. For deploying Keptn you can apply for an additional $200 credit, which you can use towards that GKE cluster needed to run Keptn.

“Google GKE helps developers run advanced apps on a secured and managed Kubernetes service” said John Tripier, Partner Co-innovation Solution Lead at Google Cloud. “By launching these new tools, Dynatrace is enabling developers to further streamline and simplify their experience on GKE, ultimately helping businesses bring new services and products to their customers more quickly”

We want to say “Thank you Google” for your support on our mission towards Autonomous Cloud and helping us grow the user base of Keptn!

After you claimed your credit follow these 5 steps to test drive Keptn:

Step 1: Setup your GKE Cluster

Just follow the Keptn sizing and setup guidelines for GKE

Step 2: Install Keptn through the Keptn CLI

Just follow the Install Keptn instructions. It’s as easy as downloading the CLI and executing the following command:

$ keptn install --platform=gke

Step 3: Setup Progressive Multi-Stage Continuous Delivery for a Service

One of Avodaq’s challenges was having to maintain many pipeline scripts that implemented the logic of build, deploy, test, validate and promote. Keptn follows a declarative approach that eliminates the need for putting processes into scripts.

In order to set up a new Keptn project and onboard a service that will then be automatically delivered through your multi-stage delivery pipeline just follow the use case instructions on onboarding a custom service or watch the following video.

Congratulations: You have a self-service platform that allows you to deploy new versions of your service through a fully automated delivery pipeline with a single command:

$ keptn send event new-artifact --project=sockshop --service=carts --image=docker.io/keptnexamples/carts --tag=0.10.1

Step 4: Add SLO-based Quality Gates to your Delivery Pipeline

Automated promotion of an artifact from one stage to the next is great. But – what if the new version has performance, scalability or architectural flaws and therefore should never reach production? Keptn has a built-in quality gate capability that will evaluate a list of SLIs (Service Level Indicators) against SLOs (Service Level Objectives). Depending on the level of quality of the new artifact Keptn will either promote or stop an artifact. Keptn currently supports Dynatrace, Prometheus, and Neoload as SLI data sources.

To add a quality gate to your delivery pipeline simply follow the use case instructions on Deployments with Quality Gates or watch the following video.

Congratulations: You have a self-service platform that stops bad artifacts based on SLOs your developers define and store as an artifact in their Git repo. This is the end of bad deployments ending in production!

Step 5: Enable Auto-Remediation of Production Issues

Not every problem can be detected prior to the production environment. Some problems occur due to change in load patterns in production, an issue of the infrastructure or because of problems with individual features that are enabled through feature flagging frameworks. Keptn has a built-in capability for automating remediations based on problems detected by your production monitoring such as Dynatrace or Prometheus.

To add auto-remediation to your Keptn project simply follow the use case instructions for Self-Healing with Keptn or watch the following video.

Congratulations: You now have a self-service platform that auto-remediates problems in production. Gone are the times where you remote into machines during off-hours to fix problems manually.

More Keptn Use Cases

If you want to learn more check out our Keptn YouTube Channel where we can find more tutorials, demos as well as recordings of our Keptn community meetings.

Automate Keptn through the CLI and API

As you walk through the Use Cases you will notice that you can automate the interaction with Keptn either through the Keptn CLI or the Keptn’s API, e.g: push a new artifact through the progressive delivery pipeline. This allows you to easily integrate Keptn with other tools your using, e.g: Jenkins or GitLab Pipelines can build artifacts and then notify Keptn to deploy that new artifact!

Extend Keptn with custom services

If you want to extend Keptn with additional services or SLI providers check out the existing Keptn contributions and read up on how you can develop your own Keptn service.

Keptns Bridge

By now you have seen the Keptn’s Bridge which gives you full transparency into every stage, every deployment, every test, every quality gate evaluation, every promotion to another stage and how problems have been remediated. The following screenshot shows the early adopter version of the new Keptn’s bridge which you can easily enable by following the instructions here. We would be more than happy to get early feedback:

Got Feedback? Got Questions?

If you run into any problems or have questions, suggestions, ideas … explore these options

1. Keptns Documentation
2. Join our conversation on Slack
3. Send us a Tweet
4. Report a problem or enhancement idea on GitHub
5. Join our future community meetings

Thanks for joining our ride to the Autonomous Cloud and Keptn as a project that gets us there faster!

The post Quickstart to Autonomous Cloud with Keptn on GKE appeared first on Dynatrace news.

Validating Deployments still seems to be a semi-automated task for most software delivery teams. Why do I say that? Because we analyzed the results of our Autonomous Cloud Survey and sat down with a handful of these companies that provided answers to questions such as: What’s your Commit Cycle Time? or What’s your Maturity of Delivery Automation?

In one example of a US national bank the average manual time spent to validate the quality of a deployment before it can go to production was 12.5 days and therefore contributing > 90% of the overall Commit Cycle Time! Most of the time is taken by quality or release engineers looking at test results, comparing them with previous builds or walking through a checklist of items that accumulated over the years in order to harden their release acceptance process.

At Neotys PAC 2019 in Chamonix, France, I presented approaches on how to solve this problem by looking at examples from companies such as Intuit, Dynatrace, Google, Netflix, T-Systems and others. The outcome of the presentation and the follow up discussions at Neotys PAC lead to Pitometer!

Introducing Pitometer: Metrics-based Deployment Validation in your CI/CD

Pitometer is a Node.js OpenSource project which is part of keptn and it provides a good solution to automate the validation of a software deployment based on a list of indicators resulting in an overall deployment score. A single indicator is defined as a query against a data source such as a monitoring, testing, security or code quality tool. The result is graded against thresholds, baselines or previous results. To calculate the overall deployment score you can define a list of Indicators, give each indicator a maximum score (=weight) and define the total score a deployment must achieve in order to be considered passed!

The following shows how to evaluate a deployment score based on metrics from Prometheus and Dynatrace. On the left you see the Pitometer spec file with the list of indicator definitions as well as the total scoring objective. When executing the validation, Pitometer

1. Queries the results through the defined data sources (Prometheus & Dynatrace)
2. Grades the results by passing it to the Grader (in this case a Threshold grader)
3. Calculates the total deployment score

Pitometer has been architected to be extensible for both data sources as well as graders. In my Neotys PAC blog, I cover more details on which other data sources we can include and what type of grading algorithms I would like to see in the future. Neotys and T-Systems MMS are amongst the first companies to build extensions.

Good news is that Pitometer is ready to be used – and ready to be extended. If you want to use it there are two options:

• As part of keptn’s Deployment Evaluation Workflow
• Standalone: call it from your existing delivery pipeline

#1: Pitometer in keptn

If you haven’t heard about keptn – well – let me give you a quick intro. Keptn is the open source control plane for continuous deployment and automated operations for cloud native applications on Kubernetes. It takes your artifacts (e.g: docker containers), and orchestrates multi-stage direct, shadow, dark, blue/green or canary deployments. At its heart it uses Istio (for traffic control) and Knative (for event driven tool orchestration) and stores all configuration in Git – following the GitOps approach. Pitometer is used to validate a deployment after it was successfully tested based on the defined testing strategy.

As of right now, keptn is officially supported on certain Kubernetes deployments (e.g: GKE). The open source community and the team at Dynatrace that drives this project is already working on support for OpenShift and other Kubernetes services such as AKS, PKS & EKS.

If you want to contribute check out the keptn community site on GitHub, follow @keptnProject, or send the team an email via keptn@dynatrace.com!

#2: Pitometer Standalone

Pitometer is set of Node.js module libraries that you can start using in your own environment and pipelines. Following animation shows the sample code from the Pitometer GitHub repo and which result it produces:

If you want to automate deployment validation based on metrics or data from tools that you already use in your CI/CD pipeline give Pitometer a try. If there is no Pitometer source implementation available for your tool no worries – check out the reference implementations of Dynatrace or Prometheus and see how easy it is. Alternatively feel free to put in a feature request on the Pitometer project.

I am also working with some of our Dynatrace customers that are using Bamboo to deploy their monolithic applications. They also have the need to automated deployment validation and are now also leveraging Pitometer as it allows them to pull in data from Dynatrace which is monitoring their test and production environment and use that for automated quality gates!

You can also checkout an open source web microservice app and an Azure function app that utilize the Keptn Pitometer Node.js modules to gather and evaluation a provided time frame and performance specification. These were developed by my colleague Rob Jahn and Daniel Semedo from Microsoft. You can learn more and get the links to the source code in his blog, Adding automated performance quality gates using Keptn Pitometer

Beyond basic metrics: Detecting Architectural Regressions

At the recent DevExperience conference in Iasi, Romania I presented on Top Performance Challenges in Distributed Architectures. I gave real life examples on N+1 Query, N+1 Call, Payload Flood, Too Tightly Coupled and a list of other architectural patterns. I concluded that most of these patterns can be automatically detected by analyzing data and metrics from monitoring or testing tools.

In that presentation I also talked about Pitometer and how we can use it to automatically each of those common architectural regression that lead to performance and scalability problems – right within your CI/CD pipeline!

The Dynatrace Pitometer Source for instance not only supports queries for Dynatrace Timeseries Data such as Response Time, Failure Rate, CPU, Memory … – it also supports to query Smartscape Metrics such as Number of Incoming and Outgoing Dependencies, Number of instances a Service runs on, and more!

On the Pitometer GitHub project you find a sample perfspec-sample.json that includes samples for the Smartscape dependency metrics. The following shows how to evaluate the number of outgoing dependencies to e.g: ensure that your service has only outgoing dependencies to a single backend service. If a code or configuration change would add a new dependency the validation would fail:

Help us make better deployment decisions

Pitometer and keptn drive the path towards better automated deployment decisions. While Dynatrace is currently driving most of the innovation we are happy to see other companies such as Neotys, T-Systems, AWS, Microsoft … already contributing to it. We invite all of you – small startup, large enterprise, schools or universities – to use these projects and help us improve it by contributing to it.

The post Automated Deployment and Architectural Validation with Pitometer and keptn! appeared first on Dynatrace news.

Monitoring as Code is a concept to define what metrics (functional, performance, architecture) are relevant for applications, services, process, containers or infrastructure. One implementation of Monitoring as Code is what my colleague Thomas Steinmaurer and my friends Matthias Fichtner and Raphael Pionke, from T-Systems MMS, have coined “Performance Signature”. The Performance Signature is a list of metrics including important thresholds stored in a JSON or YAML file. This file gets checked in with your source code and gets automatically validated as your code moves through your Continuous Delivery Pipeline. The signature acts as an automated quality gate between your different stages (dev, test, prod) as well as allows you to validate your production deployments (blue/green, canaries, …).

Matthias and Raphael recently released their implementation of the Performance Signature for Dynatrace as a Jenkins Plugin (free for everyone to download in your Jenkins). I invited both to showcase their integration and the key use cases in one of my recent Performance Clinics. Watch it on YouTube, or simply hit the play button on the following video:

Benefits of Performance Signature

During the recording, I chimed into the conversation a couple of times, re-iterating the benefits of integrating monitoring into your delivery pipeline. At Dynatrace we have been doing it for years and we have seen this approach being implemented by many organizations that are re-platforming or re-architecting towards cloud native delivery pipelines. The key benefits from my perspective are:

#1: Monitoring data capturing de-coupled from analysis

Developers can write a simple JSON file that gets automatically validated without having to worry about how the data gets collected. This basically means “Monitoring as a Pipeline Service” or better “Monitoring as a Self-Service” through the concept of “Monitoring as Code”

#2: Performance Signature for all stages of the delivery pipeline

Developers can use it in the Dev Environment to get feedback on every code change. It can be integrated into the Pull Request Approval process before merging to master. It can act as Quality Gates between staging and production after executing your load tests. And finally, it can be used to get Real-Time Production Feedback on e.g: Blue/Green, Canary of Feature Flag releases!

#3: Versioning of approved performance signature per build/release

When a build is pushed through the pipeline the actual values captured while validating your Performance Signature can be committed back to your source code repo. This not only allows you to historically see the trend of your performance – but also allows you to push these accepted values as your new production baselines. In Dynatrace we provide REST APIs to define your thresholds for your service endpoints or even for resource consumption. We can make this part of the production deployment by applying the thresholds as defined in your source code repo to your production monitoring environment!

Implementation Guide for your Jenkins Performance Signature

While the performance clinic video, as well as the documentation on the Plugins GitHub page, should give you all the information you need to implement the Performance Signature for Jenkins yourself, I thought I would write a quick step-by-step guide to better understand the use cases & benefits.

Step 0: Understand the End-to-End Continuous Delivery Workflow, Integrations & Use Cases

I created the following animation based on what Raphael & Matthias showed in their tutorial. I additionally highlighted the individual workflow steps, integrations and when which use case comes into play:

While Raphael & Matthias used a Kubernetes Cluster in their demo this also works for any other platform such as OpenShift, CloudFoundry, ECS, Fargate, Docker and even your more “classical” Java & .NET stacks! Now let’s get through all of this step-by-step

Step 1a: Install Dynatrace OneAgents in your Environment

First step is to rollout the Dynatrace OneAgent in your environment. If you don’t yet have a Dynatrace installation, simply sign up for our SaaS-based Trial and follow the instructions to deploy the OneAgent on your Linux, Windows, Solaris or AIX machines. If you run on Kubernetes, OpenShift, CloudFoundry, Azure, AWS, GCP … simply follow the instructions in the Deploy Dynatrace OneAgent section. The OneAgent will automatically take care of FullStack Monitoring.

Recommendation: Dynatrace can automatically pick up tags & metadata from the underlying PaaS or SaaS platforms, e.g: Kubernetes Namespace, Container Names, CloudFoundry Organizations, … – If you need to pass more metadata or define your own tag rules check out my session on 101 Tagging Essentials with Dynatrace and the blogs from my colleague Michael Kopp on Tagging & Naming. In my personal environment I run k8s on GCP and I have configured automated tagging to create tag for environment (from the k8s namespace) and the app (from the container name). Following screenshot shows my payment app-service deployed in the jx-staging space – fully automatically tagged through the automated tagging rules:

Step 1b: Install the Jenkins Performance Signature Plugin

Install the Performance Signature: Dynatrace plugin it via the Jenkins Plugin Manager. In the global system configuration section in Jenkins configure the endpoint to your Dynatrace Tenant. This can either be a SaaS or Managed Tenant. All you need is your Dynatrace Tenant URL, e.g: https://YOURTENANT.live.dynatrace.com and a Dynatrace API Token which you can create in Dynatrace via Settings -> Integration -> Dynatrace API.

Step 2: Push Deployment Information to Dynatrace

In the Jenkinsfile find the stage where you deploy into a certain stage, e.g: Staging. Now simply wrap this deployment with a call to createDynatraceDeploymentEvent. This Jenkins pipeline step is provided by the Performance Signature plugin. It will measure the time it takes for the deployment and upon success will push a Dynatrace Deployment Event via the Dynatrace REST API. The event will be pushed to those entities that match the tagMatchRules. Therefore, it is important to have proper tagging setup as described in Step 1a. In the following example I deploy the carts app-service into the environment jx-staging. Here is the modified Jenkinsfile:

Once the pipeline runs, we see Dynatrace not only picking up Monitoring of the deployed services, we also see the deployment events on those service entities that are tagged with app:carts & environment:jx-staging:

I believe this is the first game changer. Why? Because every time somebody runs the pipeline, either on the master or a dev branch, Dynatrace and Jenkins are automatically linked. When analyzing data in Dynatrace you immediately know which Jenkins job deployed at which time, what the Git Commit was, how long the deployment lasted, and which services were impacted by that deployment.

Step 3: Execute your Integration, API, Load or Performance Tests

This step now is straight forward. Simply execute your Selenium, JMeter, Neotys, Gatling, … tests that you normally execute in your pipeline. We do recommend that you tag your HTTP-based tests through the x-dynatrace-test HTTP Header as explained in How to I integrate Dynatrace into my load testing processes? This will make it easier to analyze, diagnose and compare individual test runs.

The following shows my modified Jenkinsfile for the Performance Testing stage. Just as we wrapped the deployment of the application in the previous step we are now wrapping the execution of the performance test into a call to recordDynatraceSession – another pipeline step that comes with the Performance Signature Jenkins Plugin. It will push a load testing event to those entities in Dynatrace once the test is complete. It also allows us to define the testCase name which is how data is grouped later on in the Jenkins Result view. You could run different tests, e.g: Load Test, Functional Test, API Test, … – and use that wrapping to pull in different metrics for different types of tests you execute.

recordDynatraceSession goes hand in hand with perfSigDynatraceReports. This is where the “real magic” behind the Performance Signature evaluation happens. Let me show you all the details about this in Step 4!

Step 4a: Understand and define your Performance Signature

Let’s have a quick look at the Performance Signature – the JSON file that we will define as an engineer for automated evaluated in our pipeline. The file is an array of timeseries definitions:

• a timeseriesId, e.g: response time, CPU, memory, failure rate …
• an aggregation: avg, min, max, count, percentile
• tags: pull metrics from those entities with these tags
• entityId: instead of tags you can also specify a specific EntityId
• upperLimit & lowerLimit: both must be specified and actually define a SEVERE and WARNING threshold. Depending on whether upperLimit > lowerLimit or lowerLimit > upperLimit the limits are validated against the upper or lower value range. See below screenshot for an example

The next screenshot shows a Performance Signature that contains two metrics: service.responsetime and service.server_side_requests (=throughput). Please not the different boundaries:

• Response Time validates against upper boundaries: We want to fail the build if the 98^th percentile of response time exceeds 1.8s and mark the build unstable if we are higher than 1.5s
• Throughput validates against lower boundaries: We want to fail the build if we have less than 100 requests / min and mark as unstable the build if we have less than 250

I hope this explains how to use upperLimit and lowerLimit. We might want to rename these though in the future to something like warningLimit and severeLimit as it makes a bit more sense.

Now let’s have a look how this gets evaluated and where we actually see the data.

Step 4a: Validating Performance Signature

In Step 3 we already saw the call to perfSigDynatraceReports. It was executed right after the performance test was executed. As a parameter it takes the references to the Performance Signature specfile:

Now – here is what happens in Dynatrace when the evaluation happens – we see the event on those entities referenced in the recordDynatraceSession step:

More important to this information are the actual performance signature results in Jenkins. We see them build over build in the Job overview:

For each individual build we see every metrics for every entity that matched and all data points for the timeframe of the test run. We also see if there were any violations to the defined thresholds (upperLimit, lowerLimit). This either creates a severe or warning incident as shown in the screen below with details about the violation: actual value vs thresholds.In case of a violation the build status would also change to either failed or unstable. Last bonus item is that the view also shows a comparison of the current to the previous build:

I believe this is a game change. WHY? Because it provides Monitoring Feedback as a Service automated into my continuous integration pipeline. As a developer I only have to define the list of metrics I want in a JSON file and everything else happens “auto-magically”. I get notified about performance, resource consumption or architectural regressions (e.g: too many unwanted dependencies) and can then easily drill down to the root cause. Which is what we are going to see next! Yeah – you heard me right – we can also extend this to architectural validations as we can also pull in Smartscape metrics such as incoming or outgoing dependencies!

Step 5: Comparing two builds and find the difference

Once we have a failed build due to the performance signature violation, we want to know what the root cause is. For HTTP-based applications & services I favor the following approach:

#1: Open the Top Web Requests View

I start in the Top Web Requests View and filter it down to those requests that came in from my load testing script. This works if you are following the Load Testing Integration for Dynatrace best practice and have request attributes configured to filter on your load tests executed from Jenkins:

In the screenshot above also see that I have filtered on the request attributed called LTN (=Load Test Name). I am only looking at those requests that come in from my specific Load Test. Now I can select one of the tests that was executed between 1:51 and 2:06 and click on Comparison of one of the tested endpoints!

In the following Comparison View I select the timeframe of the previous or next build as compare timeframe: 2:26 – 2:41. Then I open the Response Time Hotspot comparison:

In case of a code execution issue I can go directly into Method Hotspots to see which methods consumed all this wait time that wasn’t there in the previous build:

Depending on what the hotspot analysis tells us we can go off different routes of diagnosing the issue. For more details on diagnostics make sure to check out my Basic Diagnostics and Advanced Diagnostics with Dynatrace Online Tutorial.

Give us feedback on Performance Signature for Jenkins

I want to start to give feedback to Matthias and Raphael by saying: THANK YOU for implementing Monitoring as Code in such an elegant way and making it available through the Performance Signature for Jenkins plugin.

In discussion with them we already came up with a list of future improvements, e.g:

• Thresholds based on deviation of previous build vs hard coded upper & lower bounds
• Heatmap Visualization in Jenkins Build to Build Comparison Table
• Allow validating ANY percentile and not just the 98^th which is default
• Architectural validation using Smartscape data
• Promote a good performance signature as baseline for production alerting

If you have more feedback for them either leave a comment on the blog, contribute or post on the Performance Signature GitHub project or reach out via email.

The post Shift-Left in Jenkins: How to implement Performance Signature with Dynatrace appeared first on Dynatrace news.

In the recent years, I have blogged about how Dynatrace Labs approaches certain aspects of software engineering, continuous integration, continuous delivery or as we call it CDF (Continuous Delivery & Feedback).

This current blog post was triggered by a discussion I had with Dietmar Strasser, Acceptance Test Team Lead, in our lab in Linz. Our Acceptance Test team runs tests against the latest builds of Dynatrace in different test environments (dev, staging …). They primarily leverage Selenium as Functional Test tool monitor Dynatrace, the Application Under Test (AUT), with Dynatrace. So – it is Dynatrace on Dynatrace so to say. 😊

To speed up functional test result analysis Dietmar’s team found a smart and creative way to integrate Selenium with Dynatrace RUM (Real User Monitoring). In case a functional test fails or is slower as previous runs, testers and developers get a customized link in the test result report that opens up a Dynatrace RUM dashboard, showing all data of that specific test run. This data includes every single click including waterfalls, JavaScript exception details and soon a video replay through our upcoming Session Replay feature. This link eliminates the manual step of searching for the relevant RUM data, which saves time during triage and also reduces the need to educate new hires on how to navigate and search through the captured data in order to find what they are looking for!

The following screenshot shows how this looks like in the test result report the Acceptance Team generates: It’s really just a single click!

This integration is part of their fully automated Acceptance Test Environment and is therefore available for every test executed. The following diagram shows which tools they are using, how they are integrated and where Dynatrace comes into play it comes to monitoring:

While other teams within Dynatrace engineering have moved to Jenkins, the Acceptance Testing Team still uses QuickBuild, which was historically used more broadly within Dynatrace. The team has built several integrations over the years that enable the level of automation they have. In this moment there would be no real benefit for them to move to a different CI platform.

Based on the diagram above, you can see that Dynatrace OneAgent’s not only monitors the different Test Environments (Dev, Sprint, Sprint Managed). Dynatrace is also used to monitor the VMs that run the Selenium tests to detect any potential issues with the VMs. This allows them to eliminate bad test runs caused by an unhealthy test environment. I personally consider it as a best practice to monitor all your tools and systems, as you don’t want to troubleshoot issues based on a flaky test framework!

Alright! Are you ready to learn more? Ready to try it yourself?

If you are familiar with Dynatrace, the OneAgent and Dynatrace RUM, you can skip the next section where I give a quick overview of how Dynatrace works and how the Acceptance Test Team instruments their test environment. After that, we discuss integration options with Selenium – or in that respect – with ANY other functional testing tool that drives a browser.

Step #1: Enabling Dynatrace in the Acceptance Test Environment

If you do not yet have Dynatrace in your environment, simply sign up for a Dynatrace SaaS Trial or reach out to our teams and ask for a Managed (On Premises) trial. Once you have access to your Dynatrace web interface, simply follow these steps to install a OneAgent on those machines where your Application Under Test (AUT) will be running. Feel free to watch my “What is Dynatrace and How to get Started” YouTube Tutorial or any other tutorials we have on our YouTube channel that show you how to monitor applications running on physical, virtual, cloud hosts, containers, Kubernetes, OpenShift, CloudFoundry or in any other PaaS or even SaaS platform. Once installed, Dynatrace OneAgent will automatically instrument your application and automatically enables Real User Monitoring without having to modify and code or configuration files!

Validate data gets captured correctly!

I always recommend validating that Dynatrace captures RUM and backend data in the way you expect it. That can either be done by running parts of your functional test suite or by simply opening a browser and manually hitting a couple of pages of your application you want to test.

After that is done navigate to the User sessions dashboard, pick your just simulated user (it may take up to 1 minute until your user session shows up) and explore the data captured:

There are several configuration options available that allow you to optimize how and what type of data Dynatrace captures, e.g: if your web application uses certain JavaScript frameworks (Angular, jQuery, …) you can enable support for those. Have a look at the Real User Monitoring doc pages or watch my YouTube video on Getting Started with Dynatrace RUM.

Configure User Tag Capturing

If your application has a login form, you can instruct Dynatrace to capture that username and assign it to the captured user session. This makes it easier to identify a specific session, e.g: identify your test user from other users on the system. You can configure this in the application configuration settings by either specifying a CSS Selector, JavaScript variable or the name of an HTML Meta tag to be evaluated by the Dynatrace JavaScript agent. More details on how to configure user tag capturing in the documentation pages or in my YouTube video on Advanced Real User Monitoring.

From now on every test (manual or automated) that gets executed against your application under test will automatically be captured by Dynatrace as a visit and you will be able to easily identify a specific run through username.

Step #2: Integrating Dynatrace with Selenium

Having Dynatrace installed in the test environment where you deploy your AUTs automatically gives you full visibility into every single manual or automated test execution. On top of that we can enable a couple of additional integrations that will make your life even easier:

1. Pass test context (test or script names) to the captured Dynatrace user session
2. Create links to Dynatrace dashboards for easier data lookup
3. Pull data from Dynatrace into your functional test results

Let’s tackle it one by one:

1. Passing test context to a Dynatrace User Session

If you have configured User Tag capturing as explained earlier, you already have additional context information when analyzing captured data, e.g: the test user ID you use when logging in to your site during the test. If you re-use the same user ids for every test run – which I assume is very likely – but you want to do an automatic lookup of test user ABC from test run Sept 30^th on Build 256 you can do this by passing a customized user tag to the Dynatrace Visitor.

This is made possible through the Dynatrace JavaScript SDK that is automatically available for every instrumented application. Tools like Selenium allow you to execute JavaScript that runs within the browser context which allows passing this type of context from your test script to Dynatrace.

Here is a helper function I wrote using dtrum.identifyUser. Feel free to use this function for your own scripting or modify it to your needs:

/**

 * Passes the userNameTag to the Dynatrace captured Visitor via dtrum.identifyUser

 * @param browser Selenium browser object

 * @param testUserName tag value for Dynatrace Visitor Tag

 */

function setDynatraceUserTag(browser, testUserName) {

    browser.execute(function (tagValue) {

        if(dtrum) dtrum.identifyUser(tagValue);

    }, testUserName);

}

In the Dynatrace Acceptance Test Team, they generate a unique user tag for each test they run. I am sure you have noticed it earlier when looking at the screenshots, e.g: daily09_08_2018. That session identifies the daily regression suite that got executed on Aug 9^th 2018! This convention makes it very easy to open the details of the whole test run or even the waterfall of a single test step and analyze it:

At a recent implementation workshop with a customer, their automated testing team decided to pass the Jenkins Build ID to their Selenium script and add this to the user tag, e.g: LoginTest_JenkinsBuild_23. I am sure you will pick the best approach for your testing processes!

2. Create Links to Dynatrace Dashboards

When Dietmar showed me his nicely rendered test results page in QuickBuild I was intrigued to learn more about “the magic” behind the “Investigate User Session” button. Turns out, it is not that hard to do if you know what type of link you need to generate. Dietmar was gracious enough to share the function they use internally to generate that link. It gets called from their Selenium script at the end of each test case execution.

Instead of sharing their code I went ahead and created my own version which I hope serves you well as a template for your scripts. My function tries to get the rxVisitor cookie which is present in case your navigated to a page that is instrumented with Dynatrace. If that is the case, we use that value and pass it via “id=-<rxCookie>” (be aware of the leading minus symbol) to the web dashboard. If that cookie is not present we have a fallback option to search by testUserName.

Start and end time custom timeframe filters can additionally be passed. This option allows generating a general link to all visitors that were captured for your complete functional test suite.

/**

 * Generates a link to the Dynatrace Visitor Overview Dashboard for one specific user (if rxVisitor cookie is present) or filtered by username.

 * Optionally enables a timeframe filter

 * @param tenantURL Dynatrace Tenant Base URL, e.g: https://<LIVEID>.live.dynatrace.com or https://<yourdtserverurl>/e/<yourenvironmentid>

 * @param startTime If > 0 will be used as start time for custom timerange filter

 * @param endTime If > 0 will be used as end time for custom timerange filter

 * @param testUserName If rxVisitor cookie is not available the dashboard link will search for this username

 * @param browser Selenium browser object

 */

function generateDynatraceVisitorLink(tenantURL, startTime, endTime, testUserName, browser) {

    var rxVisitor = browser.getCookie("rxVisitor");

    var dashboardLink = tenantURL + "#usersearchdetailuser";

    if (rxVisitor != null && rxVisitor != "") {

        dashboardLink += ";id=-" + rsVisitor;

    } else {

        dashboardLink += ";filtrFilterSessTags=s" + testUserName;

    }

    if((startTime > 0) && (endTime > 0))

        dashboardLink += ";gtf_c" + startTime + "_" + endTime

    return dashboardLink;

}

At a recent implementation with a customer, we generated the link as described above and logged it to stdout which eventually ended up in the Jenkins output log. That made it easy for developers to scan the Jenkins console output and navigate to the dashboard that shows exactly the captured user of that test run.

The Dynatrace Acceptance Test team went a step further and produced a nice HTML formatting so that the Dynatrace and some other links, e.g: to Jira show up as buttons in the Quick Build result view. If you want to learn more about their formatting let me know and I get you in touch with them:

3. Pull Data from Dynatrace into Test Results

One of Dynatrace’s strengths lies within its automatic baselining, the anomaly detection and the fact that problem tickets get opened whenever there is an impact to your end users, availability or overall system health. This works extremely well in production or systems that see constant load where there is enough consistent data for baselining.

For environments with more “sporadic” load – such as a functional test environment – I propose to leverage the Dynatrace REST APIs at the end of your test suite execution and pull key data points from RUM via the USQL (User Session Query Language) API as well Timeseries and Smartscape data for your applications and services.

In my recent workshop with a customer we came up with a couple of interesting use cases:

#1: Add Dynatrace Real User Error and Performance metrics to test result

At the end of the test, query metrics such as JavaScript error count, failed request count, size and response time information for exactly the executed test via USQL. If you have passed a unique test user name as user tag you can filter on that value in the USQL where clause. Now we have more out-of-the-box diagnostics metrics captured by Dynatrace in your Selenium Test Result!

#2: Validate Environment Health before starting test execution

Too often test results are invalid because the application was not in a healthy state to begin with. As a pre-test check you can either query USQL or the Timeseries API for current failure rate on the application level or on individual visitor sessions. If you already see a high failure rate in the application of activity of other tests that get executed, then why even bother executing more tests? You can make this decision by simply looking at the Dynatrace RUM metrics!

#3: Query user sessions with similar errors after test failure!

A common question after a test fails is: is it just this test or are other tests failing as well for similar reasons? You can either do this manually after you see that tests are failing by looking up all visitor data captured while your tests were running and looking for similar errors – OR – you use USQL and simply query for all visitors that e.g: also saw JavaScript errors on that particular User Action. In our recent workshop one of the attendees wanted to log out all other visitors that had similar issues than the one the test ran into. Knowing how other users ended up in the same problem makes it easier to fix that issue for all the different click paths!

I am sure there are more use cases we can come up with but these 3 are a good start!

Thanks, Dynatrace Acceptance Team

I want to say Thank You to Dietmar and the rest of the Acceptance Test Team for sharing some of the internals on how they contribute to our CDF (Continuous Delivery and Feedback) philosophy. The key aspect here is that important metrics are easily accessible for every developer in the tools they are using anyway, e.g: Jenkins.

For everyone out there that wants to give this a try simply follow the steps laid out in this blog. It starts with getting your own Dynatrace Trial and installing the Dynatrace OneAgent in your test environment. If you have any questions don’t hesitate to leave a comment or ask our experts through the in-product chat channel or our online discussion forum.

The post Accelerating Functional Test Automation Analysis with Selenium Monitoring appeared first on Dynatrace news.

Building resilient systems has been a driving motivation for many architects, performance engineers and site reliability engineers (SREs) over the past years. There are plenty of blogs, conference talks and examples that show how companies such as Netflix, Google, Amazon, Microsoft, Facebook and others, baked resiliency into everything they do. Using monitoring data is a key part of building resilient systems. But instead of just focusing on keeping a system up & running in production through smart auto-remediation, auto-scaling, failover and high-availability… we at Dynatrace believe that building a reliable system must be done throughout your Continuous Delivery Pipeline: from Dev all the way to Ops.

To showcase how such an End-to-End Delivery Pipeline, that prevents bad code changes or bad situations in production from impacting the reliability of your system, looks like – I built my “Unbreakable Delivery Pipeline” tutorial. The tutorial was first given at our Dynatrace PERFORM 2018 User Conference in Las Vegas as a HOT (Hands-On-Training) session with a focus on showing the integration between Dynatrace and AWS CodePipeline:

Based on the positive feedback at PERFORM, we decided to “go on tour” with our “Unbreakable Pipeline Tutorial” workshop and bring it to cities all around the globe! Having delivered the workshop in several cities I thought it is time give a “Mid Tour Status Report”!

Feedback from cities we toured so far

We started our tour in Atlanta, GA and went on to Columbus, OH. These two cities were essential to shape the pipeline example. If you look at the GitHub repo, you can see that some of the early feedback that came in, in Atlanta and Columbus, allowed me to make some necessary improvements of my AWS Lambda functions. THANKS to all the attendees for your constructive feedback!

From there, I continued the tour and brought it to DevOps Fusion in Zurich before heading back to the US hitting Minneapolis, San Francisco (AWS Summit) and Denver:

To sum up what has happened so far:

• About 800 attendees saw the unbreakable pipeline at 2 conference sessions
• About 100 attendees implemented the unbreakable pipeline in hands-on workshops
• Several important changes were added to the GitHub repo based on attendee feedback

One important question I kept hearing in every city: “Andi – you built the Unbreakable Pipeline using AWS CodePipeline, CodeDeploy, DynamoDB, EC2 and Lambdas. Can it also be implemented with other CI/CD tools such as Jenkins, Bamboo, Concourse; deployment automation tools such as Ansible, Puppet, Chef; and testing tools such as JMeter, Neotys, Gatling …?”

The answer is: YES! All the magic happens by leveraging the Dynatrace REST API to implement concepts such as:

• Monitoring as Code: Define and Enforce Monitoring Contracts for apps and services as code
• Shift-Right: Push Deployment Information from your Pipeline to Dynatrace via Events API
• Shift-Left: Pull Performance and AI-Detected Problem details back to the Pipeline to act as automatic Quality Gates via the Problems, Smartscape and Timeseries API
• Self-Heal: Leverage Dynatrace Problem Notification integrations with ServiceNow, xMatters, PagerDuty, Ansible Tower, StackStorm or any custom integration to implement smart auto-remediation in case of issues

My team is currently working on building another “Unbreakable Pipeline” demo for Jenkins and Concourse. Until we have it ready, you can already go ahead and directly call these REST APIs from your respective tools in your CI/CD DevOps tool chain. Find all the REST API Documentation, leverage the REST API examples from Wolfgang Beer, use my Python based Dynatrace CLI or call these bash scripts I put out on GitHub!

Join US – but make sure to come prepared!

The workshop typically runs for 3 hours and we are doing a lot of hands-on work with Dynatrace, as well as AWS. While I am always happy to help people getting started with a new technology, we require attendees to have basic skills with AWS when attending this workshop. You must bring your own AWS Account and you have to be familiar with navigating through the AWS Console. We will be creating and manipulating S3 buckets, will create CloudFormation stacks, explore AWS CodePipeline and potentially look at some AWS Lambda functions and AWS API Gateway configurations. We do recommend that you walk through my 101 AWS Monitoring Tutorial for Dynatrace. If you master the first 2 labs in that tutorial, you will be READY for our AWS DevOps Tutorial for Dynatrace (aka the Unbreakable Pipeline Tutorial).

And if you really want to know more about what you are getting into – watch my 30 minute Online Performance Clinic where I give an end-to-end overview of this tutorial or read the Unbreakable Pipeline, Shift-Left, Shift-Right & Self-Healing blog.

BE PART of the Unbreakable Pipeline Movement

Thanks to all of you that have already participated in one of the workshops, downloaded or cloned the GitHub repo, watched my presentations at conferences, read my blog or watched my Observability Clinic. Please keep giving us feedback on how this concept works for you and your organization. Let us know which additional concepts we should include and share your success stories with us!

The post Unbreakable Delivery Pipeline: Mid-Tour Backstage Report appeared first on Dynatrace news.

This blog is not buzzword bingo. This is about implementing Shift-Left, Shift-Right and Self-Healing into a Continuous Delivery and Deployment Pipeline. To prove my point, I created a hands-on tutorial and uploaded it to this public GitHub Repo: https://github.com/Dynatrace/AWSDevOpsTutorial

There is also Performance Clinic Recording on our YouTube Channel.

While I chose Dynatrace for Full-Stack Monitoring and AWS services for the pipeline, deployment, orchestration, compute and storage, you should be able to apply this to other cloud platforms and CI/CD tool chains. The following animation shows the delivery pipeline flow from “Pull Request” (left) until “Production Deployment” (right). Dynatrace plays the “Guardian” between Stages (Quality Gates) and orchestrates Self-Healing in Production:

What this tutorial shows is the role of Full-Stack Monitoring in modern Delivery Pipelines. Whether you use Dynatrace or other tools I believe the following use cases MUST BE supported to implement a pipeline that prevents bad code changes from impacting your real end users:

• Automated FullStack Monitoring: Ability to auto deploy and enable monitoring for your hosts, your PaaS (CloudFoundry, OpenShift, SAP Cloud Platform …) or your Functions (Lambda, Azure Functions …). Monitoring has to auto discover all processes, services and applications regardless of the technology stack used!
• Shift-Right: Ability to push deployment information and meta data to your monitoring environment, e.g: differentiate BLUE vs GREEN deployments, push build or revision number of deployment, notify about configuration changes …
• Shift-Left: Ability to pull data for specific entities (processes, services, applications, tests …) through an Automation API and feed it into the tools that are used to decide on whether to stop the pipeline or keep it running
• Self-Healing: Ability for smart auto-remediation that addresses the root cause (e.g: FIX a bad GC setting) of a problem and not the symptom (add more server because of high CPU caused by bad GC)

Let me briefly explain these 4 use cases in detail and how I showcase them in my Dynatrace AWS DevOps Tutorial you can walk through on GitHub:

Use Case #1: Automated Full-Stack Monitoring

Dynatrace’s OneAgent technology automates the tedious deployment of otherwise different monitoring agents (Java, .NET, Web Server, Node, Lambda, …) or even different monitoring tools (Network, Log, Container, Cloud) to achieve FullStack monitoring.

In my tutorial I auto deploy the Dynatrace OneAgent with the EC2 Instances launched through AWS CloudFormation by simply adding the download and installation script step to the UserData section of my EC2 Instance Definition. The following is a snippet from my CloudFormation template – you can see that the actual download URL is parameterized so that this script works for ANY Dynatrace Tenant:

"UserData": {
  "Fn::Base64": {
     "Fn::Join": ["", [
       "#!/bin/bash -xn",
       "# First: we download and install the Dynatrace OneAgentn",
       "yum update -yn",
       "yum install ruby wget -yn",
       "wget -O Dynatrace-OneAgent-Linux.sh "", {
         "Ref": "DynatraceOneAgentURL"
       }, ""n",
       "sudo /bin/sh Dynatrace-OneAgent-Linux.sh APP_LOG_CONTENT_ACCESS=1n",
       "# Second: ...",

Once these EC2 instances launch, Dynatrace automatically monitors them FullStack as shown in the following screenshot. A single agent gives you all of this visibility:

The screenshot above shows you Dynatrace Smartscape on the left, and some of the data automatically captured for each entity on the right. All this data is also accessible through the Dynatrace Smartscape API and Timeseries API.

Use Case #2: Shift-Right

Shift-Right for me means to push context information about deployments or changes to the environment that is monitored. The more context we have the better we detect root cause and the more effective we can implement Self-Healing.

Passing Meta Data (Tags, Properties, …) to Full-Stack Monitoring

The first implementation of Shift-Right is to pass meta data as part of your automated deployment process. By default, Dynatrace captures meta data such as tags from AWS EC2 instances, Docker, CloudFoundry, OpenShift, Kubernetes … – this eliminates the need to re-tag your entities in your monitoring tool. Simply re-use what you have already specified in your platform.

Additionally to automated meta data capturing from the underlying platform Dynatrace extracts more meta data from environment variables, Java properties, etc… In my tutorial, the shell script that AWS CodeDeploy executes when launching my application will export the environment variables DT_TAGS, DT_CUSTOM_PROP and DT_CLUSTER_ID. These are special environment variables that Dynatrace OneAgent automatically converts into meta data and tags on the Process Group Instances. If you have your own environment variables or Java properties that contain build specific information you can configure Dynatrace to pick these up through custom process group detection rules. The following screenshot shows what my AWS CodeDeploy script looks like:

This is what I would call “Monitoring as Code” or “Monitoring as Configuration”. Why? Because monitoring data gets passed via your automated deployment configuration scripts! No need to configure these critical pieces of information twice! Simply RE-USE IT!

Passing Deployment Information to Full-Stack Monitoring

The second implementation of Shift-Right can be done via the Dynatrace API. The API provides the capability to push custom tags, timeseries or custom events to any monitored entity (host, process group instance, service or application). In my tutorial I used the Dynatrace Events API to push build and deployment information of the current AWS CodePipeline and AWS CodeDeploy run to the services that Dynatrace is monitoring:

Use Case #3: Shift-Left

The term Shift-Left has been used and probably abused just as often as DevOps. For me, Shift-Left means to use more data earlier in the lifecycle to answer the key question: “Is this a good or a bad change we try to push towards production?” And if it is bad, don’t push it further down the pipeline!

While monitoring data should not be the only data you look at in your pipeline (make sure you look at your code quality metrics) it is one aspect that many have tried to automatically include in the pipeline to automate build validation.

In my Dynatrace AWS DevOps Tutorial I use the Dynatrace Timeseries API to automate the Build Validation process by comparing different metrics for the services we push through the pipeline. We compare the current build that is pushed through with another baseline build or even with our current state in production. The following illustration shows how to define this type of comparison (Monitoring as Code), where to get the metrics from (which environments) and how to figure out whether this is a good or bad build (define thresholds):

Traditionally Build Validation is done manually by performance engineers that pull up different dashboards of different timeframes and either give their thumbs up or down. Many have automated the generation of these reports but still compare the final reports manually!

In my tutorial I completely automated this Build Validation Process! The Build Validation (=comparison) is triggered from the Pipeline and executed by an AWS Lambda Function which also approves or rejects the pipeline approval stage that allows or prevents it from entering the next stage. All build validation results and every single metric that was compared is also accessible through an HTML report I created:

The following animation shows how this is integrated into the Pipeline. A key aspect is that the Build Validation Process can only run when the load test has ran long enough to have enough data for a valid comparison. This “wait” is achieved through a validation request entered in a DynamoDB table and a period executing Lambda function that only executes that request when enough time has passed:

I have to give credit to Curtis Bray, who presented a similar concept at AWS re:Invent in his session called DevOps on AWS – Deep Dive on Advanced Continuous Delivery Techniques using AWS DevOps Tools. Thanks Curtis for the great inspiration!

Use Case #4: Self-Healing

No testing in the world guarantees for smooth sailing in production. Therefore, we must think about how to automate the situation if something goes wrong in production.

As you hopefully know, Dynatrace AI automatically detects problems in production and detects the root cause based on all the data captured through the FullStack monitoring approach provided by OneAgent. Every time a problem is detected, Dynatrace can notify external tools (ServiceNow, PagerDuty, xMatters, …) or call a webhook. In my tutorial I implemented an AWS Lambda function accessible through an API Gateway that can directly be called from Dynatrace in case of a problem. The Lambda function pulls additional information from Dynatrace about the problem, e.g: what was the latest deployment and when did it happen. If the latest deployment happened in close timely proximity to the detected root cause the Lambda function calls AWS CodeDeploy to deploy the previous revision that was deployed into the Production environment. A classical Rollback scenario:

I understand that deploying a previous revision is not the ultimate “self-healing” process. Depending on the actual root cause and the environment we must take different actions to “heal” the system. Check out my recent blog on Applying AI into our Digital Performance Life where I walk through 3 real life problems detected and what we would need to do to self-heal the system through something like a Lambda function or any other type of remediation script.

Your feedback is welcome

I hope some of you find the time to walk through my tutorial. Please give me feedback and let’s work together to define what an Unbreakable Pipeline must look like. Let’s make it so easy that nobody finds any excuses any longer to not do it! 😊

The post Unbreakable DevOps Pipeline: Shift-Left, Shift-Right & Self-Healing appeared first on Dynatrace news.