In today’s fast-paced IT environments, speed matters. Organizations must ensure their digital infrastructure functions optimally and that they deliver software deployments and updates rapidly and consistently. But to meet demands and stay ahead of competitors, teams are under pressure to accelerate releases as much as possible. Verifying releases using systematic quality gates is a crucial practice to speed up the pace of deployments without sacrificing quality.

What are quality gates?

Quality gates are checkpoints that require deliverables to meet specific, measurable success criteria before progressing to the next development stage. They help foster confidence and consistency throughout the entire software development lifecycle (SDLC).

Organizations can customize quality gate criteria to validate technical service-level objectives (SLOs) and business goals, ensuring early detection and resolution of code deficiencies. Automating quality gates is ideal, as it minimizes manually checking and validating key metrics throughout the SDLC. Ultimately, quality gates safeguard code viability as it advances through the delivery pipeline.

Benefits of quality gates

Quality gates provide several advantages to organizations, including the following:

• Optimized software performance: Quality gates assess code at different SDLC stages and ensure that only high-quality code progresses. This mechanism significantly boosts the likelihood of optimal functioning upon deployment. By actively monitoring metrics such as error rate, success rate, and CPU load, quality gates instill confidence in teams during software releases.
• Fewer expensive fixes. Introducing well-defined quality gates at regular checkpoints prevents the escalation of issues as software advances through the pipeline. Early identification and resolution of potential deficiencies conserves resources and mitigates the need for expensive fixes later in the SDLC.
• Continuous, informed improvement: Quality gates provide consistent feedback on key metrics. This steady input empowers teams to proactively address issues, which fosters continuous learning from the outset of the SDLC. This approach supports innovation, ambitious SLOs, DevOps scalability, and competitiveness.

Quality gates examples in Dynatrace

Quality gates hold much promise for organizations looking to release better software faster. But how do they function in practice? The following are specific examples that demonstrate quality gates in action:

Security gates

Security gates ensure code meets key security requirements defined by development and security stakeholders. In this example, we will focus on ensuring releases do not have any known vulnerabilities. In this context, before the gorouter service deploys, security gates defined in Dynatrace’s Site Reliability Guardian (SRG) validate whether the latest version of the software has introduced a vulnerability.

In this scenario, the gorouter service must meet specific key metrics relating to the following three security areas:

• New vulnerabilities: The newly detected high-profile vulnerabilities in the environment. For each of these areas, the software receives a score from one to 10. Depending on the score thresholds for each SLO, the software will either “fail” or “pass” the gate.
• Open vulnerability on process group: The total number of currently high-profile vulnerabilities related to a process group.
• Open vulnerabilities: The total number of currently open high-profile vulnerabilities in the environment.
• Vulnerability score: The highest vulnerability risk score for a process group.

To view a more in-depth breakdown of these key metrics and their thresholds, simply scroll down to the bottom:

For every new build, the SRG assigns a score for each of these SLO areas. Considering the above scores, this version of software would not pass through the security gates, and the deployment process for the gorouter service would be stalled. If even one of the key metrics receives a failing score —as it did on the open vulnerability on process group — the software cannot progress further. Adjustments must be made accordingly.

Quality gates after load/performance testing

Teams can use quality gates to evaluate performance metrics. Consider “Easytravel,” a customer-facing application used by a travel agency. Before a new version of the application is deployed, the software is subject to a series of load tests that evaluate capacity and performance under a series of simulated traffic and application demands.

Quality gates consider these metrics and determine if their values fulfill specific SLO requirements. The gates thus allow the application version to pass through if the software meets the criteria and halts the progression if not. Several tools can be used to collect metrics in load/performance testing. In this example, test results are gathered from K6, Gremlin, NeoLoad, and Dynatrace synthetics.

Per the figures above, SRG evaluates Easytravel metrics from all of these tools in one place. This way, the travel agency can easily streamline, organize, and consolidate their quality gates and metric evaluation process. The agency can also efficiently compare the newest version of Easytravel against previous versions of the software with regression testing facilitated by SRG.

The new version of the Easytravel app has passed the load/performance testing quality gates. Although there were a handful of “warnings” for the K6 metrics — this means that the metrics are close to the “failure” threshold — none of the metrics failed to meet the SLO entirely. As a result, this version of Easytravel passes through this set of quality gates and is one step closer to deployment.

Quality gates to validate the “four golden signals”

The “four golden signals” represent the most crucial metrics of a customer-facing system’s performance. These metrics are latency, traffic, errors, and saturation, all of which must be key considerations when curating user experience. Teams can use quality gates to track and evaluate such metrics. Let’s review how a quality gate can function for each of the four golden signals using the same Easytravel application. Below is a sample SRG dashboard for these signals:

• Latency
  • Latency refers to the amount of time that data takes to transfer from one point to another within a system. In the context of Easytravel, one can measure the speed at which a specific page of the application responds after a user clicks on it.
    • The passing threshold is anything below 50 ms.
      The warning threshold is 50-60 ms.
      The failure threshold is anything above 60ms.
  • In this scenario, the response time for the latest software version is 44 ms, thus, it passes through the latency quality gate.

Teams take a similar approach for the other three signals. In this example, unlike latency, the remaining three signals did not receive a “pass.” SLOs for saturation were not met, and thus, this specific build of Easytravel temporarily halts. In terms of errors and traffic, these SLOs did not receive a failing score but are close enough to the failure threshold to trigger a warning.

What’s next

Quality gates are only as effective as the tools that enable them. Dynatrace’s Site Reliability Guardian provides a platform for in-depth analysis and validation of service availability, performance, and capacity objectives throughout your entire digital environment. Equipped with these capabilities, Site Reliability Guardian can help you implement high-performing quality gates.

The post What are quality gates? How to use quality gates to deliver better software at speed and scale appeared first on Dynatrace news.

Closed-loop remediation is an IT operations process that detects issues or incidents, takes corrective actions, and verifies that the remediation action was successful. Ideally, this all occurs without human intervention.

“Closed loop” refers to the continuous feedback loop in which the system takes actions — based on monitoring and analysis — and verifies the results to ensure complete problem remediation. Closed-loop remediation is vital to effective and reliable incident management because it ensures that organizations can consistently monitor any issue until it’s fully resolved.

How closed-loop remediation works

Closed-loop remediation uses a multi-step process that goes beyond simple problem remediation. This process identifies issues, notifies teams, and verifies solution efficacy continuously until it can confirm that the original incident is remediated.

Closed-loop remediation comprises the following stages:

• Stage 1: Triage
  • Observability: First, an observability platform detects an issue by analyzing all the telemetry data from across multicloud environments and determining incident severity.
  • Ownership: Next, the platform must determine the teams or individuals responsible for managing the incident and assign ownership to them.
  • Ticket creation: Using the ticketing tool of choice, the platform creates a ticket outlining the problem details and the incident’s impact.
  • Targeted notification: Responsible parties are automatically notified that an incident has occurred.
• Stage 2: Remediate
  • Root cause analysis: The observability platform should be able to pinpoint the incident’s root cause.
  • Automated response: Root cause identification triggers remediation workflows automatically based on the type of incident. Or, to take it one step further, the platform integrates with a tool such as Ansible to trigger remediation runbooks.
  • Deployment: Once the platform has determined the most viable solution, it triggers deployment into the environment. The goal is to either improve or restore the system to its optimally functioning state.
• Stage 3: Validate
  • Observability: After solution deployment, the observability platform scans the environment to determine its efficacy in resolving the issue. If the problem is fully remediated, the platform validates the solution to be effective and resolves the ticket.
  • Closing the loop (or not): In the event of a successful solution, the platform informs the appropriate teams and closes the loop. If the solution is unsuccessful, the platform notifies the relevant teams of the persisting issue. The cycle restarts and further escalation ensues until arrival at a successful solution.

Benefits of closed-loop remediation

Beyond significantly reducing mean time to detect and mean time to resolution (MTTR), closed-loop remediation offers significant benefits, including the following:

Improved incident backlogs. Incident backlogs collect data on and record all unresolved issues. Closed loop remediation compiles important holistic data into incident backlogs, enabling teams to prioritize, monitor, organize, and analyze issues at scale. These capabilities improve resource allocation and communication among teams, breaking silos.

Improved change failure and escalation rates. Change failure rate represents the proportion of system changes that have failed to meet their intended objectives. It is also a key metric for organizations looking to improve their DevOps performance. Through the automated verification within closed-loop remediation, teams can easily determine their own change failure rate. The verification process provides teams with information regarding which kinds of system modifications are most effective, improving change failure rate over the long run.

The same concept holds true for incident escalation rate. This metric represents the proportion of system incidents resolved by escalating to a higher level of support. By continuously monitoring and verifying closed-loop remediation, teams can expect to reduce both their change failure and incident escalation rates.

Greater developer productivity. Developers often spend excessive time finding and managing solutions to incidents that arise within production environments. This time could be better spent innovating; closed–loop remediation frees up developers’ time, resources, and energy. It also facilitates workflow optimization by automating responses to specific events that trigger predefined remediation actions. Since this automation largely removes manual intervention, developers have greater bandwidth for innovation.

Auto-remediation examples

The following examples illustrate how closed-loop remediation works and its benefits.

Example #1: Stabilizing streaming load times

Consider a movie streaming application that experiences performance degradation, resulting in slow load times. The observability platform detects the anomaly and determines the root cause of the problem: increased traffic during peak usage hours, resulting in a server overload. The platform automatically creates an incident ticket that details the severity and potential impact of the issue and sends the ticket to the relevant teams. The system automatically triggers a remediation workflow that provisions additional resources to meet increased capacity requirements. After the scaling action, the platform validates the action’s efficacy by analyzing performance metrics to ensure that load times are back to normal. If successful, the system closes the loop and notifies teams. If the issue persists, it notifies the responsible teams who may need to do additional triaging to address the incident. Either way, the system updates the ticket to reflect the status of the issue.

Example #2: Prioritizing and remediating application vulnerabilities

An organization has an application that accesses confidential customer information. The organization’s observability and security platform identifies several vulnerabilities that have the potential to allow bad actors to access and leak proprietary information. However, only some of these vulnerabilities are exploitable; others are of lesser importance. The organization’s observability and security platform triages the numerous vulnerabilities and determines which ones to prioritize based on runtime context, such as adjacency to the internet or data sources. The system automatically generates tickets for critical vulnerabilities with all relevant information and assigns them to the responsible team for remediation.

Once fixes have been implemented and deployed, the platform scans for vulnerabilities once more and determines whether the vulnerabilities are still present. If the relevant vulnerabilities have been eliminated, then the fixes are validated: the system closes the loop and updates the ticket. If the platform determines that the vulnerabilities have not been eliminated, it updates the tickets to initiate further investigation. The process repeats it implements until an effective solution.

Example #3: Kafka disk resizing

A banking company has a Kafka event streaming broker whose disk usage is nearing critical levels, threatening potential disruptions or service degradation. The observability platform detects this issue as disk usage meets a predefined threshold and determines the root cause to be low disk space. Upon detection, the platform triggers an automated alert. The system executes an automated remediation workflow that dynamically adjusts the disk size by adding space or redistributing partitions. On completing these actions, the observability platform confirms that the remediation effort was successful, closes the loop, and notifies teams. If not, the platform restarts the process so teams can explore other remediation options.

To complete the cycle, platforms must be strong enough to accurately verify that remediation actions are indeed fulfilling their intended purposes. It is best practice to trigger actions to notification tools that indicate the success or failure of the remediation action.

Implementing closed-loop remediation with unified observability

At the foundation of any closed-loop remediation practice is a unified approach to observability that can fully monitor end-to-end DevOps processes. Moreover, an observability platform that harnesses AI for root-cause analysis will greatly expedite the entire remediation process, as it can quickly and intelligently determine the source of an issue without human intervention. Likewise, the observability platform can then notify the right teams and trigger automated workflows to take the appropriate remediation action.

Closed-loop remediation reduces MTTR and helps ensure a smooth, reliable customer experience. Dynatrace helps organizations realize these benefits through its unified platform that drives and optimizes auto-remediation. AutomationEngine enables DevOps teams to automate custom remediation workflows facilitated by continuous, real-time insights and precise answers. To validate the efficacy of such workflows on DevOps SLOs, Site Reliability Guardian (SRG) provides automated change impact analysis for new software versions. As such, SRG plays a crucial role in “closing the loop” of remediation workflows. SRG offers specific validation results against custom-defined SLOs for software updates across various systems. The result: teams can make more informed release decisions.

Equipped with these tools, along with unified observability, AI-driven analysis, and automation technology, Dynatrace offers all the elements necessary to optimize your closed-loop remediation practice.

The post Why unified observability is an essential auto-remediation best practice appeared first on Dynatrace news.

Testing for mishaps you can predict is essential. But with the complexity that comes with digital transformation and cloud-native architecture, teams need a way to make sure applications can withstand the “chaos” of production. Chaos engineering answers this need so organizations can deliver robust, resilient cloud-native applications that can stand up under any conditions.

What is chaos engineering?

Chaos engineering is a method of testing distributed software that deliberately introduces failure and faulty scenarios to verify its resilience in the face of random disruptions. These disruptions can cause applications to respond unpredictably and break under pressure. Chaos engineers ask why.

Practitioners subject software to a controlled, simulated crisis to test for unstable behavior. The crisis could be technical, natural, or malicious events, for example, an earthquake affecting data center availability, or a cyberattack infecting applications and websites. As software performance degrades or fails, the chaos engineers’ findings enable developers to add resiliency to the code, so the application remains intact in an emergency.

As chaos engineers grow confident in their testing, they change more variables and broaden the scope of the disaster. The many disaster scenarios and outcomes allow chaos engineers to better model what happens to applications and microservices, which gives them increasing intelligence to share with developers to perfect software and cloud-native infrastructure.

The history of chaos engineering

Netflix pioneered chaos engineering out of necessity. In 2009, the purveyor of online videos migrated to AWS cloud infrastructure to deliver its entertainment to a growing audience. But the cloud brought new complexities, such as increasing connections and dependencies. It created more uncertainty than the load-balancing issues the entertainment firm saw in its data centers. If any touchpoint in the cloud failed, the quality of the viewers’ experience could degrade. So, the organization sought to reduce complexity and raise production quality.

In 2010, Netflix introduced a technology to switch production software instances off randomly—like setting a monkey loose in a server room—to test how the cloud handled its services. Thus, the tool Chaos Monkey was born.

Chaos engineering matured at organizations such as Netflix, and gave rise to technologies such as Gremlin (2016), becoming more targeted and knowledge-based. The science has spawned specialized chaos engineers who dedicate themselves to disrupting cloud software and the on-prem systems they interact with to make them resilient. Now, chaos engineering is an established profession, stirring up managed trouble to stabilize cloud software.

How does chaos engineering work?

Chaos engineering starts with understanding the software’s expected behavior.

1. Hypothesis. Engineers ask themselves what should happen if they change a variable. If they terminate services randomly, they assume the service will continue uninterrupted. The question and the assumption form a hypothesis.
2. Testing. To test the hypothesis, chaos engineers orchestrate simulated uncertainty combined with load testing and watch for signs of upheaval in the services, infrastructure, networks, and devices that deliver the application. Any failures in the stack break the hypothesis.
3. Blast radius. By isolating and studying failures, engineers can understand what happens under unstable cloud conditions. Any damage or influence caused by the test is known as the ‘blast radius.’ Chaos engineers can manage the blast radius by controlling the tests.
4. Insights. The discoveries form inputs into the software development and delivery process, so new software and microservices will better stand up to unforeseeable events.

To mitigate damage to production environments, chaos engineers start in a non-production environment, then slowly extend to production in a controlled way. Once established, chaos engineering becomes an effective way to fine-tune service-level indicators and objectives, improve alerting, and build more efficient dashboards, so you know you are collecting all the data you need to accurately observe and analyze your environment.

Who uses chaos engineering?

Chaos engineering generally originates from small teams within DevOps, often involving applications running in both pre-production and production environments. Because it can touch many systems, chaos engineering can have broad implications, affecting groups and stakeholders across the organization.

A disruption spanning hardware, networks, and cloud infrastructure can require input and participation from network and infrastructure architects, risk experts, security teams, and even procurement officers. That’s a good thing. The greater the scope of the test, the more useful chaos engineering becomes.

Although a small team generally owns and manages the chaos engineering effort, it’s a practice that often requires input from—and provides benefits to—the village.

The benefits of chaos testing

The insights you can gain by testing the limits of your applications deliver a lot of benefits for your development teams and your overall business. Here are just a few benefits of a healthy, well-managed chaos engineering practice.

• Increases resiliency and reliability. Chaos testing enriches the organization’s intelligence about how software performs under stress and how to make it more resilient.
• Accelerates innovation. Intelligence from chaos testing funnels back to developers who can implement design changes that make software more durable and improve production quality.
• Advances collaboration. Developers aren’t the only group to see advantages. The insights chaos engineers glean from their experiments elevate the expertise of the technical group, leading to response times and better collaboration.
• Speeds incident response. By learning what failure scenarios are possible, these teams can speed up troubleshooting, repairs, and incident response.
• Improves customer satisfaction. Increased resilience and faster response times mean less downtime. Greater innovation and collaboration from development and SRE teams means better software that meets new customer demands quickly with efficiency and high performance.
• Boosts business outcomes. Chaos testing can extend an organization’s competitive advantage through faster time-to-value, saving time, money, and resources, and producing a better bottom line.

The more resilient an organization’s software is, the more consumers and business customers can enjoy its services without distraction or disappointment.

The challenges and pitfalls of chaos engineering

Although the benefits of chaos testing are clear, it is a practice that should be undertaken with deliberation. Here are the top concerns and challenges.

• Unnecessary damage. The major concern with chaos testing is the potential for unnecessary damage. Chaos engineering can lead to a real-world loss that exceeds the allowances of justifiable testing. To limit the cost of uncovering application vulnerabilities, organizations should avoid tests that overrun the designated blast radius. The goal is to control the blast radius so you can pinpoint the cause of failure without unnecessarily introducing new points of failure.
• Lack of observability. Establishing that control can be easier said than done. Lack of end-to-end observability and monitoring of all systems a blast radius might affect is a common problem. Without comprehensive observability, it can be difficult to understand critical dependencies vs non-critical dependencies, or to have adequate context to understand the true business impact of a failure or degradation in order to prioritize fixes. Lack of visibility can also make it difficult for teams to determine the exact root cause of an issue, which can complicate remediation plans.
• Unclear starting system state. Another issue is having a clear picture of the starting state of the system before the test is run. Without this clarity, teams can have difficulty understanding the true effects of the test. This can diminish the effectiveness of chaos testing and can put downstream systems at greater risk, which makes it harder to control the blast radius.

How to get started with chaos engineering

As with any scientific experiment, getting started with chaos engineering requires a little preparation, organization, and the ability to monitor and measure results.

1. Know the starting state of your environment. To plan a well-controlled chaos test, you should understand the applications, microservices, and architectural design of your environment so you can recognize the effects of the test. Having a baseline you can compare to the end state creates a blueprint for monitoring during the testing and analyzing results after.
2. Ask what could go wrong and establish a hypothesis. With a clear idea of the system’s starting state, ask what could go wrong. Understand service-level indicators and service-level objectives and use them as a basis for establishing an assumption for how the system should work under pressure.
3. Introduce chaos one variable at a time. To keep control of the blast radius, introduce only a little chaos at a time, so you can appreciate the results. Be prepared to abort the experiment under certain conditions, so no harm comes to production software, and also have a roll-back plan if something goes wrong. During the test, try to disprove the hypothesis to discover areas to focus on to improve system resilience.
4. Monitor and record the results. Monitor the experiment to record any nuances in application behavior. Analyze the results to see how the application responds and whether testing achieved the teams’ expectations. Use investigation tools to understand the precise root causes of slow-downs and failures.

Controlling the chaos

Solutions like Gremlin provide crucial management tools to plan and execute chaos engineering experiments. It makes experiments repeatable and scalable so teams can apply them to future experiments of the same or larger stacks.

Automatic and intelligent observability from Dynatrace delivers insights into the effects of chaos testing, so engineers can steer chaos experiments with care. To monitor the blast radius, Dynatrace observes the systems undergoing chaos experiments. With visibility across the full software stack, Dynatrace provides crucial contextual analysis to isolate the root cause of failures exposed by chaos testing.

Effective monitoring from Dynatrace offers an essential panoramic lens for the engineers driving chaos testing, helping them understand dependencies and predict how outages will affect the system at large. Should the chaos reach further than intended, insights from Dynatrace help teams quickly remediate any actual harm to the application’s functionality.

Organizations can achieve application resiliency in any stage of digital transformation, and chaos engineering is a great tool. However, before playing with fire, it’s critical to have the right measures in place to predict and cope with the multitude of failure scenarios this approach can bring.

The post What is chaos engineering? appeared first on Dynatrace news.

In the world of DevOps and SRE, DevOps automation answers the undeniable need for efficiency and scalability. Though the industry champions observability as a vital component, it’s become clear that teams need more than data on dashboards to overcome persistent DevOps challenges. DevOps teams need automation that goes beyond passive observations. They need event-driven automation that not only responds to events and triggers but also analyzes and interprets the context to deliver precise and proactive actions. This evolution in automation, referred to as answer-driven automation, empowers teams to address complex issues in real time, optimize workflows, and enhance overall operational efficiency.

In this blog, we will dive into the transformative power of answer-driven automation. We will also explore the evolution of DevOps automation and the significance of data-driven answers in unlocking streamlined, automated DevOps and SRE processes.

The evolution of DevOps automation

Since the concept of DevOps emerged around 2007 and 2008 in response to pain points with Agile development, DevOps automation has been continuously evolving.

Business process automation

Business process automation is the foundation for improving operational efficiency. From automating email communications and backup scheduling to streamlining invoicing and billing processes, organizations have embraced automation to save time and reduce manual effort. These initial automation endeavors paved the way for greater advancements, leading to the next evolution of event-driven automation.

Event-driven automation

Event-driven automation represents a significant leap forward. It harnesses real-time responsiveness to specific rules or events, revolutionizing business operations.

For example, consider an e-commerce website that automatically sends personalized discount codes to customers who abandon their shopping carts. This event-driven automation triggers the action of sending the discount code only when the customer abandons the cart, minimizing revenue loss and increasing conversion rates.

In the realm of cybersecurity, event-driven automation plays a crucial role in threat detection and response. A team can configure its observability platform to detect specific events, such as malicious user requests that could be part of a SQL injection attack or suspicious login attempts. When the platform identifies these events, event-driven automation can automatically trigger actions like blocking IP addresses, initiating incident response protocols, and sending alerts to security or development teams. This proactive approach strengthens the organization’s defense against cyber threats and reduces the time to detect and mitigate potential breaches.

But even the event-driven approach leaves gaps that greater detail and context could fill in. With greater detail and context, organizations can make—and automate—more nuanced decisions with greater impact to operations, security, and the business.

Answer-driven automation

With the advent of AI in operations (AIOps), DevOps teams have an opportunity to apply a more intelligent and comprehensive approach to automation driven by AI. Combining precise, causal AI-powered answers —derived from observability, security, and business data— with predictive AI and automation provides a feedback loop that makes the automation more intelligent and business-value-oriented. This is what answer-driven automation is all about.

Let’s examine a few examples of answer-driven automation using AI and context.

DevOps automation example #1: Online shopping

It has been more than 25 years since Bill Gates coined the phrase “context is king,” yet its relevance persists. Consider the aforementioned scenario of cart abandonment, where the user receives an automated email with a discount code. While this initial step may seem promising, crucial pieces of information (the context) surrounding the transaction remain missing. The absence of these details hampers decision-making. Questions arise, such as the following:

• How substantial is the shopping cart?
• How popular is the product?
• How long has it been lingering in the cart?
• Are there any existing discounts on the product?
• What is the current inventory level?
• Has the user purchased this product before?

Imagine having an AI engine that comprehends the complete context of the transaction and intelligently determines whether to send a discount code—and which one to send. This would propel event-driven automation to new heights. Full contextual awareness helps the AI engine make informed decisions. This revolutionizes the online shopping experience and leads to better conversions and enhanced user satisfaction.

DevOps automation example #2: Threat detection and response

Let’s build on the previous example. While resorting to IP address blocking as a defense measure may seem tempting, it could potentially be an excessive reaction, especially when the true intent behind the event remains unknown. This is where answer-driven automation comes into play. The AI engine becomes your trusted ally, observing the user request behavior within the application. It goes beyond surface-level observations, diving deep into the intricate details to determine outcomes, including the following:

• Is there an exploitable vulnerability?
• Does it constitute a zero-day vulnerability?
• How critical is the vulnerability?
• How adjacent is the vulnerability to the data source?
• Was the user input indeed malicious?

Armed with these precise answers, you gain a crystal-clear understanding of the situation.

The real power lies in the subsequent actions that answer-driven automation triggers. Answer-driven DevOps automation goes beyond creating tickets and extends to executing workflows designed to extract ownership information and then route the ticket to the responsible teams. This seamless handoff ensures that the right team diligently addresses the root cause of the problem, allowing teams to efficiently resolve them. Answer-driven automation mitigates alert storms and significantly reduces the number of false positives. As a result, it saves critical developer time and expedites the mean time to resolution (MTTR). Combining answer-driven automation and comprehensive threat detection and response provides a robust defense mechanism against potential threats. The combination empowers organizations to identify the true nature of events swiftly, take well-informed actions, and proactively address vulnerabilities.

DevOps automation example #3: Progressive delivery

In software development and delivery, if an organization uses feature flags to control feature releases, the marriage of observability data and answer-driven automation becomes a formidable force. Observability data provides a treasure trove of performance, stability, and user experience metrics encompassing error rates, response times, and user engagement. This rich dataset is the foundation for an answer-driven automation solution to work its magic using AI.

With swift precision, an answer-driven automation solution that uses causal AI can transform these metrics into invaluable insights. Such a solution can automatically analyze the impact of newly introduced features or changes facilitated through feature flags. For instance, in the case of poor performance, you can seamlessly toggle a feature flag and mitigate any detrimental effects. Conversely, if the feedback is overwhelmingly positive, you can dynamically expand the feature’s exposure to a wider group of users and unleash its full potential. In this way, answer-driven automation orchestrates the seamless delivery of new features to users while effectively minimizing the impact of potential downtime or negative user experiences.

This powerful synergy between observability data, causal AI insights, and answer-driven automation empowers organizations to embrace progressive delivery with unwavering confidence. It enables them to adapt to user feedback swiftly, fine-tune feature releases, and deliver exceptional user experiences, all while maintaining control and minimizing disruption.

DevOps automation example #4: Intelligent incident management and remediation

We’ve witnessed the power of event-driven automation in generating alerts, creating incident tickets, and executing runbooks to drive remediation efforts. While this level of automation is undeniably beneficial, today’s IT leaders aspire for even greater capabilities. Here, answer-driven automation enables organizations to harness the potential of answers from observability data to achieve self-healing, proactive management.

Consider an event-driven automation system designed for incident management. When a server experiences an outage, the system promptly triggers an alert and initiates actions like restarting a server or redirecting traffic to a redundant server. While this reactive approach effectively restores services, it does not address the underlying cause, which results in recurring incidents.

In contrast, answer-driven automation takes incident management to a new level. Using advanced causal AI and context-aware decision-making, it identifies the root cause behind server failures. The system analyzes performance patterns and historical data to uncover the specific software bug or resource constraint causing the issue.

Thus, in addition to generating alerts and incident tickets, the system can trigger predefined workflows or specific runbooks to deploy software patches and automatically scale up or down resources when necessary. But it doesn’t stop there. Based on all the contextual information, the system can provide predictive answers to auto-scale infrastructure and anticipate future capacity demands, effectively reducing cost, carbon footprint, and downtime. All these actions aim to avert future incidents. And finally, with an observability platform, you can effectively “close the loop” and ensure that any remediation actions fix the issue by continually observing and evaluating SLOs and business objectives. Answer-driven automation holds transformative power within the realm of remediation.

DevOps automation example #5: Safeguarding production

Imagine having the ability to instantly validate the impact of any change to a production application and receive real-time feedback. Change impact analysis is an indispensable process for effectively managing changes within an organization’s infrastructure and applications. It entails evaluating the potential consequences and risks associated with both planned and unplanned changes, ensuring smooth and successful operations.

Let’s consider a scenario where a software development team oversees the deployment of updates to a microservices-based application. Suppose a development team is working on a new feature for a specific service. They intend to roll it out to the production environment. However, they don’t have a comprehensive understanding of the intricate dependencies between the services or the potential impact of the change on the overall system.

To mitigate this uncertainty, the developers can harness causal AI to understand how the change will affect other services. They can also see how the change can affect critical objectives like SLOs and golden signals, such as traffic, latency, saturation, and error rate. This automated validation equips developers with the necessary knowledge to make informed decisions about the deployment, or even prompt them to consider a rollback if necessary. Precise answers derived from causal AI —coupled with answer-driven automation— eliminate the need for guesswork and result in a more reliable, resilient production system.

Answer-driven automation: The new frontier in DevOps automation

The use cases for answer-driven automation are as varied as the DevOps environments they can help to automate. Observability, causal AI, and predictive AI form the basis for enabling answer-driven automation. The Dynatrace platform’s unique approach to delivering precise answers —through root cause analysis from the Davis AI Engine— and intelligent automation can help your organization scale to new heights. By uniting broad, deep observability with precise security analytics capabilities in a single platform to automate DevOps, teams can deliver better, more secure software faster.

Now with AutomationEngine and Workflows, Dynatrace empowers organizations to build, customize, and deploy workflows to help teams realize the full potential of answer-driven automation.

The post DevOps automation: From event-driven automation to answer-driven automation [with causal AI] appeared first on Dynatrace news.

In today’s fast-paced digital landscape, ensuring high-quality software is crucial for organizations to thrive. Service level objectives (SLOs) provide a powerful framework for measuring and maintaining software performance, reliability, and user satisfaction. Certain service-level objective examples can help organizations get started on measuring and delivering metrics that matter. Teams can build on these SLO examples to improve application performance and reliability.

Measuring application performance is increasingly important because as organizations digitally transform, they’re also accelerating the speed of software delivery. According to the 2023 Dynatrace CIO Report, 78% of respondents deploy software updates every 12 hours or less. 54% deploy updates every two hours or less, and 20% deploy updates every minute or less. But the pressure on CIOs to innovate faster comes at a cost. More than half of CIOs confirmed that they often make tradeoffs among code quality, security, and reliability to meet the need for rapid software delivery.

SLOs are a valuable tool for organizations to ensure the health and performance of their applications. Serving as agreed-upon targets to meet service-level agreements (SLAs), SLOs can help organizations avoid downtime, improve software quality, and promote automation in the DevOps lifecycle. In this post, I’ll lay out five SLO examples that every DevOps and SRE team should consider.

Foundational service level objective examples for every industry

Organizations in every industry need their services and applications to perform responsively, safely, and reliably so they can meet customer expectations. To illustrate how our five SLO examples apply to different applications, we will explore the following two use cases:

• E-commerce websites: Whether you use Amazon, Walmart, BestBuy, or any other websites to buy and sell goods, we all expect a seamless shopping experience. E-commerce websites are a great way to explain how IT teams can apply SLOs to any use case or application. These organizations rely heavily on performance, availability, and user satisfaction to drive sales and retain customers.
• Fitness apps: During the pandemic, fitness apps boomed. While gyms and fitness studios have since reopened, these apps are still extremely popular for squeezing in short home workouts, tracking fitness goals, and receiving personalized training recommendations. Similarly, SLOs play a crucial role in ensuring a flawless, engaging user experiences.

The goal for DevOps and SRE teams is to collaborate around a common set of objectives to improve software quality and reliability. The ultimate result is that customers enjoy an exceptional user experience. Let’s dig into these foundational SLO examples.

Five SLO examples for faster, more reliable apps

Once you get started with these service-level objective examples, you can branch out to develop more targeted SLOs suited for your business.

1. Availability

Availability SLO quantifies the expected level of service availability over a specific time period. It represents the percentage of time a system or service is expected to be accessible and functioning correctly. Availability is typically expressed in 9’s, such as 99.9%. It helps organizations set clear expectations for uptime and establish a baseline for performance and reliability. Meeting or exceeding availability SLOs is crucial for providing uninterrupted services, minimizing downtime, and maintaining customer satisfaction.

E-commerce website: The e-commerce website should be available 99.9% or 99.99% of the time. This SLO example provides a standard of availability that allows customers to browse and purchase products without interruptions. Any scheduled maintenance or downtime should occur during off-peak hours to minimize the impact on users.

Fitness app: Likewise, the fitness app’s online class streaming feature should also deliver 99.9% or 99.99% availability. The availability SLO ensures that the online classes offered through the fitness app’s streaming service are accessible with minimal downtime. It aims to provide a reliable platform for users to participate in live or pre-recorded workout sessions, virtual training, or fitness tutorials without interruptions.

2. Response time

Response time refers to the total time it takes for a system to process a request or complete an operation. A low response time improves user engagement and encourages users to continue staying on the app/ website. For the response time SLO example, find the difference between the time the request was made and the time the response was received. This SLO is crucial for providing a frictionless, satisfying user experience.

E-commerce website: The e-commerce website should load within one second for each page request. This ensures that customers can quickly navigate through product listings, add items to their cart, and complete the checkout process without experiencing noticeable delays.

Fitness app: The fitness app should offer a response time of less than 500 milliseconds for exercise tracking and data recording. With this objective, the app ensures that users experience real-time feedback and immediate updates when logging workouts, recording sets and reps, or tracking performance metrics. This SLO enables a smooth and uninterrupted exercise-tracking experience.

Note: you might hear the term latency used instead of response time. Both latency and response time are critical to ensure reliability. Latency typically refers to the time it takes for a single request to travel from its source to its destination. Latency primarily focuses on the time spent in transit. Response time encompasses both the latency (transit time) and the time required for the system to process the request internally.

3. User satisfaction (ApDex

User satisfaction is a standard way of measuring user satisfaction with the performance of an application. It is also often measured using the Application Performance Index (ApDex). To calculate the user satisfaction service level objective example, identify the percentage of requests that meet a certain threshold for response time. For example, a user might expect a response time of one second or less. Thus, an ApDex score of 0.85 means that 85% of requests met that threshold.

E-commerce website: For an e-commerce website, maintain an ApDex score of 0.9 or above for the checkout process. This SLO example highlights the importance of a smooth and efficient checkout experience. It ensures that most users find the checkout process to be excellent or at least tolerable. It encompasses factors such as page loading speed, responsiveness, and overall ease of use during the checkout process, optimizing user satisfaction and minimizing cart abandonment.

Fitness app: For the fitness app, maintain an Apdex score of at least 0.85 for the workout video playback feature. The ApDex SLO ensures that users have a positive experience when watching workout videos within the fitness app. The Apdex score of 0.85 and above indicates that most users find the video playback performance excellent or at least acceptable.

4. Traffic

The traffic SLO example measures the amount of traffic or workload an application receives, either in terms of requests per second or data transfer rate. It ensures that the application can handle the expected workload without slowing down or crashing.

E-commerce website: The e-commerce site should support a peak transaction rate of 12,000 orders per minute during flash sales or limited-time promotions. The traffic SLO targets the website’s ability to handle a high volume of transactional activity during periods of high demand. By setting an SLO for 12,000 orders per minute, the website ensures that its checkout system, inventory management, and payment processing can handle the increased load efficiently and without significant delays or service interruptions.

Fitness app: The fitness app should support a concurrent user count of 10,000 during virtual fitness events or live-streaming sessions. The traffic SLO targets the app’s ability to handle many concurrent users during virtual fitness events, live-streaming workouts, or interactive sessions. By setting an SLO for 10,000 concurrent users, the app ensures that its infrastructure can handle the increased traffic and deliver a smooth and uninterrupted experience for users participating in virtual fitness events.

5. Success

Of all the service-level objective examples, success ties most closely to customer satisfaction. Success rate measures the percentage of successful operations or transactions that meet specific criteria. It sets targets for expected success rates in services or applications. Success rate SLOs ensure that critical operations are reliable and perform as expected. By defining success rate SLOs, organizations can monitor and improve the success rate of important processes to enhance user experience and overall service quality.

E-commerce website: For e-commerce, sites should maintain a success rate of 99.5% for product search requests. This means that 99.5% of search queries made by users should yield accurate and relevant results. By monitoring the success rate against this SLO, the website can ensure that its search functionality is performing optimally and delivering satisfactory results to users. It allows the website to identify and address any issues that may impact the accuracy and reliability of search results, enhancing the overall user experience and increasing the likelihood of successful product searches.

Fitness app: The fitness app should maintain a success rate of 98% for workout video playback. This means that 98% of video playback attempts should be successful, without buffering issues or playback errors. By monitoring the success rate, the fitness app can ensure that users can seamlessly access and view workout videos, providing a smooth and uninterrupted fitness experience.

Best practices to help ensure the performance and availability of your applications

In addition to leveraging these service level objective examples, consider the following best practices:

• Perform synthetic tests: Synthetic testing is a practice that applies automated simulations of user interactions with a website to help measure the website’s performance under controlled conditions. Such testing makes it easier to detect any potential issues before they impact real users. Synthetic tests can monitor performance and availability and can be run from multiple locations to simulate user traffic from different regions. Teams should conduct synthetic tests regularly, simulating common user actions such as browsing product categories, performing searches, and making purchases. In addition, teams can schedule these tests hourly or daily to ensure that critical transactions are functioning properly and meeting performance expectations.
• Maintain optimal resource utilization: Ensuring efficient usage of CPU, memory, storage, and network bandwidth is of utmost importance for any website or application. By closely monitoring and tracking resource utilization, organizations can right-size their infrastructure and optimize infrastructure spend to handle both off-peak and peak usage scenarios. This not only leads to cost savings but also guarantees a consistently smooth user experience.

Getting started with service-level objective examples using Dynatrace

Since applications in every industry need to meet the same fundamental goals for speed and reliability, creating the SLOs to meet those basic goals should be easy. In addition to these service-level objective examples, Dynatrace can jumpstart your SLO journey. To help you get started with SLO examples and beyond, Dynatrace provides pre-configured templates for SLOs out of the box that support the Site Reliability Engineering (SRE) fundamentals published by Google. You can also define your own SLOs to meet any specialized performance need or metric.

The post Service level objective examples: 5 SLO examples for faster, more reliable apps appeared first on Dynatrace news.

Keeping pace with modern digital transformation requires ensuring that applications are responsive, resilient, and always available amid increased complexity. As a result, site reliability has emerged as a critical success metric for many organizations.

Site reliability engineering (SRE) has recently become a critical discipline in recent years as the world has shifted in favor of web-based interactions. Mobile retail e-commerce spending in the U. S. surpassed $387 billion in 2022, more than double the figure of three years earlier. The volume of travel spending booked online is expected to reach nearly $1.5 trillion by 2027, up from $800 billion in 2021. With so many of their transactions occurring online, customers are becoming more demanding, expecting websites and applications to always perform perfectly. One recent report found that 32% would leave their favorite brand after just one bad experience. Website load times have been found to have a direct correlation with conversion rates.

This shift is leading more organizations to hire site reliability engineers to guarantee the reliability and resiliency of their services. But the transition to SRE maturity is not always easy.

How site reliability engineering affects organizations’ bottom line

SRE applies the disciplines of software engineering to infrastructure management, both on-premises and in the cloud. The practice uses continuous monitoring and high levels of automation in close collaboration with agile development teams to ensure applications are highly available and perform without friction.

According to the emerging trends from the global shift towards web-based interactions, IT infrastructure performance has a dramatic impact on the organizations’ bottom-line business goals. Uptime Institute’s 2022 Outage Analysis report found that over 60% of system outages resulted in at least $100,000 in total losses, up from 39% in 2019. More than one in seven outages cost more than $1 million.

Maintaining reliable uptime and consistent service quality has become more complex as organizations expand their computing footprints across multiple data centers and in the cloud. Microservices-based architectures and software containers enable organizations to deploy and modify applications with unprecedented speed. However, cloud complexity has made software delivery challenging. There are now many more applications, tools, and infrastructure variables that impact an application’s performance and availability.

Understanding the interactions between these factors heavily influences decisions about whether and when to promote a new release into production. That’s why good communication between SREs and DevOps teams is important. By automating and accelerating the service-level objective (SLO) validation process and quickly reacting to regressions in service-level indicators (SLIs), SREs can speed up software delivery and innovation.

Understanding the goal of “five-nines” availability

The guiding principle of SRE has long been “five-nines” availability, meaning systems are operative 99.999% of the time. As organizations distribute workloads among a greater number of cloud environments, that goal has become harder to attain because more variables are involved in the computing equation. The growing amount of data processed at the network edge, where failures are more difficult to prevent, magnifies complexity.

Visibility and automation are two of the most important SRE tools. The Dynatrace 2022 Global CIO Report found that 71% of top IT executives say the explosion of data produced by cloud-native technology stacks is beyond human ability to manage, and more than three-quarters say their IT environment changes once every minute or less. The takeaway is clear: IT environments are now too complex to manage without automation and AI. Without these capabilities, achieving five-nines availability will become close to impossible.

Aligning site reliability goals with business objectives

Because of this, SRE best practices align objectives with business outcomes. The following three metrics are commonly used to measure success:

• Service-level agreements (SLAs). These metrics are the product of an agreement between the service provider and customer that certain measurable levels of service will be delivered.
• Service-level objectives (SLOs). These metrics are the factors and service levels that must be achieved for each activity, function, and process to deliver on the SLA. These can include business metrics, such as conversion rates, as well as technical measures like underlying CPU availability. They’re typically expressed as percentages, such as 99.5% availability.
• Service-level indicators (SLIs). At the lowest level, SLIs provide a view of service availability, latency, performance, and capacity across systems.

5 SRE best practices

Let’s break down SRE best practices into the following five major steps:

1. Start looking for signals

Begin by monitoring the “four golden signals” that were originally outlined in Google’s SRE handbook:

• Latency: the time it takes to serve a request
• Traffic: the total number of requests across the network
• Errors: the number of requests that fail
• Saturation: the load on the network and servers

2. Identify KPIs

Next, create a list of the key performance indicators (KPIs) that are important to the business. These may include technical metrics such as response times to search query referrals, page load times, and error message frequency. They may also include business metrics influenced by performance, such as shopping cart abandonment and page views per customer.

3. Establish SLOs

Drawing on the KPIs, you can now create a list of SLOs. Remember that less is more. SLOs should directly relate to the SLA or business objective. Defining too many SLOs creates more work without a clear business impact.

Make SLOs realistic. If they’re intentionally set low to avoid SLA violations, they won’t provide an accurate picture of how systems are impacting user experience. If set too high, they can drive higher costs and effort for little incremental gain.

4. Identify key stakeholders

Once you’ve ensured SLAs and SLOs are realistic, begin recruiting stakeholders. For example, a panel of customers may occasionally provide feedback on service quality and performance. Business stakeholders can understand how SLOs relate to business results using historical data trends. Everyone needs to agree on SLO targets; otherwise, the organization will risk failing to deliver on its SLAs.

5. Automate workflows wherever possible

Automation is critical to achieving the business agility that digital transformation demands. A powerful observability solution collects relevant SLIs and evaluates SLOs automatically. It can also automatically generate alerts before SLO violation and even automatically repair many problems. Automation also enables tools to move into developers’ hands so they can make decisions about deploying code without needing to involve operations teams.

Automating SRE best practices with Site Reliability Guardian

As part of the early 2023 rollout of its new AppEngine low-code toolset for creating custom, compliant, and intelligent data-driven applications, Dynatrace introduced Site Reliability Guardian (SRG) for automated change impact analysis

AppEngine is a unique solution that consolidates observability, security, and business data with full context and dependency mapping to simplify the creation of intelligent applications and integrations. It enables teams, for the first time, to leverage causal AI for customized applications that address specific use case requirements.

SRG enriches the Dynatrace platform’s value for SREs by automating change impact analysis. It detects regressions and deviations from previously observed behavior across metrics such as latency, traffic, error rates, saturation, security coverage, vulnerability risk levels, and memory consumption.

SRG also enriches the Dynatrace platform’s value to DevOps teams who can automate release validation in pre-production environments to ensure only high-quality, highly secure software moves to production. Should the SRG detect any violations of key objectives or metrics, the CI/CD pipeline tool can halt the delivery of the artifact.

For each critical service or application, SRG can automatically monitor for golden signals, validate SLOs, and probe for security vulnerabilities before and after deployment or configuration changes. It can also send targeted notifications to the service and application owners as needed. The result is safer, more secure releases for DevOps teams and less overhead for SREs.

Keeping the focus on site reliability

SRE is a critical discipline for organizations to master as they navigate the transition to digital business powered by data-driven decisions. Tight integration with business objectives and automation now makes it possible for organizations to proactively monitor their digital presence to ensure the highest levels of availability, responsiveness, and customer experience.

The post Site reliability done right: 5 SRE best practices that deliver on business objectives appeared first on Dynatrace news.

With the world’s increased reliance on digital services and the organizational pressure on IT teams to innovate faster, the need for DevOps monitoring tools has grown exponentially.

In fact, the Dynatrace 2023 CIO Report found that 78% of respondents deploy software updates every 12 hours or less. 54% reported deploying updates every two hours or less.

This demand for rapid innovation is propelling organizations to adopt agile methodologies and DevOps principles to deliver software more efficiently and securely. The DevOps approach breaks up projects into modular components that development teams build in parallel by working closely with operations and business stakeholders. DevOps aims to move applications more quickly into production by releasing new builds frequently — often several times per day — and incorporating constant feedback and testing, both during development and after software is deployed.

But when and how does DevOps monitoring fit into the process? And how do DevOps monitoring tools help teams achieve DevOps efficiency?

What is DevOps monitoring?

DevOps monitoring is an observability practice that creates a real-time view of the status of applications, services, and infrastructure in pre-production and production environments.

DevOps monitoring tools enable teams to collect, analyze, and visualize data to ensure the smooth operation of software applications and infrastructure in a DevOps environment. The process involves monitoring various components of the software delivery pipeline, including applications, infrastructure, networks, and databases.

Also known as continuous monitoring, DevOps monitoring enables development and operations teams to detect and respond to problems quickly. The goal is to help teams respond to slowdowns, outages, and freezes before they become major problems for users. DevOps monitoring also typically involves high levels of automation that can enable self-healing before IT personnel step in.

Why DevOps monitoring is important—and challenging

DevOps monitoring is important to help teams achieve greater efficiency in their software development lifecycles. In addition, monitoring DevOps processes provide the following benefits:

• Improve system performance. Monitoring provides a clearer, more consistent picture of DevOps system performance and overall health. The result is faster and more data-driven decision making.
• Help systems meet SLAs. DevOps monitoring also gives developers and operations teams the data they need to successfully meet service-level agreements (SLAs).
• Provide metrics for improved site reliability. Monitoring also creates a baseline for continuous improvement in DevOps systems and processes, which bolsters site reliability.

However, DevOps monitoring has its challenges. Because every DevOps environment is unique and varied, teams often use many different tools at every stage of the SDLC. Moreover, most organizations use a combination of cloud-based and on-premises infrastructure. As a result, they use different tools for monitoring. These conditions can lead to the following challenges:

• Data silos. The more tools and platforms teams use, the greater the likelihood of developing data silos.
• Missed insights. This profusion of tools at different stages means teams across the organization—such as developers, site reliability engineers, and security professionals—can have difficulty sharing data and insights.
• Lost efficiency. Lack of cohesion also means DevOps efficiency can suffer as teams have to manually piece together how one system affects another.

Different types of DevOps monitoring and why they’re important

Because DevOps spans the entire SDLC from pre-production to production, DevOps monitoring includes the following four areas:

1. Application performance monitoring

Application performance monitoring tracks the availability and performance of applications in the field. It examines metrics like response times, application programming interface availability, and page load times to flag problems that affect the user experience.

2. Infrastructure monitoring

Infrastructure monitoring reviews servers, storage, network connections, virtual machines, and other data center elements that support applications. It looks at variables like central processing unit utilization rates, available bandwidth, and disk space usage to detect problems early.

3. Network monitoring

Network monitoring analyzes network traffic logs to look for anomalies such as high bandwidth usage, packet flow, and atypical patterns that might indicate a cyberattack.

4. User experience monitoring

Also known as synthetic monitoring, user experience monitoring is a practice and technology that simulates the actions users might take when using an application. User experience monitoring generates simulated behavior patterns across different scenarios, such as geography, device type, and time of day. The result is that teams can identify issues that don’t necessarily show up in other types of monitoring.

Because every DevOps environment is unique, exactly how organizations implement these monitoring types will differ depending on architecture and tools. The challenge is being able to integrate the data from these disparate monitoring efforts into insights that enable teams to automate more DevOps processes.

DevOps monitoring use cases

The goal of DevOps monitoring is to improve software quality and make processes more efficient and automatable. Common DevOps monitoring use cases include the following:

• Detect and remediate errors before they affect users. By pinpointing issues earlier in the SDLC, teams can fix problems before software is released to production. Early detection enables teams to deliver more reliable software and seamless user experiences.
• Increase system uptime. DevOps monitoring also helps teams meet their SLAs for system uptime. When the market demands five-nines uptime (a system that is up 99.999% of the time), teams need full-stack observability and pinpoint precision to meet it.
• Increase software supply chain security. As teams integrate application security into DevOps for DevSecOps, monitoring provides an organization-wide view of security issues as they occur. This view enables teams to understand software vulnerabilities and risks in the context of pre-production and production environments.
• Create a developer-operations feedback loop. An enemy of efficient DevOps is data silos that can develop because of disparate toolsets and separate teams. DevOps monitoring can improve cross-team collaboration by bringing data together in one place where both development and operations teams can gain and share new insights.
• Enable progressive delivery. DevOps monitoring allows teams to reduce the risk of failed releases by delivering new features and updates in a controlled and incremental manner. Monitoring tools allow development teams to validate key quality, performance, and functional criteria are met through the gradual introduction of new changes to production. This helps minimizes failure and improve customer satisfaction
• Enhance user experience and boost business credibility. Using real user monitoring (a performance monitoring process centered on a user’s interaction with an application), teams can gain full visibility into how users experience digital transactions across web, mobile, and custom apps. Factors such as page load times, process flows, and real-time user surveys quickly gauge user satisfaction and business impact in real time.

Advanced observability advances DevOps monitoring

DevOps monitoring is essential for delivering software efficiently, securely, and reliably. By equipping teams with the insights they need to proactively identify code issues before they escalate, DevOps monitoring tools help organizations keep up with the accelerating pace of digital transformation. But to overcome the challenges of an ever-lengthening toolchain, it helps to take an observability platform approach to DevOps monitoring.

Powered by AI and intelligent automation, Dynatrace’s scalable and comprehensive observability platform advances organizations on their DevOps monitoring initiatives. Dynatrace enables teams to build highly integrated DevOps toolchains with observability, security, and AI at the core to help further accelerate delivery speed, improve code quality, and increase DevOps throughput.

The Dynatrace platform also enables teams to monitor logs generated by applications, servers, and infrastructure components. This centralized view enables teams to gather even more insight into errors, security issues, and operational inefficiencies.

Ultimately, using the platform as a single source of truth can break down silos and increase collaboration and trust within teams.

The post DevOps monitoring tools: How to drive DevOps efficiency appeared first on Dynatrace news.

What is DevOps?

DevOps is a collection of flexible practices and processes organizations use to create and deliver applications and services by aligning and coordinating software development with IT operations.

As DevOps pioneer Patrick Debois first described it in 2009, DevOps is not a specific technology, but a tactical approach. By working together, development and operations teams can eliminate roadblocks and focus on improving how they create, deploy, and continuously monitor software.

The shift to DevOps is critical for organizations to support the ever-accelerating development speeds that customers and internal stakeholders demand. With the help of cloud-native technologies, open source solutions, and agile APIs, teams can now deliver and maintain code more efficiently than ever. Combining development with operations and the processes that support them enables organizations to keep pace with the speed of development.

The origin of DevOps

DevOps got its start in 2008 with developers Andrew Clay and Patrick Debois. Looking to overcome common issues in agile development — such as decreased collaboration as project timelines expand, and the negative impacts of incremental delivery on long-term outcomes — the pair proposed an alternative: continuous development and delivery in a combined DevOps pipeline. The term gained traction after DevOpsDays in 2009, quickly establishing itself as a new industry buzzword.

More than a decade later, it’s clear the DevOps framework is more than just hype. In practice, its biggest benefit isn’t a simple efficiency boost, but rather a cultural shift that fundamentally changes the way companies approach every stage of the software development process.
Recently, DevOps has undergone a more in-depth evolution thanks to the work of industry experts such as Gene Kim, keynote speaker at Perform 2021 and author of The DevOps Handbook and The Phoenix Project.

How does DevOps work?

Many organizations consolidate development and operations in a single team to achieve this combined process, organizing software delivery by feature rather than by job function. This approach encourages individuals to develop cross-functional skills, folding testing and application security practices into a seamless delivery lifecycle.

Implementing DevOps often goes hand in hand with continuous integration (CI), where multiple developers commit software updates to a shared repository, often many times a day. CI enables developers to discover integration issues and bugs earlier in the process and streamline code branches and builds.

With this holistic point of view, engineers can collaborate on common processes, such as defining service-level objectives (SLOs), testing, and quality gates that everyone can implement. A common set of standards and goals can streamline agile workflows and make it possible for teams to adopt a coordinated DevOps toolset so they can automate more processes in the software delivery lifecycle (SDLC).

The DevOps lifecycle

Planning

This stage involves defining the project’s goals and creating a plan for how to achieve them. It also includes identifying the risks and challenges that may be encountered. The planning stage is essential for ensuring that the DevOps process is successful. Involve all stakeholders in the planning process, including developers, operations engineers, and business leaders. The plan should be clear, concise, and achievable. 

Development

This stage is where the code is written and tested. Automate as much of the testing process as possible to ensure that the code is high quality and use a version control system to track changes to the code. The code should be unit-tested to ensure that it is working correctly. 

Deployment 

This stage is where the code is deployed to production. It is vital to have a deployment pipeline that automates the deployment process and minimizes the risk of errors. 

Monitoring

This stage involves collecting data about the application’s performance and identifying problems. It is necessary to have a monitoring system in place that can alert the team to any issues as soon as possible.  

Continuous improvement

This stage is ongoing and involves continuously evaluating the DevOps process and making improvements. Being open to feedback and willing to change the process as needed is important.  

The benefits of DevOps

In practice, DevOps offers benefits not only to creating, delivering, and maintaining software, but to every process and stakeholder from early-stage proof-of-concept to digital business analytics and customer experience.

For development teams, the goal is to recognize the process of creating code as an ongoing cycle rather than a straight line. Partnering or integrating development with operations teams helps apply agile development principles — quick, bite-sized improvements based on priority — to the entire software lifecycle. This includes initial design, proof of concept, testing, deployment, and eventual revision.

This approach is especially important as customer needs and expectations from the C-suite intensify. Development teams tasked with producing and deploying software as quickly as possible are now capable of doing just that. Meanwhile, operations teams, understandably, have concerns about the impact of rapid-fire code implementation and the changes necessary for putting that code to work, reliably and at scale.

For operations, a collaborative approach to deployment makes it possible to extend agile processes past software into platforms and infrastructure to analyze details and context of all layers within the IT environment. By applying design thinking to delivery systems, operations teams can shift their focus from managing infrastructure to delivering outstanding user experiences.

In effect, this dev-and-ops effort looks to leverage rather than limit the impact of development on operations by applying software development principles to every aspect of IT — all while maintaining operational focus on standardization and security.

The challenges of DevOps

Here are some challenges that organizations may face as they adopt DevOps:

• Security concerns: DevOps practices can introduce new security risks, such as the risk of introducing vulnerabilities into production systems through automated deployments. Organizations must have a strong security culture and implement appropriate security controls to mitigate these risks.
• Organizational silos: DevOps can be challenging to implement in organizations with strong structural silos. These silos can create communication and collaboration challenges that can hinder the adoption of DevOps practices. Organizations must break down these silos and create a more collaborative culture to adopt DevOps successfully.
• Compliance challenges: DevOps practices might introduce new compliance challenges, such as the need to comply with regulations requiring manual approvals for production system changes. Organizations must assess their compliance requirements and implement appropriate controls to remain compliant.

How to adopt DevOps

By breaking down the silos between the software development and IT operations teams, DevOps can help organizations deliver software more quickly and reliably, for example:

Start with a small pilot project.

Don’t try to implement all the DevOps practices at once. Start with a small project that is important to the organization and has senior management’s support.

Get buy-in from all stakeholders.

DevOps is a cultural change and won’t be successful unless everyone is on board. Ensure you have the buy-in of all stakeholders, including developers, operations engineers, security professionals, and business leaders.

Use the right tools.

There are several tools available to help organizations adopt DevOps practices. These tools can help with CI/CD, monitoring, and security.

Measure and improve.

It is essential to measure the results of your DevOps efforts to see what is working and what is not. This information can then improve your processes and make them more efficient.

Be patient.

It takes time to implement DevOps practices and see results. Stay positive if you don’t see immediate results.

Practices within DevOps

Continuous Integration

• Continuous integration (CI) is the practice of merging all code changes into a central repository regularly. This helps to identify and fix errors early in the development process.

Continuous delivery

• Continuous delivery (CD) is automatically deploying code changes to production after they have been tested and approved. This helps to ensure that new features and bug fixes are available to users as quickly as possible.

Microservices

• Microservices is an architectural style that divides an application into small, independent services. This makes it easier to develop, test, and deploy applications.

Infrastructure as code

• Infrastructure as code (IaC) is the practice of managing infrastructure using code. This makes it easier to automate the deployment and provisioning of infrastructure.

Monitoring

• Monitoring is the practice of collecting and analyzing data about an application’s performance. This helps to identify and troubleshoot problems before they impact users.

Automation

• Automation is the practice of automating tasks, such as testing, deployment, and monitoring. This frees up time for developers and operations engineers to focus on more strategic tasks.

Collaboration

• Collaboration is the practice of working together effectively. This is essential for DevOps, as it requires close collaboration between developers and operations engineers. This means breaking down the silos between these two teams and creating a culture where everyone works toward the same goal.

What are DevOps best practices?

Integrating disciplines, tools, and processes across the organization requires planning and coordination. Here are some best practices organizations can follow to make DevOps an enterprise-wide success.

Augment the DevOps pipeline with AI

Every stage of the DevOps pipeline requires some amount of analysis to drive decisions, responses, and automation.

For example, precise AI-based analysis can drive decisions about whether to release a piece of software, and once the software is in production, indicate whether the release is performing as it should. Or, during failed test runs, AI can provide the exact root cause down to the details of the underlying code, so developers can quickly address and remediate errors.

The ability to analyze data accurately and reliably, and provide definitive answers makes it possible for teams to automate processes throughout the software delivery lifecycle. Reliable, AI-driven answers are crucial for rapid incident response and auto-remediation so teams can understand the context behind a failure or error.

This artificial intelligence for IT operations (AIOps) is becoming a widespread practice, especially as organizations adopt cloud-native infrastructure.

Shift-left service-level objectives (SLOs)

To ensure development teams and SREs are aligned on the same success criteria, they should evaluate production SLOs against pre-production environments. By expanding quality assurance to include prevention, detection, and recoverability using production-level criteria, teams can deliver software that meets user requirements, reduces error rates, and increases overall reliability and resilience. More importantly, the cost of fixing errors in pre-production is much lower than in production.

A continuous quality mindset enables teams to architect the entire SDLC for testing. This means testing all layers of the lifecycle. It also means developing and maintaining reliable test data and test environments that developers, SREs, and IT operations teams can use at every stage of development and delivery.

One way to automatically evaluate pre-production SLOs is by establishing quality gates. A quality gate helps teams determine whether a service meets all pre-defined quality criteria. Quality gates take in key service level indicators (SLIs) or monitoring metrics and evaluates them against the quality criteria set. Code progresses to the next stage of the lifecycle only if the services meet or exceed the quality criteria.

Automate all DevOps processes

Automating DevOps pipelines allows for iterative, incremental software updates to be deployed faster and more frequently. It provides tighter feedback loops between Dev and Ops teams so they can spend more time innovating instead of performing manual processes.

Automation is what helps teams scale DevOps from a lighthouse project to an essential practice across the entire IT estate. DevOps automation typically rings the continuous integration, continuous delivery (CI/CD) bell, but automating these foundational processes can extend well beyond developing code. More advanced organizations seek to automate all stages of the DevOps lifecycle including infrastructure provisioning, deployment, monitoring, testing, remediation, and so on.

Adopt cloud-native architecture

To deliver on DevOps potential, speed and agility are key. Adopting cloud-native technologies and architecture is the best way to deliver more and richer features faster, flexibly, and at scale. These technologies include container-based computing solutions, such as Kubernetes, and serverless platforms-as-a-service (PaaS), such as AWS Lambda, Google Cloud Platform, and Azure Functions. In these environments, software runs in immutable containers using resources as needed, a setup that lends itself well to infrastructure patterns that can be easily orchestrated and automated.

Cloud-native technologies make it easier for teams to apply agile software development practices to infrastructure management. This includes automating key tasks such as version control, unit testing, continuous delivery, operations functions, and problem remediation.

Integrate security practices for DevSecOps

The diversity and flexibility of cloud-native technologies also make securing applications against vulnerabilities more challenging. As mentioned earlier, integrating application security and vulnerability assessment into a DevOps workflow is a best practice that extends the benefits of AI-driven analysis and automation to securing applications.

By automating application security testing to continuously analyze applications, libraries, and code at runtime, teams can eliminate security blind spots and false-positive alerts. Adding security-related SLOs, testing, and quality gates into all phases of the delivery lifecycle enables teams to cultivate a security mindset that eliminates another silo and results in more secure software.

Adopt a platform-driven approach with self-service processes

Achieving widespread DevOps success requires a platform approach that can make it easier for organizations to effect structural changes that optimize how teams work. A key goal is to establish self-service processes for managing different types of testing, monitoring, alerting, CI/CD workflows, internal infrastructure and development environments, and public cloud infrastructure. When teams have access to reliable data and analysis, and individuals have more autonomy to rely on their own knowledge and experience, organizations can extend the value of DevOps to the whole enterprise.

With self-service access to APIs, tools, services, and support in a single platform that leverages AI and automation, teams have a single reliable source of knowledge and coordination. This enables teams to integrate and streamline their DevOps toolchains and processes so they can spend less time maintaining the infrastructure and more time innovating.

How can DevOps transform enterprises

A successful DevOps initiative is characterized by a culture of experimentation, risk, and trust — one in which constant feedback between all members is both welcomed and utilized. But culture alone isn’t enough to transform enterprise effort; teams also need the right technologies and DevOps software to get the job done.

As tools and technologies proliferate, a crucial capability is observability: the ability to instrument and monitor telemetry data from across the cloud-native environment. This includes metrics, logs, distributed traces, as well as data from user-experiences and the latest open-source standards to measure the health of applications and their supporting infrastructure at every stage of development.

According to a recent Gartner report, leaders should consider solutions during pre-production to maximize insight into app performance, service availability, and overall environmental health.

There’s also an emerging push to include more disciplines in the DevOps effort: “DevSecOps” teams seek to integrate security testing in delivery and deployment pipelines, while “BizDevOps” efforts strive to understand app performance from a user-experience perspective.

Although the advantages of DevOps are clear, the culture and technology shift it brings can make implementing DevOps solutions an iterative journey.

What is Observability in DevOps?

DevOps combines development and operations into a unified framework that breaks down silos and fosters whole-lifecycle collaboration. In this environment, SREs can implement operations that ensure software systems’ availability, latency, performance, and resiliency, and CI/CD practices can provide well-aligned and automated development, testing, delivery, and deployment.

How observability closes the DevOps gap

What is DevOps? It’s a cultural and tactical shift that closes the gap between development efforts and operational obligations by combining teamwork with technology to streamline software delivery, standardize testing and quality gates, and automate processes and incident response. Armed with best practices and an AI-driven software intelligence platform to manage the entire DevOps tool chain, teams can maximize efficiency, minimize error rates, and deliver on continuous delivery expectations.

The post What is DevOps? Unpacking the purpose and importance of an IT cultural revolution appeared first on Dynatrace news.

As more organizations adopt cloud-based computing and the demand for digital services increases, site reliability engineering (SRE) practices have become essential. These practices help organizations meet service level agreements (SLAs) for availability, performance, user experience, and business KPIs.

But what exactly is SRE, and what do site reliability engineers do?

What is site reliability engineering?

Site reliability engineering (SRE) is the practice of applying software engineering principles to operations and infrastructure processes to help organizations create highly reliable and scalable software systems. As a discipline, SRE focuses on improving software system reliability across key categories, including availability, performance, latency, efficiency, capacity, and incident response. Those who perform the tasks involved are known as site reliability engineers.

The term “site reliability engineering” was coined in 2003 by Google VP of Engineering Ben Sloss, who famously noted on his LinkedIn profile, “If Google ever stops working, it’s my fault.” According to Google, “SRE is what you get when you treat operations as a software problem.”

Although every organization and software system is unique, it’s important to understand the fundamentals of SRE and the skills and mindset of its engineers as you think about optimizing the reliability and overall quality of your software.

The benefits of site reliability engineering

Site reliability engineering highlights reliability, scalability, and efficiency. The benefits of SRE include:

• Increased reliability and uptime: SRE focuses on preventing and mitigating incidents to ensure that systems and applications are always available and performant.
• Improved scalability: By optimizing resource usage and minimizing waste, SRE can help organizations scale their infrastructure and applications more efficiently.
• Improved user experience: SRE can ensure that applications and services are always available and responsive, which can directly impact customer satisfaction, brand reputation, and revenue.
• Continuous improvement: SRE highlights the use of data and metrics to identify areas for improvement and drive ongoing optimization and innovation.
• Increased security: SRE can help to guarantee that systems and applications are secure and compliant with industry standards and regulations.
• Predictable performance: By monitoring and analyzing usage patterns, SRE can help to predict and prevent performance issues before they occur, ensuring that systems and applications perform predictably and consistently.
• Cost savings: SRE can reduce costs by automating routine tasks and optimizing resource usage, reducing the need for manual intervention and saving time and money.
• Collaboration between development and operations teams: SRE emphasizes cross-functional teams and shared ownership of reliability and performance, promoting a culture of collaboration and accountability.

Five things to know about site reliability engineering

Implementing SRE may seem like a large undertaking, but the operational shifts it requires offer lasting benefits that promote greater efficiency and provide a positive return on investment.

1. SRE focuses on automation

A major goal of SRE is to reduce duplication or redundancy of effort as much as possible. SRE teams focus on automating manual tasks, such as provisioning access and infrastructure, setting up accounts, and building self-service tools. As a result, development teams can focus on delivering features, and operations teams can focus on managing infrastructure.

Automating processes is even more critical as organizations speed up delivery of new features into production. On one hand, speed comes from DevOps teams who leverage automation to increase continuous integration and continuous delivery (CI/CD). On the other hand, the move to microservice architectures and the adoption of cloud-native technology, containers, Kubernetes, and serverless architectures offer even more ways to delivery smaller changes faster. These methods increase efficiency and speed, but also demand consistent, repeatable processes that reduce risk and provide feedback loops for measuring operations, so teams can identify areas for improvement.

2. SRE bridges the gap between Dev and Ops

Everything the organization does in the value stream process should answer the question “how do we ensure this runs in production reliably?” SREs drive resiliency-based engineering. They can become mentors and ensure that resiliency is a top priority for both developers and operations.

Applying the DevOps mindset and skills to software reliability helps reduce silos between development and operations teams by sharing responsibility for detecting reliability and performance issues early in the development life cycle. Collaboration between developers, operations, and product owners enables site reliability engineers to define and meet uptime and availability targets.

3. SRE drives a “shift-left” mindset

SRE is a constantly evolving discipline, presenting opportunities to build methods, policies, and processes into the delivery pipeline that allow applications to “auto-remediate” or users to solve their own problems. A shift-left mindset means SREs can embed reliability principles from Dev to Ops, baking reliability and resiliency into each process, app, and code change to improve the quality of software that goes to production.

Here are some ways SRE helps to drive a “shift-left” mindset:

• Develop quality gates based on production-level service level objectives (SLOs) to detect issues earlier in the development cycle.
• Automate build testing and validation using service-level indicators (SLIs) and SLOs
• Influence architectural decisions during initial design stages to ensure resiliency and scale at the outset of software development.

The goal is to take early, proactive steps to ensure quality and reliability are built-in from the beginning. SRE can influence processes more broadly and expand to coordinating testing across the enterprise in support of CI/CD practices.

4. SRE builds services and tools to help operations and support

Traditionally, a major goal of operations teams is to improve uptime. This single-dimensional approach looks for the coveted “five nines” of uptime, or 99.999%, which translates to just over five minutes of downtime per year.

But the higher frequency of change in distributed cloud-native environments requires a multi-dimensional approach.

The goal of SRE is to enable higher change rates while maintaining resiliency and that coveted 99.999% uptime. In multicloud environments, resiliency is measured across key metrics such as performance, user experience, responsiveness, conversion rates, and so on. SRE teams need to build and implement services that improve operations and facilitate the release process across all these areas. This can be anything from adjusting monitoring and alerting to making code changes in production. Site reliability engineers often build custom tooling from scratch to meet specific needs in the software delivery or incident management workflow.

Adopting an SRE approach also requires standardizing the technologies and tools teams use. Standardization makes it easier to manage operations and reduces the burden of managing incompatible technologies, which gives teams more time to collaborate and innovate.

5. SRE requires a cultural change

Because SRE is a practice, it requires changing how teams across multiple disciplines communicate, solve problems, and implement solutions. To adopt a successful SRE culture, organizations must adopt new approaches to managing risk. It also means they must adapt governance processes, invest in hiring, and educate a collaborative workforce that’s versed in engineering and operations and learns and adapts quickly.

Organizations can then integrate these skilled engineers at key points in the DevOps lifecycle. In development and testing teams, SRE specialists develop automation that helps developers test early and often without impeding agile delivery schedules. At a system level, SRE specialists develop tooling that coordinates releases and launches, evaluates system architecture readiness, and meets system-wide SLOs. At a governance level, SRE specialists help to define and oversee enterprise architecture, establish best practices, and select tools and resources that support company-wide site reliability.

What does a site reliability engineer do?

To get an expert’s view on what site reliability engineers do, I asked our DevOps Activist, Andi Grabner.

“Site reliability engineers use good practices around software engineering to provide resilient infrastructure and resilient services to their organizations and the people that actually deliver new applications,” he explains. He also notes that SREs often come from traditional operations roles, such as systems engineers who keep systems up and running. “Site reliability engineers ensure systems stay reliable, resilient, and available,” he adds.

Typical expectations for SREs

Typically, SREs are tasked with ensuring that the speed of delivery doesn’t result in security, service, or solution interruption. But as Grabner notes, “Expectations are a little different for every company. There’s no golden rule. Many are responsible for monitoring and observability and maintaining systems — and providing automation to spin up required environments.”

Grabner highlights the role of SREs in providing the frameworks and platforms for service and application deployment. “When things go wrong, SREs often take on the role of first-line defenders if there’s an alert,” he says. “In a great organization, they don’t do it alone — they constantly work with and within individual application teams to deal with apps that are under fire.”

Perhaps the most important role of SREs is architecting resiliency. “You can’t buy resiliency as-a-service,” Grabner observes. “You have to architect for it by building systems that are resilient by design.” This architectural approach recently helped Dynatrace to withstand an AWS outage in Germany. Automatic delivery, resiliency, and auto-remediation helped to ensure critical systems weren’t affected.

Roles and responsibilities of an SRE

The role of an SRE has become invaluable as technology continues to evolve and businesses become more dependent on digital infrastructure. Here are some of the standard responsibilities of an SRE:

1. Monitoring and Alerting
   One of the primary duties of an SRE is to monitor a company’s digital infrastructure. This involves setting up monitoring tools and systems to detect concerns before they become significant problems. SREs set up alert systems that notify the appropriate people when issues are detected.
2. Incident Response
   The SRE responds quickly and effectively when issues are detected by identifying the root cause, developing and implementing a plan, and communicating with relevant stakeholders.
3. Automation and Tooling
   SREs develop and maintain the tools and systems used to manage a company’s digital infrastructure. This includes developing automation scripts to streamline processes and reduce the risk of human error. SREs also identify areas where tooling can be improved and create new tools to meet the changing needs of the business.
4. Capacity Planning
   SREs ensure that a company’s digital infrastructure can meet the needs of the business. This involves analyzing usage patterns to predict and guarantee the required capacity to meet future demand.
5. Collaboration
   SREs work closely with other teams to ensure the company’s digital infrastructure is reliable, scalable, and secure.

What makes a great SRE?

Great SREs are risk-takers, tinkerers, and innovators. They figure out what it takes to scale a system from 100 users to 100,000 users to 1,000,000 users while maintaining uptime and resiliency. They’re systems thinkers who consider how decisions made in development affect production environments, and how the needs of production systems can influence design.

This requires constant testing, accepting failure, and adapting, automating repeatable processes. Successful SREs bring a resiliency and adaptation mindset to every situation.

Grabner highlights the need for SREs to learn from their mistakes. “Some companies run ‘chaos days’ to deal with worst-case scenarios to understand what could happen and how to deal with it,” he says.

Automation is another marker of SRE success. “People excel in this role when they try to automate all the tasks that can and must be automated,” says Grabner. “This frees them up to deliver true innovation.” He notes that while anyone can innovate under the right circumstances, teams are often held back by the “toil” of manual and repetitive tasks. “Your goal is to automate yourself out of your current role and into your next role.”

Finally, Grabner made it clear that SREs can’t operate in isolation. “You need to let people educate themselves with new technologies and practices,” he says. “Show the world. Don’t keep secrets — be open and share your own learnings, along with learning from others. There are a lot of great conferences out there — it’s worth getting inspired by what others do and inspiring others with what you do.”

DevOps vs. SRE

Where DevOps teams focus on streamlining change, SREs help ensure these changes don’t increase overall failure rates. In effect, they’re two sides of the same coin: DevOps automates speed, while SRE automates reliability. “It’s a balance between speed and safety,” Grabner says.

He sees DevOps processes as moving left to right along the development life cycle, using automation to speed up new capabilities that are typically measured by deployment frequency and lead time for changes. In comparison, SRE moves right to left using production-level requirements in development, with a focus on limiting failure rates and reducing the time required to restore service. “SRE is about making sure that even though there is a lot of change, these changes don’t break things.”

Grabner sees SRE and DevOps overlapping when it comes to SLOs. “SLOs are all about supporting business goals,” he said. “Companies may need systems at 99% reliability. They may want to increase their user base or improve the end-user experience.” Satisfying these goals is the role of DevOps. “But underneath these goals are technical goals that are specific to your objectives,” he says. “They contribute to business success with the right features at the right time and help you cope with change.” Delivering on these goals is the job of SRE staff. As a result, “SLOs are a great way to bring DevOps and SREs together.”

In another blog post you can learn more about the differences and similarities of SRE vs DevOps.

Solving for site reliability

Site reliability isn’t and never will be a “solved problem”. New services and applications combined with evolving enterprise demands mean there’s always work for SRE teams, and there’s always room for improvement.

As Grabner notes, when it comes to improving SRE impact, “The biggest thing is to be open and share your own learnings, along with learning from others. There are a lot of great conferences out there — it’s worth getting inspired by what others do and inspiring others with what you do.” He also highlights the need to learn from your mistakes. “Some companies run ‘chaos days’ to deal with worst-case scenarios to understand what could happen and how to deal with it.” Finally, Grabner made it clear that SREs can’t operate in isolation. “You need to let people educate themselves with new technologies and practices. Show the world. Don’t keep secrets — don’t see it as a silo.”

Looking for a solution to up-level your SRE practices? Dynatrace can help. Providing automatic and intelligent observability for even the most complex distributed cloud environments, the Dynatrace Software Intelligence Platform empowers SRE and DevOps teams to identify problems before they occur. Driven by continuous automation with AI at its core, Dynatrace delivers precise root-cause answers to site reliability issues at every step of the software development lifecycle. From early development in pre-production environments through delivery and operations in production environments, Dynatrace helps SRE teams improve reliability, availability, and latency, and mitigate the business impact of service outages and slowdowns.

The post What is SRE (site reliability engineering)? And what do site reliability engineers do? appeared first on Dynatrace news.

Organizations everywhere are adopting site reliability engineering (SRE) to cope with the growing complexity of hybrid and cloud-native environments. Indeed, more than 1,000 solutions are now incubating in the Cloud Native Computing Foundation and modern applications comprise thousands of discrete microservices. Without SRE best practices, the observability landscape is too complex for any single organization to manage.

SRE is still in the early adoption stages. Like any evolving discipline, it is characterized by a lack of commonly accepted practices and tools. Organizations that have achieved SRE maturity have a better handle on the state of their infrastructure, the ability to tie reliability metrics more tightly to business objectives, and the means to ensure a consistent and responsive customer experience.

Dynatrace’s 2022 State of SRE Report surveyed 450 SREs across the globe. While 62% rated themselves as being well along the path toward SRE adoption, just one in five self-identified as SRE-mature. The following five key steps to success are essential for the journey to SRE maturity.

1. Make SRE accessible

More than half of all respondents cited two key SRE adoption barriers: the perceived difficulty of training existing IT professionals in SRE best practices, and the cost and difficulty of finding skilled professionals.

In a talent-constrained market, the best strategy could be to develop expertise from within the organization. Adopting tools with high levels of automation can help reduce the learning curve. Weaving DevOps and the related disciplines of DevSecOps and AIOps tightly into the development process can also accelerate the process. These reduce the need to hire specialists while providing a unified view of processes and infrastructure that make SRE more focused and effective.

2. Integrate toolchains and adopt an everything-as-code approach

Tools and practices tend to be fragmented in maturing markets. SRE is no exception, with practitioners and development teams each having their favorites. However, maintaining bespoke toolchains requires a significant investment of time and effort, and ultimately distracts SREs from more impactful tasks. Organizations should seek to standardize a single set of tools that everyone will use. Consider selecting platform-based solutions — whether open source or from a commercial vendor — that support open ecosystems.

Virtualization has revolutionized system administration by making it possible for software to manage systems, storage, and networks. By removing physical dependencies, automation can help perform SRE at scale. This can reduce labor costs and enhance reliability by enabling systems to self-heal. With the self-service features and an everything-as-code architecture, labor requirements will significantly decrease and SRE best practices will emerge.

3. Automate as much as possible

An overwhelming 85% of SREs said their ability to scale the practices requires automation and AI. Not surprisingly, more than 71% are increasing their use of automation across the lifecycle, 58% are applying automation to the continuous integration/continuous delivery (CI/CD) pipeline, and 46% are modernizing tool stacks.

Automation has the clear advantage of reducing manual labor. It also enables organizations to tie business-level objectives (BLOs) — such as user satisfaction and system responsiveness — to service-level objectives (SLOs), linking SRE best practices and bottom-line impact. Just over one in five survey respondents has gone beyond the automatic evaluation of SLOs to include BLOs. This number will likely increase as the SRE discipline matures.

4. Design, implement, and tune effective SLOs

While SLOs are the North Star of SRE, 99% of survey respondents stated they are running into challenges to define and create them. The chief inhibitors reported were information silos, incompatible toolsets, and growing complexity.

Consolidating on a single observability platform can help organizations solve these issues by reducing tool profusion and enabling the organization to create a set of performance standards that teams can be observe and manage in a unified way. SREs should agree upon SLO dashboards, error tolerances, remediation plans, and alerting tactics and test them in advance. The most popular SLOs among organizations with high levels of SRE proficiency relate to availability, user satisfaction survey scores, the ratio of failed requests to total requests, and the crash rate across all supported devices.

5. Apply AIOps for analysis and automation

AIOps — an AI-driven approach to managing IT operations such as monitoring, automation, and service desk — has become increasingly popular in the SRE lifecycle. Platforms are now offering the ability to gather data from multiple sources much faster than human operators, automatically remediate common problems, and reduce false or unnecessary alerts.

More than two-thirds of the SRE report respondents reported they’re increasing their use of AIOps across every part of the lifecycle. They’re keeping the following key drivers in mind:

• Automation of additional processes critical to ensuring service levels
• Improving the ability of teams to prioritize problems
• Identifying security vulnerabilities more quickly
• Enabling predictive monitoring to identify problems before causing a disruption.

SRE best practices may very well be in their infancy stages. But by taking these early steps now, your organization can be ahead of the competition in gaining complete insight into your entire application portfolio.

The post 5 SRE best practices you can implement today appeared first on Dynatrace news.

Site reliability engineering (SRE) continues to gain popularity as organizations embrace hybrid cloud strategies and IT automation at scale. By applying software engineering principles to operations and infrastructure practices, SRE enables organizations to streamline and automate IT processes.

In the Dynatrace State of SRE Report: 2022 Edition, 88% of surveyed site reliability engineers (SREs) said there is now greater recognition of the strategic importance of their role in business success than there was three years ago. SRE is becoming an essential discipline in organizations that use DevOps (the combination of development and operations) and agile methodologies.

SRE adoption is growing, yet gaps remain. In order to unleash the innovation organizations need to evolve their SRE approaches. The report uncovers six site reliability engineering trends that will help organizations get the most from DevOps practices.

1. MTTR reduction remains top of the list for SREs

SREs improve the reliability of production systems, and reducing mean-time-to-repair (MTTR) is their top priority. But research shows that 60% of SREs find they spend most of their time building and maintaining automation code. While increasing automation is a key goal, organizations will lose their derived efficiency if enablement is arduous and time-consuming.

Much of the problem stems from how site reliability engineering teams build automation for DevOps workflows. Often, teams handle this on a case-by-case basis because their tooling doesn’t come with automation built in and doesn’t offer everything-as-code capabilities. As a result, they must build a layer of automation on top of their tooling.

Over time, this creates a complex web of code that becomes more difficult to scale across the DevOps pipeline. If SRE teams don’t identify a more efficient approach to DevOps from day one, they will undoubtedly find more of their time drained in the future. That time sink results in developers chasing down bugs and other code issues rather than focusing on strategic, revenue-generating work.

This underscores the need for SREs to work with DevOps teams, developers, and architects to ensure that software not only meets a business need but also is resilient and automatable by default. Enabling teams to easily integrate new automation capabilities with existing tools and workflows reduces manual effort and improves engineering practices.

2. A shift to SRE-driven engineering takes hold

More than half (51%) of SREs say they dedicate significant time to influencing architectural design decisions to improve reliability. This suggests that all departments have made progress toward SRE-driven engineering across to improve reliability, resiliency, and security. But there’s still a long way to go.

The most mature organizations embrace site reliability engineering practices that include developers who have fought the battles. They understand what it takes to build systems that can scale from 10 users to 1,000, or from 1 million to 10 million users. Integrating these developers with the design process provides insight that enables architects to incorporate reliability from day one.

3. Security is a core pillar of site reliability engineering

SREs are also making progress in extending organization-wide DevSecOps approaches to ensure organizations can restore systems quickly after discovering a vulnerability. More than two-thirds (68%) of SREs say they expect their role in security to become even more central in the future. This trend will increase as organizations continue using third-party libraries for cloud-native application development.

As the Log4Shell vulnerability demonstrated when it emerged in December 2021, third-party code libraries face significant security risks. Site reliability engineering teams are critical to identifying those flaws and eliminating them for fail-safe IT protection and to minimize cloud and third-party risks.

4. SREs need the freedom to experiment

While more than half (52%) of SREs dedicate significant of time to designing experiments and tests to reduce the risk of production failure, only 1 in 10 highlights this as their top priority.

Experimentation is critical to SRE. Teams still need to make progress to ensure they have more time available for these tasks. For SREs to deliver more strategic business value, engineers must streamline tasks that involve intensive manual effort.

5. SREs need the license to prioritize strategic work

Although experimentation falls relatively low on their priority list, 51% of SREs say they’re encouraged to experiment. In addition, only a quarter (26%) of organizations see incremental project failure as OK. The weak emphasis on experimentation highlights the many distractions SREs face, limiting the time they have to focus on it.

Organizations must consider new strategies that enable site reliability engineering teams to focus on more strategic tasks. That way, teams have more time to play. Team leaders also need to foster a culture that accepts failure and understands that the principle “fail fast, fail often” provides the greatest competitive edge.

Unshackle SRE teams from traditional organizational structures that view IT as a cost center and a burden. Organizations can’t benefit from lessons gained from IT mistakes without openness to failure.

6. Organizations recognize and reward site reliability engineering teams

SREs must be free to challenge accepted norms and set new benchmarks for innovation-led design and engineering practices. Many organizations are making strides in this direction and have methods for rewarding SRE teams’ successes.

Nearly a third (31%) use hackathons to devise new ways to improve reliability, offering prizes to winning SRE teams. These approaches are key to encouraging a culture of experimentation that promotes the strategic value of site reliability engineering for the business.

All six of these SRE trends can help your organization advance its SRE practices. Embrace experimentation and unleash exciting innovation.

The post Site reliability engineering: Six SRE trends to unleash DevOps innovation appeared first on Dynatrace news.

Shift left and shift right are core testing concepts of the agile DevOps methodology. This methodology speeds up application development by releasing small builds frequently as code evolves. As part of the continuous cycle of progressive delivery, DevOps teams are also adopting shift left and shift right principles to ensure software quality in these dynamic environments.

All this shifting may sound abstract, but I’ll explain how this software verification approach benefits DevOps methods and outcomes—and makes software more reliable.

In DevOps, what is shift left? And what is shift right?

To understand shift left and shift right, consider the software development cycle as a continuum, or infinity loop, from left to right.

On the left side of the loop, teams plan, develop, and test software in pre-production. The main concern in pre-production on the left side of the loop is building software that meets design criteria.

When teams release software into production on the right side of the loop, they make it available to users. The concern in production is maintaining software that meets business goals and reliability criteria.

What does shift left mean in DevOps?

Shift left is the practice of moving testing, quality, and performance evaluation early in the development process, often before any code is written.

Shift-left testing helps teams anticipate changes affecting performance or other delivery processes that arise during the development process. With shift-left testing, teams verify APIs, container configurations, interactions between microservices, and container configurations.

Why is shift left important?

In addition to testing for functionality, shift left testing also checks that software meets customer requirements. This enables developers and stakeholders to identify improvements that could enhance customer experience and functionality. Making these changes early in the development process reduces the cost of making them after the code is released.

As a result, shift left testing has become increasingly important as teams face pressure to deliver software faster and more frequently with higher quality. Shift left speeds up development efficiency and reduces costs by detecting and addressing software defects earlier in the development cycle before they get to production.

The benefits of shift left

The concept of shift left has been gaining traction in the world of software development and testing. This approach, which introduces testing and quality assurance early in the development lifecycle, offers several considerable benefits. 

1. Early Bug Detection: By testing early in the development process, teams can identify and address bugs when they are easier and less costly to fix. This not only improves the quality of the final product but also reduces the time and resources spent on debugging and rework.
2. Faster Time to Market: With shift left, testing is integrated into every stage of development. This continuous testing allows for faster feedback and quicker iterations, which can significantly reduce the overall time to market.
3. Improved Customer Satisfaction: By ensuring that quality is a priority from the outset, shift left helps to deliver a product that meets or exceeds customer expectations. This can lead to higher customer satisfaction and loyalty.
4. Reduced Costs: While implementing shift left may require an initial investment, the cost savings from reduced rework and faster time to market can quickly offset this. In addition, by catching and fixing issues early, teams can avoid the potentially high costs associated with late-stage or post-release fixes.
5. Enhanced Collaboration: Shift left encourages collaboration between testers, developers, and other stakeholders from the beginning of the project. This collaborative approach can lead to better communication, shared understanding of the product, and a better end result.

What does shift right mean in DevOps?

Shift right is the practice of performing testing, quality, and performance evaluation in production under real-world conditions. Shift-right methods ensure that applications running in production can withstand real user load while ensuring the same high levels of quality.

With shift right, DevOps teams test a built application to ensure performance, resilience, and software reliability. The goal is to detect and remediate issues that would be difficult to anticipate in development environments.

Why is shift right important?

With shift right, teams can test code in an environment that mimics real-world production conditions that they can’t simulate in development. This practice enables teams to catch runtime issues before customers do. To automate part of the process, teams can use application programming interface (API) calls. Organizations can also apply shift-right testing to code configured or monitored in the field. The result is more comprehensive testing coverage that better addresses user experience concerns.

Like shift left testing, the objective of shift right testing is to fail small and fail fast. The assumption is that problems caught early in the pre-deployment environment are easier to solve than issues caught by customers in live production.

Shift right testing is especially useful for organizations practicing progressive delivery, wherein developers release new software features incrementally to minimize the impact of unforeseen issues. Testing in a production-ready environment is a crucial final phase before declaring features ready for prime-time.

The benefits of shift right

Shift right is a concept that extends testing into the production environment. It involves monitoring and testing systems in production to gain insights and improve quality, offering several significant advantages.

1. Realistic User Testing: Shift right testing involves testing with real users. By including them in the process, software professionals gather valuable feedback about how users interact with the software. This helps make informed decisions about future improvements.
2. Continuous Feedback Loop: Shift right testing promotes continuous monitoring and improvement throughout all phases of development. This iterative feedback loop ensures that issues are addressed promptly, leading to better product quality.
3. Test Coverage: By extending testing to real-world scenarios, shift right testing provides broader test coverage. It helps identify issues that might not appear during earlier development stages, resulting in more comprehensive testing.
4. Enhanced Monitoring: Monitoring software behavior in production environments allows for better visibility into its performance, stability, and user experience. This helps teams proactively address any anomalies or performance bottlenecks.
5. Customer-Centric Approach: Shift right testing prioritizes users’ needs and experiences. By involving real users, software teams gain insights that lead to improvements aligned with customer satisfaction.

Why shift right and shift left are critical for microservices architecture

Both shift left and shift right testing have become more important as software is increasingly built from microservices. Instead of applications being built and run as a monolith, microservices are loosely coupled functions called at runtime. How microservices-based applications perform depends on individual service responsiveness, which complicates testing in a simulated environment.

Shift left testing allows teams to observe how services interact in a simulated environment and detect potential performance impacts before deploying the code.

Shift right testing at or near the deployment point allows teams to observe real-world forces and measure their impact. Shift-right tests typically cover functionality, performance, failure tolerance, and user experience. Teams often automate such production-environment testing and translate feedback into technical specifications for developers.

With shift-left and shift-right testing, testers can isolate issues so teams can fix and tackle improvements in parallel. As an application becomes more stable, teams can start testing and optimizing performance. Notably, shifting left and right have become an important practice in Agile software development. Together, they’re part of the continuous feedback loop that characterizes DevOps.

Types of shift right tests

A shift right approach may enlist various types of test suites. The following are tests your team might find useful:

• A/B testing. Web designers commonly use this method. Users see two versions of a page, and testers measure which generates a greater response. Teams often conduct this type of test in a production environment to gather real-world feedback.
• Synthetic monitoring. Synthetic monitoring is a form of shift-right testing that uses software tools to emulate the paths users might take when engaging with an application. With synthetic monitoring, you can automatically track application uptime and how your application responds to typical user behavior. It uses scripts to generate simulated user behavior for various scenarios, geographic locations, device types, and other variables.
• Chaos engineering. With chaos engineering, also called chaos testing, developers intentionally “break” the application by introducing errors to determine how well it recovers from disruption. DevOps and IT teams set up monitoring tools to see precisely how the application responds to different types of stresses. Teams perform these tests in a controlled production environment to minimize the impact on mission-critical systems.
• Canary releases. This strategy is named for the canaries that miners use to lower into coal mines to detect toxic gases. Technology has thankfully rendered this inhumane tactic obsolete. However, the term survives to describe slowly rolling out changes to a small subset of instances for testing before teams apply them to the full infrastructure.
• Blue-green deployment. With a blue-green deployment, an organization runs two nearly identical production environments, shifting users (real or synthetic) between the two as they make small changes to one or the other. This practice is important to shift-right methodology as it can minimize downtime. It also provides a mechanism for rapid rollback should something go wrong with the latest version.

Shift right tools

Shift right tools focus on the examination of code quality after it has reached production. Such tools are important for determining and analyzing how software behaves in real-word environments, offering teams insights into its actual performance. Examples of shift right tools include: 

• Application Performance Monitoring (APM) tools: APM tools offer metrics, logs, and traces relating to application performance, giving teams insights into how to remediate issues faster and further optimize their applications 

• Digital Experience Monitoring (DEM) tools: DEM tools provide insights into the user experience (UX) of an application. These tools may offer: 

• Real-user monitoring (RUM) for analyzing the behavior and experience of real-world users. 

• Synthetic monitoring that performs artificially simulated user journeys to test for any UX bottlenecks or deficiencies 

• Session replay that provides video-like documentation of real user journeys 

• Security monitoring and incident response tools: These tools help detect, manage, and communicate security incidents and facilitate the response process. They also assist teams in prioritization processes to ensure that the most business-impacting vulnerabilities are addressed first. 

• Chaos engineering tools: These tools introduce simulated failures to test the resiliency of code and the efficacy of response processes to such failures. This helps inform teams of any code deficiencies and opens room for improvement 

• Feature flagging tools: These tools provide capabilities to toggle certain features of a deployment on and off without deploying any additional code. Such tools offer a greater level of control to developers, as they allow for more experimentation and finetuning of the user experience. Feature flagging provides increased reliability in feature rollouts and helps to ensure that a deployment provides the best user experience possible. 

Types of shift left tests

The four basic types of shift left testing are the following:

• Traditional shift left testing. The traditional approach focuses on unit and integration testing, typically involving APIs and cross-browser compatibility. This approach focuses more on unit testing small pieces of code to deliver information early and often rather than more systems-level operational and acceptance testing.
• Incremental shift left testing. This is a popular approach for teams migrating from a waterfall development approach to one that breaks complex projects into smaller pieces. In this scenario, teams conduct systems-level operational and acceptance testing on a smaller, more incremental scale. Incremental testing is popular for validating large and complex systems.
• Agile/DevOps testing. This is a series of short, continuous sprints during the development stage. This approach does not intend to address operational performance but rather validate adherence to basic requirements architecture.
• Model-based shift left. Model-based testing aims to reduce errors introduced during the requirements definition, architecture, and design phases. It tests for executable requirements, architecture, and design. The advantage of model-based testing is that it can begin almost immediately instead of after completing all other test cycles.

Shift left tools

1. Security Scanning Tools:

These tools streamline the integration of security with DevOps processes by identifying vulnerabilities early in the development cycle. They help ensure that code is secure as it is being developed, rather than waiting until the testing phase. 

Examples of security scanning tools include: 

• Static Application Security Testing (SAST): SAST tools analyze the source code without executing it. They identify potential security issues, coding standards violations, and vulnerabilities. Linters (e.g., ESLint for JavaScript, pylint for Python) fall into this category. 

• Dynamic Application Security Testing (DAST): DAST tools test the application during runtime by simulating attacks. They identify vulnerabilities that can be exploited by attackers. Examples include OWASP ZAP and Burp Suite. 

• Dependency Scanning: These tools check for vulnerabilities in third-party libraries and dependencies used in the application. 

• Secrets Detection: Tools like GitGuardian scan code repositories for exposed secrets (e.g., API keys, passwords) and prevent accidental leaks. 

• Integrated Application Security Testing (IAST): IAST tools combine static and dynamic analysis to provide comprehensive security coverage. 

2. Quality control tools

These tools help ensure and validate that code in the DevOps pipeline is not only secure but also functional and effective. Implementing such tools early in the development process helps guarantee that errors or deficiencies are caught and corrected before reaching production. 

• Static Code Analysis tools: Similar to SAST tools, Static Code Analysis tools assess the source code without executing it to determine whether the code is compliant, bug-free, and generally of good quality. 

• Code review tools: Code review tools help automate the code review process by organizing and displaying code changes, facilitating communication between developers regarding changes, and examining the efficacy of the code review process.

• Continuous integration (CI) tools: CI tools enable developer collaboration by automatically integrating code from multiple developers into a single repository, such as GitHub, GitLab, and Bitbucket.

The application security dividend of shift right and shift left

An important benefit of shifting right is improved application security. “Scanning a static image, either in a repository or in a development environment, can’t give you the same rich insights you can get if you observe the application running in production,” a Dynatrace report on security evolution in the cloud notes. “For example, you don’t get to see what libraries are actually called, how they are used, whether a process is exposed to the Internet, or whether a process interacts with sensitive corporate data.”

The growing use of software containers has complicated aspects of cybersecurity. Containers can obscure the processes running in them, and attackers even containerize exploits. Production testing exposes the behavior of container-based software, even if the contents of containers are obscured. Organizations can also use shift-right testing to test for the presence of novel zero-day exploits, such as Log4Shell.

From a shift left perspective, security testing during development helps identify vulnerabilities as early in the life cycle as possible, when they are easiest to remediate.

Shift left, shift right, or both?

As organizations modernize their application stacks around cloud-native constructs like microservices and containers, a best practice is to adopt both shift left and shift right strategies. Shift left testing reduces software defects and speeds up time to market. Shift right better ensures reliability in production by testing under real-world conditions to affirm that software is doing its job where it matters most: serving the user. The combination of the two moves the organization closer to the ultimate goal of continuous integration and continuous delivery using automation and interdisciplinary communication. The result is a faster and more agile organization.

An essential capability for comprehensive shift left and shift right testing is automated full-stack observability and monitoring. With the capacity for end-to-end analysis at all phases of development, DevOps teams can monitor all requests and processes. Automatic analysis of all services across a sprawling complex of multicloud applications from a single interface enables teams to automate. Testers can push deployment information and metadata to the monitoring environment using scripts to track builds, revisions, and configuration changes.

The better an automated DevOps platform understands the full context of an issue, the better it can detect root causes, flag with the proper parties, and even implement self-healing measures.

Whether your organization has shifted testing left to the development phase or right in production — or simply wants to monitor performance in the field—an AI-driven, full-stack observability solution can take your software development to the next level.

The post Shift left vs shift right: A DevOps mystery solved appeared first on Dynatrace news.

Congratulations! You have set up a DevOps practice. Now, with the hard work done and DevOps metrics and DevOps KPIs in place, you can sit back, relax, and witness the collaboration between your Dev and Ops teams as they deliver better quality software faster.

If only it were that easy.

Why are metrics important in DevOps?

As we look at today’s applications, microservices, and DevOps teams, we see leaders are tasked with supporting complex distributed applications using new technologies spread across systems in multiple locations. Because of this, the way we measure and understand critical services and applications has also changed.

What are DevOps metrics?

DevOps metrics and DevOps KPIs are essential for ensuring your DevOps processes, pipelines, and tooling meet their intended goal. Like any IT or business project, you’ll need to track critical key metrics.

Here are nine key DevOps metrics and DevOps KPIs that will help you meet your goals.

What are the four main DevOps metrics? DORA’s Four Keys

Let’s start with the four most common metrics Google’s DevOps Research and Assessment (DORA) team established known as “The Four Keys.” Through six years of research, the DORA team identified these four key metrics as those that indicate the performance of a DevOps team, ranking them from “low” to “elite,” where elite teams are twice as likely to meet or exceed their organizational performance goals. Let’s dive into how these DevOps KPIs can help your team perform better and deliver better code.

1. Deployment frequency

Deployment frequency measures how often a team successfully releases to production.

As more organizations adopt continuous integration/continuous delivery (CI/CD), teams can release more frequently, often multiple times per day. A high deployment frequency helps organizations deliver bug fixes, improvements, and new features more quickly. It also means developers can receive valuable real-world feedback more quickly, which enables them to prioritize fixes and new features that will have the most impact.

Deployment frequency measures both long-term and short-term efficiency. For example, by measuring deployment frequency daily or weekly, you can determine how efficiently your team is responding to process changes. Tracking deployment frequency over a longer period can indicate whether your deployment velocity is improving over time. It can also indicate any bottlenecks or service delays that need to be addressed.

2. Lead time for changes

Lead time for changes measures the amount of time it takes for committed code to get into production.

This metric is important for understanding how quickly your team responds to specific application-related issues. Shorter lead times are generally better, but a longer lead time doesn’t always indicate an issue. It could just indicate a complex project that naturally takes more time. Lead time for changes helps teams understand how effective their processes are.

To measure lead time for changes, you need to capture when the commit happened and when deployment happened. Two important ways to improve this metric are to implement quality assurance testing throughout multiple development environments and to automate testing and DevOps processes.

3. Change failure rate

Change failure rate measures the percentage of deployments that result in a failure in production that requires a bug fix or roll-back.

Change failure rate looks at how many deployments were attempted and how many of those deployments resulted in failures when released into production. This metric gauges the stability and efficiency of your DevOps processes. To calculate the change failure rate, you need the total count of deployments, and the ability to link them to incident reports resulting from bugs, labels on GitHub incidents, issue management systems, and so on.

A change failure rate above 40% can indicate poor testing procedures, which means teams will need to make more changes than necessary, eroding efficiency.

The goal behind measuring change failure rate is to automate more DevOps processes. Increased automation means released software that’s more consistent and reliable and more likely to be successful in production.

4. Mean time to restore service

Mean time to restore (MTTR) service measures how long it takes an organization to recover from a failure in production.

In a world where 99.999% availability is the standard, measuring MTTR is a crucial practice to ensure resiliency and stability. In the case of unplanned outages or service degradations, MTTR helps teams understand what response processes need improvement. To measure MTTR, you need to know when an incident occurred and when it was effectively resolved. For a clearer picture, it’s also helpful to know what deployment resolved the incident and to analyze user experience data to understand whether service has been restored effectively.

For most systems, an optimum MTTR could be less than one hour while others have an MTTR of less than one day. Anything that takes more than a day could indicate poor alerting or poor monitoring and can result in a larger number of affected systems.

To achieve quick MTTR metrics, deploy software in small increments to reduce risk and deploy automated monitoring solutions to preempt failure.

Five supplemental DevOps KPIs

DORA’s Four Keys make a good foundation to improve the performance of your development practices, but they are only a start. Here are five more DevOps KPIs to help your team perform more optimally.

5. Defect escape rate

Defect escape velocity measures the number of bugs that “escape” testing and are released into production.

This metric helps you determine the effectiveness of your testing procedures and the overall quality of your software. A high defect escape rate indicates processes need improvement and more automation, and a lower rate (preferably close to zero) indicates a well-functioning testing program and high-quality software.

To gain visibility into this metric, you need to track all defects found in your released code and software. This means looking at defects in development/QA and in production so you get insight into any defects that got through to production from development and QA. In general, organizations should strive to find 90% of defects in QA before a release hits production.

6. Mean time to detect

Mean time to detect (MTTD) measures the average time between when an incident starts and when it’s discovered.

Of the other DevOps metrics, this measurement helps determine the effectiveness of your monitoring and detection capabilities in support of system reliability and availability. Measuring MTTD is influenced by other DevOps KPI metrics, including mean time to failure (MTTF) and mean time to recovery (MTTR). To calculate MTTD, add all the incident detection times for a given team or project and divide by the total number of incidents.

The challenge of MTTD is having accurate knowledge of when an IT incident starts, which requires the ability to analyze and evaluate historical infrastructure KPI data.

7. Percentage of code covered by automated tests

Percentage of code covered by automated tests measures the proportion of code subject to automated testing.

Automated testing generally indicates greater code stability, although manual testing can still play a role in software development. A higher percentage of code covered by automated testing is the goal, although it is healthy to always have some broken tests–it is important that teams write code to work as intended, not just to pass tests.

8. App availability

Application availability measures the proportion of time an application is fully functioning and accessible to meet end-user needs.

A highly available system is designed to meet the gold standard KPI of five 9s (99.999%). To accurately measure application availability, first make sure you can accurately measure the true end-user experience, not just network statistics. While teams don’t always expect downtime, they often plan for it because of maintenance. Planned downtime makes communication between DevOps and SRE team members crucial to resolve unforeseen failures and ensure both the frontend and backend operate seamlessly.

9. Application usage and traffic

Application usage and traffic monitors the number of users accessing your system and informs many other metrics, including system uptime.

Once you’ve deployed your software, you’ll want to know how many users are accessing your system and the number of transactions occurring to ensure everything is operating normally.

For example, if your application gets too much traffic and usage, it could fail under the pressure. Similarly, these metrics can be useful for indirect feedback on deployments – new and existing. If there’s a dip in usage and/or traffic, this could be feedback that a change you’ve made hasn’t been well received by the end-user.

Having DevOps KPIs such as these application usages and traffic metrics allows you to see if something is wrong and when, for example, there’s a strange spike in traffic or other abnormal usage or traffic metrics. Similarly, you can monitor usage and traffic against microservices that are specifically supporting critical applications. As a result, your DevOps team can use these metrics to ensure systems operate as they should and to take the appropriate action, for example, revert to a previous version to keep end-users happy.

How to monitor DevOps metrics and KPIs for cloud resources and distributed systems

A successful DevOps practice requires teams to monitor a consistent and meaningful set of DevOps KPIs to ensure that processes, pipelines, and tooling meet the intended goal of delivering better software faster.

To help teams gain visibility into DevOps tools and processes, Dynatrace provides automatic full-stack observability into multi-cloud environments. Dynatrace DevOps solutions automatically makes sense of data from across the DevOps life cycle with AI at its core — from development to production. This ability to provide precise answers and integrate with more than 500 technologies enables teams to customize and fine-tune DevOps metrics, automate more DevOps processes, and drive greater efficiency for great user experiences.

The post 9 key DevOps metrics for success appeared first on Dynatrace news.

The surge in demand for digital services since 2020 is a trend that is here to stay and only expected to accelerate. To compete, organizations have to achieve both speed and reliability when bringing new products and services to market. To meet this demand, organizations are adopting DevOps practices, such as continuous integration and continuous delivery, and the related practice of continuous deployment, referred to collectively as CI/CD.

CI/CD is a series of interconnected processes that empower developers to build quality software through well-aligned and automated development, testing, delivery, and deployment. Together, these practices ensure better collaboration and greater efficiency for DevOps teams throughout the software development life cycle.

Here’s what you need to know about these software development practices, how they relate to each other, and how they benefit DevOps teams as they optimize and automate more processes to achieve ever-faster time to value for customers.

Continuous integration streamlines development

Continuous integration (CI) is a software development practice that streamlines the internal process of creating software. With CI, multiple software developers can work on different features or modules of the same application and individually commit their updates to a shared code repository as they complete them, often many times a day. When they check in their code, the build management system automatically creates a build and tests it. If the test fails, the system notifies the team to fix the code. This practice helps software teams quickly detect and resolve any bugs that come up during the development process.

Continuous integration also prevents “merge hell,” which can happen when two or more developers inadvertently make conflicting changes that break the build when the lines are merged back into the master branch. Continuous integration also avoids teams having to reconcile substantial amounts of conflicting or redundant code, which can require code freezes or even a dedicated integration stage in the pipeline.

When software development teams continuously integrate incremental changes, each developer is free to make changes without worrying about throwing someone else’s work off track or about their own work being stepped on by another. Smooth, regular merging helps software development teams complete projects more quickly and efficiently. It also ensures there is always a testable, up-to-date build that will properly compile, which is critical for frequent and rigorous application testing.

This approach saves time and resources that would otherwise be spent fixing issues later in the software development life cycle, or worse, after release, when issues are much more difficult to address.

Continuous delivery ensures code is always ready to deploy

Continuous delivery (CD) is a process in which DevOps teams develop and deliver complete portions of software to a repository, such as GitHub or a container registry, in short, controlled cycles. Continuous delivery seeks to make releases regular and predictable events for DevOps staff, and seamless for end-users. Another goal of continuous delivery is to always keep code in a deployable state so updates can go live at a moment’s notice with little or no issues.

DevOps teams can automate CI/CD pipelines to move code through the appropriate environments with no human input, which accelerates the build, test, and deployment stages of software development — or any additional stages they have in place, depending on the processes they use.

For example, when a feature is ready for client demonstration, the DevOps team can have a CD tool automatically deploy it to a test server so the client can see how it works and provide feedback before it is released to the production server.

Continuous deployment ensures customers always have the latest

Continuous deployment (also CD) is an extension of continuous delivery in which builds that pass testing are automatically deployed directly to production environments on a daily, or even hourly, basis.

Continuous deployment hastens the feedback loop with your customers and eases the burdens on operations teams by automating the next stage in the pipeline.

Once an organization’s continuous delivery practices are stable and mature, they often adopt automated continuous deployment and testing in a tiered “blue-green” sequence. In this scenario, the new build (green) is deployed in parallel with the existing build (blue) to ensure it works before the blue build is retired. Teams can also use a phased “canary” deployment approach, where the new build is gradually phased in to replace the existing build.

Continuous delivery vs. continuous deployment: which is it in CI/CD?

Continuous delivery (CD) often gets confused with continuous deployment (CD). If CD stands for continuous delivery and continuous deployment, which is it when people refer to CI/CD practices?

In short, it’s either continuous delivery, or continuous delivery and continuous deployment. Although related, the two terms refer to automated processes that happen at different points in the delivery workflow.

You can think of adopting these practices as three steps in a continuum of DevOps automation:

1. Continuous integration to streamline development and automate building and testing. It’s a way for developers to work on the same code base at the same time.
2. Continuous delivery to automate the process of delivering completed code blocks into the main branch, where it can be deployed to a production environment by the operations team. It’s a way for the development part of a DevOps team to automate the process of testing and committing code.
3. Continuous deployment to automate the process of deploying completed, tested code into a production environment. It’s a way for the operations part of a DevOps team to automate the process of deploying new code to customers.

When an organization first adopts agile DevOps practices, they often start out with continuous integration, and mature quickly into continuous delivery, thus achieving CI/CD. Many organizations stop here, preferring to release production code manually. Others progress into continuous deployment so they can automate the entire software development, delivery, and deployment pipeline.

The benefits of CI/CD

The main benefit of continuous integration and continuous delivery is that it reduces the time required to develop applications and features. This gives companies a competitive advantage over businesses that still take a manual approach to software development.

Besides the competitive advantage, CI/CD enables agile development teams to constantly iterate and release new features. These practices ensure everyone is working with the same version of the code as they develop and refine features and functions. With the logistics of integrating and testing builds automated, engineers can focus on what they do best: coding.

If teams have implemented continuous deployment, code can spend less time waiting for testing and deployment and more time in production, which accelerates feedback from users and, ultimately, improves business outcomes.

Increased efficiency, reduced time to market, and quicker innovation make CI/CD an attractive proposition for organizations of all types.

The drive to optimize and automate more CI/CD processes

To meet competitive demands, organizations know they must release new products and services more quickly than ever without compromising quality or reliability. They know basic CI/CD practices are fundamental to DevOps and DevSecOps initiatives that look to develop applications with more effective collaboration and greater precision.

Teams that implement CI/CD practices successfully rely on many tools and methods to manage features, versions, testing, and builds. Automation happens at every stage of the pipeline, from building, packaging, and testing to pushing applications to different production environments.

To accelerate the development pipeline for ever-faster releases with less risk, teams need continuous automation and advanced AI-driven observability across all the tools in their DevOps toolchain so they can automate manual steps and identify quality issues earlier in the software lifecycle.

If your organization uses or is considering adopting CI/CD DevOps processes, a best practice is to implement a full-stack observability platform that can provide code-level visibility of all software builds, apps, and services in your environment, whether they’re in development or deployed to end-users.

The Dynatrace Software Intelligence Platform continuously and automatically monitors the performance of DevOps tools, and seamlessly integrates with CI/CD workflows. With improved collaboration on a single platform and a shared data model, you can ensure your entire team has continuous situational awareness across the lifecycle.

To take a deeper look at continuous delivery, see my next post, What is continuous delivery and what are best practices for implementing it?

The post Continuous integration and continuous delivery (CI/CD): How it enhances DevOps and continuous deployment appeared first on Dynatrace news.

The events of 2020 accelerated the trend of organizations shifting to cloud-native technologies in response to the dramatic increase in demand for online services. Cloud-native environments bring speed and agility to software development and operations (DevOps) practices. But with that speed and agility comes new complications and complexity, all while maintaining performance and reliability with less than 1% down-time per year. That’s where SRE comes in.

So which is it: SRE vs. DevOps, or SRE and DevOps? DevOps is focused on optimizing software development and delivery, and SRE is focused on operations processes. Both practices live by the same overarching tenets. Here we’ll cover the purpose of each and explore how they interact.

DevOps as a philosophy

DevOps has been around longer than SRE, so let’s start there. DevOps is not a specific process, but rather a general collection of flexible software creation and delivery practices that looks to close the gap between software development and IT operations. Its goal is to develop and deliver better software faster by using agile processes, automating as much as possible, and horizontally aligning service-oriented teams instead of siloed by organization.

That’s not always easy. Teams can get entrenched and siloed in familiar manual processes and piecemeal solutions as they roll out new applications. If this is the case, leaders should support their DevOps teams to develop and adopt methods to streamline and automate as many processes as possible.

DevOps is more than processes and automation—it’s a mindset to align teams on a common set of objectives. Organizations often need to do some team building to synthesize the objectives of developers and operations. In general, developers want to create with greater agility, while operations teams want to maintain stability and predictability. The goal is to align teams on this cultural convergence with some reassurance they can both get what they need by joining forces.

SRE as an application of DevOps

DevOps is best thought of as a practical approach to speeding up new software development and delivery. SRE is the transformation of traditional operations practices by using software engineering and DevOps principles to improve the availability, performance, and scalability of releases by building resiliency into apps and infrastructure. As Google’s Ben Treynor explains, “Fundamentally, it’s what happens when you ask a software engineer to design an operations function.”

Site reliability engineers carry forth the DevOps mission by following core practices, such as:

• Using software engineering to solve the operations problem
• Designating and managing Service Level Objectives (SLOs) as availability targets for a service
• Investing in automation and tooling to avoid toil
• Encouraging a shift-left approach, testing earlier in the development lifecycle

What are the differences between SRE and DevOps?

If SRE applies software engineering principles to the operations side of DevOps, then it’s reasonable to expect the two will share a lot in common. As Google explains, both SRE and DevOps require teams to:

• Understand that change is necessary for improvement
• Share values that extend across single teams
• Adopt an “everything as code” philosophy using APIs and version-controlled artifacts
• Value measurability as critical to doing their jobs
• Aim to achieve better velocity through the software delivery pipeline

SRE vs. DevOps

SRE and DevOps don’t compete so much as operate with different scopes in different areas. DevOps doesn’t concern itself with how to run operations at a detailed level. SRE’s purpose, on the other hand, is to apply DevOps principles to enhance the operational processes.

Yes, DevOps is a broad philosophy whose tenets can be applied in other areas (e.g. in Security for DevSecOps or in software delivery to make CI/CD), but in its purest form, it’s associated more with software development and delivery.

SRE applies DevOps principles to enhance operational processes specifically, including the following:

• Availability
• Reduced latency
• Efficiency
• Streamlined change management
• Robust emergency response
• Accurate capacity planning

How can SREs support DevOps?

Many organizations that have already started a DevOps journey are now turning to SRE to optimize operations. If you haven’t implemented either, a best practice to get started is to develop a strategy that incorporates both DevOps and SRE practices.

Adopting these practices is a culture shift. Success requires executive vision and buy-in, internal discussion, and management coordination to support a more integrated way of developing and delivering applications.

At Dynatrace, we believe in advancing customers’ and the industry’s movement toward highly automated, AI-driven DevOps so teams can automate mundane tasks and deliver innovation faster and with less risk. The Dynatrace Software Intelligence Platform helps organizations transform how they develop and manage cloud-native applications, enabling DevOps teams to automate the CI/CD pipeline, and SRE teams to automate testing and remediation. This enables DevOps and SRE teams to identify quality issues earlier in the software life cycle, automate more processes, and build resilience into code on its way to production.

Learn more about how Dynatrace cloud automation can support your DevOps and SRE teams, or activate your free trial today.

The post SRE vs. DevOps: The differences—and similarities appeared first on Dynatrace news.