As more organizations adopt cloud-based computing and the demand for digital services increases, site reliability engineering (SRE) practices have become essential. These practices help organizations meet service level agreements (SLAs) for availability, performance, user experience, and business KPIs.

But what exactly is SRE, and what do site reliability engineers do?

What is site reliability engineering?

Site reliability engineering (SRE) is the practice of applying software engineering principles to operations and infrastructure processes to help organizations create highly reliable and scalable software systems. As a discipline, SRE focuses on improving software system reliability across key categories, including availability, performance, latency, efficiency, capacity, and incident response. Those who perform the tasks involved are known as site reliability engineers.

The term “site reliability engineering” was coined in 2003 by Google VP of Engineering Ben Sloss, who famously noted on his LinkedIn profile, “If Google ever stops working, it’s my fault.” According to Google, “SRE is what you get when you treat operations as a software problem.”

Although every organization and software system is unique, it’s important to understand the fundamentals of SRE and the skills and mindset of its engineers as you think about optimizing the reliability and overall quality of your software.

The benefits of site reliability engineering

Site reliability engineering highlights reliability, scalability, and efficiency. The benefits of SRE include:

• Increased reliability and uptime: SRE focuses on preventing and mitigating incidents to ensure that systems and applications are always available and performant.
• Improved scalability: By optimizing resource usage and minimizing waste, SRE can help organizations scale their infrastructure and applications more efficiently.
• Improved user experience: SRE can ensure that applications and services are always available and responsive, which can directly impact customer satisfaction, brand reputation, and revenue.
• Continuous improvement: SRE highlights the use of data and metrics to identify areas for improvement and drive ongoing optimization and innovation.
• Increased security: SRE can help to guarantee that systems and applications are secure and compliant with industry standards and regulations.
• Predictable performance: By monitoring and analyzing usage patterns, SRE can help to predict and prevent performance issues before they occur, ensuring that systems and applications perform predictably and consistently.
• Cost savings: SRE can reduce costs by automating routine tasks and optimizing resource usage, reducing the need for manual intervention and saving time and money.
• Collaboration between development and operations teams: SRE emphasizes cross-functional teams and shared ownership of reliability and performance, promoting a culture of collaboration and accountability.

Five things to know about site reliability engineering

Implementing SRE may seem like a large undertaking, but the operational shifts it requires offer lasting benefits that promote greater efficiency and provide a positive return on investment.

1. SRE focuses on automation

A major goal of SRE is to reduce duplication or redundancy of effort as much as possible. SRE teams focus on automating manual tasks, such as provisioning access and infrastructure, setting up accounts, and building self-service tools. As a result, development teams can focus on delivering features, and operations teams can focus on managing infrastructure.

Automating processes is even more critical as organizations speed up delivery of new features into production. On one hand, speed comes from DevOps teams who leverage automation to increase continuous integration and continuous delivery (CI/CD). On the other hand, the move to microservice architectures and the adoption of cloud-native technology, containers, Kubernetes, and serverless architectures offer even more ways to delivery smaller changes faster. These methods increase efficiency and speed, but also demand consistent, repeatable processes that reduce risk and provide feedback loops for measuring operations, so teams can identify areas for improvement.

2. SRE bridges the gap between Dev and Ops

Everything the organization does in the value stream process should answer the question “how do we ensure this runs in production reliably?” SREs drive resiliency-based engineering. They can become mentors and ensure that resiliency is a top priority for both developers and operations.

Applying the DevOps mindset and skills to software reliability helps reduce silos between development and operations teams by sharing responsibility for detecting reliability and performance issues early in the development life cycle. Collaboration between developers, operations, and product owners enables site reliability engineers to define and meet uptime and availability targets.

3. SRE drives a “shift-left” mindset

SRE is a constantly evolving discipline, presenting opportunities to build methods, policies, and processes into the delivery pipeline that allow applications to “auto-remediate” or users to solve their own problems. A shift-left mindset means SREs can embed reliability principles from Dev to Ops, baking reliability and resiliency into each process, app, and code change to improve the quality of software that goes to production.

Here are some ways SRE helps to drive a “shift-left” mindset:

• Develop quality gates based on production-level service level objectives (SLOs) to detect issues earlier in the development cycle.
• Automate build testing and validation using service-level indicators (SLIs) and SLOs
• Influence architectural decisions during initial design stages to ensure resiliency and scale at the outset of software development.

The goal is to take early, proactive steps to ensure quality and reliability are built-in from the beginning. SRE can influence processes more broadly and expand to coordinating testing across the enterprise in support of CI/CD practices.

4. SRE builds services and tools to help operations and support

Traditionally, a major goal of operations teams is to improve uptime. This single-dimensional approach looks for the coveted “five nines” of uptime, or 99.999%, which translates to just over five minutes of downtime per year.

But the higher frequency of change in distributed cloud-native environments requires a multi-dimensional approach.

The goal of SRE is to enable higher change rates while maintaining resiliency and that coveted 99.999% uptime. In multicloud environments, resiliency is measured across key metrics such as performance, user experience, responsiveness, conversion rates, and so on. SRE teams need to build and implement services that improve operations and facilitate the release process across all these areas. This can be anything from adjusting monitoring and alerting to making code changes in production. Site reliability engineers often build custom tooling from scratch to meet specific needs in the software delivery or incident management workflow.

Adopting an SRE approach also requires standardizing the technologies and tools teams use. Standardization makes it easier to manage operations and reduces the burden of managing incompatible technologies, which gives teams more time to collaborate and innovate.

5. SRE requires a cultural change

Because SRE is a practice, it requires changing how teams across multiple disciplines communicate, solve problems, and implement solutions. To adopt a successful SRE culture, organizations must adopt new approaches to managing risk. It also means they must adapt governance processes, invest in hiring, and educate a collaborative workforce that’s versed in engineering and operations and learns and adapts quickly.

Organizations can then integrate these skilled engineers at key points in the DevOps lifecycle. In development and testing teams, SRE specialists develop automation that helps developers test early and often without impeding agile delivery schedules. At a system level, SRE specialists develop tooling that coordinates releases and launches, evaluates system architecture readiness, and meets system-wide SLOs. At a governance level, SRE specialists help to define and oversee enterprise architecture, establish best practices, and select tools and resources that support company-wide site reliability.

What does a site reliability engineer do?

To get an expert’s view on what site reliability engineers do, I asked our DevOps Activist, Andi Grabner.

“Site reliability engineers use good practices around software engineering to provide resilient infrastructure and resilient services to their organizations and the people that actually deliver new applications,” he explains. He also notes that SREs often come from traditional operations roles, such as systems engineers who keep systems up and running. “Site reliability engineers ensure systems stay reliable, resilient, and available,” he adds.

Typical expectations for SREs

Typically, SREs are tasked with ensuring that the speed of delivery doesn’t result in security, service, or solution interruption. But as Grabner notes, “Expectations are a little different for every company. There’s no golden rule. Many are responsible for monitoring and observability and maintaining systems — and providing automation to spin up required environments.”

Grabner highlights the role of SREs in providing the frameworks and platforms for service and application deployment. “When things go wrong, SREs often take on the role of first-line defenders if there’s an alert,” he says. “In a great organization, they don’t do it alone — they constantly work with and within individual application teams to deal with apps that are under fire.”

Perhaps the most important role of SREs is architecting resiliency. “You can’t buy resiliency as-a-service,” Grabner observes. “You have to architect for it by building systems that are resilient by design.” This architectural approach recently helped Dynatrace to withstand an AWS outage in Germany. Automatic delivery, resiliency, and auto-remediation helped to ensure critical systems weren’t affected.

Roles and responsibilities of an SRE

The role of an SRE has become invaluable as technology continues to evolve and businesses become more dependent on digital infrastructure. Here are some of the standard responsibilities of an SRE:

1. Monitoring and Alerting
   One of the primary duties of an SRE is to monitor a company’s digital infrastructure. This involves setting up monitoring tools and systems to detect concerns before they become significant problems. SREs set up alert systems that notify the appropriate people when issues are detected.
2. Incident Response
   The SRE responds quickly and effectively when issues are detected by identifying the root cause, developing and implementing a plan, and communicating with relevant stakeholders.
3. Automation and Tooling
   SREs develop and maintain the tools and systems used to manage a company’s digital infrastructure. This includes developing automation scripts to streamline processes and reduce the risk of human error. SREs also identify areas where tooling can be improved and create new tools to meet the changing needs of the business.
4. Capacity Planning
   SREs ensure that a company’s digital infrastructure can meet the needs of the business. This involves analyzing usage patterns to predict and guarantee the required capacity to meet future demand.
5. Collaboration
   SREs work closely with other teams to ensure the company’s digital infrastructure is reliable, scalable, and secure.

What makes a great SRE?

Great SREs are risk-takers, tinkerers, and innovators. They figure out what it takes to scale a system from 100 users to 100,000 users to 1,000,000 users while maintaining uptime and resiliency. They’re systems thinkers who consider how decisions made in development affect production environments, and how the needs of production systems can influence design.

This requires constant testing, accepting failure, and adapting, automating repeatable processes. Successful SREs bring a resiliency and adaptation mindset to every situation.

Grabner highlights the need for SREs to learn from their mistakes. “Some companies run ‘chaos days’ to deal with worst-case scenarios to understand what could happen and how to deal with it,” he says.

Automation is another marker of SRE success. “People excel in this role when they try to automate all the tasks that can and must be automated,” says Grabner. “This frees them up to deliver true innovation.” He notes that while anyone can innovate under the right circumstances, teams are often held back by the “toil” of manual and repetitive tasks. “Your goal is to automate yourself out of your current role and into your next role.”

Finally, Grabner made it clear that SREs can’t operate in isolation. “You need to let people educate themselves with new technologies and practices,” he says. “Show the world. Don’t keep secrets — be open and share your own learnings, along with learning from others. There are a lot of great conferences out there — it’s worth getting inspired by what others do and inspiring others with what you do.”

DevOps vs. SRE

Where DevOps teams focus on streamlining change, SREs help ensure these changes don’t increase overall failure rates. In effect, they’re two sides of the same coin: DevOps automates speed, while SRE automates reliability. “It’s a balance between speed and safety,” Grabner says.

He sees DevOps processes as moving left to right along the development life cycle, using automation to speed up new capabilities that are typically measured by deployment frequency and lead time for changes. In comparison, SRE moves right to left using production-level requirements in development, with a focus on limiting failure rates and reducing the time required to restore service. “SRE is about making sure that even though there is a lot of change, these changes don’t break things.”

Grabner sees SRE and DevOps overlapping when it comes to SLOs. “SLOs are all about supporting business goals,” he said. “Companies may need systems at 99% reliability. They may want to increase their user base or improve the end-user experience.” Satisfying these goals is the role of DevOps. “But underneath these goals are technical goals that are specific to your objectives,” he says. “They contribute to business success with the right features at the right time and help you cope with change.” Delivering on these goals is the job of SRE staff. As a result, “SLOs are a great way to bring DevOps and SREs together.”

In another blog post you can learn more about the differences and similarities of SRE vs DevOps.

Solving for site reliability

Site reliability isn’t and never will be a “solved problem”. New services and applications combined with evolving enterprise demands mean there’s always work for SRE teams, and there’s always room for improvement.

As Grabner notes, when it comes to improving SRE impact, “The biggest thing is to be open and share your own learnings, along with learning from others. There are a lot of great conferences out there — it’s worth getting inspired by what others do and inspiring others with what you do.” He also highlights the need to learn from your mistakes. “Some companies run ‘chaos days’ to deal with worst-case scenarios to understand what could happen and how to deal with it.” Finally, Grabner made it clear that SREs can’t operate in isolation. “You need to let people educate themselves with new technologies and practices. Show the world. Don’t keep secrets — don’t see it as a silo.”

Looking for a solution to up-level your SRE practices? Dynatrace can help. Providing automatic and intelligent observability for even the most complex distributed cloud environments, the Dynatrace Software Intelligence Platform empowers SRE and DevOps teams to identify problems before they occur. Driven by continuous automation with AI at its core, Dynatrace delivers precise root-cause answers to site reliability issues at every step of the software development lifecycle. From early development in pre-production environments through delivery and operations in production environments, Dynatrace helps SRE teams improve reliability, availability, and latency, and mitigate the business impact of service outages and slowdowns.

The post What is SRE (site reliability engineering)? And what do site reliability engineers do? appeared first on Dynatrace news.

Shift left and shift right are core testing concepts of the agile DevOps methodology. This methodology speeds up application development by releasing small builds frequently as code evolves. As part of the continuous cycle of progressive delivery, DevOps teams are also adopting shift left and shift right principles to ensure software quality in these dynamic environments.

All this shifting may sound abstract, but I’ll explain how this software verification approach benefits DevOps methods and outcomes—and makes software more reliable.

In DevOps, what is shift left? And what is shift right?

To understand shift left and shift right, consider the software development cycle as a continuum, or infinity loop, from left to right.

On the left side of the loop, teams plan, develop, and test software in pre-production. The main concern in pre-production on the left side of the loop is building software that meets design criteria.

When teams release software into production on the right side of the loop, they make it available to users. The concern in production is maintaining software that meets business goals and reliability criteria.

What does shift left mean in DevOps?

Shift left is the practice of moving testing, quality, and performance evaluation early in the development process, often before any code is written.

Shift-left testing helps teams anticipate changes affecting performance or other delivery processes that arise during the development process. With shift-left testing, teams verify APIs, container configurations, interactions between microservices, and container configurations.

Why is shift left important?

In addition to testing for functionality, shift left testing also checks that software meets customer requirements. This enables developers and stakeholders to identify improvements that could enhance customer experience and functionality. Making these changes early in the development process reduces the cost of making them after the code is released.

As a result, shift left testing has become increasingly important as teams face pressure to deliver software faster and more frequently with higher quality. Shift left speeds up development efficiency and reduces costs by detecting and addressing software defects earlier in the development cycle before they get to production.

The benefits of shift left

The concept of shift left has been gaining traction in the world of software development and testing. This approach, which introduces testing and quality assurance early in the development lifecycle, offers several considerable benefits. 

1. Early Bug Detection: By testing early in the development process, teams can identify and address bugs when they are easier and less costly to fix. This not only improves the quality of the final product but also reduces the time and resources spent on debugging and rework.
2. Faster Time to Market: With shift left, testing is integrated into every stage of development. This continuous testing allows for faster feedback and quicker iterations, which can significantly reduce the overall time to market.
3. Improved Customer Satisfaction: By ensuring that quality is a priority from the outset, shift left helps to deliver a product that meets or exceeds customer expectations. This can lead to higher customer satisfaction and loyalty.
4. Reduced Costs: While implementing shift left may require an initial investment, the cost savings from reduced rework and faster time to market can quickly offset this. In addition, by catching and fixing issues early, teams can avoid the potentially high costs associated with late-stage or post-release fixes.
5. Enhanced Collaboration: Shift left encourages collaboration between testers, developers, and other stakeholders from the beginning of the project. This collaborative approach can lead to better communication, shared understanding of the product, and a better end result.

What does shift right mean in DevOps?

Shift right is the practice of performing testing, quality, and performance evaluation in production under real-world conditions. Shift-right methods ensure that applications running in production can withstand real user load while ensuring the same high levels of quality.

With shift right, DevOps teams test a built application to ensure performance, resilience, and software reliability. The goal is to detect and remediate issues that would be difficult to anticipate in development environments.

Why is shift right important?

With shift right, teams can test code in an environment that mimics real-world production conditions that they can’t simulate in development. This practice enables teams to catch runtime issues before customers do. To automate part of the process, teams can use application programming interface (API) calls. Organizations can also apply shift-right testing to code configured or monitored in the field. The result is more comprehensive testing coverage that better addresses user experience concerns.

Like shift left testing, the objective of shift right testing is to fail small and fail fast. The assumption is that problems caught early in the pre-deployment environment are easier to solve than issues caught by customers in live production.

Shift right testing is especially useful for organizations practicing progressive delivery, wherein developers release new software features incrementally to minimize the impact of unforeseen issues. Testing in a production-ready environment is a crucial final phase before declaring features ready for prime-time.

The benefits of shift right

Shift right is a concept that extends testing into the production environment. It involves monitoring and testing systems in production to gain insights and improve quality, offering several significant advantages.

1. Realistic User Testing: Shift right testing involves testing with real users. By including them in the process, software professionals gather valuable feedback about how users interact with the software. This helps make informed decisions about future improvements.
2. Continuous Feedback Loop: Shift right testing promotes continuous monitoring and improvement throughout all phases of development. This iterative feedback loop ensures that issues are addressed promptly, leading to better product quality.
3. Test Coverage: By extending testing to real-world scenarios, shift right testing provides broader test coverage. It helps identify issues that might not appear during earlier development stages, resulting in more comprehensive testing.
4. Enhanced Monitoring: Monitoring software behavior in production environments allows for better visibility into its performance, stability, and user experience. This helps teams proactively address any anomalies or performance bottlenecks.
5. Customer-Centric Approach: Shift right testing prioritizes users’ needs and experiences. By involving real users, software teams gain insights that lead to improvements aligned with customer satisfaction.

Why shift right and shift left are critical for microservices architecture

Both shift left and shift right testing have become more important as software is increasingly built from microservices. Instead of applications being built and run as a monolith, microservices are loosely coupled functions called at runtime. How microservices-based applications perform depends on individual service responsiveness, which complicates testing in a simulated environment.

Shift left testing allows teams to observe how services interact in a simulated environment and detect potential performance impacts before deploying the code.

Shift right testing at or near the deployment point allows teams to observe real-world forces and measure their impact. Shift-right tests typically cover functionality, performance, failure tolerance, and user experience. Teams often automate such production-environment testing and translate feedback into technical specifications for developers.

With shift-left and shift-right testing, testers can isolate issues so teams can fix and tackle improvements in parallel. As an application becomes more stable, teams can start testing and optimizing performance. Notably, shifting left and right have become an important practice in Agile software development. Together, they’re part of the continuous feedback loop that characterizes DevOps.

Types of shift right tests

A shift right approach may enlist various types of test suites. The following are tests your team might find useful:

• A/B testing. Web designers commonly use this method. Users see two versions of a page, and testers measure which generates a greater response. Teams often conduct this type of test in a production environment to gather real-world feedback.
• Synthetic monitoring. Synthetic monitoring is a form of shift-right testing that uses software tools to emulate the paths users might take when engaging with an application. With synthetic monitoring, you can automatically track application uptime and how your application responds to typical user behavior. It uses scripts to generate simulated user behavior for various scenarios, geographic locations, device types, and other variables.
• Chaos engineering. With chaos engineering, also called chaos testing, developers intentionally “break” the application by introducing errors to determine how well it recovers from disruption. DevOps and IT teams set up monitoring tools to see precisely how the application responds to different types of stresses. Teams perform these tests in a controlled production environment to minimize the impact on mission-critical systems.
• Canary releases. This strategy is named for the canaries that miners use to lower into coal mines to detect toxic gases. Technology has thankfully rendered this inhumane tactic obsolete. However, the term survives to describe slowly rolling out changes to a small subset of instances for testing before teams apply them to the full infrastructure.
• Blue-green deployment. With a blue-green deployment, an organization runs two nearly identical production environments, shifting users (real or synthetic) between the two as they make small changes to one or the other. This practice is important to shift-right methodology as it can minimize downtime. It also provides a mechanism for rapid rollback should something go wrong with the latest version.

Shift right tools

Shift right tools focus on the examination of code quality after it has reached production. Such tools are important for determining and analyzing how software behaves in real-word environments, offering teams insights into its actual performance. Examples of shift right tools include: 

• Application Performance Monitoring (APM) tools: APM tools offer metrics, logs, and traces relating to application performance, giving teams insights into how to remediate issues faster and further optimize their applications 

• Digital Experience Monitoring (DEM) tools: DEM tools provide insights into the user experience (UX) of an application. These tools may offer: 

• Real-user monitoring (RUM) for analyzing the behavior and experience of real-world users. 

• Synthetic monitoring that performs artificially simulated user journeys to test for any UX bottlenecks or deficiencies 

• Session replay that provides video-like documentation of real user journeys 

• Security monitoring and incident response tools: These tools help detect, manage, and communicate security incidents and facilitate the response process. They also assist teams in prioritization processes to ensure that the most business-impacting vulnerabilities are addressed first. 

• Chaos engineering tools: These tools introduce simulated failures to test the resiliency of code and the efficacy of response processes to such failures. This helps inform teams of any code deficiencies and opens room for improvement 

• Feature flagging tools: These tools provide capabilities to toggle certain features of a deployment on and off without deploying any additional code. Such tools offer a greater level of control to developers, as they allow for more experimentation and finetuning of the user experience. Feature flagging provides increased reliability in feature rollouts and helps to ensure that a deployment provides the best user experience possible. 

Types of shift left tests

The four basic types of shift left testing are the following:

• Traditional shift left testing. The traditional approach focuses on unit and integration testing, typically involving APIs and cross-browser compatibility. This approach focuses more on unit testing small pieces of code to deliver information early and often rather than more systems-level operational and acceptance testing.
• Incremental shift left testing. This is a popular approach for teams migrating from a waterfall development approach to one that breaks complex projects into smaller pieces. In this scenario, teams conduct systems-level operational and acceptance testing on a smaller, more incremental scale. Incremental testing is popular for validating large and complex systems.
• Agile/DevOps testing. This is a series of short, continuous sprints during the development stage. This approach does not intend to address operational performance but rather validate adherence to basic requirements architecture.
• Model-based shift left. Model-based testing aims to reduce errors introduced during the requirements definition, architecture, and design phases. It tests for executable requirements, architecture, and design. The advantage of model-based testing is that it can begin almost immediately instead of after completing all other test cycles.

Shift left tools

1. Security Scanning Tools:

These tools streamline the integration of security with DevOps processes by identifying vulnerabilities early in the development cycle. They help ensure that code is secure as it is being developed, rather than waiting until the testing phase. 

Examples of security scanning tools include: 

• Static Application Security Testing (SAST): SAST tools analyze the source code without executing it. They identify potential security issues, coding standards violations, and vulnerabilities. Linters (e.g., ESLint for JavaScript, pylint for Python) fall into this category. 

• Dynamic Application Security Testing (DAST): DAST tools test the application during runtime by simulating attacks. They identify vulnerabilities that can be exploited by attackers. Examples include OWASP ZAP and Burp Suite. 

• Dependency Scanning: These tools check for vulnerabilities in third-party libraries and dependencies used in the application. 

• Secrets Detection: Tools like GitGuardian scan code repositories for exposed secrets (e.g., API keys, passwords) and prevent accidental leaks. 

• Integrated Application Security Testing (IAST): IAST tools combine static and dynamic analysis to provide comprehensive security coverage. 

2. Quality control tools

These tools help ensure and validate that code in the DevOps pipeline is not only secure but also functional and effective. Implementing such tools early in the development process helps guarantee that errors or deficiencies are caught and corrected before reaching production. 

• Static Code Analysis tools: Similar to SAST tools, Static Code Analysis tools assess the source code without executing it to determine whether the code is compliant, bug-free, and generally of good quality. 

• Code review tools: Code review tools help automate the code review process by organizing and displaying code changes, facilitating communication between developers regarding changes, and examining the efficacy of the code review process.

• Continuous integration (CI) tools: CI tools enable developer collaboration by automatically integrating code from multiple developers into a single repository, such as GitHub, GitLab, and Bitbucket.

The application security dividend of shift right and shift left

An important benefit of shifting right is improved application security. “Scanning a static image, either in a repository or in a development environment, can’t give you the same rich insights you can get if you observe the application running in production,” a Dynatrace report on security evolution in the cloud notes. “For example, you don’t get to see what libraries are actually called, how they are used, whether a process is exposed to the Internet, or whether a process interacts with sensitive corporate data.”

The growing use of software containers has complicated aspects of cybersecurity. Containers can obscure the processes running in them, and attackers even containerize exploits. Production testing exposes the behavior of container-based software, even if the contents of containers are obscured. Organizations can also use shift-right testing to test for the presence of novel zero-day exploits, such as Log4Shell.

From a shift left perspective, security testing during development helps identify vulnerabilities as early in the life cycle as possible, when they are easiest to remediate.

Shift left, shift right, or both?

As organizations modernize their application stacks around cloud-native constructs like microservices and containers, a best practice is to adopt both shift left and shift right strategies. Shift left testing reduces software defects and speeds up time to market. Shift right better ensures reliability in production by testing under real-world conditions to affirm that software is doing its job where it matters most: serving the user. The combination of the two moves the organization closer to the ultimate goal of continuous integration and continuous delivery using automation and interdisciplinary communication. The result is a faster and more agile organization.

An essential capability for comprehensive shift left and shift right testing is automated full-stack observability and monitoring. With the capacity for end-to-end analysis at all phases of development, DevOps teams can monitor all requests and processes. Automatic analysis of all services across a sprawling complex of multicloud applications from a single interface enables teams to automate. Testers can push deployment information and metadata to the monitoring environment using scripts to track builds, revisions, and configuration changes.

The better an automated DevOps platform understands the full context of an issue, the better it can detect root causes, flag with the proper parties, and even implement self-healing measures.

Whether your organization has shifted testing left to the development phase or right in production — or simply wants to monitor performance in the field—an AI-driven, full-stack observability solution can take your software development to the next level.

The post Shift left vs shift right: A DevOps mystery solved appeared first on Dynatrace news.

Since the term artificial intelligence for IT operations (AIOps) was coined by Gartner in 2016, organizations have considered it a good strategy to adopt an AIOps platform or AIOps tools. These tools can help manage and automate anomaly detection and incident response for IT operations in production environments.

But as organizations adopt CI/CD practices and layer in a growing array of cloud-native solutions and open-source technologies to their DevOps toolchains, it’s becoming clear that AIOps can unlock value along the entire digital value chain.

Data in modern cloud-native environments is a continuum—from software development through service delivery all the way to customer interactions. Everything that happens provides telemetry to help teams discover root causes, inform decisions, and automate processes.

In this increasingly integrated landscape, organizations benefit not just from an AIOps platform, but an all-in-one observability, deterministic AI, and analytics platform—a software intelligence platform—to continuously automate, analyze, predict, and remediate IT issues while navigating their digital acceleration journey.

AI applied to cloud ops

As the size and complexity of distributed cloud computing systems continue to grow, typical AIOps approaches to monitoring, diagnosing, and repairing software are not scaling. Traditional AIOps relies on correlating data in order to reduce alerts, which is slow and inaccurate and does little to identify root causes.

An intelligent AIOps platform with end-to-end observability can leverage AI-based algorithms and real-time data analytics to automate triaging, response, and remediation for common IT issues, including unexpected downtime, system latency, or determining why a Kubernetes pod was terminated.

An integrated platform that includes AIOps, observability, and analytics can consume and analyze the increasing volume of cloud data to automate and optimize these routine monitoring and management tasks. Such an observability-based AIOps platform can also provide advanced insight for IT and DevOps, while reducing mean time to resolution (MTTR) and speeding up mean time to discovery (MTTD). With end-to-end visibility into multicloud environments, an intelligent AIOps platform with advanced analytics enables faster innovation, higher quality, more efficiency, and ultimately, better business outcomes.

In addition to driving enterprise automation, there are six key capabilities an all-in-one AIOps platform approach delivers:

1. Alert management
   • Replaces monitoring tool alert storms with accurate, reliable root-cause analysis.
   • Eliminates up to 90% of false alarms and reduces noise with deterministic AI fault tree analysis.
   • Observes, analyzes, and enables automated response in near-real time.
2. Automation
   • Contextualizes and processes large volumes of operational data.
   • Uses this high-fidelity, context-rich collected data to create real-time topology and service flow maps.
   • Provides analysis and AI-powered insights across the application lifecycle.
   • Continuously discovers changes to environments, apps, and services.
3. Incident prioritization and routing
   • Delivers relevant insights to the right people at the right time, providing precise answers with root-cause determination, prioritized by business impact.
4. Event causation
   • Uses causation-based AI to point directly to the root cause and impact of a failed test run, an application slowdown, or system outage, or to drive decisions about whether to release a piece of software.
5. Predictive analytics
   • Monitors the entire technology stack end-to-end to predict and prevent future disruptions before they occur.
6. Auto-remediation
   • Automates anomaly detection, problem notification, and self-remediation with full-stack monitoring and integration with workflow automation platforms, such as ServiceNow and Jenkins.

Why AIOps needs to “shift-left”

With the volume of data increasing, and the demand for services rocketing upward, the need for AIOps is no longer limited to IT operations. As DevOps and SRE practices mature, pre-production workflows need AIOps capabilities just as acutely.

A typical continuous integration/continuous delivery (CI/CD) pipeline follows the following sequence:

1. Source — creating source code
2. Build — compiling the application
3. Test — testing code for functionality
4. Release — pushing code to the repository
5. Deploy — moving code to production

The term “shift-left” refers to the practice of performing a task at an earlier stage of development before it goes to production, such as automated testing at the source phase instead of when code is ready to be released.

Shift-left applied to AIOps integrates AI into the full DevOps lifecycle, including data ingestion, building code, and testing for enhanced software quality and deeper root-cause analysis before code is deployed to production. A software intelligence platform that includes end-to-end observability in its approach to AIOps delivers continuous alert and incident management, automatically observes and identifies anomalies in CI/CD pipelines, and prevents issues from reaching the production stage, resulting in more efficient builds and quicker, higher-quality releases of new versions of software.

Shifting AIOps left means development teams can easily leverage production service-level objectives (SLOs) as criteria for building quality gates into acceptance testing earlier in the development cycle. It also means teams can initiate auto-remediation for CI/CD workflows by integrating with software configuration and deployment management technologies, such as Chef, Puppet, and Ansible.

Faster time-to-value with an intelligent AIOps platform

An AIOps platform based on continuous discovery and end-to-end observability can detect anomalies before they affect the CI/CD pipeline or impact customer experience and SLOs. It can automate validation processes by using SLO-based quality gates with events, tags, and APIs integrating seamlessly with existing CI/CD workflows — for faster automated deployments and time-to-value.

AI-enhanced alerting and escalation using advanced algorithms based on deterministic fault-tree analysis can automatically route incidents to the appropriate team, empowering them with metadata and context that results in accurate, reliable, and precise root-cause analysis. If automated processes are unable to address a slowdown or outage, DevOps is then given a clear path to remediation, eliminating time spent on “problem triage,” which drives faster innovation and better quality.

A shift-left AIOPs platform approach mitigates the cost of IT downtime

Every CIO and CFO knows IT downtime is costly — potentially adding up to thousands of dollars per minute or more depending on the organization’s size and reach of services. But what may be as significant is the impact downtime and performance issues can have on already pushed-to-the-limit DevOps and IT teams. They are under increasing pressure to maintain system reliability and prevent outages of highly complex, distributed multicloud operations. The constant context switching required to hop from issue to issue also derails focus on mission-critical concerns and distracts teams from their core functions.

By shifting AIOps left using production-based performance criteria as a quality gate earlier in the development cycle, teams can release more resilient software. If an issue arises anywhere in the DevOps workflow, an integrated AIOps platform with all-in-one observability, deterministic AI, and analytics can automatically remediate issues and optimize performance based on system health and user demands — preventing or minimizing the duration of outages and reducing costs. It also combats IT teams’ top stressors by reducing the need for manual intervention in IT operations and DevOps workflows, helping to prevent burnout and costly employee churn.

Shift AIOps left with an integrated software intelligence platform

Teams undergoing digital transformation are discovering that shifting AIOps left into DevOps and SRE workflows can accelerate and increase the effectiveness of their DevOps and SRE initiatives.

To help developers pinpoint problems and automate more processes during the development, test, and delivery phases of DevOps, Dynatrace seamlessly integrates AIOps into the CI/CD pipeline, bringing fault-tree analysis to pre-production workflows. Shifting AIOps left enables developers to discover and auto-remediate issues in pre-production so they can optimize processes and deliver higher quality code to production.

Using the Cloud Automation control plane—powered by Keptn, an open-source technology for cloud-native application life-cycle orchestration—Dynatrace provides release analysis, version awareness, and SLO-based quality gates so teams can automate releases at all stages of the DevOps pipeline. By integrating with DevOps tools like Chef, Puppet and Ansible, Dynatrace can execute closed-loop remediation workflows or orchestrate ITSM tools to trigger incident management workflows.

To learn more about how Dynatrace approaches AIOps, see the eBook: AIOps Done Right.

Dynatrace was also named a leader in AIOps in the Forrester Wave. Read the report here.

The post How an AIOps platform can shift left–and why it should appeared first on Dynatrace news.

Site Reliability Engineering is an exciting discipline in our industry. There are many aspects to building reliable and resilient systems which I won’t be able to cover in a single blog post. What I learned from many conversations with enterprise organizations is that the term “self-healing” is getting everybody excited. That’s also why I put it into the title, to get you “excited”! However – I have at least two major problems with the way self-healing is sold and thought of:

#1: Self-healing as a term is misleading!

I don’t think that there are any real self-healing systems. I think we have gotten better to automate remediation actions – and by leveraging modern monitoring tools we can execute specific remediation actions in a much smarter and efficient way. Hence, I believe we should talk about Smart- or Auto-Remediation and not necessarily Self-Healing as it is misleading!

#2: Self-healing is not just an operations activity!

In most of my discussions around self-healing, I talk with Ops teams that try to fix issues that materialized in production. While it is very important to keep our systems running even under “chaotic” conditions in production, I strongly believe we should invest just as much time and effort in preventing these issues ever making it into production. That’s where my “Shift-Left SRE” term comes from as I believe we must bake resiliency into our delivery pipelines.

I presented my thoughts on this topic at AWS re:Invent 2018. It was called Shift-Left SRE: Self-Healing with Lambda!

While my talk was targeted for an AWS crowd the concepts, I explained are applicable for any platform, cloud or technology stack.

I structured my presentation in 4 parts which is the same way I’d like to organize this blog:

• Remediation Use Cases
• PREVENT in CI/CD vs REPAIR in Prod
• “Auto Remediation as Code”
• The “Unbreakable Delivery Pipeline”

Let’s dig into it!

Remediation Use Cases

I have to be honest. I never worked in Operations, neither did I work as an SRE. I always worked in Performance Engineering, where I ended up breaking a lot of environments for different reasons, e.g: hitting the system with too much load, not cleaning up log directories before a new run or improper configuration of my test or test environment. When talking to my friends, customers and partners that work in Operations, they tell me similar root causes they see in a production environment.

Let me therefore start with a quick overview of common problem scenarios that impact production environments and the remediation actions that can be taken to bring the system back to normal:

This is a very short list of remediation actions – but – a list that addresses very common and basic issues in todays environments: Process Restarts, Resource (e.g: Disk) Cleanup, Revert Bad Configuration Changes, Scale Up or Down, Switch Blue vs Green, …

While these use cases have been known for years by many teams I talk with, only a handful have invested in some sort of automation. While many have built scripts – these typically get executed manually when a problem is detected. That means – there is a lot of room for improvement!

PREVENT in CI/CD vs REPAIR in Prod

While we must invest in automating these remediation scripts in production we should learn from every problem we detect in production, figure out what the root cause is, how to replicate, measure or detect that root cause in a lower level environment and automate that into the CI/CD Delivery Pipeline as a Quality Gate:

This approach of adding these checks in a pre-production environment requires the teams (SRE Teams, Ops, …) that currently work on auto-remediation scripts for production to also think about how these situations can be detected in a lower level environment as part of the automated delivery pipeline. Here are the tasks that from my perspective define Shifting-Left SRE:

• Take the lessons learned from production
• Reproduce them in Pre-Production through e.g: Chaos Monkey Tests
• Automate the checks into CI/CD Quality Gates

In my presentation at re:Invent, I listed examples on how to detect certain use cases, what the metrics are to watch out for and how to capture these metrics:

Once we have the metrics and the tests to replicate these scenarios in a lower level environment we can add it to the pipeline as quality gate checks:

If you are interested in integrating Dynatrace captured metrics into your Jenkins, Azure, Concourse, Bamboo … pipelines check out our my recent blog posts and YouTube tutorials: Jenkins Performance Signature, Azure DevOps for Dynatrace, AWS DevOps Tutorial!

Auto-Remediation As Code

When using Dynatrace for full-stack monitoring of platforms we do not just receive alerts based on individual metrics that violated a static threshold or baseline, e.g: CPU is high or Disk is Full. The Smartscape Model, the PurePaths and the high-fidelity monitoring data we ingest and forward to our AI allows Dynatrace to detect problems and correlate all relevant events and data points to a single problem instance. All correlated information on that problem is made available to the auto-remediation as code scripts that SRE engineers wrote to handle specific problem root causes. The auto-remediation scripts do not have to do any correlation work but simply focus on fixing the root cause of a single problem vs chasing many loose ends of unrelated events you get from other monitoring or alerting tools.

The following animation visualizes the full problem/incident response workflow Once Dynatrace detects a problem we set the correct incident response actions in motion to inform the impacted teams as well as call auto-remediation as code scripts that address the root causes rather than just symptoms:

In my demo at re:Invent, I showed the self-healing part of my AWS DevOps Tutorial where I deploy a faulty version of a Service that results in a high failure rate impacting end users. During the deployment I pass the link to an AWS Lambda function that the developer (in this case me) wrote to be called in case of an issue in this service (=Auto-Remediation As Code). Once Dynatrace AI detects the issue, it calls the Lambda function that is associated with the latest deployment of the faulty service!

I encourage Operations or SRE Teams to collaborate with the engineering teams so that they can start writing their own remediation code. This code should be battle tested in a pre-prod environment when those problems get simulated that have been seen in production before. This is a great way to build resilience into the code package that comes from engineering and as it has been battle tested in CI/CD the chances of a real problem in production are very small 😊

The “Unbreakable Delivery Pipeline”

Now – if we put all the discussed concepts together in an automated delivery pipeline we can speak of what I coined “The Unbreakable Delivery Pipeline”:

1. We stop potential negative impacting changes early
2. We auto-validate “Auto-Remediation as Code” via a Quality Gate
3. We use “Auto-Remediation as Code” to “self-heal” production

We “Shift-Left” every problem we haven’t yet auto-remediated

What do you “Auto-Remediate as Code” now?

I hope you got a couple of new ideas on how to Shift-Left SRE (Site Reliability Engineering). Working closely with engineers, providing them the frameworks to develop and extend the delivery pipeline to automatically validate and deploy your auto-remediation scripts allows you to build more and better resilient systems.

Leave me a message, drop me an email or message me on LinkedIn or Twitter in case you want to have a share your thoughts on this topic.

The post Shift-Left SRE: Building Self-Healing into your Cloud Delivery Pipeline appeared first on Dynatrace news.