Developers are key stakeholders in modern observability. Their responsibilities have stretched dramatically in the last few years, together with their tasks and challenges. Historically, the industry focused more on operational-driven observability but has since begun to include—amplified by shift-left movements—developers’ unique needs. In this blog post, we will see how Dynatrace harnesses the power of observability and analytics to tailor a new experience to easily extend to the left, allowing developers to solve issues faster, build more efficient software, and ultimately improve developer experience!

What developers want

Developers want to own their code in a distributed, ephemeral, cloud, microservices-based environment. This ownership starts with understanding how their code behaves in all environments, resolving issues, and writing and optimizing code in a high-quality, secure, and timely manner.

Building the dream package

Observability for Developers, the newly introduced offering from Dynatrace, is designed to cater to developers’ specific needs and challenges. It packages the existing Dynatrace capabilities needed by developers in their day-to-day work—such as logs, distributed traces, profiling data, exceptions, and more. It combines all of these with dynamic instrumentation capabilities and experience to tackle production bugs and optimize developer tasks and resources.

The Observability for Developers initial offering includes the following:

This package provides development and cloud-native teams with powerful capabilities:

• Easy data access and exploration: Intuitive dashboarding and advanced log and trace analytics powered by Davis AI make it easy to track and optimize application performance, monitor health, analyze end-user interactions, view historical data, and provide forecasting all in one place.

• Enhanced troubleshooting power: Dynatrace troubleshooting capabilities—backed with AI power and automatic root cause analysis—are now further supported with live production debugging capabilities.
• Live Debugger enables developers to access real-time insights from runtime environments without requiring issue reproduction or redeployments, extract debugging information without performance impact, and leverage contextual insights for rapid problem resolution.
• Enterprise adoption with self-service: To facilitate enterprise adoption while minimizing tool sprawl and data silos, Dynatrace allows observability teams and platform engineers to implement a self-service model for developers. Tailored entry points and integrations with developer portals and integrated development environments (IDEs) unlock easy access to all functionalities while maintaining a curated and compliant approach, adhering to company standards.

Dashboards: Insights at a glance

While writing code is the primary focus of most developers, ensuring that their services and applications maintain a stable and reliable state is essential for their peace of mind.

Dashboards are a great tool for gaining real-time insights into applications by transforming complex data into dynamic, interactive visualizations. In a single view, developers get an instant overview of application performance, system health, logs, problems, deployment status, user interactions, and much more.

They can even combine this with data from external sources or add custom code to address bespoke requirements. Quickly access your data through seamless integration with other Dynatrace® Apps such as Kubernetes, Logs, Clouds, Services, and more. Benefit from dynamic filtering and cross-data-type visualizations to never lose track of your application’s health.

To get started, use one of many ready-made dashboards or start building custom, interactive dashboards within minutes.

Davis AI is your CoPilot

The Problems app offers a comprehensive, AI-assisted problem-triage and remediation solution tailored for the entire software development lifecycle, from development to operations. Visually guiding users and automatically surfacing critical details allows for faster mean time to repair (MTTR). This efficiency is achieved using Davis^® AI to identify root causes, pinpoint relevant details, and visually reproduce them in charts, highlighting the affected deployment.

When a large-scale incident occurs, Davis AI identifies the root cause and connects all relevant log lines. As every developer knows, logs are crucial for uncovering insights and detecting fundamental flaws, such as process crashes or exceptions. When identifying a root cause, Davis causal AI collects all relevant logs and automatically connects them to the underlying issue, making them easily accessible in the Problems app. Additionally, Dynatrace integrates relevant trace data, providing full visibility into complex, microservices-based architectures.

Read our blog post about recent enhancements to Problems, including using Davis CoPilot to get a written summary of issues, as well as tips for problem resolution and how to use generative AI for automatic remediation.

Log Analytics: Get desired logs on the fly, enriched with contextual data

Dynatrace Log Management & Analytics empowers developers and enhances their productivity by supporting them with easy-to-use yet powerful filtering capabilities. The DQL Editor allows power users to craft their own queries, or to translate the filters into DQL snippets.

In any case, far less manual filtering is required as Developers are provided with logs in the context of related entities, traces, and services. This eliminates time-consuming manual log file correlation between otherwise siloed application-, cluster-, and service logs.

Your desired log on the fly, enriched with contextual and AI data

Distributed tracing: Quickly understand dependencies, interactions, and bottlenecks

Dynatrace distributed traces offer a wealth of information, offering nearly anything you can imagine, helping you navigate and understand complex microservices architecture. Traces combine multiple layers of spans and logs in a structured manner, providing you with method names, call arguments, exceptions, execution times, version information, and so much more. Dynatrace provides a detailed view of application transactions across microservices architectures, enabling you to track requests from start to finish. This visibility helps quickly identify bottlenecks and performance issues, reducing mean time to resolution (MTTR). Developers gain insights into service interactions and dependencies, allowing for faster debugging and improved application performance. Distributed tracing enhances user experience and application reliability by optimizing workflows and ensuring seamless operation, empowering developers to deliver high-quality, efficient software solutions.

Live Debugger: Get code-level data from production in real time

The new Dynatrace Live debugger app provides real-time access to code wherever it’s running, from dev to prod environments. Developers can set a non-breaking breakpoint without interfering with the runtime. The captured debug data includes stack trace, variable values, process, and more, drastically reducing debug cycles and enhancing efficiency as error reproduction and time-consuming redeployments are now a thing of the past. Developers can access this capability directly from the Dynatrace platform or by using our native Visual Studio Code and JetBrains IDE plugins. With the ability to analyze live application behavior and track down issues as they occur, developers gain immediate visibility into their code’s execution. This leads to faster resolution of bugs and improved application performance, ultimately empowering teams to deliver high-quality software with greater confidence and improved experience.

Read more about the Live Debugger application in our blog post – Dare to debug production

Streamline your work

As a real-life example, let’s take a look at a situation where some developers are debugging an issue that manifests as an alert for a high CPU issue on a server and is suspected to be a code issue. We start with the newly created Dynatrace Problems app, where we get problem alerts and first impressions from existing logs that are embedded in the Problems app. (Alternatively, we could start our analysis of this high CPU issue in the Services or Logs app.) Upon realizing there is a need for code-level data, we follow the link to Live Debugger to debug the code. Using Live Debugger, we immediately get insights into the running code, including variable values, process and thread information, and even a trace ID for the captured transaction.

Using the trace ID, we can inspect the captured trace directly in the Dynatrace Tracing app, giving us a full view of the end-to-end transaction as it flows through our systems. From there, any service or process can be selected to get more details on the CPU, memory, or running threads for that service, giving insight into low-level details of what is potentially the root cause of the issue.

For a detailed explanation of this example and walkthrough, visit our blog post, Debug complex performance issues in production.

Flatten the learning curve

Dynatrace allows central and platform engineering teams to easily provide curated access to the most critical Dynatrace use cases, ensuring a minimal impact on processes and reduced cognitive load on development teams.

Tailor-made entry points can be achieved using the aforementioned IDE plugins or by extending the Backstage developer portal with the Dynatrace plugin. Additionally, Dynatrace Launchpad offers an easy-to-use, configurable launch page. Follow the guidance provided in our documentation to ensure a quick start.

Summary

It doesn’t matter if you shift left to fix bugs early in your Software Development Life Cycle or shift right to ensure code quality, testing, and release of a new feature.
Enhance your developers’ experience with analytics and automation throughout the software delivery process. This will allow them to understand exactly what’s happening in their code at any given moment, wherever it’s running, from Dev to Production and from bare metal to Kubernetes.

Observability for Developers is generally available except for Traces powered by Grail and Live Debugger, which are currently in preview and will be generally available within the next 90 days.

Useful Resources

• Press Release
• Observability for Developers webpage
• Live Debugger blog
• Documentation
• Jetbrain IDE plugin
• VSCode IDE plugin

The post Dynatrace Observability for Developers saves time with real-time data appeared first on Dynatrace news.

Keeping pace with modern digital transformation requires ensuring that applications are responsive, resilient, and always available amid increased complexity. As a result, site reliability has emerged as a critical success metric for many organizations.

Site reliability engineering (SRE) has recently become a critical discipline in recent years as the world has shifted in favor of web-based interactions. Mobile retail e-commerce spending in the U. S. surpassed $387 billion in 2022, more than double the figure of three years earlier. The volume of travel spending booked online is expected to reach nearly $1.5 trillion by 2027, up from $800 billion in 2021. With so many of their transactions occurring online, customers are becoming more demanding, expecting websites and applications to always perform perfectly. One recent report found that 32% would leave their favorite brand after just one bad experience. Website load times have been found to have a direct correlation with conversion rates.

This shift is leading more organizations to hire site reliability engineers to guarantee the reliability and resiliency of their services. But the transition to SRE maturity is not always easy.

How site reliability engineering affects organizations’ bottom line

SRE applies the disciplines of software engineering to infrastructure management, both on-premises and in the cloud. The practice uses continuous monitoring and high levels of automation in close collaboration with agile development teams to ensure applications are highly available and perform without friction.

According to the emerging trends from the global shift towards web-based interactions, IT infrastructure performance has a dramatic impact on the organizations’ bottom-line business goals. Uptime Institute’s 2022 Outage Analysis report found that over 60% of system outages resulted in at least $100,000 in total losses, up from 39% in 2019. More than one in seven outages cost more than $1 million.

Maintaining reliable uptime and consistent service quality has become more complex as organizations expand their computing footprints across multiple data centers and in the cloud. Microservices-based architectures and software containers enable organizations to deploy and modify applications with unprecedented speed. However, cloud complexity has made software delivery challenging. There are now many more applications, tools, and infrastructure variables that impact an application’s performance and availability.

Understanding the interactions between these factors heavily influences decisions about whether and when to promote a new release into production. That’s why good communication between SREs and DevOps teams is important. By automating and accelerating the service-level objective (SLO) validation process and quickly reacting to regressions in service-level indicators (SLIs), SREs can speed up software delivery and innovation.

Understanding the goal of “five-nines” availability

The guiding principle of SRE has long been “five-nines” availability, meaning systems are operative 99.999% of the time. As organizations distribute workloads among a greater number of cloud environments, that goal has become harder to attain because more variables are involved in the computing equation. The growing amount of data processed at the network edge, where failures are more difficult to prevent, magnifies complexity.

Visibility and automation are two of the most important SRE tools. The Dynatrace 2022 Global CIO Report found that 71% of top IT executives say the explosion of data produced by cloud-native technology stacks is beyond human ability to manage, and more than three-quarters say their IT environment changes once every minute or less. The takeaway is clear: IT environments are now too complex to manage without automation and AI. Without these capabilities, achieving five-nines availability will become close to impossible.

Aligning site reliability goals with business objectives

Because of this, SRE best practices align objectives with business outcomes. The following three metrics are commonly used to measure success:

• Service-level agreements (SLAs). These metrics are the product of an agreement between the service provider and customer that certain measurable levels of service will be delivered.
• Service-level objectives (SLOs). These metrics are the factors and service levels that must be achieved for each activity, function, and process to deliver on the SLA. These can include business metrics, such as conversion rates, as well as technical measures like underlying CPU availability. They’re typically expressed as percentages, such as 99.5% availability.
• Service-level indicators (SLIs). At the lowest level, SLIs provide a view of service availability, latency, performance, and capacity across systems.

5 SRE best practices

Let’s break down SRE best practices into the following five major steps:

1. Start looking for signals

Begin by monitoring the “four golden signals” that were originally outlined in Google’s SRE handbook:

• Latency: the time it takes to serve a request
• Traffic: the total number of requests across the network
• Errors: the number of requests that fail
• Saturation: the load on the network and servers

2. Identify KPIs

Next, create a list of the key performance indicators (KPIs) that are important to the business. These may include technical metrics such as response times to search query referrals, page load times, and error message frequency. They may also include business metrics influenced by performance, such as shopping cart abandonment and page views per customer.

3. Establish SLOs

Drawing on the KPIs, you can now create a list of SLOs. Remember that less is more. SLOs should directly relate to the SLA or business objective. Defining too many SLOs creates more work without a clear business impact.

Make SLOs realistic. If they’re intentionally set low to avoid SLA violations, they won’t provide an accurate picture of how systems are impacting user experience. If set too high, they can drive higher costs and effort for little incremental gain.

4. Identify key stakeholders

Once you’ve ensured SLAs and SLOs are realistic, begin recruiting stakeholders. For example, a panel of customers may occasionally provide feedback on service quality and performance. Business stakeholders can understand how SLOs relate to business results using historical data trends. Everyone needs to agree on SLO targets; otherwise, the organization will risk failing to deliver on its SLAs.

5. Automate workflows wherever possible

Automation is critical to achieving the business agility that digital transformation demands. A powerful observability solution collects relevant SLIs and evaluates SLOs automatically. It can also automatically generate alerts before SLO violation and even automatically repair many problems. Automation also enables tools to move into developers’ hands so they can make decisions about deploying code without needing to involve operations teams.

Automating SRE best practices with Site Reliability Guardian

As part of the early 2023 rollout of its new AppEngine low-code toolset for creating custom, compliant, and intelligent data-driven applications, Dynatrace introduced Site Reliability Guardian (SRG) for automated change impact analysis

AppEngine is a unique solution that consolidates observability, security, and business data with full context and dependency mapping to simplify the creation of intelligent applications and integrations. It enables teams, for the first time, to leverage causal AI for customized applications that address specific use case requirements.

SRG enriches the Dynatrace platform’s value for SREs by automating change impact analysis. It detects regressions and deviations from previously observed behavior across metrics such as latency, traffic, error rates, saturation, security coverage, vulnerability risk levels, and memory consumption.

SRG also enriches the Dynatrace platform’s value to DevOps teams who can automate release validation in pre-production environments to ensure only high-quality, highly secure software moves to production. Should the SRG detect any violations of key objectives or metrics, the CI/CD pipeline tool can halt the delivery of the artifact.

For each critical service or application, SRG can automatically monitor for golden signals, validate SLOs, and probe for security vulnerabilities before and after deployment or configuration changes. It can also send targeted notifications to the service and application owners as needed. The result is safer, more secure releases for DevOps teams and less overhead for SREs.

Keeping the focus on site reliability

SRE is a critical discipline for organizations to master as they navigate the transition to digital business powered by data-driven decisions. Tight integration with business objectives and automation now makes it possible for organizations to proactively monitor their digital presence to ensure the highest levels of availability, responsiveness, and customer experience.

The post Site reliability done right: 5 SRE best practices that deliver on business objectives appeared first on Dynatrace news.

As more organizations adopt cloud-based computing and the demand for digital services increases, site reliability engineering (SRE) practices have become essential. These practices help organizations meet service level agreements (SLAs) for availability, performance, user experience, and business KPIs.

But what exactly is SRE, and what do site reliability engineers do?

What is site reliability engineering?

Site reliability engineering (SRE) is the practice of applying software engineering principles to operations and infrastructure processes to help organizations create highly reliable and scalable software systems. As a discipline, SRE focuses on improving software system reliability across key categories, including availability, performance, latency, efficiency, capacity, and incident response. Those who perform the tasks involved are known as site reliability engineers.

The term “site reliability engineering” was coined in 2003 by Google VP of Engineering Ben Sloss, who famously noted on his LinkedIn profile, “If Google ever stops working, it’s my fault.” According to Google, “SRE is what you get when you treat operations as a software problem.”

Although every organization and software system is unique, it’s important to understand the fundamentals of SRE and the skills and mindset of its engineers as you think about optimizing the reliability and overall quality of your software.

The benefits of site reliability engineering

Site reliability engineering highlights reliability, scalability, and efficiency. The benefits of SRE include:

• Increased reliability and uptime: SRE focuses on preventing and mitigating incidents to ensure that systems and applications are always available and performant.
• Improved scalability: By optimizing resource usage and minimizing waste, SRE can help organizations scale their infrastructure and applications more efficiently.
• Improved user experience: SRE can ensure that applications and services are always available and responsive, which can directly impact customer satisfaction, brand reputation, and revenue.
• Continuous improvement: SRE highlights the use of data and metrics to identify areas for improvement and drive ongoing optimization and innovation.
• Increased security: SRE can help to guarantee that systems and applications are secure and compliant with industry standards and regulations.
• Predictable performance: By monitoring and analyzing usage patterns, SRE can help to predict and prevent performance issues before they occur, ensuring that systems and applications perform predictably and consistently.
• Cost savings: SRE can reduce costs by automating routine tasks and optimizing resource usage, reducing the need for manual intervention and saving time and money.
• Collaboration between development and operations teams: SRE emphasizes cross-functional teams and shared ownership of reliability and performance, promoting a culture of collaboration and accountability.

Five things to know about site reliability engineering

Implementing SRE may seem like a large undertaking, but the operational shifts it requires offer lasting benefits that promote greater efficiency and provide a positive return on investment.

1. SRE focuses on automation

A major goal of SRE is to reduce duplication or redundancy of effort as much as possible. SRE teams focus on automating manual tasks, such as provisioning access and infrastructure, setting up accounts, and building self-service tools. As a result, development teams can focus on delivering features, and operations teams can focus on managing infrastructure.

Automating processes is even more critical as organizations speed up delivery of new features into production. On one hand, speed comes from DevOps teams who leverage automation to increase continuous integration and continuous delivery (CI/CD). On the other hand, the move to microservice architectures and the adoption of cloud-native technology, containers, Kubernetes, and serverless architectures offer even more ways to delivery smaller changes faster. These methods increase efficiency and speed, but also demand consistent, repeatable processes that reduce risk and provide feedback loops for measuring operations, so teams can identify areas for improvement.

2. SRE bridges the gap between Dev and Ops

Everything the organization does in the value stream process should answer the question “how do we ensure this runs in production reliably?” SREs drive resiliency-based engineering. They can become mentors and ensure that resiliency is a top priority for both developers and operations.

Applying the DevOps mindset and skills to software reliability helps reduce silos between development and operations teams by sharing responsibility for detecting reliability and performance issues early in the development life cycle. Collaboration between developers, operations, and product owners enables site reliability engineers to define and meet uptime and availability targets.

3. SRE drives a “shift-left” mindset

SRE is a constantly evolving discipline, presenting opportunities to build methods, policies, and processes into the delivery pipeline that allow applications to “auto-remediate” or users to solve their own problems. A shift-left mindset means SREs can embed reliability principles from Dev to Ops, baking reliability and resiliency into each process, app, and code change to improve the quality of software that goes to production.

Here are some ways SRE helps to drive a “shift-left” mindset:

• Develop quality gates based on production-level service level objectives (SLOs) to detect issues earlier in the development cycle.
• Automate build testing and validation using service-level indicators (SLIs) and SLOs
• Influence architectural decisions during initial design stages to ensure resiliency and scale at the outset of software development.

The goal is to take early, proactive steps to ensure quality and reliability are built-in from the beginning. SRE can influence processes more broadly and expand to coordinating testing across the enterprise in support of CI/CD practices.

4. SRE builds services and tools to help operations and support

Traditionally, a major goal of operations teams is to improve uptime. This single-dimensional approach looks for the coveted “five nines” of uptime, or 99.999%, which translates to just over five minutes of downtime per year.

But the higher frequency of change in distributed cloud-native environments requires a multi-dimensional approach.

The goal of SRE is to enable higher change rates while maintaining resiliency and that coveted 99.999% uptime. In multicloud environments, resiliency is measured across key metrics such as performance, user experience, responsiveness, conversion rates, and so on. SRE teams need to build and implement services that improve operations and facilitate the release process across all these areas. This can be anything from adjusting monitoring and alerting to making code changes in production. Site reliability engineers often build custom tooling from scratch to meet specific needs in the software delivery or incident management workflow.

Adopting an SRE approach also requires standardizing the technologies and tools teams use. Standardization makes it easier to manage operations and reduces the burden of managing incompatible technologies, which gives teams more time to collaborate and innovate.

5. SRE requires a cultural change

Because SRE is a practice, it requires changing how teams across multiple disciplines communicate, solve problems, and implement solutions. To adopt a successful SRE culture, organizations must adopt new approaches to managing risk. It also means they must adapt governance processes, invest in hiring, and educate a collaborative workforce that’s versed in engineering and operations and learns and adapts quickly.

Organizations can then integrate these skilled engineers at key points in the DevOps lifecycle. In development and testing teams, SRE specialists develop automation that helps developers test early and often without impeding agile delivery schedules. At a system level, SRE specialists develop tooling that coordinates releases and launches, evaluates system architecture readiness, and meets system-wide SLOs. At a governance level, SRE specialists help to define and oversee enterprise architecture, establish best practices, and select tools and resources that support company-wide site reliability.

What does a site reliability engineer do?

To get an expert’s view on what site reliability engineers do, I asked our DevOps Activist, Andi Grabner.

“Site reliability engineers use good practices around software engineering to provide resilient infrastructure and resilient services to their organizations and the people that actually deliver new applications,” he explains. He also notes that SREs often come from traditional operations roles, such as systems engineers who keep systems up and running. “Site reliability engineers ensure systems stay reliable, resilient, and available,” he adds.

Typical expectations for SREs

Typically, SREs are tasked with ensuring that the speed of delivery doesn’t result in security, service, or solution interruption. But as Grabner notes, “Expectations are a little different for every company. There’s no golden rule. Many are responsible for monitoring and observability and maintaining systems — and providing automation to spin up required environments.”

Grabner highlights the role of SREs in providing the frameworks and platforms for service and application deployment. “When things go wrong, SREs often take on the role of first-line defenders if there’s an alert,” he says. “In a great organization, they don’t do it alone — they constantly work with and within individual application teams to deal with apps that are under fire.”

Perhaps the most important role of SREs is architecting resiliency. “You can’t buy resiliency as-a-service,” Grabner observes. “You have to architect for it by building systems that are resilient by design.” This architectural approach recently helped Dynatrace to withstand an AWS outage in Germany. Automatic delivery, resiliency, and auto-remediation helped to ensure critical systems weren’t affected.

Roles and responsibilities of an SRE

The role of an SRE has become invaluable as technology continues to evolve and businesses become more dependent on digital infrastructure. Here are some of the standard responsibilities of an SRE:

1. Monitoring and Alerting
   One of the primary duties of an SRE is to monitor a company’s digital infrastructure. This involves setting up monitoring tools and systems to detect concerns before they become significant problems. SREs set up alert systems that notify the appropriate people when issues are detected.
2. Incident Response
   The SRE responds quickly and effectively when issues are detected by identifying the root cause, developing and implementing a plan, and communicating with relevant stakeholders.
3. Automation and Tooling
   SREs develop and maintain the tools and systems used to manage a company’s digital infrastructure. This includes developing automation scripts to streamline processes and reduce the risk of human error. SREs also identify areas where tooling can be improved and create new tools to meet the changing needs of the business.
4. Capacity Planning
   SREs ensure that a company’s digital infrastructure can meet the needs of the business. This involves analyzing usage patterns to predict and guarantee the required capacity to meet future demand.
5. Collaboration
   SREs work closely with other teams to ensure the company’s digital infrastructure is reliable, scalable, and secure.

What makes a great SRE?

Great SREs are risk-takers, tinkerers, and innovators. They figure out what it takes to scale a system from 100 users to 100,000 users to 1,000,000 users while maintaining uptime and resiliency. They’re systems thinkers who consider how decisions made in development affect production environments, and how the needs of production systems can influence design.

This requires constant testing, accepting failure, and adapting, automating repeatable processes. Successful SREs bring a resiliency and adaptation mindset to every situation.

Grabner highlights the need for SREs to learn from their mistakes. “Some companies run ‘chaos days’ to deal with worst-case scenarios to understand what could happen and how to deal with it,” he says.

Automation is another marker of SRE success. “People excel in this role when they try to automate all the tasks that can and must be automated,” says Grabner. “This frees them up to deliver true innovation.” He notes that while anyone can innovate under the right circumstances, teams are often held back by the “toil” of manual and repetitive tasks. “Your goal is to automate yourself out of your current role and into your next role.”

Finally, Grabner made it clear that SREs can’t operate in isolation. “You need to let people educate themselves with new technologies and practices,” he says. “Show the world. Don’t keep secrets — be open and share your own learnings, along with learning from others. There are a lot of great conferences out there — it’s worth getting inspired by what others do and inspiring others with what you do.”

DevOps vs. SRE

Where DevOps teams focus on streamlining change, SREs help ensure these changes don’t increase overall failure rates. In effect, they’re two sides of the same coin: DevOps automates speed, while SRE automates reliability. “It’s a balance between speed and safety,” Grabner says.

He sees DevOps processes as moving left to right along the development life cycle, using automation to speed up new capabilities that are typically measured by deployment frequency and lead time for changes. In comparison, SRE moves right to left using production-level requirements in development, with a focus on limiting failure rates and reducing the time required to restore service. “SRE is about making sure that even though there is a lot of change, these changes don’t break things.”

Grabner sees SRE and DevOps overlapping when it comes to SLOs. “SLOs are all about supporting business goals,” he said. “Companies may need systems at 99% reliability. They may want to increase their user base or improve the end-user experience.” Satisfying these goals is the role of DevOps. “But underneath these goals are technical goals that are specific to your objectives,” he says. “They contribute to business success with the right features at the right time and help you cope with change.” Delivering on these goals is the job of SRE staff. As a result, “SLOs are a great way to bring DevOps and SREs together.”

In another blog post you can learn more about the differences and similarities of SRE vs DevOps.

Solving for site reliability

Site reliability isn’t and never will be a “solved problem”. New services and applications combined with evolving enterprise demands mean there’s always work for SRE teams, and there’s always room for improvement.

As Grabner notes, when it comes to improving SRE impact, “The biggest thing is to be open and share your own learnings, along with learning from others. There are a lot of great conferences out there — it’s worth getting inspired by what others do and inspiring others with what you do.” He also highlights the need to learn from your mistakes. “Some companies run ‘chaos days’ to deal with worst-case scenarios to understand what could happen and how to deal with it.” Finally, Grabner made it clear that SREs can’t operate in isolation. “You need to let people educate themselves with new technologies and practices. Show the world. Don’t keep secrets — don’t see it as a silo.”

Looking for a solution to up-level your SRE practices? Dynatrace can help. Providing automatic and intelligent observability for even the most complex distributed cloud environments, the Dynatrace Software Intelligence Platform empowers SRE and DevOps teams to identify problems before they occur. Driven by continuous automation with AI at its core, Dynatrace delivers precise root-cause answers to site reliability issues at every step of the software development lifecycle. From early development in pre-production environments through delivery and operations in production environments, Dynatrace helps SRE teams improve reliability, availability, and latency, and mitigate the business impact of service outages and slowdowns.

The post What is SRE (site reliability engineering)? And what do site reliability engineers do? appeared first on Dynatrace news.

Service level objectives (SLOs) are a common method to set and maintain agreed-upon service levels of applications in modern cloud environments. SLOs have evolved beyond basic target measurements; they are powerful guidance tools for site reliability engineers (SREs) and DevOps platform teams to help direct areas of improvement in both CI/CD as well as production processes of every organization.

However, creating effective SLOs can be difficult. According to the 2022 State of SRE Report from Dynatrace, 99% of SREs say they encounter challenges when defining and creating SLOs. Identifying and implementing effective SLOs requires a thoughtful and structured approach for success. Here are our recommended steps for implementing the right SLOs for your platform and services.

Step 1: Understand service level agreements, internal business goals, and external stakeholders

Service level agreements (SLAs) are contractual financial agreements between vendors and their customers. These agreements define the service levels that customers and end-users expect, making them a great starting point to understand how IT can ensure the overall business objectives are being met. SLA violations lead to financial penalties, impact revenue, and damage your company’s reputation. Therefore, it is critical to align your SLOs to meet customers’ needs.

Step 2: Identify and prioritize critical services that affect SLAs

Pinpoint the necessary services to meet SLAs, especially the ones that customers frequently interact with or could cause the most amount of pain if failures occur. It’s important to then prioritize these services in order of customer and financial impact. For example, a “checkout” service for purchasing a product is a higher priority than a “compare service” for comparing products.

Step 3: Identify internal stakeholders and align with the different teams

Identify which team or person will own the responsibility of ensuring that services are running as expected. This includes identifying who creates the services, who is responsible for monitoring the service, and who is responsible for remediation. Documented, agreed-upon roles and responsibilities between stakeholders are key to avoiding finger-pointing or confusion when problems occur. Internal alignment ensures that teams know which services they own and, more importantly, who needs to fix something when it breaks.

Step 4: Identify key metrics to use as service-level indicators (SLIs)

Once you have an internal process established, you can then start measuring services. SLO measurements are based on service-level indicators (SLIs), which are quantifiable measurements that help you determine whether a service is working. Work with your SRE and Operations teams to understand what key metrics your observability platform provides and which ones you would need to track. There are many types of SLIs to choose from, such as Google’s Four Golden Signals, RED metrics (Rate, Error, Durability), or USE metrics (Utilization, Saturation, Errors).

Step 5: Identify key SLOs

Once you have identified the critical services and SLIs, you can create your SLOs. Ensure that each objective is measurable with realistic, attainable thresholds set for a particular timeframe (example: hour, week, month). Unrealistically high thresholds for SLOs will face constant violations. Conversely, easily achievable, low SLO thresholds make it difficult to know when service disruptions occur. SLOs must be meaningful and drive business outcomes, not exist as mere targets to reach. A good way to determine thresholds is by looking at historical trends of how the service has performed.

Step 6: Define your error budgets

Error budgets define the amount of tolerable service failure with no contractual consequence. If you burn all of your error budget, customers will likely start complaining and be unhappy with the service. Defining the error budget is a powerful way to proactively measure the health of your SLO and avoid the shock of an SLO turning from green to red instantly.

Step 7: Ensure proactive SLO monitoring and alerting

Monitoring is the final step to ensuring you are meeting your SLAs and business objectives. In addition to receiving alerts when SLOs violations occur, a better and more proactive approach would be receiving alerts when error budget burn rates appear faster than normal. This method allows you to address potential issues before they cause problems. Either way, alerts should route to the right team or individual to speed up triaging issues and reduce MTTR.

Ensure resilience through proactive SLO monitoring and automatic remediation with Dynatrace

Dynatrace provides native and proactive SLO monitoring. With AI-powered observability, organizations achieve meaningful insight into how their key business services and applications are performing. Teams can visualize and display SLIs on dashboards while Dynatrace automatically monitors them against SLO targets without extra manual oversight.

The post 7 steps to identify and implement effective Service Level Objectives (SLOs) appeared first on Dynatrace news.

Site reliability engineering (SRE) continues to gain popularity as organizations embrace hybrid cloud strategies and IT automation at scale. By applying software engineering principles to operations and infrastructure practices, SRE enables organizations to streamline and automate IT processes.

In the Dynatrace State of SRE Report: 2022 Edition, 88% of surveyed site reliability engineers (SREs) said there is now greater recognition of the strategic importance of their role in business success than there was three years ago. SRE is becoming an essential discipline in organizations that use DevOps (the combination of development and operations) and agile methodologies.

SRE adoption is growing, yet gaps remain. In order to unleash the innovation organizations need to evolve their SRE approaches. The report uncovers six site reliability engineering trends that will help organizations get the most from DevOps practices.

1. MTTR reduction remains top of the list for SREs

SREs improve the reliability of production systems, and reducing mean-time-to-repair (MTTR) is their top priority. But research shows that 60% of SREs find they spend most of their time building and maintaining automation code. While increasing automation is a key goal, organizations will lose their derived efficiency if enablement is arduous and time-consuming.

Much of the problem stems from how site reliability engineering teams build automation for DevOps workflows. Often, teams handle this on a case-by-case basis because their tooling doesn’t come with automation built in and doesn’t offer everything-as-code capabilities. As a result, they must build a layer of automation on top of their tooling.

Over time, this creates a complex web of code that becomes more difficult to scale across the DevOps pipeline. If SRE teams don’t identify a more efficient approach to DevOps from day one, they will undoubtedly find more of their time drained in the future. That time sink results in developers chasing down bugs and other code issues rather than focusing on strategic, revenue-generating work.

This underscores the need for SREs to work with DevOps teams, developers, and architects to ensure that software not only meets a business need but also is resilient and automatable by default. Enabling teams to easily integrate new automation capabilities with existing tools and workflows reduces manual effort and improves engineering practices.

2. A shift to SRE-driven engineering takes hold

More than half (51%) of SREs say they dedicate significant time to influencing architectural design decisions to improve reliability. This suggests that all departments have made progress toward SRE-driven engineering across to improve reliability, resiliency, and security. But there’s still a long way to go.

The most mature organizations embrace site reliability engineering practices that include developers who have fought the battles. They understand what it takes to build systems that can scale from 10 users to 1,000, or from 1 million to 10 million users. Integrating these developers with the design process provides insight that enables architects to incorporate reliability from day one.

3. Security is a core pillar of site reliability engineering

SREs are also making progress in extending organization-wide DevSecOps approaches to ensure organizations can restore systems quickly after discovering a vulnerability. More than two-thirds (68%) of SREs say they expect their role in security to become even more central in the future. This trend will increase as organizations continue using third-party libraries for cloud-native application development.

As the Log4Shell vulnerability demonstrated when it emerged in December 2021, third-party code libraries face significant security risks. Site reliability engineering teams are critical to identifying those flaws and eliminating them for fail-safe IT protection and to minimize cloud and third-party risks.

4. SREs need the freedom to experiment

While more than half (52%) of SREs dedicate significant of time to designing experiments and tests to reduce the risk of production failure, only 1 in 10 highlights this as their top priority.

Experimentation is critical to SRE. Teams still need to make progress to ensure they have more time available for these tasks. For SREs to deliver more strategic business value, engineers must streamline tasks that involve intensive manual effort.

5. SREs need the license to prioritize strategic work

Although experimentation falls relatively low on their priority list, 51% of SREs say they’re encouraged to experiment. In addition, only a quarter (26%) of organizations see incremental project failure as OK. The weak emphasis on experimentation highlights the many distractions SREs face, limiting the time they have to focus on it.

Organizations must consider new strategies that enable site reliability engineering teams to focus on more strategic tasks. That way, teams have more time to play. Team leaders also need to foster a culture that accepts failure and understands that the principle “fail fast, fail often” provides the greatest competitive edge.

Unshackle SRE teams from traditional organizational structures that view IT as a cost center and a burden. Organizations can’t benefit from lessons gained from IT mistakes without openness to failure.

6. Organizations recognize and reward site reliability engineering teams

SREs must be free to challenge accepted norms and set new benchmarks for innovation-led design and engineering practices. Many organizations are making strides in this direction and have methods for rewarding SRE teams’ successes.

Nearly a third (31%) use hackathons to devise new ways to improve reliability, offering prizes to winning SRE teams. These approaches are key to encouraging a culture of experimentation that promotes the strategic value of site reliability engineering for the business.

All six of these SRE trends can help your organization advance its SRE practices. Embrace experimentation and unleash exciting innovation.

The post Site reliability engineering: Six SRE trends to unleash DevOps innovation appeared first on Dynatrace news.

Software reliability and resiliency don’t just happen by simply moving your software to a modern stack, or by moving your workloads to the cloud. There is no “Resiliency as a Service” you can connect to via an API that makes your service withstand chaotic situations. The fact is, Reliability and Resiliency must be rooted in the architecture of a distributed system. The path to “Architected for Resiliency” is long, but it clearly pays off in the long run, especially when outages occur, as I want to show you in this blog post.

This article was inspired by an email I received from Thomas Reisenbichler, Director of Autonomous Cloud Enablement on Friday, June 11^th. The subject line said: “Success Story: Major Issue in single AWS Frankfurt Availability Zone!” The email walked through how our Dynatrace self-monitoring notified users of the outage but automatically remediated the problem thanks to our platform’s architecture. This meant there was no negative impact on our end-users, Service Level Objectives (SLOs), or Service Level Agreements (SLAs). And the last sentence of the email was what made me want to share this story publicly, as it’s a testimonial to how modern software engineering and operations should make you feel. It read: “Observing this issue, it was an honor to have the possibility to ‘just’ sit next to it and do a little bit of babysitting knowing that we are coping very well with this failure!”

What Thomas meant by saying this, was the dashboard showed how well the Dynatrace architecture automatically redirected traffic to the remaining nodes that were not impacted by the issue – thanks to our multi-availability zone deployment – as you can see below:

Ready to learn more? Then read on!

Fact #1: AWS EC2 outage properly documented

Let’s start with some facts. There really was an outage and AWS did a great job notifying their users about any service disruptions via their AWS Service Health Dashboard. The following screenshot shows the problem reported on June 10^th for the EC2 Services in one of their Availability Zones in Frankfurt, Germany:

The problem started at 1:24PM PDT, with the services starting to become available again about 3 hours later. The final status update was at 6:54PM PDT with a very detailed description of the temperature rise that caused the shutdown initially, followed by the fire suppression system dispersing some chemicals which prolonged the full recovery process.

Fact #2: No significant impact on Dynatrace Users

There are several ways Dynatrace monitors and alerts on the impact of service disruption. Let me start with the end-user impact.

Dynatrace provides both Real User Monitoring (RUM) as well as Synthetic Monitoring as part of our Digital Experience Solution. Through the RUM data, Dynatrace’s AI engine, Davis, detected seven users were impacted by the outage when they tried to access the Web Interface. This number was so low because the automatic traffic redirect was so fast it kept the impact so low. The screenshot below shows the opened problem ticket and the root cause information:

Note to our Dynatrace users: This story triggered a feature request that will benefit every Dynatrace user in the future. The team wants to enrich root-cause information in the Dynatrace problem ticket with external or third-party status details such as the AWS Service Health Status. This will eliminate the need to cross-check whether an existing outage of your third-party providers is going on right now.

Fact #3: Minimum impact detected through synthetics

Besides real user analytics, we also use Dynatrace Synthetic Monitoring, which continuously validates successful logins to our SaaS tenants on each cluster. Those tests get executed from two locations (Paris and London) hosted by different cloud vendors (Azure & AWS).

For the outage, Dynatrace Synthetic detected a very short one-time connection timeout, as you can see below:

As a general best practice, Synthetic Tests are great to validate your core use cases are always working as expected. In our case that includes the login to our SaaS tenants and exploring captured data. If those use cases don’t work as expected, we want to get alerted.

Tip: We see more of our users started to Shift-Left and GitOps-ify Dynatrace Synthetic. This means that Synthetic Tests are not just used in production but also in pre-production environments to validate environment stability, e.g., do I have a stable build in a QA or Test environment or not? Thanks to our Automation APIs and our open-source project Monaco (Monitoring as Code) the creation and updates of those synthetic tests are fully embedded into their GitOps automation. Dynatrace Synthetic Test definitions are version control in Git, as YAML gets automatically rolled out as part of their delivery automation, e.g.: via Jenkins, GitLab, Azure DevOps, Keptn

Fact #4: Multi-node, multi-availability zone deployment architecture

I already mentioned at the beginning of this blog that resiliency and reliability do not come for free – they must be part of your architecture. And that’s true for Dynatrace as well. You can find a lot of information about the Dynatrace architecture online, both for our SaaS and Managed deployments.

I wanted to highlight a couple of essential elements that are key for Dynatrace’s resilience against a data center (=AWS Availability Zone) outage:

1. High availability due to multi-AZ Dynatrace cluster node deployments
2. Rack-aware Cassandra deployments

Let’s have a quick look at Dynatrace Smartscape to see how our cluster node services are truly distributed across multiple EC2 hosts in different Availability Zones:

The health-based load balancing of incoming traffic automatically redirects traffic to healthy nodes. In case of host unavailability, consumers of Dynatrace services (via Web UI or API) never experience any issues. This deployment is also super resilient to full data center (e.g., Availability Zone) outages.

As for Cassandra, a 3-node Dynatrace SaaS deployment, we deploy 9 Cassandra nodes with a rack-aware deployment. The rack is linked to the AWS Availability Zone. In case one zone goes down, the traffic gets redirected to the remaining Cassandra nodes. The following chart shows the distribution of nodes before, during, and after the outage:

The Dynatrace deployment also contains our Active Gates. But – thanks to our multi-node and multi-datacenter deployments, all these components provide the same high availability and resiliency. That’s why the complete Dynatrace Software Intelligence Platform is “Architected for Resiliency”

Tip: Our managed customers have the same high availability and resiliency features. For more information check out our documentation on fault domain awareness such as rack aware managed deployments.

Conclusion: Investing in resilient architecture is CONTINUOUS

This story proves that high availability and resiliency must be features and considerations you plan from the start when designing a distributed system. Built-in monitoring is the only way to validate these systems work as designed, and alerting is the insurance that you get notified in corner cases to reduce the risk of negative end-user impact.

I was also reminded that resilient architecture is not a “one-time investment”. It needs continuous attention and focus. At Dynatrace we built our current architecture years ago, and to ensure it still withstands challenging situations every new feature gets evaluated against non-functional requirements such as resiliency or performance. Our dynamic growth in engineering also made us invest in continuous training for new and existing hires. And to give them feedback on the potential impact of code changes we have an automated continuous performance environment that battle tests new versions before admitting them to production.

Before saying goodbye, let me say thanks to our Dynatrace Engineering and everyone involved in designing and building such a resilient system architecture. I also want to say thank you to Thomas Reisenbichler for bringing this story to my attention, to Thomas Steinmaurer for giving me additional background information, and to Giulia Di Pietro for helping me finalize the blog post.

For more on Chaos Engineering & Observability be sure to register for my upcoming webinar with Gremlin. 

The post Architected for resiliency: How Dynatrace withstands data center outages appeared first on Dynatrace news.

Our customers are increasingly transitioning to agile software development, DevOps, and progressive continuous delivery to deliver business value faster. To speed up release frequency, they’re investing in delivery-pipeline automation. The flip side of speeding up delivery, however, is that each software release comes with the risk of impacting your goals of availability, performance, or any business KPIs.

To speed up delivery, you need to automate not just delivery but also the process of deciding whether it is safe and risk-free to deliver the next change into production (which might include a wide range of changes, such as deploying a new microservice, enabling a feature flag, scaling your pods on Kubernetes, or including a new third-party service). Confidence for a safe release is provided by more validation of how the changes impact quality, for example, before deploying to production or with impact analysis during blue/green deployments. In these circumstances, Site Reliability Engineering teams face two big challenges:

• Measuring uptime is not enough anymore.

We are moving from static to very dynamic infrastructure and applications. The modern IT landscape is driven by the automatic scaling of compute nodes and highly dynamic provisioning capabilities within Kubernetes clusters. Just measuring the uptime of service tells you nothing about health or quality.

• Which metrics are relevant for your business, anyway?

Modern observability tools provide many metrics, but which ones are really important for your business? Automating any decision-making based on release risks requires domain knowledge and an understanding of business impact. You need to consider what services are provided by the software to be released, what legal service-level agreements (SLAs) are set up, and how any software release or configuration change could impact SLAs. (Read more about the basics of Site Reliability Engineering below.)

We’re happy to announce that Dynatrace now provides guidance and templates for setting up Service-Level Objectives (SLOs) with the right metrics, gives you all the facts, and combines this with the powerful analytics of problem root-cause detection. All you need is to make a confident decision about whether to push the release button—with respect to release risks or any quality degradation.

Scale and automate SRE into your delivery processes with Dynatrace

The Dynatrace Software Intelligence Platform delivers all the necessary Service-Level Indicators (SLIs) for defining your Service-Level Objectives (SLOs) and monitors the status and error budgets of those objectives. Whether you want to define an SLO for real user experience or for measuring your service availability, your Dynatrace monitoring environment already comes with the right tools for your Site Reliability Teams. You can get all the facts about the health of your critical business services with respect to Business and Service-Level Objectives (SLO), Service-Level Agreements (SLAs), and end-user experience:

• Get more than 2000 different metrics ready for use as dedicated SLIs

Depending on the service, an SRE team’s responsibility is to find the best indicators that prove the successful delivery of the service to the customer. Dynatrace offers more than 2000 different metrics that are ready for use as dedicated SLIs.

• Choose from a range of ready-to-use SLO use cases

As you might find it hard to select a good indicator metric for various use cases, Dynatrace offers a collection of out-of-the-box SLO templates that support major use cases for your monitoring environment.

Typical Dynatrace use cases cover SLOs for service availability, web application performance, mobile application availability, and synthetic availability. Each action button in the SLO wizard selects one of these important templates.

• Get Davis AI-powered notifications about detected anomalies with actionable root causes

Other solutions on the market trigger alerts for SLO status and error budget burn-down rates. As SLOs are typically defined over longer periods of time, this approach gives teams very late notification of an existing problem in production. Also, once an SLO turns red, the SRE team’s immediate question about the cause of the issue is still unanswered when the SLO metric alone doesn’t offer much by way of an answer.

On the other hand, the Dynatrace AI causation engine Davis® provides both quick notifications for detected anomalies as well as actionable root causes. This also means that when your SLO turns red, it’s very likely that you’ve already received a Davis notification that informs you of the problem as well as shows what the root cause of the situation is.

Note: We will soon improve the drill-downs to Davis problems directly from existing SLO definitions so that SRE teams can benefit more from the Dynatrace Davis AI engine.

• Build your own Site Reliability Engineering dashboards

Once you’ve selected your SLI metrics and successfully defined your list of SLOs in Dynatrace, you can build your own Site Reliability Engineering (SRE) dashboards by adding SLO tiles alongside your other important monitoring data.

Now let’s dive deeper into how Dynatrace can be used to support your Site Reliability Engineering process.

Scale Site Reliability Engineering with SLOs in Dynatrace

Selecting metrics of interest and creating SRE dashboards has already been possible in Dynatrace for a while (for example, for public APIs with Synthetic monitors and for automated SRE-driven performance engineering). With our latest updates, we’ve put Service-Level Objectives front and center by making SLO monitoring a new, top-level capability accessible from the Dynatrace navigation menu (shown in the screenshot below). The Dynatrace SLO page gives your SRE team full access to defining and observing your SLOs.

The SLO list in a Dynatrace monitoring environment displays the most important information such as the current status, error budget, defined target, and an additional warning target along with the evaluation window.

By default, each SLO is evaluated according to its defined time window, but you have the ability to temporarily switch the evaluation to use the global timeframe instead. This helps to perform dynamic what-if analysis with different timeframes and to look back in time. The screenshot below shows how to switch between evaluation timeframes:

Select Create a new SLO to launch the SLO-creation wizard that guides you through the definition of a new Service-Level Objective. The SLO wizard gives you guidance on creating SLOs based on the metrics (SLIs) Dynatrace monitors for your environment.

The SLO wizard allows you to select either one SLI percentage metric such as the Service success rate (Single rate metric) or two metrics that represent the “Good count” divided by the “Total count” such as the Number of service calls without error divided by the Total number of service calls (Metrics for good vs. total count).

As you might find it hard to select a good indicator metric for various use cases, Dynatrace offers a collection of out-of-the-box SLO templates that support major use cases—service availability, web application performance, mobile app availability, and synthetic availability. (See the action buttons in the screenshot above for each of these important templates.)

Once you’ve selected your SLI metrics and successfully defined the list of your SLOs, you can show SLO statuses directly in your Dynatrace dashboards. Build your own Site Reliability Engineering (SRE) dashboards by adding Service-level objective tiles next to your other important monitoring data.

We ourselves have a long history of using Dynatrace for progressive delivery at cloud scale, leveraging SRE concepts to make faster decisions on whether “To Release or Not Release” a new product version to our customer base. Over the past months, we’ve also worked very closely with Google to bring new SLO capabilities into our Software Intelligence Platform to enable all our users to scale their SRE practices.

Site Reliability Engineering 101

Site Reliability Engineering was a term introduced by Ben Treynor Sloss, Google’s VP for 24/7 Operations, who first claimed that reliability was the most fundamental feature of any product.

Google has a long history of shaping SRE processes for their global-scale services that are dedicated to making their services more scalable, reliable, and efficient. Read more:

• Google’s definition of the term Site Reliability Engineering
• Release decision making with Service-Level Objectives (SLOs)
• Site reliability engineering: 5 things to you need to know

Finding good Service-Level Indicators (SLIs)

The purpose of an SRE team is to define and monitor the level of service fulfillment towards the customer and to manage the risk around that fulfillment.

In principle, it doesn’t really matter if the focus is a software service, such as an online payment process, or buying a Coke at a traditional vending machine. What matters is how reliably this service is delivered to the customer.

An SRE team’s means of measuring the delivery of a service to a customer is to find dedicated indicators, so-called Service-Level Indicators (SLI), that can prove successful service delivery. So the SRE team is responsible for finding good SLIs for a given service that closely monitors the reliable delivery of a service to the customer.

SLIs can be different per service, as not all services are equally critical in terms of time and error constraints.

Here are some examples:

• The successful request of a payment service depends highly on not returning an error on valid data and on finishing within three seconds.
• A music-streaming service depends on the continuous delivery of media packages while the loss of some of those packages is not overly critical.

Dynatrace offers more than 2000 different metrics that are ready for use as dedicated SLIs. While this is great, it might be overwhelming for many (“Which SLIs are right for me?”).

Dynatrace has already prepared a selection of SLO use cases along with recommended SLI metrics in our SLO-creation wizard. See more about our SLO wizard for the Dynatrace-recommended, ready-to-use SLO use cases. Once the SRE team has found and agreed upon a collection of good indicator metrics, what remains is to define and monitor the operational goal with a Service-Level Objective (SLO).

Defining your target by setting a Service-Level Objective (SLO)

Choosing the right target for your Service-Level Objective (SLO) is a tedious task and requires expert knowledge about the measured service. As services are, of course, highly dependent on the user context, not all indicators are equally appropriate in very strict SLO target definitions.

For example, the service response time is heavily dependent on the customer’s location, browser, and client device. A service that delivers perfect results within milliseconds in most cases could take ages on an outdated mobile device running in low bandwidth geolocation.

Dynatrace offers a lot of expertise in terms of measuring the real user experience of delivered services directly on the client-side. Dynatrace metrics such as the Apdex (Application Performance Index) or the User experience score can directly be used in an SLO definition (as we’ll show in the examples below).

As defined by Google, a typical SLO consists of the following fundamental information:

• Service-Level Indicator (SLI): The indicator used to measure successful service delivery—SLIs typically refer to metrics such as service success rate, crash-free mobile app users, successful synthetic test runs, or response time.
• Target: The target defines the planned goal to achieve in terms of service delivery. A target could be that 99.99% of all service calls must return without an error or that 95% of all service requests must be fulfilled in under 2 seconds’ response time.
• Evaluation period: The evaluation period is necessary to standardize communication concerning the SLO result. Without a defined evaluation period, one person could talk about 100% availability within the last 24 hours, while another person comes to a different conclusion of 95% availability looking at the same SLO over the period of the last week.

Given the fundamental ingredients above, each SLO definition can be evaluated by following two metrics:

• SLO status: The current evaluation result of the SLO, expressed as a percentage—The semantics of this percentage (for example, 99.3% of all service requests are successful, or 99.99% of all website users are “satisfied” in terms of Apdex rating) and the target defined for this percentage is up to the SRE team.

Again, the SLO status must be normalized towards a percentage value in the range of 0–100%!

• SLO error budget: The error budget represents the buffer remaining until the defined SLO target is considered as failed. For example, if an SLO defines a 95% target and its current SLO status is evaluated at 98%, the remaining error budget is the difference between the status and target.

Support for release decisions: Shift-Left of SLO evaluation for new software releases

While “never change a running system” might be the precept when SLOs are in place, we might still want to release new features with confidence, without the risk of violating SLOs in production after a release. A new software release can always impact SLOs, and this comes with even more risks if there is only a small error budget remaining in production. A release rollout should be stopped automatically if newer versions are worse with regard to SLO evaluation. This can be detected during any canary deployment or blue/green traffic routing to a new version.

However, SLOs at risk with a new release can also be detected in pre-production environments by evaluating the SLOs as part of your delivery pipeline’s quality gate. In the end, SLOs are just metrics compared against your objectives. This concept can also be applied to continuous delivery where a response-time objective might be “Login must be faster than 100 ms” or “Login should not be slower as compared to a previous build!”

To learn more about SLO-driven quality gates, read and watch SRE-inspired DevOps automation via Keptn, integrated with Dynatrace.

Summary

Site Reliability Engineering represents an important methodology for embracing and managing risk while deploying and providing software services at a global scale.

Dynatrace, as the leading platform for software intelligence, delivers all the necessary Service-Level Indicators (SLIs) for defining your Service-Level Objectives (SLOs) and offers tools for monitoring the status and error budgets of those objectives. Whether you want to define an SLO for real user experience or for measuring your service availability, your Dynatrace monitoring environment already comes with the right tools for your Site Reliability Teams.

Get started with Dynatrace version 1.207 defining your first SLO using our templates, enrich your dashboards with SLO tiles, and create your dedicated SRE dashboards!

Stay tuned for more release advice as well as automation provided by Dynatrace for release risk management and automated release decision making based on SLOs.

The post Accelerate DevOps and Scale SRE with Service Level Objectives (SLOs) appeared first on Dynatrace news.

Site Reliability Engineering is an exciting discipline in our industry. There are many aspects to building reliable and resilient systems which I won’t be able to cover in a single blog post. What I learned from many conversations with enterprise organizations is that the term “self-healing” is getting everybody excited. That’s also why I put it into the title, to get you “excited”! However – I have at least two major problems with the way self-healing is sold and thought of:

#1: Self-healing as a term is misleading!

I don’t think that there are any real self-healing systems. I think we have gotten better to automate remediation actions – and by leveraging modern monitoring tools we can execute specific remediation actions in a much smarter and efficient way. Hence, I believe we should talk about Smart- or Auto-Remediation and not necessarily Self-Healing as it is misleading!

#2: Self-healing is not just an operations activity!

In most of my discussions around self-healing, I talk with Ops teams that try to fix issues that materialized in production. While it is very important to keep our systems running even under “chaotic” conditions in production, I strongly believe we should invest just as much time and effort in preventing these issues ever making it into production. That’s where my “Shift-Left SRE” term comes from as I believe we must bake resiliency into our delivery pipelines.

I presented my thoughts on this topic at AWS re:Invent 2018. It was called Shift-Left SRE: Self-Healing with Lambda!

While my talk was targeted for an AWS crowd the concepts, I explained are applicable for any platform, cloud or technology stack.

I structured my presentation in 4 parts which is the same way I’d like to organize this blog:

• Remediation Use Cases
• PREVENT in CI/CD vs REPAIR in Prod
• “Auto Remediation as Code”
• The “Unbreakable Delivery Pipeline”

Let’s dig into it!

Remediation Use Cases

I have to be honest. I never worked in Operations, neither did I work as an SRE. I always worked in Performance Engineering, where I ended up breaking a lot of environments for different reasons, e.g: hitting the system with too much load, not cleaning up log directories before a new run or improper configuration of my test or test environment. When talking to my friends, customers and partners that work in Operations, they tell me similar root causes they see in a production environment.

Let me therefore start with a quick overview of common problem scenarios that impact production environments and the remediation actions that can be taken to bring the system back to normal:

This is a very short list of remediation actions – but – a list that addresses very common and basic issues in todays environments: Process Restarts, Resource (e.g: Disk) Cleanup, Revert Bad Configuration Changes, Scale Up or Down, Switch Blue vs Green, …

While these use cases have been known for years by many teams I talk with, only a handful have invested in some sort of automation. While many have built scripts – these typically get executed manually when a problem is detected. That means – there is a lot of room for improvement!

PREVENT in CI/CD vs REPAIR in Prod

While we must invest in automating these remediation scripts in production we should learn from every problem we detect in production, figure out what the root cause is, how to replicate, measure or detect that root cause in a lower level environment and automate that into the CI/CD Delivery Pipeline as a Quality Gate:

This approach of adding these checks in a pre-production environment requires the teams (SRE Teams, Ops, …) that currently work on auto-remediation scripts for production to also think about how these situations can be detected in a lower level environment as part of the automated delivery pipeline. Here are the tasks that from my perspective define Shifting-Left SRE:

• Take the lessons learned from production
• Reproduce them in Pre-Production through e.g: Chaos Monkey Tests
• Automate the checks into CI/CD Quality Gates

In my presentation at re:Invent, I listed examples on how to detect certain use cases, what the metrics are to watch out for and how to capture these metrics:

Once we have the metrics and the tests to replicate these scenarios in a lower level environment we can add it to the pipeline as quality gate checks:

If you are interested in integrating Dynatrace captured metrics into your Jenkins, Azure, Concourse, Bamboo … pipelines check out our my recent blog posts and YouTube tutorials: Jenkins Performance Signature, Azure DevOps for Dynatrace, AWS DevOps Tutorial!

Auto-Remediation As Code

When using Dynatrace for full-stack monitoring of platforms we do not just receive alerts based on individual metrics that violated a static threshold or baseline, e.g: CPU is high or Disk is Full. The Smartscape Model, the PurePaths and the high-fidelity monitoring data we ingest and forward to our AI allows Dynatrace to detect problems and correlate all relevant events and data points to a single problem instance. All correlated information on that problem is made available to the auto-remediation as code scripts that SRE engineers wrote to handle specific problem root causes. The auto-remediation scripts do not have to do any correlation work but simply focus on fixing the root cause of a single problem vs chasing many loose ends of unrelated events you get from other monitoring or alerting tools.

The following animation visualizes the full problem/incident response workflow Once Dynatrace detects a problem we set the correct incident response actions in motion to inform the impacted teams as well as call auto-remediation as code scripts that address the root causes rather than just symptoms:

In my demo at re:Invent, I showed the self-healing part of my AWS DevOps Tutorial where I deploy a faulty version of a Service that results in a high failure rate impacting end users. During the deployment I pass the link to an AWS Lambda function that the developer (in this case me) wrote to be called in case of an issue in this service (=Auto-Remediation As Code). Once Dynatrace AI detects the issue, it calls the Lambda function that is associated with the latest deployment of the faulty service!

I encourage Operations or SRE Teams to collaborate with the engineering teams so that they can start writing their own remediation code. This code should be battle tested in a pre-prod environment when those problems get simulated that have been seen in production before. This is a great way to build resilience into the code package that comes from engineering and as it has been battle tested in CI/CD the chances of a real problem in production are very small 😊

The “Unbreakable Delivery Pipeline”

Now – if we put all the discussed concepts together in an automated delivery pipeline we can speak of what I coined “The Unbreakable Delivery Pipeline”:

1. We stop potential negative impacting changes early
2. We auto-validate “Auto-Remediation as Code” via a Quality Gate
3. We use “Auto-Remediation as Code” to “self-heal” production

We “Shift-Left” every problem we haven’t yet auto-remediated

What do you “Auto-Remediate as Code” now?

I hope you got a couple of new ideas on how to Shift-Left SRE (Site Reliability Engineering). Working closely with engineers, providing them the frameworks to develop and extend the delivery pipeline to automatically validate and deploy your auto-remediation scripts allows you to build more and better resilient systems.

Leave me a message, drop me an email or message me on LinkedIn or Twitter in case you want to have a share your thoughts on this topic.

The post Shift-Left SRE: Building Self-Healing into your Cloud Delivery Pipeline appeared first on Dynatrace news.