Introduction

Welcome to SRE From Scratch, a no-nonsense guide to establishing Site Reliability Engineering practices within any organization.

Who This Book Is For

This book speaks directly to teams and individuals eager to embrace SRE principles but facing the challenge of minimal top-down initiatives or executive backing. You might be navigating these waters without the luxury of a dedicated SRE budget or the support of a formal organizational structure. Whether you find yourself in the dynamic environment of a startup, the evolving landscape of a mid-sized business, or the complex web of a large enterprise devoid of formal SRE frameworks, this guide offers you the practical strategies needed to build SRE functions from the ground up.

What You’ll Learn

Throughout this book, you will explore how to define and implement SRE practices that resonate with your business objectives. You will delve into practical methodologies for incident response, observability, and automation, equipping yourself with the tools necessary to enhance system reliability. Additionally, you will gain insights into fostering organizational support in the absence of formal approval, as well as real-world techniques for scaling SRE as your organization expands.

The Challenge

The journey to adopting SRE is often fraught with hurdles. Engineers frequently find themselves trapped in a cycle of firefighting, leaving little room to prioritize reliability. Leadership may struggle to appreciate the value of SRE beyond the basic requirement of keeping systems operational. The absence of a structured implementation roadmap further complicates the adoption process, while budget constraints make it difficult to assemble specialized teams.

This book serves as your playbook for overcoming these challenges.

Let’s embark on this journey by unraveling what SRE truly entails in practice.

What is SRE?

Site Reliability Engineering (SRE) is Google’s innovative method for managing production systems at scale. More fundamentally, it’s a comprehensive set of principles and practices that any team can embrace to enhance the reliability of their systems.

The Core Principle

SRE is what happens when you treat operations as a software engineering problem.

In the realm of SRE, the traditional approach of manually handling incidents and scaling systems is transformed. Instead, your team will engage in writing code to automate operational tasks, making the process more efficient and less prone to human error. You will measure reliability through precise metrics, such as Service Level Indicators (SLIs) and Service Level Objectives (SLOs), which provide a clear picture of how well your system is performing. By balancing feature velocity with system stability, you ensure that while new features are rolled out, the system remains robust. Furthermore, failures are treated as learning opportunities through blameless postmortems, fostering a culture of continuous improvement.

The Three Pillars

SRE practices are structured around three interconnected pillars, each vital to the overall success of the system.

1. Incident Response & Culture

In this pillar, you will establish well-defined on-call rotations and escalation procedures to ensure a swift and organized response to incidents. The postmortem processes are designed not just to identify what went wrong, but to drive real change and prevent future occurrences. A blameless culture is cultivated, focusing on learning and growth rather than assigning blame. Error budgets play a crucial role here, striking a balance between reliability and innovation, allowing your team to push boundaries while maintaining system health.

2. Observability & Golden Signals

Effective observability is about monitoring the right metrics, specifically latency, traffic, errors, and saturation. By setting meaningful SLIs and SLOs, you can ensure that the metrics align with your business goals. Alerting should be focused on what truly matters, avoiding unnecessary noise and allowing you to understand system behavior through comprehensive data analysis. This pillar empowers you with the insights needed to make informed decisions about system performance and reliability.

3. Tools & Scaling

The third pillar emphasizes the power of automation in eliminating toil. By adopting practices such as infrastructure as code, you streamline processes and reduce manual intervention. Capacity planning and performance optimization ensure your systems are prepared to handle growth efficiently. Building platforms that scale with minimal human intervention allows your team to focus on strategic improvements rather than operational firefighting.

Why SRE Matters

In the absence of SRE principles, teams often find themselves trapped in a cycle of firefighting, with little time left for building and innovating. Reliability issues tend to persist, unresolved and chronic, because there is no structured way to measure or improve system health. On-call duties become overwhelming and unsustainable.

However, by incorporating SRE principles, incidents are transformed into valuable learning opportunities. Reliability is not just maintained but improves measurably over time. Teams gain clarity in their priorities and boundaries, enabling them to direct their engineering efforts toward high-impact improvements.

In the following chapters, we will explore each of these pillars in greater detail, providing you with the knowledge and tools needed to implement them effectively within your organization.

Incident Response & Culture

Incident response is the lifeblood of Site Reliability Engineering, the pulse by which you measure your team’s effectiveness in maintaining stability and trust. The way your team navigates the storm of production incidents speaks volumes about your underlying reliability culture.

What Makes Effective Incident Response

To orchestrate a successful incident response, you must first establish clear roles and responsibilities. The Incident Commander is the linchpin who coordinates the response and makes critical decisions. Meanwhile, the Communications Lead is charged with keeping stakeholders informed and managing external communications. Your Technical Responders are the detectives and problem-solvers, diving deep to debug and resolve the issue at hand. Lastly, the Scribe plays an essential role by meticulously documenting the timeline and actions taken for review in the postmortem.

Understanding the gravity of an incident is crucial, and this is where defined severity levels come into play. Not all incidents demand the same urgency. A SEV-1 incident is a complete outage, severely impacting customers and requiring an all-hands-on-deck approach. In contrast, a SEV-2 is a degraded service with a partial impact, warranting a focused team response. SEV-3 incidents involve minor issues with minimal customer impact, allowing them to be addressed during regular business hours. Lastly, SEV-4 incidents have no direct customer impact and can be resolved during the normal course of development.

Runbooks and playbooks are indispensable tools in your incident response arsenal. Runbooks provide step-by-step procedures for routine operational tasks, while playbooks offer guided troubleshooting for known failure modes. These documents are living entities that must evolve in tandem with your systems to remain relevant and effective.

Building a Blameless Culture

The true aim of incident response is not to assign blame but to ensure that incidents are less likely to recur. Achieving this requires a cultural shift towards a blameless mindset. Start by assuming good intent; recognize that individuals made the best decisions possible with the information available at the time. Focus on systems rather than individuals, acknowledging that if human error was a factor, then the system permitted it to occur. Near-misses are not failures but opportunities to avert future incidents. Sharing these learnings widely through accessible postmortems ensures the entire organization benefits from the insights gained.

The Postmortem Process

A significant incident warrants a thorough postmortem. Begin by constructing a factual timeline of events, detailing exactly what transpired. Dive into the root cause analysis to uncover why it happened, accepting that multiple contributing factors are often at play. Assess the impact on both customers and the business. Highlight what went well during the response, celebrating effective strategies and actions. Finally, identify specific action items aimed at improvement, ensuring they are assigned to individuals with clear deadlines.

Making Action Items Stick

The key differentiator between competent and exceptional SRE teams is their ability to follow through on action items. Each item should be assigned to a responsible person and accompanied by a realistic deadline, typically within 30 days. Track the completion of these items in your issue tracker, reviewing outstanding tasks in team meetings. Do not consider a postmortem complete until all critical action items have been addressed.

On-Call Rotations

Maintaining sustainable on-call rotations is pivotal for long-term reliability. Clearly communicate the rotation schedule, whether weekly or biweekly shifts, to ensure everyone is prepared. Handoff procedures must be seamless, facilitating smooth transitions between on-call engineers. Establish clear escalation paths, providing guidance on when and how to escalate issues. Maintain alert hygiene by ensuring you only page for actionable, urgent issues. Recognize the burden of on-call duty through compensation, whether it be time off, pay, or other forms of recognition.

Common Pitfalls

There are several common pitfalls to be wary of in incident response. Avoid fostering a hero culture that glorifies firefighting rather than prevention. Ensure that action items from postmortems are not ignored, as they are essential for improvement. Be vigilant against alert fatigue by avoiding unnecessary pages for minor issues. Shift the focus of reviews from assigning blame to improving systems. Finally, cultivate an environment of psychological safety where team members feel free to admit mistakes and seek help without fear.

In the next section, we will delve into how observability and golden signals can enhance your ability to detect and diagnose incidents more swiftly.

Observability & Golden Signals

In the realm of Site Reliability Engineering, the adage “You can’t improve what you don’t measure” holds significant weight. Observability is about gaining a comprehensive understanding of what your systems are doing—and more critically, why they are doing it.

The Golden Signals

Google’s SRE book delineates four essential metrics that every service must monitor. These are known as the Golden Signals.

1. Latency

Latency answers the question: How long does it take to serve a request? When measuring latency, it is crucial to differentiate between successful and failed requests. To gain a more accurate picture, utilize percentiles—such as p50, p95, and p99—rather than relying solely on averages. Track latency meticulously by endpoint, region, or customer segment to pinpoint areas for improvement.

For example, you might find that “95% of API requests complete in under 200ms.”

2. Traffic

Traffic measures how much demand your system is currently handling. This metric can vary depending on the type of system you are working with. For web services, you assess requests per second; for databases, transactions per second; for queues, messages per second; and for real-time systems, active connections. Monitoring these metrics helps you ensure your system can handle peak loads, such as “Processing 10,000 requests per minute during peak hours.”

3. Errors

Errors reflect the percentage of requests that are failing. In web services, this might manifest as HTTP 5xx errors, while in application code, it could be exception rates or failed background jobs. Client-side errors also fall under this category. Monitoring error rates, such as “Error rate of 0.1% (10 failed requests per 10,000),” allows you to quickly identify and rectify issues affecting system reliability.

4. Saturation

Saturation indicates how “full” your system is, measuring resources like CPU and memory utilization, disk I/O, storage capacity, database connection pool usage, and queue depth. Understanding saturation levels, such as “Database connection pool at 70% capacity,” enables you to proactively manage resource allocation and avoid bottlenecks.

From Metrics to SLIs and SLOs

After collecting metrics, the next step is to translate them into Service Level Indicators (SLIs) and Service Level Objectives (SLOs).

Service Level Indicators (SLIs)

SLIs are precise measurements of service behavior, and effective SLIs should be user-centric, measurable based on real data, and actionable. For instance, you might track “Percentage of API requests completed in < 200ms” or “Percentage of login attempts that succeed.”

Service Level Objectives (SLOs)

SLOs are the target values set for your SLIs, defining acceptable reliability. An example SLO might state, “99.9% of API requests will complete in under 200ms.” It’s crucial to recognize that 100% reliability is neither achievable nor desirable, as perfect uptime often indicates a lack of risk-taking and innovation.

Error Budgets

Error budgets represent the amount of unreliability you can “spend” while still meeting your SLOs. If your SLO is 99.9% uptime, you have a 0.1% error budget, equating to approximately 43 minutes of downtime per month. Should you exceed this budget, prioritize reliability over new feature launches. Conversely, if you are well within your error budget, you have the flexibility to innovate and take calculated risks.

Alerting That Actually Works

The objective of alerting is to provide actionable, urgent notifications rather than noise.

Alert Design Principles

1. Alert on symptoms, not causes: For example, instead of alerting on “Disk usage > 80%,” focus on symptoms like “API latency p99 > 500ms AND error rate > 1%.”

2. Every alert should require action: If an alert doesn’t necessitate action at 3am, reconsider its urgency. Differentiate between levels of urgency, such as page, ticket, or email.

3. Include context in alerts: Ensure alerts clearly convey what’s broken, the severity of the impact, and include relevant links to runbooks or dashboards. Suggest first troubleshooting steps to streamline response times.

Alert Fatigue is a Reliability Issue

An overload of alerts leads to ignored alerts and missed incidents. To combat this, employ strategies to reduce noise: aggregate related alerts, use intelligent thresholds like dynamic baselines, suppress alerts during known maintenance, and regularly review and prune low-value alerts.

Observability Tools

To effectively monitor and analyze your systems, leverage a suite of observability tools:

Metrics

• Prometheus: For time-series metrics collection and querying.
• Grafana: For visualization and dashboarding.
• StatsD: For application-level metric aggregation.

Logging

• ELK Stack: Comprising Elasticsearch, Logstash, and Kibana for comprehensive logging.
• Loki: Prometheus-style log aggregation.
• CloudWatch/Stackdriver: Cloud-native logging solutions.

Tracing

• Jaeger: For distributed tracing in microservices environments.
• Zipkin: To visualize request flows.
• OpenTelemetry: Provides a unified observability standard.

Common Pitfalls

Be vigilant to avoid common pitfalls such as tracking vanity metrics that appear impressive but offer no real value, creating too many dashboards that result in information overload, optimizing for averages while missing the long tail of poor user experiences, not setting SLOs, and allowing alert sprawl that leads to a flood of ignored notifications.

In the following sections, we will delve into the tools and automation strategies that enable scalable SRE practices.

Tools & Scaling

In the realm of Site Reliability Engineering (SRE), your primary mission is to eliminate toil—that repetitive manual labor which fails to yield enduring value. Harnessing the right tools and automation strategies empowers even small teams to efficiently manage vast, intricate systems.

What is Toil?

Google characterizes toil as any task fitting these criteria: it is manual, necessitating human intervention; repetitive, involving recurring actions; automatable, capable of being delegated to software; tactical, being reactive and interrupt-driven; devoid of enduring value, thus offering no permanent improvement to the system; and it scales linearly with service growth, meaning that as your user base expands, so does the workload.

Examples of Toil:

Consider these scenarios as typical manifestations of toil: manually restarting services that have crashed, logging into servers via SSH to inspect logs, executing database cleanup scripts by hand, provisioning new infrastructure manually, and responding to identical alerts repeatedly.

Not Toil:

Conversely, tasks that transcend toil include writing automation scripts to eliminate manual labor, designing new monitoring frameworks, planning for capacity expansion, conducting postmortems, and refactoring code to enhance reliability.

The Automation Hierarchy

Level 1: No Automation

At this level, every task is performed manually. Such a strategy is inherently unscalable.

Level 2: Documentation

Runbooks are introduced to guide human operators. While an improvement, the system remains fundamentally manual.

Level 3: Partial Automation

Scripts exist but still require human judgment for execution. This reduces toil but doesn’t eradicate it completely.

Level 4: Full Automation

Your systems reach a point where they can self-heal without human intervention. This is the goal.

Level 5: Autonomous Systems

In the ideal scenario, systems proactively predict and prevent issues before they occur, achieving a state of autonomy.

Essential SRE Tools

Infrastructure as Code

Treat infrastructure like software

Implement Infrastructure as Code (IaC) with tools like Terraform for multi-cloud provisioning, Ansible for configuration management and orchestration, cloud-native options such as CloudFormation or ARM, and Pulumi which leverages real programming languages for IaC.

Benefits include version-controlled infrastructure, reproducible environments, automated disaster recovery, and the ability to easily roll back changes.

CI/CD Pipelines

Automate the path from code to production

Employ CI/CD solutions such as GitHub Actions for an integrated workflow, GitLab CI for a comprehensive DevOps platform, Jenkins for flexible, self-hosted automation, and CircleCI for quick, cloud-based CI/CD.

Key practices involve automated testing at every phase, deploying canary releases to mitigate risks, automatic rollback in case of failures, and progressive delivery using feature flags.

Container Orchestration

Manage services at scale

Leverage container orchestration with tools like Kubernetes, the industry standard, Docker Swarm for simpler deployments, AWS-native options like ECS/EKS, or Nomad for lightweight orchestration.

The significance of these tools lies in their ability to maintain a declarative desired state, self-heal systems, enable easy horizontal scaling, and optimize resources.

Chaos Engineering

Test reliability by breaking things intentionally

Adopt chaos engineering platforms such as Gremlin for managed chaos testing, Chaos Monkey for random instance termination, LitmusChaos for Kubernetes-native chaos experiments, and conduct GameDay Exercises to simulate failures.

The purpose is to uncover weaknesses before customers do, build confidence in system resilience, train teams for better failure response, and validate monitoring and alerting systems.

Scaling Your SRE Practice

Small Team (1-3 people)

Focus: Foundation and automation

For smaller teams, establish basic monitoring and alerting, automate the most excruciating manual tasks, set up an incident response process, and document everything meticulously.

Medium Team (4-10 people)

Focus: Embedded SRE and specialization

As your team grows, embed SREs within product teams, develop platform tools to address common needs, implement comprehensive Service Level Objectives (SLOs), and build self-service infrastructure.

Large Team (10+ people)

Focus: Platform engineering and scale

For expansive teams, align SRE teams with business domains, create a centralized platform team for shared services, pursue advanced automation and machine learning-driven operations, and establish a production readiness review process.

Measuring Automation Success

To assess your automation success, track these metrics:

• Toil percentage: Compare the time spent on toil versus engineering work, aiming for less than 50%.
• Time to deploy: Measure how quickly you can deliver changes to production.
• Mean time to recovery (MTTR): Evaluate the speed at which incidents are resolved.
• Change failure rate: Determine the percentage of changes that result in incidents.
• Manual intervention rate: Monitor how frequently human intervention becomes necessary.

Common Pitfalls

Beware of these pitfalls in your automation journey:

• Premature optimization: Automating tasks before fully understanding the problem.
• Over-engineering: Creating overly complex solutions for simple tasks.
• Automation debt: Neglecting script maintenance, leading to failures.
• Tool sprawl: Accumulating too many overlapping tools without a clear strategy.
• Ignoring the human factor: Designing automation that is too opaque to debug effectively.

The Right Amount of Automation

Recognize that not every task requires automation.

• Automate: Tasks that are repetitive and occur more than once per week.
• Document: Tasks that are infrequent yet have clear execution steps.
• Eliminate: Processes that needn’t occur at all.

Now that you have a solid understanding of SRE, let’s delve into the steps necessary to establish an SRE program within your organization.

Founding the Program

Embarking on the journey to establish a Site Reliability Engineering (SRE) program in an environment devoid of executive sponsorship or a dedicated budget demands a blend of ingenuity, determination, and strategic foresight. Throughout this chapter, you will discover pragmatic strategies to lay the groundwork for SRE in settings where resources are limited.

Start Where You Are

In your quest to enhance reliability, you don’t need anyone’s permission to begin. Take the initiative to track metrics related to system reliability, refine your runbooks, conduct informal postmortems, automate repetitive tasks, and share your findings with your team. These actions speak volumes. By demonstrating value through tangible efforts rather than mere proposals, you set the stage for future success.

The Grassroots Approach

1. Pick One High-Impact Problem

Resist the temptation to solve every issue at once. Instead, focus on one significant and visible problem. This could be frequent pager alerts for the same issue, a manual deployment process causing outages, a lack of visibility into production health, or slow incident response due to inadequate documentation. By concentrating on a single pain point, you can make meaningful progress.

2. Fix It Measurably

Begin by documenting the current state of the problem, including the time spent, frequency, and impact. Implement a solution and measure the improvements. Once you have concrete results, share them widely to highlight the success. For instance, consider the example of automating a database backup process. Previously, this task consumed two hours of manual work per week, but with automation, it now requires none, freeing up over 100 hours annually.

3. Build Momentum

Success is a powerful catalyst for support. After securing your first victory, it’s time to address another high-impact issue. Involve others in crafting solutions, document your approach for easy replication, and present your findings to leadership. This momentum can drive further buy-in and resource allocation for your SRE initiatives.

Getting Organizational Buy-In

Speak the Language of Business

Technical excellence alone rarely garners attention. To secure organizational buy-in, frame SRE concepts in business terms. Instead of stating, “We need better observability,” articulate it as, “We’re losing $X per incident due to slow detection. Improved monitoring could reduce our Mean Time to Recovery (MTTR) by 50%.” Similarly, replace “We should implement SLOs” with “We can’t prioritize reliability work without clear targets. Service Level Objectives (SLOs) provide an objective means to balance features and stability.”

Build Relationships

SRE cannot thrive in isolation. Forge cross-functional relationships to demonstrate the benefits of SRE to various stakeholders. Show engineering teams how SRE simplifies their tasks, highlight the reliability impact on user experience to product managers, quantify the cost of incidents and automation return on investment for finance teams, and present reliability as a competitive advantage to leadership.

Creative Tactics

When traditional budget requests fall short, employ creative tactics to advance your SRE goals:

Team Dinners and Informal Talks

Host brown-bag lunch sessions centered on SRE topics. Share postmortems and insights over team dinners to foster a community focused on reliability.

Borrow Time from Existing Projects

Integrate reliability improvements into ongoing feature work, treating SRE efforts as “infrastructure” or “tech debt.” Build automation alongside new features to maximize efficiency.

Leverage Open Source Tools

Utilize free tiers of commercial tools like PagerDuty and Datadog, deploy self-hosted open source alternatives, and construct internal tools only when necessary.

Start with Volunteers

Identify engineers with a keen interest in reliability, form an informal working group, and hold regular meetings to discuss practices and share tools.

Defining Your Mission

A well-defined mission guides your efforts and communicates value effectively. Here are some example SRE mission statements:

Example SRE Mission Statements

For a startup, the mission might be, “Ensure customer-facing services maintain 99.9% uptime while enabling rapid feature development.” In an enterprise setting, it could be, “Reduce incident impact on business operations by 50% through automation, monitoring, and cultural change.” A platform team’s mission may focus on providing “reliable, self-service infrastructure that enables product teams to deploy confidently.”

Key Elements

• Measurable outcomes: Set specific reliability targets.
• Stakeholder alignment: Clearly demonstrate value to the business.
• Cultural component: Outline how you’ll transform work practices.
• Scope boundaries: Define what’s included and what’s outside your purview.

Common Obstacles and Solutions

“We don’t have budget”

Start with free or open source tools and demonstrate return on investment through pilot projects. Quantify the cost of incidents to justify further investment.

“We don’t have headcount”

Begin by allocating 20% of time from interested engineers, proving value before seeking dedicated roles. Rotate SRE responsibilities across teams to manage workload.

“Leadership doesn’t see the value”

Track and share the costs of incidents, including downtime, engineering hours, and customer impact. Present case studies from similar companies and start small, letting results speak for themselves.

“Our culture resists change”

Identify allies and champions in other teams. Make SRE practices optional yet appealing, and lead by example rather than enforcing mandates.

Measuring Early Success

To validate your efforts, track key metrics such as incident frequency, MTTR, toil reduction, deploy frequency, and team satisfaction. These indicators help prove the value of your SRE initiatives.

Timeline for Bootstrapping

Month 1-3: Foundation

Begin by identifying a critical reliability problem, implementing a solution, and measuring its impact. Document your approach and share the results with stakeholders to build credibility.

Month 4-6: Expansion

Tackle 2-3 additional high-impact issues, recruit volunteers for an SRE working group, establish basic on-call and incident response practices, and start tracking Service Level Indicators (SLIs) for critical services.

Month 7-12: Formalization

Present a business case for formal SRE investment, define team structure and responsibilities, implement a comprehensive SLO framework, and develop self-service tools for common needs.

Next, we’ll explore how to hire and build your SRE team as the program matures.

Hiring the Team

Building a successful Site Reliability Engineering (SRE) team is not just about assembling individuals with impressive resumes. It requires finding those who possess a unique blend of software engineering expertise, operational experience, and the right mindset for reliability work. These qualities are crucial for ensuring that your systems run smoothly and efficiently, minimizing downtime and enhancing user satisfaction.

What Makes a Great SRE?

Technical Skills

To be an effective SRE, you must possess certain technical skills that go beyond basic engineering. First and foremost, you should be able to write production-quality code. This means your coding is not just functional, but also efficient, maintainable, and scalable. A deep understanding of systems is also essential. You should be familiar with networking, operating system internals, and distributed systems, as these form the backbone of reliable infrastructure. Additionally, being a methodical troubleshooter is crucial. You need to be able to diagnose and resolve issues under pressure, when time is of the essence. Lastly, you must have an automation mindset. You should view repetitive manual tasks as opportunities to create tools that not only save time but also reduce the margin for error.

Non-Technical Skills

While technical prowess is important, non-technical skills are equally vital. As an SRE, you need to communicate complex technical issues clearly to both technical and non-technical stakeholders. This ensures that everyone understands the problems and solutions involved. Blameless thinking is another key attribute; you should focus on improving systems rather than assigning blame to individuals when things go wrong. Customer empathy is crucial as well; you must understand the user impact of reliability issues and strive to enhance their experience. Furthermore, a great SRE should have the ability to teach. Sharing knowledge and building team capabilities are essential for the continuous improvement of the team. Finally, emotional resilience is necessary to handle the stress of being on-call and managing incidents without succumbing to burnout.

The SRE Mindset

A great SRE embodies a particular mindset that sets them apart. Curiosity about failure drives them to understand why things break and how to prevent future incidents. They possess a bias toward automation, constantly seeking to eliminate toil rather than simply accepting it as part of the job. Decisions are data-driven, relying on measurable outcomes rather than assumptions. Healthy paranoia is a trait that allows them to anticipate what could go wrong and plan accordingly. Lastly, continuous learning is a cornerstone of their professional development, keeping them current with evolving technologies and industry practices.

SRE vs. DevOps vs. Platform Engineering

To target your hiring efforts effectively, it’s important to understand the distinctions between SRE, DevOps, and Platform Engineering roles.

Site Reliability Engineer

As an SRE, your primary focus is on production reliability and incident response. This involves managing Service Level Objectives (SLOs), monitoring systems, being on-call, conducting postmortems, and eliminating toil. Typically, SREs come from a software engineering background with a keen interest in operations.

DevOps Engineer

DevOps Engineers concentrate on Continuous Integration/Continuous Deployment (CI/CD) and deployment automation. Their activities often include building pipelines, provisioning infrastructure, and developing tools that improve developer productivity. They usually have a mix of development and operations experience.

Platform Engineer

Platform Engineers focus on creating internal developer platforms and facilitating self-service infrastructure. They build tools, APIs, and abstractions to support product teams. Strong software engineering skills are a common background for this role.

Note: Job titles can vary widely, so it’s more effective to focus on the responsibilities and skills associated with each role rather than the titles themselves.

Building Your First SRE Team

Option 1: Internal Conversion

One approach to building your SRE team is to identify existing engineers who already exhibit SRE behaviors. These are individuals who volunteer for on-call responsibilities, build automation tools in their spare time, lead postmortems, and have a deep concern for production reliability.

Advantages:

• They already understand your systems.
• They are a proven cultural fit.
• Onboarding is faster.

Challenges:

• They may lack formal SRE training.
• Time is needed to develop new skills.
• The team may lose a valued engineer from their current role.

Option 2: External Hiring

Alternatively, you can recruit experienced SREs from other organizations.

Advantages:

• They bring fresh perspectives and best practices.
• They can quickly apply SRE fundamentals.
• They can mentor internal team members.

Challenges:

• They require a longer ramp-up time to learn your systems.
• Cultural fit may be uncertain.
• They might have expectations from larger companies.

Option 3: Hybrid Approach

A hybrid approach often proves to be the most effective strategy. By hiring an experienced SRE to lead the team and developing other members internally, you can leverage the expertise and direction from the external hire while benefiting from the system knowledge of internal engineers. This approach fosters knowledge transfer and creates mentorship opportunities.

Interview Process

Technical Assessment

During the interview process, focus on realistic scenarios rather than trivia. This ensures you are assessing candidates based on practical skills and thinking.

Troubleshooting Exercise

Present candidates with a production incident scenario, providing incomplete information to mimic real-life situations. Observe their debugging methodology, assess how they communicate their findings, and evaluate their prioritization of actions.

System Design

Ask candidates to design a reliable system. Inquire how they would ensure high availability for a component, identify potential failure modes, monitor the system, and make necessary trade-offs.

Coding Challenge

Give candidates a practical automation task, such as parsing logs and extracting metrics, writing a health check system, or building a simple deployment tool. Look for code quality, readability, error handling, testing approach, and documentation skills.

Cultural Assessment

Blameless Thinking

Ask candidates about a production incident they caused. A good response will focus on system failures, lessons learned, and improvements made, while a red flag response will deflect blame and focus on individual mistakes.

Handling Pressure

Inquire how candidates handle being paged at 3am for a critical incident. A good response will describe a methodical approach, knowing when to escalate, and learning from incidents, whereas a red flag response will exhibit a hero complex and ignore the impact on team sustainability.

Collaboration

Ask candidates to describe a time they disagreed with a decision. A good response will involve respectful disagreement, data-driven arguments, and acceptance of team decisions, while a red flag response will dismiss others and show stubbornness without evidence.

Team Structure Models

Embedded SRE

In this model, SREs are assigned to specific product teams.

Pros:

• They gain deep knowledge of specific services.
• There’s close collaboration with developers.
• There is clear ownership and accountability.

Cons:

• They can become siloed.
• It can be harder to share knowledge across teams.
• They may get pulled into feature work.

Centralized SRE

A centralized SRE model involves a single SRE team supporting all services.

Pros:

• There’s shared knowledge and best practices.
• Resource utilization is better.
• There is a clear SRE identity and culture.

Cons:

• The team can become a bottleneck.
• There is less deep knowledge of individual services.
• They may be perceived as “gatekeepers.”

Hybrid Model

The hybrid model combines the best of both worlds by maintaining a central SRE team with embedded rotations. SREs rotate through product teams for 3-6 months, which allows for deep knowledge without permanent silos while maintaining shared practices and a cohesive team culture.

Onboarding New SREs

Week 1: Foundation

During the first week, ensure new SREs have access to all systems and tools, shadow incident response teams, read key postmortems, and meet stakeholders across engineering.

Week 2-4: Shadowing

In the following weeks, have them pair with experienced SREs on troubleshooting tasks, join the on-call rotation as a secondary responder, participate in postmortems, and begin small automation projects.

Week 5-8: Independence

By weeks five to eight, new SREs should take on-call shifts as primary responders (with backup), own specific services or systems, lead postmortems, and propose and implement improvements.

Month 3+: Full Contribution

Beyond the initial onboarding period, SREs should be fully integrated into team rotations, driving SRE projects independently, mentoring newer team members, and influencing SRE strategy.

Growing Senior SRE Talent

Career Progression

It’s crucial to define clear paths for career growth within your SRE team. As a Junior SRE, you learn systems, assist with incidents, and execute defined projects. As an SRE, you own services, lead incidents, and design automation solutions. A Senior SRE sets technical direction, mentors the team, and drives strategic initiatives. Finally, a Staff/Principal SRE influences organization-wide reliability and architects platforms.

Retention Strategies

To keep your SRE team engaged, offer variety by rotating responsibilities to prevent burnout. Ensure they work on problems that matter, support their learning through conferences, training, and certifications, include them in architectural decisions, and respect on-call boundaries and time off.

Common Hiring Mistakes

Avoid these common pitfalls when hiring for your SRE team. Do not hire only operations people, as SRE requires strong coding skills. Similarly, avoid hiring only developers, as production operations is a distinct skillset. Overvaluing credentials can lead you to miss candidates with demonstrated abilities, while ignoring culture fit means technical skills alone won’t suffice. Lastly, ensure the current SRE team is involved in evaluating candidates to ensure a good fit.

Next, we’ll explore how to size your SRE team appropriately for your organization’s scale and needs.

Sizing SRE to Your Org

Determining the appropriate number of Site Reliability Engineers (SREs) for your organization is a nuanced endeavor. It hinges on various factors such as the size and complexity of your organization, the frequency and severity of incidents, and your maturity in reliability practices. While there isn’t a one-size-fits-all formula, there are tested approaches that can guide your decision-making process.

The Google Rule of Thumb

Google offers a foundational guideline suggesting the employment of 1 SRE for every 10 software engineers. This rule serves as a starting point rather than a definitive answer. The actual number of SREs you require will be shaped by the complexity and distribution of your services, the frequency and severity of incidents, the level of automation maturity, the sustainability of on-call rotations, and the business criticality of your systems.

Factors That Increase SRE Needs

High Incident Volume

If your team finds itself constantly in firefighting mode, you will require more SREs to maintain sustainable on-call rotations. However, your focus should be on reducing toil rather than just increasing headcount.

Complex Distributed Systems

Managing microservices, multi-region deployments, and heterogeneous infrastructure demands highly specialized knowledge. These systems are inherently harder to troubleshoot and maintain, which might necessitate having embedded SREs for each major service.

Rapid Growth

When your organization is scaling quickly, capacity planning becomes crucial. Frequent changes in infrastructure demand that you stay ahead with automation to keep up with growth.

Regulatory Requirements

Compliance and auditing introduce additional documentation and process overhead. You will need more monitoring and controls, which might lead to separate SRE focus areas to address these requirements.

Factors That Decrease SRE Needs

High Automation Maturity

As your systems evolve to become self-healing, the operational burden decreases. With reduced need for manual intervention, fewer pages and incidents occur, allowing SREs to focus on more strategic work.

Simple Architecture

In environments with monolithic or well-understood systems, there are fewer failure modes. Troubleshooting is easier, and there is less coordination overhead.

Strong DevOps Culture

When developers take ownership of production, shared on-call responsibilities become the norm. Here, SREs act more as consultants, focusing on platform work, and a smaller centralized team suffices.

Managed Services

Extensive use of Platform as a Service (PaaS) or Software as a Service (SaaS) means less infrastructure to manage. The provider takes care of underlying reliability, allowing SREs to focus on the application layer.

Team Size by Organization Scale

Startup (<50 engineers)

For a startup, the typical SRE team consists of 0-2 people. Initially, you may not have dedicated SREs, with developers handling on-call and operations. The first SRE hire should concentrate on automation and standards to lay the groundwork for future scaling. Watch out for red flags indicating the need for SREs, such as frequent production incidents, developers spending over 50% of their time on operations, scary or slow deployments, and a general lack of understanding about how production works.

Small Company (50-200 engineers)

In a small company, you might find a centralized SRE team of 2-10 people. This team should focus on platforms and self-service, embed with critical services, and establish a Service Level Objective (SLO) framework. The team composition typically includes 1-2 Senior SREs for leadership, 2-5 SREs for execution, and 1-2 Platform Engineers for tooling.

Mid-Size Company (200-1000 engineers)

Mid-size companies generally have SRE teams ranging from 10-50 people. Here, a mix of embedded and centralized teams often works best, with SRE teams aligned to product areas and a central platform team for shared services. Formal production readiness reviews become part of the approach. The team structure usually consists of 2-4 SREs per major product area, 5-10 SREs on the platform team, and 1-2 SREs focusing on incident response and coordination.

Large Company (1000+ engineers)

In large companies, SRE teams can range from 50 to over 200 people. A fully federated SRE organization is common, with SRE leadership distinct from the engineering teams. Specialized roles such as chaos engineering and capacity planning emerge, and SREs act as a consulting service to product teams. The team structure typically includes multiple SRE teams per business unit, dedicated SRE leadership and program management, centers of excellence for specific domains, and organized SRE rotations and career development programs.

The On-Call Constraint

Ensuring a sustainable on-call rotation is crucial and requires a minimum team size.

Minimum Viable On-Call: 4 people

With a team of four, you can maintain weekly rotations where each person is on-call once per month. This allows for vacation and sick time, assuming you have a secondary on-call backup.

Comfortable On-Call: 6-8 people

A team of six to eight people allows for biweekly rotations, meaning each person is on-call every six to eight weeks. This setup provides a better work-life balance and is more resilient to turnover.

Multi-timezone Coverage: 12+ people

For 24/7 support across regions, a team of twelve or more enables regional on-call rotations, significantly reducing off-hours pages.

If your team cannot sustain the minimum on-call size, consider using external on-call services, sharing on-call duties with development teams, reducing service scope, or focusing on automation to decrease pages.

Efficiency Metrics

To guide your decisions regarding team sizing, you should measure SRE efficiency.

Toil Percentage

Your goal should be for less than 50% of SRE time to be spent on toil. If the team is spending more than half of their time on manual operations, you need to either hire more SREs or invest significantly in automation.

Services per SRE

A good benchmark is managing 5-10 services per SRE. In highly automated environments, this number can increase to 10-20 services per SRE, while in manual environments, it may decrease to 2-5 services.

Incident Load

Aim for less than two incidents per SRE per week. If incident rates are higher, this suggests a need for more SREs, improvements in reliability, or both.

When to Grow Your Team

Signals You Need More SREs

1. Burnout and turnover: The team is unsustainably stressed.
2. Incident backlog: You can’t keep up with response and follow-up.
3. Abandoned projects: There’s no time for automation or improvements.
4. Coverage gaps: You can’t support on-call or new services.
5. Toil creep: More time is being spent on manual work over time.

Signals You Can Wait

1. Low incident load: The team is handling the current volume easily.
2. Automation working: Toil is decreasing over time.
3. Healthy on-call: Rotations are sustainable and fair.
4. Project velocity: You are able to execute on strategic initiatives.
5. Team satisfaction: SREs feel engaged and valued.

Alternatives to Hiring

Before deciding to grow your team, consider these alternatives:

Automation First

Invest in eliminating toil, building self-service platforms, and implementing better observability.

Shared Responsibility

Train developers on production operations and implement a “you build it, you run it” culture, allowing SREs to shift toward consulting rather than execution.

Outsourcing

Consider managed services for infrastructure, specialized SRE consultants for projects, or an external Network Operations Center (NOC) for tier-1 support.

Process Improvements

Focus on better postmortem follow-through, improved runbooks and documentation, and streamlined incident response processes.

Common Sizing Mistakes

Beware of common pitfalls such as hiring too early before understanding actual needs, hiring too late after the team is already burned out, assuming SRE needs grow linearly with the number of engineers, ignoring automation by adding personnel to handle toil, having the wrong skill mix (all senior or all junior engineers), and lacking a clear scope, which leads to SREs being pulled into everything.

Making the Business Case

When you need to request additional headcount, you should be prepared to quantify the problem, show the return on investment (ROI), and present options.

Quantify the Problem

For instance, “We had 50 incidents last quarter, costing X engineering hours” or “Current on-call rotation is every two weeks, causing burnout.”

Show the ROI

Explain potential gains, such as “Adding two SREs would reduce Mean Time to Recovery (MTTR) by 50%, saving $X in downtime.”

Present Options

Lay out the possibilities: “Option A: Hire 2 SREs, focus on automation ($XXX/year),” “Option B: Implement follow-the-sun on-call with the current team (unsustainable),” or “Option C: Continue as-is, accept a 20% engineer turnover risk.”

Next, we’ll explore how to move fast while maintaining reliability—the core tension SRE exists to balance.

Moving Fast

In the realm of Site Reliability Engineering (SRE), you often grapple with the fundamental tension of shipping features rapidly while maintaining the reliability of your systems. This balance might seem elusive, but by leveraging error budgets and implementing thoughtful practices, you can achieve both speed and stability.

The Reliability vs. Velocity Trade-off

Traditional Approaches (Both Wrong)

You might be familiar with the mantra “Move fast and break things.” This approach encourages you to continuously ship features, accepting frequent outages as collateral damage. However, this inevitably leads to a loss of customer trust and eventually slows down progress as incidents accumulate.

On the other hand, there is the approach of “Extreme stability at all costs,” where exhaustive testing and review processes lead to sluggish release cycles. Your features may lag behind competitors, and engineers often find themselves mired in frustrating bureaucracy.

The SRE Approach

The SRE methodology seeks to balance reliability and velocity through data-driven decision making. A key insight here is recognizing that aiming for 100% reliability is not only impractical but also counterproductive. It is an unattainable goal that proves too costly and stifles innovation. Most importantly, users typically do not perceive a noticeable difference between 99.99% and 100% reliability.

Error Budgets: The Game Changer

What is an Error Budget?

An error budget defines the permissible level of unreliability your service can withstand before violating its Service Level Objective (SLO). For instance, if your SLO dictates 99.9% uptime, your error budget allows for 0.1% downtime, translating to approximately 43 minutes per month or around 8.5 hours annually.

How Error Budgets Drive Decisions

When You Have Budget Remaining

When your error budget is intact, you can afford to ship faster. This means you can accept more risk in deployments, experiment with new features, try architectural changes, and deploy with greater frequency.

When You’ve Spent Your Budget

Conversely, if you have depleted your error budget, the focus must shift to reliability. You should freeze feature launches, address underlying issues, enhance testing and automation, and pay down technical debt.

Error Budget Policy

Establishing a transparent error budget policy ensures everyone is on the same page:

When error budget is:
- >75% remaining: Normal feature velocity
- 50-75% remaining: Increase caution, review change process
- 25-50% remaining: Focus on stability, defer risky features
- <25% remaining: Feature freeze, all-hands on reliability

Such a policy eliminates emotional debates about whether to proceed with a release, as data becomes the decisive factor.

Deployment Strategies for Speed + Safety

Continuous Deployment

Automating the entire path to production is key in continuous deployment. This strategy offers the benefit of small, incremental changes, which are inherently safer. You gain the ability to roll back swiftly when issues occur, reduce manual errors, and increase deployment frequency. However, to succeed, you need comprehensive automated testing, robust observability, feature flags for risk mitigation, and rollback automation.

Progressive Delivery

Progressive delivery emphasizes not releasing to everyone simultaneously.

Canary Deployments

This involves deploying to a small percentage of your traffic, typically 1-5%, and monitoring for errors or performance issues. If the deployment proves healthy, you can gradually increase exposure to 100%, with automatic rollback in case of issues.

Blue/Green Deployments

In this setup, you maintain two identical environments. Initially, traffic routes to “blue” (the current version), while you deploy changes to “green” (the new version). Once satisfied, you switch traffic to “green,” with an easy rollback by reverting the traffic back to “blue.”

Ring Deployments

Ring deployments enable you to deploy in stages to different user groups. Start with Ring 0 (internal employees), progress to Ring 1 (beta users or canary customers), and subsequently to Ring 2 (a broader rollout) before reaching Ring 3 (everyone). This staged approach helps you catch issues before they impact all users.

Feature Flags

Feature flags allow you to decouple deployment from release. You can deploy code to production without activating features immediately. This enables gradual rollouts to a percentage of users and provides the ability to disable features instantly if problems arise. Additionally, feature flags facilitate A/B testing and experimentation.

if feature_flag.is_enabled("new_checkout_flow", user):
    return new_checkout()
else:
    return legacy_checkout()

Testing Strategies

Test Pyramid

A well-structured test pyramid helps ensure code quality.

Unit Tests (70% of tests) are fast, isolated, and numerous, focusing on individual functions or components. They should run on every code change.

Integration Tests (20% of tests) verify interactions between components and check the contracts between services, helping catch integration bugs.

End-to-End Tests (10% of tests) focus on critical user journeys. While they are slow and brittle, they are essential for verifying the most important flows.

Production Testing

Testing should not cease after deployment.

Synthetic Monitoring

This involves automated checks that run continuously, simulating real user behavior to detect issues before customers do.

Chaos Engineering

Chaos engineering intentionally injects failures into the system to verify resilience, test disaster recovery procedures, and build confidence in the system’s reliability.

Load Testing

Load testing assesses system behavior under stress, helping you identify capacity limits, find performance bottlenecks, and validate autoscaling.

Change Management

Production Readiness Reviews

Before launching new services, ensure production readiness by verifying:

Monitoring & Alerting: Instrument golden signals, configure and test alerts, create dashboards, and write on-call runbooks.

Reliability: Define SLOs, load test at expected scale, identify failure modes, and document a disaster recovery plan.

Operations: Ensure deployment automation works, test rollback procedures, document dependencies, and complete capacity planning.

Change Review Process

Not all changes require the same level of scrutiny.

Low-risk changes can be auto-approved and include configuration updates, documentation changes, feature flag toggles, and minor bug fixes.

Medium-risk changes require peer review and encompass new features, database migrations, dependency updates, and performance optimizations.

High-risk changes necessitate SRE review and approval, covering architectural changes, security updates, database schema changes, and infrastructure migrations.

Incident Response That Enables Speed

Blameless Postmortems

When incidents occur, adopt a mindset of assuming good intent. Engineers make reasonable decisions based on available information. Focus on systems, as human errors indicate systemic weaknesses. Fix the system to prevent similar mistakes in the future, fostering psychological safety that encourages speed.

Fast Feedback Loops

Quickly identifying issues is crucial for maintaining momentum:

• Automated testing provides feedback in minutes, not hours.
• Canary deployments limit impact to 1% of users, not 100%.
• Good monitoring detects issues in seconds.
• Clear alerts help you understand exactly what’s wrong.
• Practiced runbooks enable swift resolution.

Mean Time to Recovery > Mean Time Between Failures

Accept that incidents will occur and prioritize quick recovery. Aim for recovery times measured in minutes, not hours. Automate common fixes, practice incident response, and understand that perfection is unattainable.

Cultural Enablers of Speed

Trust Between Teams

Trust between SRE and development teams is essential. SRE trusts developers to write reliable code, respond to incidents, and implement postmortem action items. Developers trust SRE to avoid unnecessary feature blocks, make data-driven decisions, and support velocity goals.

Shared Responsibility

The principle of “You build it, you run it” ensures developers experience the impact of their code in production. This responsibility incentivizes building reliability and provides faster feedback on design decisions, leading to better incident response. The SRE role shifts to platform and tooling, consulting and advice, setting standards, and handling cross-cutting concerns.

Learning Culture

A culture that celebrates learning rather than just success is crucial. Share postmortems widely, conduct “near-miss” reviews, run game days and simulations, and encourage experimentation.

Common Pitfalls

Avoid these common pitfalls:

• Ignoring error budgets by making them advisory instead of binding
• Perfectionism, which leads to the futile elimination of all risk
• A blame culture that punishes failures instead of learning from them
• No rollback plan, assuming deployments will always succeed
• Manual bottlenecks that slow progress with required approvals
• Ignoring toil by accepting manual work instead of automating it

Metrics That Matter

Monitor these metrics to assess your balance between speed and reliability:

• Deployment frequency: How often you ship
• Lead time: From code commit to production
• Change failure rate: Percentage of deployments causing incidents
• MTTR: How quickly you recover from failures
• Error budget consumption: Are you staying within SLOs?

High-performing teams achieve multiple deployments per day, maintain a change failure rate below 15%, and keep MTTR under one hour.

As we move forward, let’s explore the future of SRE and the emerging trends shaping this discipline.

The Future

As the field of Site Reliability Engineering (SRE) evolves, keeping pace with emerging trends is crucial for honing your practice and preparing for the future of complex systems. This journey involves not only understanding new technologies but also adapting your skills and mindset to thrive in an ever-maturing industry.

Emerging Trends

AI and Machine Learning in SRE

AI and machine learning are increasingly being integrated into operations, a movement often referred to as AIOps. Currently, machine learning models are adept at identifying anomalies in metrics, effectively reducing noise by correlating related alerts, forecasting capacity needs before they become critical, and suggesting probable causes during root cause analysis based on historical data. As you look to the future, anticipate advancements like autonomous remediation, where AI could resolve common issues independently, and predictive incident management, which aims to preempt problems before they lead to outages. Additionally, intelligent on-call systems may soon triage alerts, escalating only when necessary. However, remember that AI is a tool to enhance, not replace, your judgment. It should augment human decision-making, not supplant it.

Platform Engineering Evolution

The focus of SRE is shifting from merely maintaining service uptime to constructing self-service platforms that empower developers to deploy reliably. This evolution involves creating developer portals that serve as a single interface for all infrastructure needs, establishing “golden paths” with opinionated, pre-configured templates for common development patterns, and maintaining a centralized service catalog of all services and dependencies. By automating guardrails through policy as code, you can prevent reliability issues before they arise.

Observability 2.0

The realm of observability is expanding beyond traditional metrics, logs, and traces into new paradigms. Continuous profiling offers always-on performance analysis, while eBPF-based observability provides deep kernel-level insights without the need for instrumentation. OpenTelemetry emerges as a vendor-neutral instrumentation standard, and real user monitoring shifts focus to understanding actual user experiences over synthetic tests. The overarching goal is to comprehend any system state without the need to predict what to monitor in advance.

Chaos Engineering at Scale

Chaos engineering is transitioning from isolated experiments to a continuous validation process. This evolution includes automating chaos testing within CI/CD pipelines, conducting chaos tests in staging and canary deployments rather than production, and shifting from one-off tests to ongoing resilience validation. The focus has broadened from infrastructure failures to include application-level fault injection. Tools like Chaos Mesh, AWS Fault Injection Simulator, Gremlin, and LitmusChaos are maturing to support these efforts.

Serverless and SRE

In a serverless landscape, SRE faces new challenges such as reduced control over infrastructure, unique failure modes, cold start performance issues, and the need for cost optimization as a reliability concern. Your focus should pivot toward API design and client resilience, enhancing observability in ephemeral environments, managing distributed states, and efficiently overseeing third-party dependencies.

GitOps and Infrastructure as Code

Embracing GitOps means managing everything through Git, eliminating manual interventions. This approach benefits SRE by ensuring all changes are auditable, enabling easy rollbacks via Git revert, and mandating that changes undergo code review. It supports a declarative approach to desired state management. Popular tools like ArgoCD, Flux, Terraform Cloud, and Crossplane facilitate this transition.

Evolving Team Structures

From SRE Teams to SRE Practices

The future may see a shift from dedicated SRE teams to SRE practices where SRE serves as a role rather than a standalone team. This can involve rotating responsibilities among engineers, adopting an SRE consulting model with central experts advising product teams, embedding SRE principles in self-service platform teams, and encouraging distributed ownership where developers hold reliability responsibilities with SRE support.

The Rise of Production Engineering

Production engineering broadens the scope beyond traditional SRE by encompassing developer experience and productivity, build and release engineering, developer tooling, and infrastructure. It positions SRE as a subset of production engineering, reflecting the reality that reliability depends on the entire development and deployment pipeline.

Skills SREs Will Need

Core Fundamentals (Still Critical)

To remain effective, you must continue to hone essential skills in software engineering, distributed systems, networking, infrastructure, and incident response.

Emerging Skills

As the field evolves, new skills become increasingly vital. Proficiency in Kubernetes and cloud-native technologies is becoming the norm. Understanding security engineering as it converges with SRE, acquiring data engineering expertise for large-scale observability, and developing product thinking to align with user needs and business impact are crucial. Additionally, cost optimization is emerging as a core reliability concern.

Soft Skills (More Important Than Ever)

Given SRE’s pervasive influence across engineering, developing soft skills is more important than ever. Cross-functional collaboration, teaching and mentoring, strategic thinking to align SRE work with business outcomes, and effective communication of complex technical topics to diverse audiences are paramount.

Industry Standardization

SRE Best Practices Becoming Standards

The industry is moving toward standardized practices. Concepts like OpenSLO for SLO definitions, OpenTelemetry for unified observability instrumentation, DORA metrics for standardized DevOps performance measurement, and common postmortem formats are becoming widespread expectations rather than Google-specific practices.

Certifications and Training

SRE is gaining recognition as a formal discipline. Certification programs by Google, university courses on site reliability, SRE bootcamps, and dedicated conference tracks are proliferating, signaling the maturation of the field.

Challenges Ahead

Complexity Continues to Grow

Despite advances, systems are becoming more complex. Microservices sprawl, multi-cloud deployments, edge computing, and IoT devices contribute to this complexity. Your SRE practice must adapt to manage these challenges without overwhelming your team.

Sustainability and Environmental Impact

Reliability now encompasses environmental responsibility. Future considerations should include carbon-aware scheduling, energy-efficient architectures, right-sizing to minimize waste, and geographic distribution for enhanced efficiency.

Economic Pressures

Economic constraints require doing more with less. Cost optimization becomes a first-class concern, requiring justification of SRE investments, even in downturns. Automation can maintain reliability with leaner teams, and strategic use of managed services can ease resource constraints.

Security and Reliability Convergence

Security and reliability share a common goal: availability. The overlap between SRE and security is growing, encompassing DDoS protection, zero-trust networking, secrets management, compliance, and auditing. This convergence might evolve SRE into “Secure Reliability Engineering.”

Advice for the Future

Stay Focused on Fundamentals

While trends shift, core principles endure. Measure reliability objectively, automate toil, learn from failures, and balance reliability with velocity. These fundamentals remain your guiding compass.

Invest in Platform Thinking

The future of SRE lies in making the right thing the easy thing by building self-service infrastructure and embedding reliability into developer workflows.

Keep Learning

SRE, as a young discipline, continues to evolve. Engage with industry thought leaders, experiment with new tools and practices, share insights with the community, and stay curious about emerging technologies.

Remember the Mission

SRE’s mission is unwavering: enable business value delivery, empower engineers to ship confidently, and ensure excellent customer experiences. While technology evolves, this mission remains at the core of SRE.

Closing Thoughts

Implementing SRE principles doesn’t require Google’s resources, team size, or systems. What you need are clear goals, data-driven decision-making, automation to eliminate toil, and a culture that embraces blameless learning from failures. Balance reliability with velocity and start small. Measure, iterate, and share your progress to build momentum. SRE is about cultivating a culture of reliability, one step at a time.

Additional Resources

Books

• Site Reliability Engineering (Google)
• The Site Reliability Workbook (Google)
• Seeking SRE (Community perspectives)
• Database Reliability Engineering (Laine Campbell)

Websites & Communities

• SRE Weekly Newsletter
• Reddit r/sre
• SREcon Conference
• USENIX LISA Conference

Tools & Platforms

• Prometheus & Grafana (Observability)
• PagerDuty (Incident response)
• Terraform (Infrastructure as code)
• OpenTelemetry (Observability standard)

Blogs Worth Following

• Google Cloud Blog (SRE practices)
• Netflix Tech Blog
• Uber Engineering Blog
• GitHub Engineering Blog

Thank you for reading SRE From Scratch. Now go build reliable systems.

Templates & Checklists

This appendix provides ready-to-use templates and checklists you can adapt for your organization. Copy, customize, and put them to work immediately.

Available Templates

Incident Response Templates

Postmortem and runbook templates for documenting and responding to incidents. Use these to establish consistent incident practices across your team.

SLO & SLI Templates

Service Level Objective documentation template with SLI definitions, error budget policies, and compliance tracking.

Checklists

Production readiness, weekly team health checks, new SRE onboarding, and change review checklists to ensure nothing gets missed.

Reference Cards

Golden signals and incident severity quick reference tables.

Communication Templates

Email templates for requesting SRE tools budget and proposing an SRE program.

These templates are starting points. Adapt them to your organization’s needs, culture, and tooling. The best templates are living documents that evolve with your team.

Incident Response Templates

These templates help establish consistent incident documentation and response procedures across your organization.

Incident Postmortem Template

Use this template after every significant incident to capture learnings and prevent recurrence.

# Incident Postmortem: [Brief Description]

**Date**: [YYYY-MM-DD]
**Severity**: [SEV-1/2/3/4]
**Duration**: [Start time - End time] ([Total duration])
**Impact**: [Customer/business impact description]

## Summary
[2-3 sentence overview of what happened]

## Timeline (all times in [timezone])
- **HH:MM** - [Event description]
- **HH:MM** - [Event description]
- **HH:MM** - [Detection/Alert fired]
- **HH:MM** - [Response began]
- **HH:MM** - [Mitigation applied]
- **HH:MM** - [Incident resolved]

## Root Cause
[Detailed explanation of the underlying cause. Include technical details.]

## Contributing Factors
- [Factor 1: What allowed this to happen?]
- [Factor 2: Why wasn't it caught earlier?]
- [Factor 3: What made it worse?]

## Impact
- **Customers affected**: [Number/percentage]
- **Revenue impact**: [$Amount or N/A]
- **Engineering time**: [Hours spent responding]
- **User-facing symptoms**: [What users experienced]

## What Went Well
- [Positive aspect 1]
- [Positive aspect 2]

## What Went Poorly
- [Issue 1]
- [Issue 2]

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| [Fix underlying issue] | @person | YYYY-MM-DD | Open |
| [Improve monitoring] | @person | YYYY-MM-DD | Open |
| [Update runbook] | @person | YYYY-MM-DD | Open |

## Lessons Learned
- [Key takeaway 1]
- [Key takeaway 2]

Incident Response Runbook Template

Create runbooks for your critical services and common failure modes. A good runbook enables any on-call engineer to respond effectively.

# Runbook: [Service/Issue Name]

**Last Updated**: [Date]
**Maintained By**: [Team/Person]

## Overview
[What this runbook covers]

## When to Use This Runbook
- [Symptom/alert 1]
- [Symptom/alert 2]

## Severity Assessment
- **SEV-1 (Critical)**: [Criteria]
- **SEV-2 (High)**: [Criteria]
- **SEV-3 (Medium)**: [Criteria]

## Quick Actions (Do This First)
1. [Immediate action to reduce impact]
2. [Second immediate action]
3. [Communication: Who to notify]

## Diagnostic Steps

### Check System Health
# [Command to check service status]
# [Command to check logs]
# [Command to check metrics]

### Common Causes
1. **[Issue type 1]**
   - Symptoms: [How to identify]
   - Fix: [Resolution steps]

2. **[Issue type 2]**
   - Symptoms: [How to identify]
   - Fix: [Resolution steps]

## Escalation
- **L1**: [On-call engineer] - [Response actions]
- **L2**: [Senior engineer/team lead] - [Escalate after X minutes or if Y condition]
- **L3**: [Engineering manager] - [Escalate for severity or duration thresholds]

## Recovery Steps
1. [Step 1]
2. [Step 2]
3. [Verification: How to confirm resolution]

## Communication Templates

**Internal Slack Message**:
🚨 Incident: [Brief description]
Severity: [SEV-X]
Impact: [What's affected]
Status: [Investigating/Mitigating/Resolved]
Next update: [Time]

**Customer Status Page**:
We are currently experiencing [issue]. 
Our team is actively investigating. 
We will provide updates every [X minutes].

## Related Documentation
- [Link to architecture docs]
- [Link to monitoring dashboard]
- [Link to previous incidents]

SLO & SLI Templates

This template helps you document Service Level Objectives for your critical services, including SLI definitions, targets, and error budget policies.

Service Level Objective (SLO) Document

# SLO: [Service Name]

**Owner**: [Team/Person]
**Last Review**: [Date]
**Next Review**: [Date]

## Service Description
[What this service does and why it matters]

## SLIs (Service Level Indicators)

### 1. Availability SLI
**Definition**: Percentage of successful requests
**Measurement**: sum(requests{status!~"5.."}) / sum(requests) * 100
**Data Source**: [Prometheus/CloudWatch/etc.]

### 2. Latency SLI
**Definition**: 95th percentile response time
**Measurement**: p95 of request duration < 200ms
**Data Source**: [Monitoring system]

### 3. Error Rate SLI
**Definition**: Percentage of requests returning errors
**Measurement**: (5xx errors / total requests) * 100
**Data Source**: [Logging/metrics system]

## SLOs (Service Level Objectives)

| SLI | Target | Measurement Window | Error Budget |
|-----|--------|-------------------|--------------|
| Availability | 99.9% | 30 days | 0.1% (43 min) |
| Latency p95 | < 200ms | 7 days | 5% above threshold |
| Error Rate | < 0.1% | 30 days | 10 errors per 10k requests |

## Error Budget Policy

### Budget Remaining: > 75%
- ✅ Normal feature development velocity
- ✅ Accept reasonable risks in deployments
- ✅ Experiment with new features

### Budget Remaining: 50-75%
- ⚠️ Increase caution in changes
- ⚠️ Review deployment processes
- ⚠️ Defer risky features

### Budget Remaining: 25-50%
- 🛑 Focus on stability improvements
- 🛑 Limit feature work to low-risk changes
- 🛑 Increase monitoring and testing

### Budget Remaining: < 25%
- 🚨 Feature freeze
- 🚨 All hands on reliability
- 🚨 Root cause all recent incidents
- 🚨 Implement comprehensive fixes

## Dashboard Links
- [Production monitoring]
- [Error budget tracking]
- [SLO compliance reports]

Checklists

Practical checklists for common SRE activities. Print them, adapt them, and use them to ensure nothing gets missed.

Production Readiness Checklist

Use this before launching any new service:

Monitoring & Observability

• Golden signals instrumented (latency, traffic, errors, saturation)
• Application logs structured and centralized
• Distributed tracing implemented
• Dashboards created for key metrics
• SLIs defined and measured
• SLOs documented with error budgets

Alerting

• Critical alerts configured and tested
• Alert runbooks written
• On-call rotation assigned
• Alert thresholds validated (not too noisy)
• Escalation procedures documented
• Test alert sent and verified

Reliability & Resilience

• Load tested at 2x expected peak traffic
• Failure modes identified and documented
• Circuit breakers implemented for dependencies
• Graceful degradation tested
• Rate limiting configured
• Retry logic with exponential backoff
• Timeout policies set on all external calls

Deployment & Rollback

• Automated deployment pipeline tested
• Canary deployment strategy implemented
• Rollback procedure tested successfully
• Feature flags for major changes
• Database migrations are reversible
• Zero-downtime deployment verified

Capacity & Scaling

• Autoscaling configured and tested
• Resource limits set (CPU, memory, connections)
• Capacity plan documented for next 6 months
• Database connection pooling configured
• Caching strategy implemented

Security

• Authentication and authorization implemented
• Secrets managed securely (not in code)
• Input validation on all endpoints
• Rate limiting to prevent abuse
• Security scan passed
• Least privilege access configured

Documentation

• Architecture diagram up to date
• API documentation complete
• Runbooks for common operations
• Disaster recovery plan documented
• Dependencies mapped
• Contact information for dependencies

Disaster Recovery

• Backup strategy implemented and tested
• Recovery Time Objective (RTO) defined
• Recovery Point Objective (RPO) defined
• DR drill completed successfully
• Backup restoration tested

Weekly SRE Team Health Check

Run this with your team every week:

Toil Assessment

• Tracked time spent on toil this week
• Toil percentage: ____% (target: <50%)
• Top 3 toil sources identified
• Action items to reduce toil created

Incident Review

• All incidents from last week have postmortems
• Action items from postmortems assigned
• Overdue action items escalated
• Patterns across incidents identified

On-Call Health

• On-call load acceptable (< 2 pages/shift average)
• All pages were actionable and necessary
• Runbooks updated based on this week’s pages
• On-call engineers have time to recover

SLO Compliance

• All services within error budget
• Services approaching budget limits flagged
• Error budget policy followed for decisions
• SLO dashboard reviewed with team

Team Morale

• Team members feel supported
• Workload is sustainable
• Learning opportunities available
• Work-life balance maintained

New SRE Onboarding Checklist

Week 1: Foundation

• Access to all necessary systems granted
• Laptop setup and tooling installed
• Meet the team (1:1s scheduled)
• Read last 10 incident postmortems
• Review service architecture diagrams
• Shadow on-call engineer for a shift
• Join incident response as observer

Week 2-4: Learning

• Complete production access training
• Deploy a test change to production
• Write or update a runbook
• Participate in a postmortem as scribe
• Shadow 3+ incident responses
• Review SLOs for all critical services
• Complete security training

Week 5-8: Contributing

• Join on-call rotation as secondary
• Lead a postmortem
• Complete a toil reduction project
• Update or create monitoring dashboard
• Improve alerting for one service
• Present learning to team

Month 3+: Independent

• On-call as primary (with backup)
• Own a service or system area
• Propose and execute SRE improvement
• Mentor next new hire
• Contribute to SRE strategy

Change Review Checklist

Use this for production changes:

Before Deployment

• Code reviewed by at least 2 people
• All tests passing
• Deployment plan documented
• Rollback plan documented and tested
• Monitoring confirms current system health
• Error budget status checked
• High-risk change identified (if applicable)
• Stakeholders notified of change window

During Deployment

• Monitoring dashboard open
• Communication channel active
• Canary deployed and healthy
• Gradual rollout (10% → 50% → 100%)
• Metrics watched for anomalies
• Ready to rollback if needed

After Deployment

• All metrics within normal ranges
• Error rates unchanged or improved
• Latency unchanged or improved
• No new alerts fired
• Deployment marked successful
• Change documented in change log

Reference Cards

Quick reference tables for common SRE concepts.

Golden Signals

Incident Severity Guide

Communication Templates

Email templates for common SRE scenarios.

Requesting Budget for SRE Tools

Subject: ROI Analysis: [Tool Name] for Improved Reliability

Hi [Manager],

I'd like to propose adopting [Tool Name] to address [specific problem].

**Current State**:
- We spend [X hours/week] on [manual task]
- Recent incidents cost us [Y hours] in response time
- [Current limitation]

**Proposed Solution**: [Tool Name]
- Cost: $[amount]/month
- Reduces [task] time by [%]
- Improves MTTR by [estimate]

**ROI Analysis**:
- Engineering time saved: [X hours/month] = $[value]
- Reduced incident impact: [estimate]
- Payback period: [months]

**Trial Plan**:
Can we start with a 30-day trial to validate these benefits?

I'd be happy to discuss further.

[Your name]

Proposing an SRE Program

Subject: Proposal: Formal SRE Program

Hi [Leadership],

I'd like to propose establishing a formal SRE program to improve 
our reliability and reduce operational burden.

**Problem**:
- [X incidents] in the last quarter
- Engineering spends [Y%] of time firefighting
- Customer complaints about [reliability issue]

**Proposed Solution**:
- Dedicate [N people] to SRE practices
- Implement SLOs and error budgets
- Build automation to reduce toil

**Expected Outcomes** (6 months):
- 50% reduction in incident frequency
- 30% reduction in MTTR
- [X hours] of engineering time freed for features
- Improved customer satisfaction

**Next Steps**:
- Month 1: Define SLOs for critical services
- Month 2-3: Implement basic monitoring and automation
- Month 4-6: Scale to all services

Can we schedule time to discuss this proposal?

[Your name]

Case Studies

Real-world examples of SRE implementation in resource-constrained environments. Names and details have been changed to protect confidentiality, but the challenges and solutions are authentic.

Case Study 1: Bootstrapping SRE at a 50-Person Startup

Background

Company: B2B SaaS platform
Team Size: 12 engineers, no dedicated ops
Problem: Weekly production incidents, 3am pages, engineer burnout
Budget: $0 for new tools or headcount

Initial State

• Deployments required manual steps and often failed
• No monitoring beyond basic uptime checks
• Incidents handled by whoever was available
• No postmortem process
• Developers afraid to deploy on Fridays

Approach Taken

Month 1: Stop the Bleeding

One person (Sarah, senior engineer) spent 20% time on reliability

Actions:

1. Documented the current pain

   • Tracked all incidents for one month
   • Calculated engineering time spent firefighting: 40 hours/week
   • Presented data to leadership showing cost

2. Quick wins with free tools

   • Set up Prometheus (self-hosted) for basic metrics
   • Created simple Grafana dashboards
   • Configured PagerDuty free tier for on-call
   • Wrote runbooks for the 3 most common incidents

Results: MTTR dropped from 2 hours to 45 minutes

Month 2-3: Build Momentum

Recruited 2 volunteers from engineering team

Actions:

1. Implemented basic SLOs

   • Defined: “99% of API requests complete in < 500ms”
   • Tracked error budget in a spreadsheet
   • Used error budget to say “no” to a risky Friday deploy

2. Automated the most painful toil

   • Created deployment automation (GitHub Actions)
   • Eliminated manual database backups (cronjob + monitoring)
   • Automated most common incident fix (restart stuck workers)

3. Started blameless postmortems

   • Used template for all SEV-1 and SEV-2 incidents
   • Tracked action items in existing project management tool
   • Shared learnings in weekly all-hands

Results:

• Incidents dropped from 12/month to 6/month
• Engineering time on ops: 40 hours → 20 hours per week
• Successful deploys increased from 60% to 95%

Month 4-6: Formalize and Scale

Leadership approved 1 dedicated SRE hire

Actions:

1. Hired experienced SRE (converted from inside the company)
2. Established on-call rotation (6 engineers, weekly)
3. Production readiness reviews for new services
4. Invested in observability (Datadog, justified by ROI)

Results After 6 Months:

• Incidents: 12/month → 2/month (83% reduction)
• MTTR: 2 hours → 15 minutes (87% improvement)
• Deploy frequency: 2/week → 15/week
• Engineering satisfaction up significantly
• Customer complaints about reliability down 90%

Key Lessons

1. Start with data: Quantify the problem before proposing solutions
2. Quick wins build credibility: Free tools + automation proved value
3. Grassroots worked: Didn’t wait for top-down initiative
4. ROI convinced leadership: Showed hours saved and customer impact
5. Culture change takes time: Blameless postmortems felt awkward at first

What Would They Do Differently?

• Start SLOs earlier: Wished they had objective targets from day one
• Document more: Early runbooks were too sparse
• Involve product earlier: Took too long to get PM buy-in for error budgets

Case Study 2: Enterprise SRE Without Executive Support

Background

Company: 200-person financial services firm
Team Size: 50 engineers across 5 product teams
Problem: Each team ran ops differently, no standards, constant firefighting
Challenge: IT leadership skeptical of “Google practices”

Initial State

• Each team had different monitoring tools (Nagios, New Relic, CloudWatch)
• No unified on-call process
• Production changes required manual approval from IT
• Incidents blamed on individuals, not systems
• Teams competed for limited ops resources

Approach Taken

Phase 1: Build Coalition (Months 1-3)

Started with one sympathetic team lead

Actions:

1. Pilot program with one product team

   • Implemented SLOs for their service
   • Set up modern monitoring (Prometheus + Grafana)
   • Ran first blameless postmortem
   • Automated their deployment pipeline

2. Measured and shared results

   • Created monthly reports showing improvement
   • Presented at engineering all-hands
   • Offered to help other teams do the same

Results: 2 other teams asked to join

Phase 2: Create Shared Platform (Months 4-9)

3 teams now collaborating on SRE practices

Actions:

1. Built shared observability platform

   • Central Prometheus + Grafana instance
   • Common SLO tracking dashboard
   • Shared alert manager

2. Created SRE working group

   • 1 engineer from each participating team
   • Met weekly to share practices
   • Maintained shared runbook repository
   • Cross-team on-call for critical services

3. Evangelized through lunch-and-learns

   • Monthly presentations on SRE topics
   • Invited skeptical teams to observe
   • Shared postmortems (anonymized)

Results:

• 4 of 5 product teams now using shared platform
• IT leadership started asking questions
• Director-level sponsor emerged

Phase 3: Formal Recognition (Months 10-12)

Working group became official “SRE Practice Team”

Actions:

1. Got executive approval for SRE initiative

   • Presented year of data showing impact
   • Demonstrated cost savings from automation
   • Showed competitive advantage of reliability

2. Hired 2 dedicated SREs

   • Focused on platform and standards
   • Embedded with product teams on rotation
   • Ran production readiness reviews

3. Established SRE standards

   • Company-wide SLO framework
   • Unified incident response process
   • Standard observability stack
   • Production readiness checklist

Results After 1 Year:

• All 5 product teams adopted SRE practices
• Incidents across company: Down 60%
• Average MTTR: 3 hours → 30 minutes
• Deployment frequency: Up 4x
• IT approval process streamlined (went from days to hours)
• 2-person SRE team supporting 50 engineers

Key Lessons

1. Coalition beats mandate: Bottom-up adoption more sustainable than top-down
2. Show, don’t tell: Pilot results convinced skeptics
3. Find a sponsor: Eventually needed director-level support
4. Patience required: Took a full year to get formal recognition
5. Shared tooling accelerated adoption: Easier to join than build alone

Challenges Faced

• Political resistance: “Not invented here” from some teams
• Budget constraints: Shared costs across teams to afford tools
• Competing priorities: SRE work often deprioritized for features
• Cultural shift: Blameless postmortems were uncomfortable initially

Case Study 3: Solo SRE Supporting 20 Microservices

Background

Company: Mid-size e-commerce platform
Team Size: 1 SRE (Marcus), 25 developers
Problem: Rapid microservices adoption, no operational maturity
Constraint: Couldn’t hire more SREs for 12 months

Initial State

• 20+ microservices with inconsistent practices
• Each team “owned” their service but lacked ops expertise
• On-call rotations dysfunctional (developers didn’t know how to respond)
• SRE (Marcus) was bottleneck for all production issues
• Marcus paged constantly, burning out

Approach Taken

Strategy: Multiply Through Enablement

Goal: Make developers self-sufficient, not dependent on SRE

Actions:

1. Created “Golden Path” templates

   • Standard microservice template with:
     • Pre-configured monitoring and logging
     • CI/CD pipeline built-in
     • Default SLOs and dashboards
     • Production readiness checklist
   • Made it easier to do things right than wrong

2. Self-service observability platform

   • Centralized logging (ELK stack)
   • Automated dashboard creation
   • Alert templates developers could customize
   • One-click integration with new services

3. Embedded training program

   • 2-hour “Production Operations 101” for all developers
   • Quarterly “GameDay” exercises (chaos testing)
   • Paired with developers during incidents
   • Recorded troubleshooting sessions as training

4. Tiered support model

   • Tier 1: Developers (first responders for their services)
   • Tier 2: SRE (escalation for complex issues)
   • Tier 3: Engineering leadership
   • Clear escalation criteria in runbooks

5. Office hours instead of interrupts

   • Set 2 hours/day for ad-hoc developer questions
   • Rest of time protected for project work
   • Reduced constant context switching

Results After 6 Months:

• Developers handled 80% of incidents without SRE involvement
• Marcus’s page load: 15/week → 2/week
• MTTR actually improved (developers closer to their code)
• 18 of 20 services using golden path templates
• Freed Marcus to work on strategic automation

Key Lessons

1. Can’t scale by doing: Had to scale by teaching
2. Make the right thing easy: Templates and automation key
3. Developers can do ops: They just needed training and tools
4. Office hours are boundaries: Protected focus time critical
5. Self-service > tickets: Eliminated request bottleneck

What Enabled Success

• Management support: Leadership backed “developers own production” model
• Good documentation: Invested heavily in runbooks and training materials
• Automation: Golden path reduced toil for both SRE and developers
• Psychological safety: Developers felt safe admitting what they didn’t know

Case Study 4: Legacy Monolith to SRE Practices

Background

Company: Healthcare tech company
Team Size: 8 engineers maintaining 15-year-old monolith
Problem: System too fragile to change, technical debt overwhelming
Constraint: Can’t rewrite, must improve in place

Initial State

• Monolithic PHP application
• Deployments required 6-hour maintenance windows
• No meaningful tests
• Manual deployment process (40-step runbook)
• Database schema migrations terrifying
• Outages during every deploy

Approach Taken

Phase 1: Observe Before Changing

Spent 2 months just understanding the system

Actions:

1. Instrumented existing code

   • Added logging at boundaries
   • Instrumented critical flows
   • Tracked performance metrics
   • Identified bottlenecks

2. Mapped the actual architecture

   • Discovered undocumented dependencies
   • Found “mystery cronjobs” nobody understood
   • Documented data flows
   • Identified single points of failure

3. Established baseline SLIs

   • Page load times (currently p95: 3 seconds)
   • Error rates (currently 2%)
   • Transaction success rates
   • Batch job completion times

No changes made yet, just measurement

Phase 2: Safe Improvements

Made system more observable and deployable

Actions:

1. Deployment automation

   • Automated the 40-step deploy process
   • Added deployment verification tests
   • Reduced deployment from 6 hours to 45 minutes
   • Zero-downtime deploys (blue-green)

2. Improved monitoring without code changes

   • Set up log aggregation (ELK)
   • Created dashboards for key flows
   • Configured meaningful alerts
   • Stopped monitoring infrastructure, started monitoring user impact

3. Strangler fig pattern for new features

   • New functionality as microservices
   • Old monolith gradually hollowed out
   • Each new service built with SRE practices
   • Reduced risk of breaking legacy code

Results After 6 Months:

• Deployment risk dramatically reduced
• Can deploy during business hours
• Visibility into production unprecedented
• Began setting realistic SLOs
• Confidence in changing code improved

Key Lessons

1. Observe first: Can’t improve what you don’t understand
2. Non-invasive improvements: Monitoring and deployment before code changes
3. Strangler fig works: Don’t rewrite, gradually replace
4. SLIs from baseline: Started with “better than 3 seconds” not “under 200ms”
5. Team confidence matters: Visibility reduced fear of change

Unexpected Wins

• Found optimization opportunities: Monitoring revealed low-hanging fruit
• Justified tech debt work: SLOs gave data to prioritize improvements
• Reduced tribal knowledge: Documentation and runbooks critical
• Enabled new hires: Observability made onboarding possible

Common Patterns Across Case Studies

What Worked Everywhere

1. Start with measurement: Data drives decisions and proves value
2. Quick wins build momentum: Early successes create credibility
3. Automation multiplies impact: Especially in resource-constrained environments
4. Blameless culture is hard but essential: Takes time and consistent leadership
5. Teaching scales better than doing: Enable others rather than being a bottleneck

What Failed Repeatedly

1. Waiting for permission: Grassroots approaches were more successful
2. Perfect solutions: Better to ship something imperfect than wait
3. Blame-focused postmortems: Destroyed psychological safety
4. Ignoring culture: Tools alone didn’t fix organizational issues
5. Hero culture: Celebrating firefighting instead of prevention

Resource Constraints Drove Innovation

• Limited budget forced creative use of open source
• Small teams required automation and self-service
• No headcount forced “teach to scale” approaches
• Lack of buy-in required proving value through data

Your Turn

These case studies show SRE is achievable regardless of resources. The common thread: starting small, proving value, and building momentum.

What’s your constraint? Budget? Headcount? Cultural resistance? There’s a path forward.

The question isn’t whether you can afford to implement SRE.
The question is whether you can afford not to.

Resources & Tools

A curated collection of books, tools, communities, and resources for building and scaling your SRE practice.

Essential Reading

Core SRE Books

Site Reliability Engineering (Google, 2016)
The book that started it all

• Foundational SRE concepts from Google
• Real-world examples from Google’s scale
• Free online: sre.google/sre-book

The Site Reliability Workbook (Google, 2018)
Practical implementation guidance

• Hands-on exercises and examples
• More accessible than the first book
• Free online: sre.google/workbook

Seeking SRE (David Blank-Edelman, 2018)
Community perspectives beyond Google

• Essays from SREs at various companies
• Different approaches to SRE
• Real-world implementation stories

Database Reliability Engineering (Laine Campbell & Charity Majors, 2017)
SRE principles for database systems

• Treating databases as distributed systems
• Operational best practices
• Essential for data-heavy services

The Phoenix Project (Gene Kim et al., 2013)
DevOps novel that inspired many SRE teams

• Fictional story of IT transformation
• Makes concepts accessible
• Great for explaining SRE value to leadership

Operations & Incident Response

Incident Management for Operations (Rob Schnepp et al., 2017)

• Practical incident response frameworks
• Communication strategies
• Postmortem best practices

The Field Guide to Understanding ‘Human Error’ (Sidney Dekker, 2014)

• Why blame doesn’t improve systems
• Foundation for blameless culture
• Human factors in incidents

Chaos Engineering (Casey Rosenthal & Nora Jones, 2020)

• Building confidence through controlled experiments
• Netflix’s approach to resilience
• Practical chaos engineering implementation

Monitoring & Observability

Observability Engineering (Charity Majors et al., 2022)

• Modern approach to understanding systems
• Beyond metrics, logs, and traces
• High-cardinality data strategies

Practical Monitoring (Mike Julian, 2017)

• Effective monitoring strategies
• Alerting best practices
• Avoiding common pitfalls

Tools by Category

Monitoring & Observability

Open Source (Free)

Prometheus

• Time-series metrics collection
• Pull-based monitoring
• PromQL query language
• prometheus.io

Grafana

• Visualization and dashboarding
• Multiple data source support
• Alerting capabilities
• grafana.com

Elastic Stack (ELK)

• Log aggregation and search
• Elasticsearch, Logstash, Kibana
• Good for text log analysis
• elastic.co

Jaeger

• Distributed tracing
• Request flow visualization
• Performance debugging
• jaegertracing.io

OpenTelemetry

• Unified observability framework
• Vendor-neutral instrumentation
• Metrics, traces, and logs
• opentelemetry.io

Commercial (with free tiers)

Datadog - All-in-one observability platform
New Relic - APM and infrastructure monitoring
Honeycomb - Observability for complex systems
Sentry - Error tracking and performance
Better Stack - Logs, uptime, and incident management

Incident Response & On-Call

Open Source

Rootly (Free tier available)

• Incident management platform
• Slack integration
• Postmortem automation
• rootly.com

Grafana OnCall

• On-call management
• Alert routing
• Schedule management
• grafana.com/products/oncall

Commercial

PagerDuty (Free tier: 5 users)

• Industry standard for on-call
• Alert aggregation and routing
• Incident response workflows
• pagerduty.com

Opsgenie

• Alert and on-call management
• Atlassian integration
• Flexible routing rules
• atlassian.com/software/opsgenie

FireHydrant

• Modern incident management
• Retros and learning focus
• firehydrant.com

Infrastructure as Code

Terraform (Free & Open Source)

• Multi-cloud infrastructure provisioning
• Declarative configuration
• State management
• terraform.io

Ansible (Free & Open Source)

• Configuration management
• Agentless architecture
• Simple YAML syntax
• ansible.com

Pulumi (Free tier available)

• IaC using real programming languages
• TypeScript, Python, Go support
• pulumi.com

CloudFormation / ARM Templates / Cloud Deployment Manager

• Cloud-native IaC tools
• AWS, Azure, GCP respectively
• Good for single-cloud strategies

CI/CD & Deployment

GitHub Actions (Free for public repos, generous for private)

• Integrated with GitHub
• Simple YAML workflows
• Large ecosystem of actions
• github.com/features/actions

GitLab CI (Free tier available)

• Built into GitLab
• Auto DevOps features
• docs.gitlab.com/ee/ci

CircleCI (Free tier available)

• Fast cloud-based CI/CD
• Docker-native
• circleci.com

Jenkins (Free & Open Source)

• Self-hosted automation server
• Massive plugin ecosystem
• Ultimate flexibility
• jenkins.io

Container Orchestration

Kubernetes (Free & Open Source)

• Industry-standard orchestration
• Complex but powerful
• Large ecosystem
• kubernetes.io

Docker Swarm (Free & Open Source)

• Simpler alternative to K8s
• Built into Docker
• Good for smaller deployments
• docs.docker.com/engine/swarm

Nomad (Free & Open Source)

• HashiCorp’s orchestrator
• Lighter than Kubernetes
• Multi-workload support
• nomadproject.io

Chaos Engineering

Gremlin (Free tier available)

• Managed chaos engineering platform
• Safety controls built-in
• gremlin.com

Chaos Mesh (Free & Open Source)

• Kubernetes-native chaos
• Wide variety of fault types
• chaos-mesh.org

LitmusChaos (Free & Open Source)

• Cloud-native chaos engineering
• ChaosHub for pre-built experiments
• litmuschaos.io

SLO & Error Budget Tracking

Sloth (Free & Open Source)

• SLO generator for Prometheus
• Standardized SLO definitions
• github.com/slok/sloth

Pyrra (Free & Open Source)

• SLO tracking and alerting
• Kubernetes-native
• github.com/pyrra-dev/pyrra

Nobl9 (Commercial)

• SLO platform
• Multi-data source support
• nobl9.com

Online Communities & Learning

Newsletters

SRE Weekly - sreweekly.com
The best weekly SRE newsletter

• Curated articles and tools
• Incident write-ups
• Free, delivered every Sunday

DevOps Weekly - devopsweekly.com

• Broader DevOps perspective
• Tools and techniques
• Industry news

Reddit Communities

r/sre - reddit.com/r/sre

• Active SRE community
• Career advice and discussions
• Tool recommendations

r/devops - reddit.com/r/devops

• DevOps practices and tools
• Large, active community

r/kubernetes - reddit.com/r/kubernetes

• Kubernetes-specific discussions
• Troubleshooting help

Slack Communities

SRE Community - sre.xyz

• Large SRE-focused Slack
• Job board and resources
• Vendor-neutral

Rands Leadership Slack

• Engineering leadership
• Many SREs and managers
• randsinrepose.com/welcome-to-rands-leadership-slack

Discord Servers

DevOps/SRE Discord

• Real-time discussions
• Help and advice
• Search for current invite links

Conferences & Events

Major SRE Conferences

SREcon (USENIX)

• Premier SRE conference
• Americas, EMEA, APAC editions
• Technical deep dives
• usenix.org/conferences/byname/925

KubeCon + CloudNativeCon

• Kubernetes and cloud-native ecosystem
• Large SRE presence
• Multiple locations yearly
• cncf.io/kubecon-cloudnativecon-events

LISA (USENIX)

• Large Installation System Administration
• Operations and SRE focus
• usenix.org/conferences/byname/5

Monitorama

• Monitoring and observability
• Intimate, single-track format
• monitorama.com

Vendor-Specific (Free to attend)

ObservabilityCON (Grafana Labs)
Dash (Datadog)
FutureStack (New Relic)
re:Invent (AWS)

Blogs Worth Following

Company Engineering Blogs

Netflix Tech Blog - netflixtechblog.com
Pioneers of chaos engineering and resilience

Uber Engineering - uber.com/blog/engineering
Scale and operational challenges

Stripe Engineering - stripe.com/blog/engineering
Reliability at scale

GitHub Engineering - github.blog/category/engineering
Platform and infrastructure

Google Cloud Blog - cloud.google.com/blog
SRE practices from the source

LinkedIn Engineering - engineering.linkedin.com
SRE and infrastructure at scale

Individual Blogs

Charity Majors - charity.wtf
Observability, engineering management, on-call

Julia Evans - jvns.ca
Debugging, learning, making complex topics accessible

Cindy Sridharan - copyconstruct.medium.com
Distributed systems, observability

Brendan Gregg - brendangregg.com
Performance analysis, Linux systems

Training & Certification

Certifications

Google Professional Cloud DevOps Engineer

• Google Cloud SRE practices
• cloud.google.com/certification/cloud-devops-engineer

AWS Certified DevOps Engineer

• AWS operational best practices
• aws.amazon.com/certification/certified-devops-engineer-professional

Certified Kubernetes Administrator (CKA)

• Kubernetes operations
• Hands-on practical exam
• cncf.io/certification/cka

Online Courses

LinkedIn Learning SRE Courses

• Various SRE and DevOps topics
• Often free through library memberships

Coursera - Site Reliability Engineering

• University and company courses
• Some free to audit

Udemy SRE Courses

• Practical, hands-on courses
• Often on sale

Podcasts

Software Engineering Daily

• Broad engineering topics
• Regular SRE and infrastructure episodes
• softwareengineeringdaily.com

The Cloudcast

• Cloud and DevOps
• Industry news and interviews
• thecloudcast.net

Screaming in the Cloud

• AWS and cloud infrastructure
• Corey Quinn’s insights
• lastweekinaws.com/podcast

YouTube Channels

GOTO Conferences

• Conference talks on SRE and DevOps
• High-quality presentations

USENIX

• SREcon and LISA talks
• Technical deep dives

Google Cloud Tech

• SRE practices and GCP
• Regular SRE-focused content

DevOps Toolkit

• Viktor Farcic’s practical guides
• Kubernetes, GitOps, DevOps

GitHub Resources

Awesome Lists

awesome-sre - github.com/dastergon/awesome-sre
Curated list of SRE resources

awesome-chaos-engineering - github.com/dastergon/awesome-chaos-engineering
Chaos engineering tools and articles

awesome-incident-response - github.com/meirwah/awesome-incident-response
Incident response tools and resources

Example Code & Configs

Search GitHub for:

• “SLO prometheus” - Example SLO configurations
• “incident postmortem template” - Postmortem examples
• “production readiness checklist” - PR checklists
• “sre runbook” - Runbook examples

Standards & Specifications

OpenSLO

• Standard SLO specification format
• openslo.com

OpenTelemetry

• Observability instrumentation standard
• opentelemetry.io

DORA Metrics

• DevOps Research and Assessment
• Standard performance metrics
• cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance

Getting Help

Where to Ask Questions

1. Your company’s internal Slack/Teams - Start here
2. r/sre and r/devops - Reddit communities
3. SRE Slack communities - Real-time help
4. Stack Overflow - Tag: [sre] or [site-reliability]
5. Tool-specific forums - Prometheus, Kubernetes, etc.

Tips for Getting Good Answers

• Be specific: Include error messages, configs, and context
• Show what you’ve tried: Demonstrate effort before asking
• Make it reproducible: Minimal example that shows the problem
• Follow up: Share what worked to help others

Staying Current

SRE practices evolve quickly. Here’s how to stay up to date:

1. Subscribe to 2-3 newsletters (SRE Weekly is essential)
2. Follow 5-10 practitioners on Twitter/LinkedIn
3. Attend 1 conference per year (virtual counts!)
4. Read incident postmortems from major companies
5. Experiment with new tools in non-production environments

Final Advice

You don’t need all of these tools and resources to do SRE.

Start with:

• One monitoring tool (Prometheus + Grafana)
• One incident management tool (PagerDuty free tier)
• SRE Weekly newsletter
• The Google SRE books (free online)

Build from there based on your actual needs, not what’s trendy.

The best tools are the ones you’ll actually use.