Hiring the Team

Building a successful Site Reliability Engineering (SRE) team is not just about assembling individuals with impressive resumes. It requires finding those who possess a unique blend of software engineering expertise, operational experience, and the right mindset for reliability work. These qualities are crucial for ensuring that your systems run smoothly and efficiently, minimizing downtime and enhancing user satisfaction.

What Makes a Great SRE?

Technical Skills

To be an effective SRE, you must possess certain technical skills that go beyond basic engineering. First and foremost, you should be able to write production-quality code. This means your coding is not just functional, but also efficient, maintainable, and scalable. A deep understanding of systems is also essential. You should be familiar with networking, operating system internals, and distributed systems, as these form the backbone of reliable infrastructure. Additionally, being a methodical troubleshooter is crucial. You need to be able to diagnose and resolve issues under pressure, when time is of the essence. Lastly, you must have an automation mindset. You should view repetitive manual tasks as opportunities to create tools that not only save time but also reduce the margin for error.

Non-Technical Skills

While technical prowess is important, non-technical skills are equally vital. As an SRE, you need to communicate complex technical issues clearly to both technical and non-technical stakeholders. This ensures that everyone understands the problems and solutions involved. Blameless thinking is another key attribute; you should focus on improving systems rather than assigning blame to individuals when things go wrong. Customer empathy is crucial as well; you must understand the user impact of reliability issues and strive to enhance their experience. Furthermore, a great SRE should have the ability to teach. Sharing knowledge and building team capabilities are essential for the continuous improvement of the team. Finally, emotional resilience is necessary to handle the stress of being on-call and managing incidents without succumbing to burnout.

The SRE Mindset

A great SRE embodies a particular mindset that sets them apart. Curiosity about failure drives them to understand why things break and how to prevent future incidents. They possess a bias toward automation, constantly seeking to eliminate toil rather than simply accepting it as part of the job. Decisions are data-driven, relying on measurable outcomes rather than assumptions. Healthy paranoia is a trait that allows them to anticipate what could go wrong and plan accordingly. Lastly, continuous learning is a cornerstone of their professional development, keeping them current with evolving technologies and industry practices.

SRE vs. DevOps vs. Platform Engineering

To target your hiring efforts effectively, it’s important to understand the distinctions between SRE, DevOps, and Platform Engineering roles.

Site Reliability Engineer

As an SRE, your primary focus is on production reliability and incident response. This involves managing Service Level Objectives (SLOs), monitoring systems, being on-call, conducting postmortems, and eliminating toil. Typically, SREs come from a software engineering background with a keen interest in operations.

DevOps Engineer

DevOps Engineers concentrate on Continuous Integration/Continuous Deployment (CI/CD) and deployment automation. Their activities often include building pipelines, provisioning infrastructure, and developing tools that improve developer productivity. They usually have a mix of development and operations experience.

Platform Engineer

Platform Engineers focus on creating internal developer platforms and facilitating self-service infrastructure. They build tools, APIs, and abstractions to support product teams. Strong software engineering skills are a common background for this role.

Note: Job titles can vary widely, so it’s more effective to focus on the responsibilities and skills associated with each role rather than the titles themselves.

Building Your First SRE Team

Option 1: Internal Conversion

One approach to building your SRE team is to identify existing engineers who already exhibit SRE behaviors. These are individuals who volunteer for on-call responsibilities, build automation tools in their spare time, lead postmortems, and have a deep concern for production reliability.

Advantages:

  • They already understand your systems.
  • They are a proven cultural fit.
  • Onboarding is faster.

Challenges:

  • They may lack formal SRE training.
  • Time is needed to develop new skills.
  • The team may lose a valued engineer from their current role.

Option 2: External Hiring

Alternatively, you can recruit experienced SREs from other organizations.

Advantages:

  • They bring fresh perspectives and best practices.
  • They can quickly apply SRE fundamentals.
  • They can mentor internal team members.

Challenges:

  • They require a longer ramp-up time to learn your systems.
  • Cultural fit may be uncertain.
  • They might have expectations from larger companies.

Option 3: Hybrid Approach

A hybrid approach often proves to be the most effective strategy. By hiring an experienced SRE to lead the team and developing other members internally, you can leverage the expertise and direction from the external hire while benefiting from the system knowledge of internal engineers. This approach fosters knowledge transfer and creates mentorship opportunities.

Interview Process

Technical Assessment

During the interview process, focus on realistic scenarios rather than trivia. This ensures you are assessing candidates based on practical skills and thinking.

Troubleshooting Exercise

Present candidates with a production incident scenario, providing incomplete information to mimic real-life situations. Observe their debugging methodology, assess how they communicate their findings, and evaluate their prioritization of actions.

System Design

Ask candidates to design a reliable system. Inquire how they would ensure high availability for a component, identify potential failure modes, monitor the system, and make necessary trade-offs.

Coding Challenge

Give candidates a practical automation task, such as parsing logs and extracting metrics, writing a health check system, or building a simple deployment tool. Look for code quality, readability, error handling, testing approach, and documentation skills.

Cultural Assessment

Blameless Thinking

Ask candidates about a production incident they caused. A good response will focus on system failures, lessons learned, and improvements made, while a red flag response will deflect blame and focus on individual mistakes.

Handling Pressure

Inquire how candidates handle being paged at 3am for a critical incident. A good response will describe a methodical approach, knowing when to escalate, and learning from incidents, whereas a red flag response will exhibit a hero complex and ignore the impact on team sustainability.

Collaboration

Ask candidates to describe a time they disagreed with a decision. A good response will involve respectful disagreement, data-driven arguments, and acceptance of team decisions, while a red flag response will dismiss others and show stubbornness without evidence.

Team Structure Models

Embedded SRE

In this model, SREs are assigned to specific product teams.

Pros:

  • They gain deep knowledge of specific services.
  • There’s close collaboration with developers.
  • There is clear ownership and accountability.

Cons:

  • They can become siloed.
  • It can be harder to share knowledge across teams.
  • They may get pulled into feature work.

Centralized SRE

A centralized SRE model involves a single SRE team supporting all services.

Pros:

  • There’s shared knowledge and best practices.
  • Resource utilization is better.
  • There is a clear SRE identity and culture.

Cons:

  • The team can become a bottleneck.
  • There is less deep knowledge of individual services.
  • They may be perceived as “gatekeepers.”

Hybrid Model

The hybrid model combines the best of both worlds by maintaining a central SRE team with embedded rotations. SREs rotate through product teams for 3-6 months, which allows for deep knowledge without permanent silos while maintaining shared practices and a cohesive team culture.

Onboarding New SREs

Week 1: Foundation

During the first week, ensure new SREs have access to all systems and tools, shadow incident response teams, read key postmortems, and meet stakeholders across engineering.

Week 2-4: Shadowing

In the following weeks, have them pair with experienced SREs on troubleshooting tasks, join the on-call rotation as a secondary responder, participate in postmortems, and begin small automation projects.

Week 5-8: Independence

By weeks five to eight, new SREs should take on-call shifts as primary responders (with backup), own specific services or systems, lead postmortems, and propose and implement improvements.

Month 3+: Full Contribution

Beyond the initial onboarding period, SREs should be fully integrated into team rotations, driving SRE projects independently, mentoring newer team members, and influencing SRE strategy.

Growing Senior SRE Talent

Career Progression

It’s crucial to define clear paths for career growth within your SRE team. As a Junior SRE, you learn systems, assist with incidents, and execute defined projects. As an SRE, you own services, lead incidents, and design automation solutions. A Senior SRE sets technical direction, mentors the team, and drives strategic initiatives. Finally, a Staff/Principal SRE influences organization-wide reliability and architects platforms.

Retention Strategies

To keep your SRE team engaged, offer variety by rotating responsibilities to prevent burnout. Ensure they work on problems that matter, support their learning through conferences, training, and certifications, include them in architectural decisions, and respect on-call boundaries and time off.

Common Hiring Mistakes

Avoid these common pitfalls when hiring for your SRE team. Do not hire only operations people, as SRE requires strong coding skills. Similarly, avoid hiring only developers, as production operations is a distinct skillset. Overvaluing credentials can lead you to miss candidates with demonstrated abilities, while ignoring culture fit means technical skills alone won’t suffice. Lastly, ensure the current SRE team is involved in evaluating candidates to ensure a good fit.

Next, we’ll explore how to size your SRE team appropriately for your organization’s scale and needs.