Observability & Golden Signals

In the realm of Site Reliability Engineering, the adage “You can’t improve what you don’t measure” holds significant weight. Observability is about gaining a comprehensive understanding of what your systems are doing—and more critically, why they are doing it.

The Golden Signals

Google’s SRE book delineates four essential metrics that every service must monitor. These are known as the Golden Signals.

1. Latency

Latency answers the question: How long does it take to serve a request? When measuring latency, it is crucial to differentiate between successful and failed requests. To gain a more accurate picture, utilize percentiles—such as p50, p95, and p99—rather than relying solely on averages. Track latency meticulously by endpoint, region, or customer segment to pinpoint areas for improvement.

For example, you might find that “95% of API requests complete in under 200ms.”

2. Traffic

Traffic measures how much demand your system is currently handling. This metric can vary depending on the type of system you are working with. For web services, you assess requests per second; for databases, transactions per second; for queues, messages per second; and for real-time systems, active connections. Monitoring these metrics helps you ensure your system can handle peak loads, such as “Processing 10,000 requests per minute during peak hours.”

3. Errors

Errors reflect the percentage of requests that are failing. In web services, this might manifest as HTTP 5xx errors, while in application code, it could be exception rates or failed background jobs. Client-side errors also fall under this category. Monitoring error rates, such as “Error rate of 0.1% (10 failed requests per 10,000),” allows you to quickly identify and rectify issues affecting system reliability.

4. Saturation

Saturation indicates how “full” your system is, measuring resources like CPU and memory utilization, disk I/O, storage capacity, database connection pool usage, and queue depth. Understanding saturation levels, such as “Database connection pool at 70% capacity,” enables you to proactively manage resource allocation and avoid bottlenecks.

From Metrics to SLIs and SLOs

After collecting metrics, the next step is to translate them into Service Level Indicators (SLIs) and Service Level Objectives (SLOs).

Service Level Indicators (SLIs)

SLIs are precise measurements of service behavior, and effective SLIs should be user-centric, measurable based on real data, and actionable. For instance, you might track “Percentage of API requests completed in < 200ms” or “Percentage of login attempts that succeed.”

Service Level Objectives (SLOs)

SLOs are the target values set for your SLIs, defining acceptable reliability. An example SLO might state, “99.9% of API requests will complete in under 200ms.” It’s crucial to recognize that 100% reliability is neither achievable nor desirable, as perfect uptime often indicates a lack of risk-taking and innovation.

Error Budgets

Error budgets represent the amount of unreliability you can “spend” while still meeting your SLOs. If your SLO is 99.9% uptime, you have a 0.1% error budget, equating to approximately 43 minutes of downtime per month. Should you exceed this budget, prioritize reliability over new feature launches. Conversely, if you are well within your error budget, you have the flexibility to innovate and take calculated risks.

Alerting That Actually Works

The objective of alerting is to provide actionable, urgent notifications rather than noise.

Alert Design Principles

  1. Alert on symptoms, not causes: For example, instead of alerting on “Disk usage > 80%,” focus on symptoms like “API latency p99 > 500ms AND error rate > 1%.”

  2. Every alert should require action: If an alert doesn’t necessitate action at 3am, reconsider its urgency. Differentiate between levels of urgency, such as page, ticket, or email.

  3. Include context in alerts: Ensure alerts clearly convey what’s broken, the severity of the impact, and include relevant links to runbooks or dashboards. Suggest first troubleshooting steps to streamline response times.

Alert Fatigue is a Reliability Issue

An overload of alerts leads to ignored alerts and missed incidents. To combat this, employ strategies to reduce noise: aggregate related alerts, use intelligent thresholds like dynamic baselines, suppress alerts during known maintenance, and regularly review and prune low-value alerts.

Observability Tools

To effectively monitor and analyze your systems, leverage a suite of observability tools:

Metrics

  • Prometheus: For time-series metrics collection and querying.
  • Grafana: For visualization and dashboarding.
  • StatsD: For application-level metric aggregation.

Logging

  • ELK Stack: Comprising Elasticsearch, Logstash, and Kibana for comprehensive logging.
  • Loki: Prometheus-style log aggregation.
  • CloudWatch/Stackdriver: Cloud-native logging solutions.

Tracing

  • Jaeger: For distributed tracing in microservices environments.
  • Zipkin: To visualize request flows.
  • OpenTelemetry: Provides a unified observability standard.

Common Pitfalls

Be vigilant to avoid common pitfalls such as tracking vanity metrics that appear impressive but offer no real value, creating too many dashboards that result in information overload, optimizing for averages while missing the long tail of poor user experiences, not setting SLOs, and allowing alert sprawl that leads to a flood of ignored notifications.

In the following sections, we will delve into the tools and automation strategies that enable scalable SRE practices.