Incident Response Templates

These templates help establish consistent incident documentation and response procedures across your organization.

Incident Postmortem Template

Use this template after every significant incident to capture learnings and prevent recurrence.

# Incident Postmortem: [Brief Description]

**Date**: [YYYY-MM-DD]
**Severity**: [SEV-1/2/3/4]
**Duration**: [Start time - End time] ([Total duration])
**Impact**: [Customer/business impact description]

## Summary
[2-3 sentence overview of what happened]

## Timeline (all times in [timezone])
- **HH:MM** - [Event description]
- **HH:MM** - [Event description]
- **HH:MM** - [Detection/Alert fired]
- **HH:MM** - [Response began]
- **HH:MM** - [Mitigation applied]
- **HH:MM** - [Incident resolved]

## Root Cause
[Detailed explanation of the underlying cause. Include technical details.]

## Contributing Factors
- [Factor 1: What allowed this to happen?]
- [Factor 2: Why wasn't it caught earlier?]
- [Factor 3: What made it worse?]

## Impact
- **Customers affected**: [Number/percentage]
- **Revenue impact**: [$Amount or N/A]
- **Engineering time**: [Hours spent responding]
- **User-facing symptoms**: [What users experienced]

## What Went Well
- [Positive aspect 1]
- [Positive aspect 2]

## What Went Poorly
- [Issue 1]
- [Issue 2]

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| [Fix underlying issue] | @person | YYYY-MM-DD | Open |
| [Improve monitoring] | @person | YYYY-MM-DD | Open |
| [Update runbook] | @person | YYYY-MM-DD | Open |

## Lessons Learned
- [Key takeaway 1]
- [Key takeaway 2]

Incident Response Runbook Template

Create runbooks for your critical services and common failure modes. A good runbook enables any on-call engineer to respond effectively.

# Runbook: [Service/Issue Name]

**Last Updated**: [Date]
**Maintained By**: [Team/Person]

## Overview
[What this runbook covers]

## When to Use This Runbook
- [Symptom/alert 1]
- [Symptom/alert 2]

## Severity Assessment
- **SEV-1 (Critical)**: [Criteria]
- **SEV-2 (High)**: [Criteria]
- **SEV-3 (Medium)**: [Criteria]

## Quick Actions (Do This First)
1. [Immediate action to reduce impact]
2. [Second immediate action]
3. [Communication: Who to notify]

## Diagnostic Steps

### Check System Health
# [Command to check service status]
# [Command to check logs]
# [Command to check metrics]

### Common Causes
1. **[Issue type 1]**
   - Symptoms: [How to identify]
   - Fix: [Resolution steps]

2. **[Issue type 2]**
   - Symptoms: [How to identify]
   - Fix: [Resolution steps]

## Escalation
- **L1**: [On-call engineer] - [Response actions]
- **L2**: [Senior engineer/team lead] - [Escalate after X minutes or if Y condition]
- **L3**: [Engineering manager] - [Escalate for severity or duration thresholds]

## Recovery Steps
1. [Step 1]
2. [Step 2]
3. [Verification: How to confirm resolution]

## Communication Templates

**Internal Slack Message**:
🚨 Incident: [Brief description]
Severity: [SEV-X]
Impact: [What's affected]
Status: [Investigating/Mitigating/Resolved]
Next update: [Time]

**Customer Status Page**:
We are currently experiencing [issue]. 
Our team is actively investigating. 
We will provide updates every [X minutes].

## Related Documentation
- [Link to architecture docs]
- [Link to monitoring dashboard]
- [Link to previous incidents]