On-Call Culture & Runbooks

Also Known As

TL;DR

Explanation

Healthy on-call culture requires: fair rotation (spread the load, compensate for on-call time), actionable alerts (every alert requires human action — no noise), runbooks for every alert (responder should never need to improvise), blameless postmortems (incidents are systemic failures, not individual failures), and time-boxed escalation (know when to escalate). Engineering metrics: MTTA (Mean Time to Acknowledge < 15 minutes), MTTR (Mean Time to Recover), alert volume per on-call shift. Red flags: >10 alerts per night, same incident recurring, on-call team burning out.

Common Misconception

Why It Matters

Common Mistakes

• No runbooks — responders improvise under pressure, increasing MTTR and mistakes.
• Same engineer on-call every week — burnout and bus factor.
• No compensation for on-call time — engineers resent the additional burden.
• Postmortems that blame individuals — systemic fixes prevent recurrence; blame does not.

Code Examples

# Unsustainable on-call:
# Alert: CPU > 70% for 1 minute — pages on-call
# Alert: any 500 error — pages on-call
# Alert: disk > 80% — pages on-call
# Alert: memory > 75% — pages on-call
# Result: 40 pages per night, all noise
# MTTR for real incidents: 2 hours (fatigue + no runbooks)
# Engineer turnover: high

# Sustainable on-call:
# Alert criteria: user-impacting only, 3-month review to prune noise
# Every alert: links to runbook with exact diagnostic steps
# Rotation: weekly, 5 engineers, no back-to-back
# Compensation: 1 day off per on-call week
# Postmortem: every P1/P2, blameless, action items tracked
# MTTA: < 5 minutes | MTTR: < 30 minutes
# Alert volume: < 5 pages per shift
# Engineer turnover: low

References

• ↗ https://sre.google/sre-book/being-on-call/

Tags

Related Terms