Alert Fatigue
debt(d8/e7/b7/t6)
Closest to 'silent in production until users hit it' (d9), -1. Tools like PagerDuty, Datadog, and Grafana can surface metrics like alert volume and silence rates, but detection is explicitly marked as 'automated: no'. The code_pattern describes behavioral signals (alerts silenced without action, high noise-to-signal ratio) that require deliberate human audit to notice. There's no automated tool that flags 'you have alert fatigue' — it manifests as on-call engineers ignoring pages until a real incident is missed. Scoring d8 because monitoring platforms can at least surface the raw data (alert counts, acknowledgment rates) even if they don't flag the problem automatically.
Closest to 'cross-cutting refactor across the codebase' (e7). The quick_fix says 'audit your alerts monthly' which sounds simple, but the common_mistakes reveal the real scope: pruning stale alerts, rewriting threshold-based alerts to symptom-based ones, adding runbooks, separating critical from informational channels — this touches monitoring configs, runbooks, on-call processes, and team culture across the entire system. It's not architectural rework (e9) but it's clearly cross-cutting effort spanning infrastructure, application code, and organizational processes.
Closest to 'strong gravitational pull' (b7). Alert fatigue is a cultural and structural problem that shapes how teams respond to incidents. It applies across web and CLI contexts, and the tags (observability, devops, monitoring, culture) indicate it's cross-cutting. Once alert fatigue sets in, it degrades the entire monitoring investment — every new alert added risks making it worse, and every incident response is shaped by the team's relationship with the alerting system. It doesn't quite define the system's shape (b9) but it exerts strong gravitational pull on operational practices.
Closest to 'notable trap' (t5), +1. The misconception is explicit: 'More alerts means better monitoring coverage.' This is a genuinely common and intuitive wrong belief — developers naturally equate more coverage with better safety. However, it's not as severe as t7 (contradicts similar concepts elsewhere) since experienced ops engineers do learn this lesson, and the concept of signal-to-noise is broadly understood. The +1 is because the trap is insidious: the damage from acting on the misconception is gradual and the feedback loop is delayed, making it harder to recognize than a typical documented gotcha.
Also Known As
TL;DR
Explanation
Alert fatigue occurs when noise overwhelms signal: too many non-actionable alerts, alerts for things that resolve themselves, alerts at inappropriate severity, or alerts with no clear remediation steps. The consequence: real alerts are missed or dismissed. Remediation: audit all alerts for actionability (is there anything a human should do?), eliminate or downgrade non-actionable alerts, use multi-condition alerts (error rate AND high latency, not just error rate), and add runbook links to every alert. Alert on symptoms (user-facing impact) not causes (CPU spike).
Common Misconception
Why It Matters
Common Mistakes
- Alerting on every 5xx response — alert on error rate above threshold, not individual errors.
- CPU threshold alerts that don't correlate with user impact — alert on latency or error rate instead.
- Alerts with no runbook — 2am page for an alert nobody knows how to respond to.
- Never reviewing and pruning stale alerts — monitoring debt accumulates just like code debt.
Avoid When
- Do not alert on every transient spike — short bursts that self-resolve train engineers to ignore alerts.
- Avoid creating alerts without a defined runbook action; an alert with no clear response step adds noise.
- Do not duplicate alerts across multiple channels for the same event — multiple pages for one incident multiply fatigue.
When To Use
- Alert on user-facing symptoms (error rate, latency, availability) rather than internal resource metrics.
- Set alert thresholds that require a human decision — if the response is scripted, automate it instead of paging.
- Use separate channels: critical page-worthy alerts vs informational ones that appear in a dashboard only.
Code Examples
# Alert configuration causing fatigue:
# Alert: any 500 error → page on-call
# Alert: CPU > 70% for 1 minute → page on-call
# Alert: memory > 80% → page on-call
# Alert: any slow query > 100ms → page on-call
# Alert: disk > 85% → page on-call
# Result: 40 pages per night, all noise
# Real incident at 3am: ignored like the previous 39# Alert on user-facing symptoms with thresholds:
# CRITICAL (page): error rate > 1% for 5min → user impact, human needed
# CRITICAL (page): p99 latency > 3s for 5min → user impact
# WARNING (Slack): error rate 0.1-1% → watch, may self-resolve
# INFO (dashboard only): CPU, memory, disk trends
# Every alert has:
# - Clear title: 'Checkout error rate elevated'
# - Runbook link: 'See runbook/checkout-errors.md'
# - Auto-resolution when metric recovers