Prometheus & Grafana
debt(d8/e5/b4/t6)
Closest to 'silent in production' (d8) — missing metrics or high-cardinality labels won't surface until production incidents or Prometheus itself crashes; no automated tool flags absence of a /metrics endpoint.
Closest to 'simple parameterised fix' to 'multiple files' (e5) — adding promphp/prometheus_client_php and exposing /metrics is straightforward, but instrumenting meaningful business metrics (histograms, counters, gauges) touches multiple components across web/cli/queue contexts.
Closest to 'localised tax' to 'persistent tax' (b4) — metrics instrumentation is mostly additive and isolated to instrumentation points, but cardinality discipline and dashboard maintenance impose ongoing tax across the codebase per applies_to scope.
Closest to 'serious trap' (t6) — per misconception, devs assume Prometheus is K8s-only; plus the high-cardinality label trap (user_id as label) is catastrophic and counterintuitive since labels feel like normal tags, slightly less than t7.
Also Known As
TL;DR
Explanation
Prometheus pulls metrics from /metrics endpoints exposed by applications (pull model). Metric types: Counter (monotonically increasing, e.g. request count), Gauge (current value, e.g. active connections), Histogram (distribution, e.g. request duration percentiles), Summary. PromQL queries aggregate and alert on metrics. Grafana connects to Prometheus as a data source and provides dashboards. For PHP, the prometheus_client_php library exposes metrics; Blackfire and Datadog offer hosted alternatives.
Diagram
flowchart LR
APP[PHP App] -->|expose /metrics| PROM[(Prometheus<br/>scrapes every 15s)]
PROM -->|PromQL queries| GRAF[Grafana<br/>dashboards]
PROM -->|alert rules| ALERT[Alertmanager]
ALERT -->|PagerDuty Slack| ONCALL[On-call engineer]
subgraph Metrics_Flow
COUNTER[Counter<br/>requests_total]
HIST[Histogram<br/>request_duration]
GAUGE[Gauge<br/>active_connections]
end
APP --> COUNTER & HIST & GAUGE
style PROM fill:#d29922,color:#fff
style GRAF fill:#1f6feb,color:#fff
style ALERT fill:#f85149,color:#fff
style ONCALL fill:#238636,color:#fff
Common Misconception
Why It Matters
Common Mistakes
- High-cardinality label values — using user_id or request_id as Prometheus labels creates millions of time series, crashing Prometheus.
- Not using histograms for latency — average latency hides tail latency; histograms expose P95/P99.
- Scrape interval too long — 60s interval misses sub-minute spikes; 15s is typical.
- Not setting alert thresholds based on historical data — arbitrary thresholds create alert fatigue.
Code Examples
// High-cardinality label — DO NOT DO THIS:
$histogram->observe($duration, ['user_id' => $userId]); // 1M users = 1M time series
// Prometheus OOMs and crashes with too many series// Low-cardinality labels only:
$histogram->observe($duration, [
'route' => '/api/users', // ~20 routes — OK
'method' => 'GET', // ~5 methods — OK
'status' => '200', // ~10 status codes — OK
]);
// NOT: user_id, request_id, session_id — too many unique values