P50/P95/P99 Latency Percentiles

TL;DR

Explanation

Average latency is misleading — a 100ms average can mask 5% of requests taking 2 seconds. Percentiles: P50 (median — half faster), P95 (95% faster — almost everyone), P99 (99% faster — worst 1%), P99.9 (worst 0.1%). P50 ≈ typical user. P99 = power users or large-data users. P99.9 = outliers (usually infrastructure issues). Implementation: histogram metrics in Prometheus. histogram_quantile(0.99, rate(http_duration_bucket[5m])). Aggregating percentiles: can't average percentiles across instances — must use histogram buckets. Set SLO on P99, not average.

Common Misconception

Why It Matters

Common Mistakes

• Monitoring average instead of percentiles.
• Aggregating percentiles from different instances — statistically invalid.
• Setting SLO on P50 — only half of users satisfy it.

Code Examples

// Average latency metric — hides outliers:
Gauge::set('latency_avg', $totalTime / $count);
// 100ms average, but 1% of requests take 5s

// Prometheus histogram — correct percentiles:
$histogram = $meter->createHistogram('http.request.duration');
$histogram->record($durationMs, ['route' => $route]);

// Query P99:
// histogram_quantile(0.99, rate(http_request_duration_bucket[5m]))

// SLO: P99 < 500ms

References

• ↗ https://prometheus.io/docs/practices/histograms/

Tags

Related Terms