Tail Latency (p95, p99)

Also Known As

TL;DR

Explanation

Average latency hides the experience of slow requests. Percentile metrics reveal tail behaviour: p50 (median), p95 (95% of requests are faster), p99, p99.9. At scale, tail latency is user-impacting — if p99 is 2 seconds and you serve 1,000 requests per second, 10 users per second experience that delay. Tail latency causes include: garbage collection pauses, lock contention, slow database queries, cold OPcache, and resource starvation. Monitor percentiles in production (Prometheus, Datadog, New Relic), set SLOs against p99, and alert when they breach thresholds rather than when averages rise.

Common Misconception

Why It Matters

Common Mistakes

• Monitoring only average latency — averages hide tail latency completely.
• Not setting per-request timeouts — one slow downstream call raises P99 for all requests waiting on it.
• Unbounded retries that amplify tail latency — a slow response retried 3 times triples the latency.
• Not hedging requests — sending a duplicate request to a second server after a threshold eliminates most tail latency.

Code Examples

// Monitoring average — hides P99 problems:
$avg = array_sum($latencies) / count($latencies); // 50ms average looks fine
// But P99 might be 2000ms — 1% of users see 2s loads

// Measure percentiles:
sort($latencies);
$p99 = $latencies[(int)(count($latencies) * 0.99)];

// Hedged requests — if P99 latency is high, send a second request
// after a short delay and use whichever responds first
function hedgedFetch(string $url, int $hedgeAfterMs = 100): mixed {
    $primary   = asyncFetch($url);
    $secondary = delay($hedgeAfterMs)->then(fn() => asyncFetch($url));
    return race([$primary, $secondary]); // first to resolve wins
}

// Measure percentiles — averages hide tail latency
// p50=10ms, p99=800ms means 1% of users wait 80x longer
// Use Prometheus histograms or StatsD timers to expose percentiles
$histogram->observe($responseTimeMs); // not just average

// Common tail latency causes: GC pauses, lock contention, network jitter
// Mitigation: timeouts + retries with jitter, connection pooling, async I/O

References

• ↗ https://brooker.co.za/blog/2021/04/19/latency.html

Tags

Related Terms