Log Sampling Strategies

Also Known As

TL;DR

Explanation

Log sampling is the practice of writing only a fraction of eligible log events to disk or a downstream aggregator, chosen so the retained subset still reflects system behaviour. High-traffic services produce logs faster than humans, storage, or search indexes can absorb - sampling caps that firehose while preserving the signals engineers actually query. Common strategies fall into three families. (1) Head-based (upfront) sampling: decide at request entry whether to log this trace or drop it. Probabilistic head sampling (e.g. keep 1 in 100) is trivial to implement and gives predictable volume, but can miss rare errors. (2) Tail-based sampling: buffer events for a request and decide after completion - keep everything for errors, slow requests, or specific tenants, drop routine successes. This preserves debugging value but requires buffering infrastructure. (3) Deterministic/consistent sampling: hash a trace or user id and keep events whose hash falls in a chosen bucket. Two services making the same decision on the same id means correlated logs stay together across a distributed trace. Additional techniques layer on top: rate limiting per log key (log the first N of each error signature per minute), reservoir sampling for unbounded streams, priority sampling that always keeps ERROR/WARN and samples INFO/DEBUG, and dynamic sampling that adjusts rate based on current volume or cost budget. The critical design rule: always retain 100% of errors, security events, and audit records. Sample the boring stuff. Record the effective sample rate as a field on every emitted event so aggregations can multiply back up to true counts. Without that, dashboards silently under-report.

Common Misconception

Why It Matters

Common Mistakes

• Applying a flat probabilistic rate to all events, so rare errors and security-relevant events get dropped alongside routine traffic.
• Not recording the sample rate on emitted events, causing dashboards and alerts to report artificially low counts.
• Using inconsistent sampling decisions across services in a trace, so logs for the same request appear in one service but not the next.
• Sampling before applying rate limits, so a single noisy log line still floods the pipeline just at a lower percentage.
• Head sampling without any tail-based override for errors, missing the exact requests that engineers need to investigate.

Avoid When

• Audit, compliance, or security event logs where regulators require complete records.
• Low-volume services where full logging is affordable and every event has debugging value.
• Financial or transactional flows where every state transition must be reconstructible.

When To Use

• High-throughput HTTP or RPC services where per-request INFO logs dominate ingest cost.
• Distributed traces where consistent sampling across services keeps correlated events together.
• Debug or verbose logging in production where a small representative sample is enough for trend analysis.
• When observability spend is capped and you must choose between sampling or dropping entire log streams.

Code Examples

// Flat 1% sampling - drops 99% of errors too
if (Math.random() < 0.01) {
    logger.info('request', { path, status, duration });
}
// Errors get dropped at the same rate as successes.
// No sample_rate field, so downstream counts are wrong.
if (Math.random() < 0.01) {
    logger.error('payment_failed', { orderId, err });
}

// Priority + consistent sampling with rate metadata
// hash() = any stable 32-bit hash, e.g. FNV-1a or xxhash of the string traceId
function sampleRate(level, traceId) {
    if (level === 'error' || level === 'warn') return 1.0;
    if (level === 'info')  return 0.1;
    return 0.01; // debug
}

function shouldEmit(level, traceId) {
    const rate = sampleRate(level, traceId);
    // Deterministic: same traceId => same decision across services
    const bucket = (hash(traceId) % 10000) / 10000;
    return { keep: bucket < rate, rate };
}

const { keep, rate } = shouldEmit(level, ctx.traceId);
if (keep) {
    logger.log(level, msg, { ...fields, sample_rate: rate, trace_id: ctx.traceId });
}
// Downstream: true_count = count(*) * avg(1/sample_rate)

Tags

Related Terms

References

• https://opentelemetry.io/docs/concepts/sampling/
• https://sre.google/workbook/monitoring/
• https://www.honeycomb.io/blog/dynamic-sampling-by-example
• https://vector.dev/docs/reference/configuration/transforms/sample/