Trace Sampling Strategies
debt(d9/e3/b5/t7)
Closest to 'silent in production until users hit it' (d9), bad sampling config (missing errors, too expensive) is invisible until incidents occur or the cloud bill arrives. OpenTelemetry config doesn't flag suboptimal sampling rates.
Closest to 'simple parameterised fix' (e3), per quick_fix it's a config change: set 5-10% head sampling plus OTel Collector tail sampling rules for errors. Centralised in collector config, not code-wide.
Closest to 'persistent productivity tax' (b5), sampling applies across web/cli/queue contexts and shapes what's debuggable forever after — every incident investigation depends on whether the right traces were kept.
Closest to 'serious trap' (t7), per misconception developers equate sampling rate with coverage of problems, not realising error-priority sampling captures 100% of errors at 1% overall rate. Contradicts intuition about statistical sampling.
TL;DR
Explanation
100% trace capture is expensive at scale. Sampling strategies: (1) Head-based: decision at trace start. Simple but may miss rare errors. Fixed-rate (1%), rate-limiting (100 traces/sec), probabilistic. (2) Tail-based: decision at trace end (after all spans collected). Can prioritise errors and high-latency traces. Requires OTel Collector with tail sampling processor. (3) Priority sampling: always sample errors, sample 1% of success. (4) Adaptive: adjust rate based on traffic volume. Recommended: 1-10% head-based for normal traffic, 100% for errors and slow traces. Trace ID-based sampling: all requests with same trace ID sampled consistently across services.
Common Misconception
Why It Matters
Common Mistakes
- 100% sampling in production — too expensive at scale.
- 0.1% sampling — misses too many errors.
- Not prioritising error traces — errors sampled the same rate as successes.
Code Examples
# 0.1% sampling — misses most errors:
sampling:
type: probabilistic
sampling_percentage: 0.1# OTel Collector tail sampling — 100% errors, 5% success:
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: errors-policy
type: status_code
status_code: {status_codes: [ERROR]}
- name: slow-policy
type: latency
latency: {threshold_ms: 1000}
- name: base-policy
type: probabilistic
probabilistic: {sampling_percentage: 5}