Choosing a Trace Sampling Strategy: Head, Tail, and Dynamic Sampling

Sampling rate is the tracing setting that is easiest to overlook and shows up most directly on the bill. Ingesting 100% of traces multiplies storage and query cost; dropping to 1% means missing the exact slow request you need during an incident. Get it wrong and you either drown in storage bills or fly blind when something breaks. The good news is that the decision usually comes down to one of three well-understood designs, and each maps cleanly to a different cost-versus-coverage posture.

Head-based sampling: predictable cost, but slow requests slip through

Head-based sampling decides at the moment a request enters the service, usually by fixed random ratio. TraceIdRatioBased in the OpenTelemetry SDK is the canonical example:

sampler:
  parentbased_traceidratio:
    root:
      traceidratio: 0.1

The advantage is simplicity and precise cost control: no matter how traffic grows, you always capture a fixed percentage. One subtlety worth knowing is that parent-based samplers inherit the root decision, so every child span follows the same keep-or-drop choice. That keeps trace graphs complete instead of leaving them half-finished, which matters a lot when you are staring at a broken waterfall view.

The drawback is that random is not the same as useful. A slow or failing request is sampled with exactly the same probability as a healthy one, so a one-off 30-second timeout may be precisely the trace that was dropped, leaving you with nothing to inspect afterwards. For systems where rare-but-severe events matter, head sampling alone is usually the wrong tool.

Tail-based sampling: slightly more expensive, but errors and slow requests rarely escape

Tail-based sampling postpones the decision until after the request finishes. The Collector buffers the spans of an entire trace in memory, then decides whether to keep or drop it based on status code, duration, and error state:

processors:
  tail_sampling:
    policies:
      - name: keep-errors
        type: status_code
        status_code: [ERROR]
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: default
        type: probabilistic
        sampling_percentage: 5

With this configuration every errored request and every request slower than one second is kept in full, while the rest are sampled at 5%. The trade-off is that the Collector must hold spans in a buffer, usually for 10 to 30 seconds, so memory usage and Collector cluster size grow accordingly. A rough sizing rule: buffer capacity equals peak spans per second multiplied by the wait window, plus headroom for slow-finishing requests. High-cardinality workloads can exhaust that buffer, so always set an upper bound and alert on buffer evictions.

Dynamic sampling: follow traffic and error rate

Dynamic sampling adjusts the rate based on live traffic and error rate: raise it during low-traffic hours, lower it during peaks. OBSERVE supports per-service sampling rates and can switch to full capture automatically when the error rate spikes, so no trace is lost during an incident window. It suits services with pronounced peaks and troughs where you would rather not pay for extra Collector capacity just to run tail sampling. The one thing to design for is a small control loop that occasionally recomputes the rate; that state should be cheap to store and cheap to distribute to all the services that read it.

How to choose: a decision table you can act on

• Purely cost-sensitive, traces used only for macro traffic visibility: head-based sampling at 1% to 5%.
• Slow requests and errors must stay traceable: tail-based sampling, keeping errors plus slow requests plus a 5% fallback.
• Pronounced peaks and troughs, balancing cost and coverage: dynamic sampling.
• Critical transaction paths (payment, checkout, reconciliation): force 100% capture by route or custom label regardless of the global rate.

You can also combine approaches: many teams run head sampling on a noisy internal service while forcing full capture on a few high-value endpoints. The key is to be deliberate about which traffic you can afford to lose, and to write that decision down so the next engineer understands why the rates are what they are.

Common mistakes to avoid

A few traps show up repeatedly. First, sampling at the SDK level and again at the Collector level multiplies the effective rate without anyone noticing. Second, tail-sampling policies ordered wrong can shadow each other, so keep the keep-errors and keep-slow rules before the probabilistic fallback. Third, changing the sampling rate without re-evaluating storage means you either waste money or lose coverage silently.

One final reminder: sampling works in concert with logs and metrics. A dropped trace is fine, but your logs must still carry the traceId so you can pivot from a single log line back to the surrounding request context during an investigation, instead of losing the trail entirely.