OpenTelemetry Collector in Practice: Sampling, Batching and Troubleshooting Common Errors

Why you need a Collector

Having every service push data straight to the platform is simple but fragile: each service must know the ingestion endpoint, a network hiccup drops data, and changing a sampling policy means redeploying every service. Putting an OpenTelemetry Collector in between solves this — services talk only to a local or in-network Collector, which handles sampling, batching, retries and forwarding in one place.

Agent or gateway: two deployment modes

The Collector can run as an agent or sidecar next to each service, or as a central gateway that every service forwards to. An agent keeps per-node overhead low and isolates noisy neighbors, while a gateway gives you a single place to apply sampling, redaction and rate limits. Many teams run both: a lightweight agent per host forwarding to a central gateway, which then applies tail sampling and exports to Observe. Start with a gateway when you have only a handful of services; add agents when per-node resource contention becomes a problem.

A config you can use today

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    send_batch_size: 512
    timeout: 5s
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-and-slow
        type: and
        and:
          and_sub_policy:
            - name: keep-errors
              type: status_code
              status_code: { status_codes: [ERROR] }
            - name: keep-slow
              type: latency
              latency: { threshold_ms: 500 }

exporters:
  otlphttp:
    endpoint: https://ob.example.com/v1/traces
    headers:
      Authorization: 'Bearer <token>'

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [otlphttp]

Three things that matter

• Put memory_limiter first. It absorbs memory spikes and keeps the Collector from being OOM-killed by a burst of traffic.
• batch merges requests. It accumulates individual spans into batches before sending, cutting network and ingestion load substantially.
• Tail sampling decides after the fact. Head sampling drops spans randomly at the entry point; tail sampling waits for the whole trace to finish and keeps only the ones that errored or ran slow, so you never discover mid-incident that the critical trace was sampled away.

Sampling and cost

Log and trace volume is the biggest cost driver. A common policy keeps 100% of errors and requests over 500ms while sampling normal traffic at 10%. Tail sampling holds storage cost down while guaranteeing that every slow or failing request is preserved. Start from these defaults, then tune based on what you actually query — if the team only ever investigates errors and slow requests, sampling healthy traffic more aggressively rarely hurts debugging.

Redact sensitive data before it leaves

The Collector is the right place to strip secrets before telemetry ever reaches the platform. Use the redaction processor to mask card numbers, tokens and passwords from log records and span attributes, so sensitive values never leave your network:

processors:
  redaction:
    blocked_values: ['password', 'token', 'authorization']

Wire it into the pipeline ahead of the exporter, and verify with a sample payload that masked fields arrive as ***.

The three errors you will actually hit

1. context deadline exceeded: the Collector timed out reaching the platform. Check that the endpoint is reachable and that no proxy is in the way — start with curl -v.
2. 401 unauthorized: a wrong or expired token. Regenerate it in the console, and whatever you do, do not commit it to the repository.
3. Data arrives but queries return nothing: almost always a timezone mismatch between span timestamps and your query window. Standardize on UTC or configure the timezone explicitly.

Scaling and reliability

Run at least two Collector instances behind a load balancer and configure failover on the service side, so one instance dying does not drop telemetry. Watch the Collector's own queue depth and refused-span count: a rising queue means the exporter cannot keep up, and you should either increase send_batch_size or scale out. Because every service depends on the Collector, it deserves the same monitoring care as any production service.

Verify the pipeline end to end

Start the Collector, fire a single trace from a minimal demo service, then search for it by trace id in the console. If you see the full call chain with span attributes and durations, the pipeline is wired correctly.