Ingesting K8s Logs, Metrics and Traces with One OpenTelemetry Collector Config

Why a Collector instead of wiring the SDK into every service

Plugging an SDK into every service and configuring an endpoint for each one becomes a nightmare as your service count grows: change the reporting endpoint and you touch dozens of services, sampling policy can't be enforced uniformly, and each SDK retries differently when the network hiccups. Worse, logs, metrics and traces often target three different receivers, so the configuration is duplicated and easy to get wrong. Secrets — the ingestion tokens — end up scattered across every repository and environment.

The OpenTelemetry Collector closes all of that into one place. It runs as a resident collector (or a set of them) inside the cluster, receives OTLP data from every service, applies sampling, batching and scrubbing, then forwards it to Observe. Applications only care about one thing: sending data to the local Collector. Everything after that — authentication, retries, batching, even redaction of sensitive fields — is centralized in one config file the platform team owns.

Once it is in place, adding a new signal, applying consistent sampling, or scrubbing API keys from logs becomes a one-line change in a single file instead of a coordinated rollout across every service. That single choke point is also where you enforce data governance — dropping fields you must not store, or routing PII to a separate pipeline.

Deploying the Collector: three Helm steps

The official Helm chart is the least-friction path. Three steps:

# 1. Add the repository
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# 2. Install in DaemonSet mode (one instance per node, for local log collection)
helm install otel-collector open-telemetry/opentelemetry-collector \
  --set mode=daemonset \
  --set config.exporters.otlp.endpoint=ob.example.com:4317

# 3. Check the state
kubectl get pods -l app.kubernetes.io/name=opentelemetry-collector

DaemonSet rather than Deployment matters because log collection usually reads container log files off each node, so one instance per node is more direct. For pure metrics/traces workloads, a Deployment with load balancing also works fine. If you run a large cluster, you can even split roles: DaemonSet collectors for logs, and a Deployment of gateway collectors for metrics and traces.

Configuring ingestion and forwarding for all three signals

The heart of the setup is config.yaml. This configuration receives logs, metrics and traces simultaneously and forwards them all to Observe over OTLP:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
  filelog:
    include: [/var/log/pods/*/*/*.log]
    operators:
      - type: container
        id: container-parser

processors:
  batch:
    send_batch_size: 1024
    timeout: 1s
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlp:
    endpoint: ob.example.com:4317
    headers:
      Authorization: "Bearer <your-ingestion-token>"

service:
  pipelines:
    logs:
      receivers: [otlp, filelog]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Key points: memory_limiter goes first to keep an upstream traffic burst from crushing the Collector; batch packs scattered data into larger chunks to reduce downstream pressure and amortize network round-trips; authentication uses a token in the headers, configured under exporters.otlp.headers. The filelog receiver tails the standard pod log directory, and the container operator enriches each entry with Kubernetes metadata like pod name, namespace and container name — the fields you'll later filter and group by in Observe.

Verification and common pitfalls

After deployment, generate a few log lines and a request, then verify in the Observe console with a SQL-like query:

SELECT * FROM logs WHERE service = 'your-service' ORDER BY ts DESC LIMIT 20;

Common pitfalls:

• Nothing arrives: check the Collector's own logs for exporter errors — it's usually a mistyped token or endpoint
• Duplicate or missing logs: confirm the DaemonSet actually scheduled one pod per node and that the filelog include path matches your layout
• Timezone drift: if container and host timezones differ, log timestamps shift in bulk; standardize on UTC or set TZ consistently
• Memory spikes: when memory_limiter isn't in effect, peak traffic can OOM the pod — make sure limit_mib is realistic

Finally, set a resource budget for the Collector early. Two CPUs and 1 GiB per node is a reasonable starting point for most clusters; watch the exporter queue length and scale up before it becomes the bottleneck.

Treat the Collector as cluster infrastructure: pin its version, keep the config in one place, and monitor its own health (its own metrics can be scraped like any other service). When it's stable, every upstream service's data is reliable.