Unified Log and Metric Collection on Kubernetes with the OpenTelemetry Collector

Why collect with a single Collector

A Kubernetes cluster runs dozens of services. If every application configures its own exporter, then tokens, ingestion endpoints and sampling rates end up scattered across a dozen repositories, and changing any of them becomes a chore. The value of the OpenTelemetry Collector is that it centralizes receive, process and forward in one place: applications send their data only to the Collector running on the local node, and the Collector handles filtering, redaction and batching before forwarding to the backend.

Observe speaks OTLP natively, so a single OTLP exporter on the Collector can ship logs, metrics and traces to the platform in one stream. This guide walks through the whole setup, from the DaemonSet manifest to verification.

Deployment: one DaemonSet

Run the Collector as a DaemonSet on every node, so each instance can collect the stdout logs of all containers and the host metrics on its own node:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  config.yaml: |
    receivers:
      filelog:
        include: [/var/log/pods/*/*/*.log]
        operators:
          - type: json_parser
      hostmetrics:
        scrapers: [cpu, memory, disk, network]
    exporters:
      otlphttp:
        endpoint: https://ob.example.com/otlp
        headers:
          Authorization: Bearer <your-token>
    service:
      pipelines:
        logs:    {receivers: [filelog], exporters: [otlphttp]}
        metrics: {receivers: [hostmetrics], exporters: [otlphttp]}

Deploy the DaemonSet with kubectl apply -f, and mount the host's /var/log/pods and /var/log/containers directories into the Collector container. The filelog receiver tails those paths, and the json_parser operator extracts structured fields from each line so you can filter on them later in the platform.

The application only does one thing: OTLP export

After adding the OpenTelemetry SDK to your Java, Go or Python service, set OTEL_EXPORTER_OTLP_ENDPOINT to point at the local Collector (localhost:4318). The Collector then attaches cluster metadata — node, namespace and pod labels — before forwarding, so application code stays untouched while the data arriving in Observe becomes more complete and easier to query.

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=order-service

Because every pod talks only to the Collector on its own node, there is no cross-node traffic, and the exporters in your applications never need to know the Observe endpoint or token at all. Rotating a token later becomes a one-place change rather than a redeploy of every service.

Processing at the edge: filtering, redaction and sampling

Collector processors transform data before it reaches the backend. Redact phone numbers and ID numbers from access logs with a redaction processor:

processors:
  redaction:
    allow_all_keys: true
    blocked_values: [password, token]

Combine that with a filter processor to drop debug-level logs and a batch processor to merge small batches and cut network overhead. Traces are the noisiest of the three signals, so gate them with a sampling processor that keeps every error span and a small fraction of healthy traffic:

processors:
  probabilistic_sampler:
    sampling_percentage: 10

Handling this at the edge, before data leaves the cluster, can reduce backend storage costs by more than 30 percent and keeps sensitive fields out of the platform entirely — which also makes compliance reviews much easier.

Verification and troubleshooting

After deployment, query one recent log in the console and confirm fields such as k8s.namespace and k8s.pod are present; on the metrics page, confirm host metrics are flowing in. The most common issues, in order: forgetting to mount /var/log/pods so filelog sees nothing; mixing up the OTLP protocol (4317 is gRPC, 4318 is HTTP); and putting the token in the wrong header name. Give the DaemonSet a generous but bounded memory limit — the Collector buffers data in memory, and under a traffic spike a tight limit causes dropped spans rather than a crash that takes the whole pipeline down with it. Work through those checks and the pipeline almost always comes up within a few minutes.

A single Collector config, versioned in your repo, is a small investment that pays off every time you add a service, rotate a token, or need to redact a new sensitive field — you change it in one place and every node picks it up on the next rollout.