Ingest Kubernetes Logs and Metrics with the OpenTelemetry Collector

Getting Kubernetes logs and metrics into your observability platform is usually the first step of any observability build-out. This guide walks you through it with the OpenTelemetry Collector: one DaemonSet, one config, and your logs and metrics land in Observe with pod metadata attached — filterable by namespace and correlatable by trace_id.

Architecture and prerequisites

Deploy the OpenTelemetry Collector as a DaemonSet so one instance runs per node, collecting the stdout/stderr logs and cAdvisor metrics of every container on that node, then shipping everything to Observe's OTLP endpoint. Compared with injecting a sidecar into every pod, a DaemonSet shares resources node-wide, costs less, is easier to upgrade, and gives the whole cluster one consistent pipeline. The trade-off is losing per-pod customization, which logs and metrics rarely need. We choose the Collector over a hand-rolled script because it standardizes the messy parts — parsing, filtering, redaction, retries — and the community maintains it.

Before you start, confirm three things: Kubernetes is 1.22 or newer, the Collector's nodes can reach Observe's OTLP endpoint (open egress rules in restricted networks), and you already have the service's ingestion token. On a mixed-architecture cluster where some nodes are ARM, set a nodeSelector on the Collector so every node runs the image built for its architecture.

Deploying the Collector

The key pieces: the filelog receiver reads container logs, the k8s_cluster receiver pulls node and pod metrics, the k8sattributes processor enriches logs with pod name, namespace and labels, and the otlphttp exporter pushes to Observe. The type: container operator parses the container runtime's log format, so the message and timestamp come out clean instead of wrapped in an extra layer of JSON.

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector
data:
  config.yaml: |
    receivers:
      filelog:
        include: [/var/log/containers/*.log]
        operators:
          - type: container
      k8s_cluster:
        auth_type: serviceAccount
    processors:
      k8sattributes:
        extract:
          metadata: [k8s.pod.name, k8s.namespace.name]
    exporters:
      otlphttp:
        endpoint: https://ob.example.com/otlp
        headers:
          Authorization: Bearer ${OBSERVE_TOKEN}
    service:
      pipelines:
        logs:
          receivers: [filelog]
          processors: [k8sattributes]
          exporters: [otlphttp]
        metrics:
          receivers: [k8s_cluster]
          processors: [k8sattributes]
          exporters: [otlphttp]

You'll also need a ServiceAccount with RBAC so the collector can list pods and namespaces, and resource requests and limits on the DaemonSet so a noisy neighbor can't starve it. On containerd clusters, confirm the include path actually resolves to the log files — it's a symlink on most distributions, but verify it. If your applications print multi-line stack traces, add a multiline operator so a single exception isn't split into a dozen partial log lines that don't line up during an investigation.

Configuring the service and token

Create a service in the Observe console and grab its ingestion token, then inject it into the Collector's environment through a Secret rather than a plain-text ConfigMap. Rotate the token on a schedule; after rotation, restarting the Collector is enough to pick up the new value. On high-volume clusters, configure batching and retries on the exporter so per-line pushes don't saturate the network. If your logs carry sensitive fields, a redaction processor can strip them before they leave the cluster — especially important in finance and government settings.

Verification and common errors

After deploying, check the Collector's own logs first and confirm the exporter isn't returning 401s or timeouts. Then confirm data is flowing with a SQL-like query:

SELECT * FROM logs WHERE k8s.namespace.name = 'production' AND ts > now() - 5m LIMIT 20

Check metrics too — if you can query node CPU and memory, the k8s_cluster pipeline is working. Common issues: an expired token causes 401s; an empty filelog usually means the include path doesn't match the container runtime's log directory (containerd and Docker differ); missing metrics point to incomplete RBAC for the k8s_cluster receiver; duplicate logs mean the collector and an existing log agent are tailing the same files.

Getting data flowing is just the first step. Two things to do next: correlate logs with trace_id so logs, traces and metrics line up into one story, and set per-environment retention — keep production logs longer and test logs shorter to control cost without hurting investigations.