Onboarding Kubernetes to OBSERVE: Deploying the OpenTelemetry Collector for Logs and Metrics

Architecture: DaemonSet + Deployment

Onboarding a Kubernetes cluster into OBSERVE works best with the OpenTelemetry Collector split across two shapes, because they serve two very different jobs:

• DaemonSet: one per node, collecting node logs, container stdout, and kubelet metrics. It reads log files via hostPath and reports locally, so node-level data never has to travel across the cluster network. This is the workhorse for anything tied to a specific machine;
• Deployment: a central set of Collector replicas that receive application OTLP traffic (traces, metrics, logs), apply batching and load balancing, then forward to OBSERVE. This is the single ingress point for everything an application pushes.

This layering keeps node data local while giving application data one place where you can add sampling, redaction, and filtering — instead of sprinkling those policies across dozens of DaemonSets. If you run a large cluster, you can also scale the Deployment layer independently of the DaemonSet layer, since the two have very different CPU and memory profiles.

Deploying the Collector

Install the official Helm chart and override the key config:

mode: daemonset
config:
  receivers:
    filelog:
      include: [/var/log/pods/*/*/*.log]
      operators:
        - type: json_parser
    otlp:
      protocols: { grpc: {}, http: {} }
    prometheus:
      config:
        scrape_configs:
          - job_name: kubelet
            scheme: https
  exporters:
    otlp:
      endpoint: "otel.example.com:4317"
  service:
    pipelines:
      logs: [filelog, otlp]
      metrics: [prometheus, otlp]
      traces: [otlp, otlp]

The important bits: filelog.include points at /var/log/pods, and json_parser splits the container log into fields so downstream queries see structured data rather than one long string. The OTLP exporter points at the OBSERVE gateway. If your gateway requires TLS (it should in production), set the exporter's tls block accordingly rather than downgrading to plaintext.

Collecting application logs

Container logs go to stdout, which the collector reads from the node's log directory — no application code changes. That's the default path for Kubernetes workloads and covers most services. For apps that write files to a persistent volume (older apps, or a logging framework that won't go to stdout), mount that volume as hostPath in the DaemonSet and add another filelog rule pointing at the mount. Multi-line stacks — Java exceptions or Go panics — are merged with a multiline config keyed on the first-line regex, so a single stack trace stays a single log entry instead of being split into hundreds of lines.

A note on volume: parsing every container log on every node costs CPU. If a namespace generates huge volumes of debug logs, consider a filter in the pipeline to drop or rate-limit those lines before they're exported, which saves both bandwidth and storage.

Metrics and traces

Applications join tracing with no code changes by adding environment variables to the Pod:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector:4317"
  - name: OTEL_SERVICE_NAME
    value: order-service

Paired with the Java, Python, or Go auto-instrumentation agents, trace data flows to the in-cluster Collector Service and is forwarded by the Deployment layer. Point the endpoint at the Collector, not directly at OBSERVE — that way sampling, batching, and retries happen inside the cluster, and a network blip doesn't cause every application to buffer and retry on its own. Metrics from kubelet and the API server are scraped by the DaemonSet's Prometheus receiver, so you get node and control-plane visibility without a separate Prometheus install.

Verifying and debugging

After deploying, check the Collector's own metrics — otelcol_receiver_accepted_spans, otelcol_exporter_sent_spans, and the equivalents for logs and metrics — to confirm data is actually moving through the pipeline. Then look in OBSERVE for your service.name values. Common gotchas, in order of frequency: DaemonSet hostPath permissions (the filelog receiver can't read the pod logs), an OTLP endpoint pointing outside the cluster and timing out across the network boundary, and Pods missing OTEL_SERVICE_NAME so every service shows up as unknown. Fix the last one early — retroactively renaming services after traces have accumulated is messy.