Onboarding Kubernetes with Observe: DaemonSet Log Collection and OpenTelemetry Tracing

Why a DaemonSet instead of a sidecar

There are two mainstream ways to collect logs in Kubernetes: inject a sidecar container into every Pod, or run a collector DaemonSet on every node. Sidecars sit closer to the application and can transform logs in process, but they multiply the container count, compete with the workload for CPU and memory, and become nearly impossible to maintain across dozens or hundreds of Pods. Every new deployment has to remember to add the sidecar, and every version bump has to be rolled out cluster-wide.

Observe recommends the DaemonSet approach. One collector per node reads container logs under /var/log/containers, tags them from Pod labels, and routes them to the correct service and namespace automatically. A node-level failure never affects other nodes, the collector is decoupled from the application lifecycle, and there is exactly one component per node to upgrade. For the vast majority of clusters this is the right trade-off, and it is the default the Helm chart ships with.

Step 1: Add the Helm repo and install

helm repo add jjhub https://charts.jjhub.cn
helm repo update
helm install observe-agent jjhub/observe-agent \
  --set endpoint=https://ob.example.com \
  --set token=$OBSERVE_TOKEN \
  --set cluster=prod-k8s

Once installed, each node runs an observe-agent DaemonSet that collects container stdout/stderr logs, cAdvisor metrics and Kubernetes events out of the box. endpoint, token and cluster are the only required values; everything else has sensible defaults. If your cluster enforces RBAC or runs an admission policy such as Pod Security Standards, the chart ships the required ServiceAccount and security context, but check that your admission webhooks are not blocking the agent namespace. The agent also reports its own health metrics, so you can confirm collection is live before touching any application.

Step 2: Configure log parsing

The DaemonSet reports each log line as a raw message by default. To make logs structurally searchable, define parsers that match parsing rules by container image:

parsers:
  - name: nginx-json
    match: image=~"nginx"
    type: json
    time_key: time_local
  - name: java-stacktrace
    match: image=~"java"
    type: multiline
    pattern: '^\d{4}-\d{2}-\d{2}'

Containers matching the nginx image get their logs parsed as JSON, while Java applications are recognized by their leading timestamps so multi-line stack traces are merged into a single entry instead of dozens of fragments. Multi-line merging is the step most often skipped and the one that hurts the most when you are debugging a Java or Python exception — a split stack trace is almost unusable in search results. Keep regex patterns anchored and simple; an unbounded lookahead on a busy node can quietly burn CPU.

Step 3: Inject OpenTelemetry tracing

Logs alone do not give you a call chain — you need to inject trace context on the application side. Use the OpenTelemetry Operator to auto-instrument workloads:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Annotate a namespace with instrumentation.opentelemetry.io/inject: "true" and the Operator injects an agent into each Pod automatically, sending traces and log correlation data to observe-agent, which forwards everything to Observe. Because the agent attaches trace context to log records as well, you can jump from a single log line straight into the full trace that produced it. For languages the auto-instrumentation does not cover, you still want to export traces directly to observe-agent's OTLP endpoint — the correlation works the same way either path.

Verify the full pipeline

After the first few requests hit your workload, run a quick check in the console:

SELECT * FROM logs WHERE cluster = 'prod-k8s' ORDER BY ts DESC LIMIT 50

If logs appear, pick any line with a trace id and open its trace — you should see the full span waterfall with the log line attached. If the trace is empty or truncated, the problem is almost always a hop in the middle dropping context, which the next section addresses.

Troubleshooting checklist

• No logs arriving: run kubectl -n observe get pods to check whether the agent is Running, then inspect the agent logs for endpoint or token errors. A wrong token is the most common cause and shows up immediately in the agent's own output.
• Logs present but unparsed: check whether the parser match hits the image name. The image name must include the full repository path — example.com/team/nginx:1.2 will not match a rule written against the bare nginx tag.
• Broken traces: confirm the injection annotation is on the correct namespace and that downstream callers also propagate context. A trace that stops at the gateway almost always means a hop in the middle dropped the headers.

DaemonSet collection plus Operator injection is the most cost-effective way to onboard Kubernetes: one configuration brings logs, metrics and traces online at the same time, and the same chart scales from a three-node dev cluster to a multi-region production fleet.