Collecting Kubernetes Logs: DaemonSet Deployment and Multiline Logs

Why a DaemonSet instead of changing the app

There are two ways to collect logs in Kubernetes: change each service to push logs itself, or collect centrally on the nodes. The first means touching every service's code; the second means adding one agent. OBSERVE recommends the latter: run one agent per node that reads container stdout from /var/log/containers, with zero application changes.

The benefits are clear. New services are collected automatically with no per-service config. Collection lives at the node level, so it doesn't compete with your workloads for resources. And when a pod crashes, its logs are still there — stdout is managed by the container runtime, not the application process, so a SIGKILL doesn't lose the last lines before death.

Deploying: one Helm command

helm repo add observe https://charts.ob.jjhub.cn
helm install observe-agent observe/agent \
  --set endpoint="https://ob.jjhub.cn/ingest" \
  --set token="replace-with-your-ingest-token"

Underneath it is a DaemonSet that mounts the host log directories read-only into the pod:

volumeMounts:
  - name: varlog
    mountPath: /var/log
    readOnly: true
  - name: containers
    mountPath: /var/lib/docker/containers
    readOnly: true

After installing, kubectl -n observe get pods should show one agent per node in Running state. One caveat: containerd stores logs under /var/log/pods, not the Docker path, so if your cluster runs containerd you need to adjust the mounts. Most managed clusters moved to containerd years ago, so this is the first thing to check when no logs show up.

Injecting Kubernetes metadata

Raw container logs have content and a timestamp but no notion of “which namespace, which pod, which container”. The agent enables metadata injection by default, stamping namespace, pod, container, and node as labels. You can then filter in OBSERVE with namespace="prod" AND pod=~"order-*", and the labels line up with Prometheus pod dimensions so logs and metrics share the same vocabulary.

Multiline logs: merge Java stack traces

A Java exception prints its stack across many lines. If each line becomes its own log record, the traceback gets shredded and a search shows you half an exception. The agent recognizes multiline blocks with a regex: a line matching the “start of record” pattern begins a new log, and everything else appends to the previous one.

multiline:
  firstline: '^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}'

The pattern here matches a leading timestamp, which is the most reliable marker because stack frames never start with a timestamp. Once configured, a full stack trace is one log record in OBSERVE, and clicking it shows every frame. If your app logs in a format without a leading timestamp, use the log level (ERROR, WARN) as the first-line marker instead — stack frames don't start with those either.

From raw lines to queryable fields

Shipping the lines is only half the job — the other half is making them queryable. If your services already emit JSON logs, the agent parses the fields automatically and you can filter on status, latency, and user_id immediately. If they emit plain text, configure a parse template in OBSERVE so the same fields get extracted at ingest time rather than at query time. Parsing at ingest costs a little CPU on the node; parsing at query time costs it on every search. For hot paths, do it once on the way in.

A useful rule of thumb: any field you expect to filter, group, or alert on more than a handful of times a day is worth extracting into a column. Leave genuinely free-form text — error messages, stack traces — as text and rely on full-text search for those. Resist the urge to regex-parse the free-form parts into columns; that's where regex maintenance costs quietly pile up and where most of the field-extraction bugs you'll ever fight will live.

What happens when the agent falls behind

The agent buffers locally and retries, so a brief network blip doesn't lose logs. If the endpoint is down long enough to fill the buffer, it drops oldest-first rather than blocking your pods — your application keeps running even if the observability pipe is saturated. Watch the agent's own dropped metric; if it's non-zero outside a known outage, you have a capacity or network problem to chase down before it silently eats the logs you'll need during the next incident.

Troubleshooting checklist

• No logs arriving: run kubectl -n observe logs on the agent first; nine times out of ten it's an expired token or an unreachable endpoint.
• High latency: check the agent's buffer size and flush interval; increase the buffer under network jitter so short outages don't backpressure the pipeline.
• Wrong fields: confirm whether the logs are JSON or plain text; plain text needs a parse template configured in OBSERVE to extract fields.
• Timestamps off by hours: if the container doesn't run UTC, the collector will parse times wrong; standardize on UTC and parse accordingly, otherwise every chart is shifted by eight hours.