Kubernetes Log Ingestion: From DaemonSet to Structured Search

Two traps in container logging

Collecting logs from containers is not like collecting from VMs. First, logs land on each node's filesystem by default, so when a Pod reschedules to another node, the local logs stop following it. Second, the moment a Pod restarts — even during a routine rolling update — the log files inside the container can simply disappear, and by the time you want the history it is too late.

That is why the standard Kubernetes approach is to run one collection agent per node that streams stdout and log files out in real time, so logs land centrally on the platform instead of staying on the node. The agent runs as a DaemonSet, so it automatically follows nodes as they come and go, and you never have to think about "which node is this Pod on right now".

Deploying the agent as a DaemonSet

Run the agent as a DaemonSet — one Pod per node, mounting the host's log directories:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: jjhub-agent
spec:
  selector:
    matchLabels: { app: jjhub-agent }
  template:
    metadata:
      labels: { app: jjhub-agent }
    spec:
      containers:
      - name: agent
        image: jjhub/agent:2.5
        env:
        - name: TOKEN
          valueFrom:
            secretKeyRef: { name: jjhub-agent, key: token }
        - name: ENDPOINT
          value: https://ob.example.com
        volumeMounts:
        - { name: varlog, mountPath: /var/log }
        - { name: varlibdocker, mountPath: /var/lib/docker/containers, readOnly: true }
      volumes:
      - { name: varlog, hostPath: { path: /var/log } }
      - { name: varlibdocker, hostPath: { path: /var/lib/docker/containers } }

Keep the token in a Secret, not hard-coded in the YAML — a token that ships inside a manifest is a credential leak waiting to happen. The agent mounts /var/log for system logs and the Docker/containerd container directory for stdout, and pushes both to the platform over a single path.

Two details worth checking before you roll this out broadly. The container log directory differs by runtime: Docker uses /var/lib/docker/containers, but containerd (the default on most managed clusters) writes to /var/log/pods, so adjust the second mount to match your runtime. And mount it read-only — an agent should never need to write to the log directory it is collecting.

Injecting Pod metadata: so search can point to a person

Collecting logs is not enough — you also need to know which Pod, namespace and deployment each line belongs to. The agent attaches these labels automatically:

namespace=prod, pod=order-service-7f8c9, container=order, deployment=order-service

With those dimensions you can filter by namespace, Pod or container, and when something breaks you look up the Pod name from the alert and query its logs directly instead of digging through dozens of nodes. Metadata is what turns a wall of log lines into a searchable index — without it, every incident starts with "which of these thirty nodes should I even look at?".

There is one requirement on the application side: write logs to stdout, not to a file inside the container. stdout is picked up by the container runtime automatically, while log files need extra collection paths and are easy to miss. If you have legacy apps that insist on writing files, add a sidecar or an explicit collection rule — but push the team toward stdout for anything new.

Troubleshooting

1. Incomplete logs — check that the agent's local buffer is on, so logs queue to disk while the platform is briefly unreachable
2. No history after a Pod restart — confirm logs are streamed to the platform in real time rather than depending on node-local files
3. Split multi-line stacks — Java exceptions span multiple lines; enable multi-line merging or one exception becomes dozens of entries
4. Disk filled by logs — set rotation and caps on the node log directories so the agent itself does not fill the disk

Rollout advice

Start with one namespace and get collection, search and alerting working end to end, then roll out cluster-wide. Above all, verify the three metadata dimensions (namespace, pod, container) are present — they are the foundation for every search and alert that comes after. Agree on field names using the platform's convention so teams do not invent their own, and revisit the field catalog when onboarding a new service rather than letting naming drift year over year.