Onboarding Log Collection in 5 Minutes: DaemonSet Deployment and Parsing Rules

Why deploy as a DaemonSet

Container logs usually go to stdout and are collected by the runtime. But in production a lot of logs live in files — a Java app's application.log, Nginx's access.log, business-rolled log files. To collect those, the most reliable approach is running one agent per node via a DaemonSet, mounting the host's log directories with hostPath so the agent reads them locally. Each node gets one agent, logs are collected close to the source with no network hop, and there's no single point of failure in collection. Two more reasons to prefer a DaemonSet over a sidecar or a central push: you collect node-level logs (kubelet, container runtime) that no pod can see, and you don't pay for an extra container per pod. Do set resource requests and limits on the agent container — a log spike shouldn't let the collector starve your workloads of CPU.

Step 1: Deploy the agent

Jujing's collection agent supports one-click DaemonSet deployment. Generate an access token first (console → Access Management → Data Ingestion), fill in the placeholder, and apply:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: jujing-agent
spec:
  selector:
    matchLabels: { app: jujing-agent }
  template:
    metadata:
      labels: { app: jujing-agent }
    spec:
      containers:
      - name: agent
        image: registry.jjhub.cn/jujing/agent:latest
        env:
        - name: JJHUB_TOKEN
          value: "<your-token>"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlib
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath: { path: /var/log }
      - name: varlib
        hostPath: { path: /var/lib/docker/containers }

/var/log is for host log files; /var/lib/docker/containers links container stdout. readOnly: true is mandatory — the agent only reads logs, and denying write access is a security baseline. If you run a non-Docker runtime, swap the second hostPath to the matching container-log directory (e.g. /var/log/pods on containerd) or drop it and rely on the runtime's own forwarding.

Step 2: Configure sources and parsing rules

After deployment, sources and parsing rules are configured in the console, not in YAML. For Nginx, point a source at /var/log/nginx/access.log and set parsing rules:

• Multiline merge: Java stack traces and Python tracebacks span lines. Set a multiline mode with a regex matching the start line (e.g. ^\d{4}-\d{2}-\d{2}), and following lines attach to the same entry. Get the start-line regex right, or a single stray match will glue unrelated lines together.
• JSON extraction: when the log line is already JSON, pick JSON parsing and fields become searchable key-values with no hand-written regex.
• Field extraction: for unstructured logs, use grok/regex to pull key fields — extract status, latency, uri from access.log so you can filter and aggregate by field afterwards.

Parsing rules are retroactive-safe: changing a rule only affects newly ingested logs; history stays as-is, no re-collection needed.

Step 3: Verify ingestion and watch for pitfalls

Go to Log Search and filter by service or host.name to check for new entries. If nothing shows, troubleshoot in this order: confirm the agent is Running (kubectl get pods -l app=jujing-agent), check agent logs for token errors (401/403 means a bad token), then confirm the path actually exists inside the container (ls /var/log/nginx/). Two pitfalls worth calling out: log rotation — the agent tracks file inode offsets, so configure it to follow renames, not just the original path; and backpressure — if the agent can't keep up, it buffers to disk rather than dropping lines, so watch its queue depth. A third one for tailing: if a file is being actively written, the last line can arrive half-written, which is exactly why the multiline start-line regex matters even for otherwise single-line logs. Once data flows, set retention and hot/cold tiering, and your day-to-day triage loop is complete.