Shipping Kubernetes Logs and Metrics in 10 Minutes: A Collector Setup Guide

The first step to observability on Kubernetes is getting logs and metrics out reliably. This article walks through onboarding Jujing OBSERVE from scratch — DaemonSet deployment, filtering, field conventions, resource limits, and a few high-frequency pitfalls.

Deploy the collector: one DaemonSet for the whole cluster

The collector runs as a DaemonSet — one replica per node — collecting that node's container stdout/stderr logs, cAdvisor metrics, and Kubernetes events. Helm installs it in two steps:

helm repo add jjhub https://charts.ob.jjhub.cn
helm upgrade --install ob-collector jjhub/ob-collector \
  --set endpoint=ob.example.com:4317 \
  --set tenant=platform \
  --set token=$OB_TOKEN

After installing, confirm every node's Pod is Running, then confirm the backend is receiving data. A common mistake is collecting logs only to the local disk without a working upload endpoint — you collect a pile of data but can't search it. Check the collector's own logs for successful upload records first.

Two security notes worth handling up front. First, use a dedicated service-account token with the narrowest permissions rather than an admin token, so a compromised collector can't do anything beyond shipping data. Second, put the token in a Secret and reference it by name, never inline in the Helm values, and rotate it on a schedule.

Filtering and labels: don't ship all the noise

Collecting everything by default causes two problems: data volume explodes and search slows down. We recommend filtering by namespace, excluding system namespaces like kube-system and temporary test namespaces, and collecting only business namespaces:

namespaceFilter:
  include: ["prod-*", "staging"]
  exclude: ["kube-system", "kube-public"]

At the same time, tag every log line with Kubernetes metadata: namespace, pod, container, node, labels, and annotations. These labels are the foundation for search and alert routing later — missing one makes "search logs by service" painful. Business labels like team or tier pass through from Pod labels, so agree on a naming convention up front to avoid rework.

Field conventions: only structured logs are searchable

The collector should parse logs into structured fields rather than dumping everything into one message string. For JSON-format application logs, the collector parses fields automatically; for plain-text logs, use a regex parser to extract timestamp, level, and trace_id. It's better for the application to emit JSON logs directly — zero parsing cost on the collector side is the most reliable path.

Two must-dos: first, standardize the time field — decide whether to use collection time or the log's own timestamp, or cross-container ordering breaks. Second, define trace_id as a dedicated, indexed field to prepare for log-trace correlation later. A cheap way to verify your setup end to end: after onboarding, run a request that logs something, then search for its trace_id and confirm the exact line shows up with the right pod and namespace tags.

Multiline logs and resource limits

Java stack traces and Python tracebacks are multiline logs; collecting them line-by-line shatters them. Configure multiline merging so a start-line regex glues the following lines back together:

multiline:
  pattern: '^\d{4}-\d{2}-\d{2}'   # a line starting with a date begins a new entry
  maxLines: 200

Resource limits are equally important. The collector is a resident process on the node, so cap its CPU and memory or it will steal resources from your workloads on busy nodes. Start with 200m CPU / 512Mi memory, adjust after observing, and set size and retention policies on the local log spool to avoid filling the disk.

Troubleshooting common errors

• Upload failure: check endpoint and token, and read the collector logs for connection errors.
• Logs not searchable: confirm the namespace filter isn't excluding your target, and that the time field parses correctly.
• Multiline entries split apart: verify the multiline regex matches the actual start line of your logs.
• Data volume abnormal: check whether you're collecting system namespaces or noisy containers, and turn them off by label.

Collecting reliably, tagging accurately, and structuring clearly is the foundation of Kubernetes observability. Once the foundation is solid, search, alerting, and tracing on top of it all become meaningful.