A log platform is only as good as each stage of its pipeline. We walk through ingestion, parsing, indexing and search in Observe, with practical advice on field design, hot/cold tiering and cost control.
The first hurdle of any log platform is ingestion. Applications are scattered across hosts, their formats differ, and if ingestion is inconsistent, no amount of search power downstream can save you. Observe supports three ways in: a host agent, collection plugins compatible with Filebeat and Fluentd, and native OTLP ingestion. On Kubernetes, run the collector as a DaemonSet so it auto-discovers pods and tails stdout logs, and keeps collecting across container restarts without dropping a line.
One piece of advice on choosing: if you're heavily containerized, prefer OTLP direct ingestion and skip the middle layer entirely. For traditional VMs and bare metal, run an agent close to the workloads and let it handle buffering, compression and resumable delivery, so a network blip doesn't cost you logs. Keep the collection side to two jobs — collect as much as you can, and retry on failure. Don't push parsing work onto the application; that's what the platform is for.
Raw logs are just text; they only become searchable once they're split into fields. The platform supports three parsers: automatic JSON, regex/Grok extraction, and key=value parsing. The rule of thumb is to preserve the raw message at ingest time and define fields at parse time — always keep the original message field so you can go back and re-examine anything.
Field design matters more than people expect. Fields you query frequently — trace_id, user_id, service, level — should be indexed; fields you only occasionally inspect, like a full stack trace, can be stored without an index. Standardize field names across teams: if one service logs level, another severity, and a third log_level, every search has to write three conditions, and that friction shows up as engineers quietly giving up on searching altogether.
Search speed comes down to how indexes are built. The platform uses an inverted index for string fields and a columnar index for numeric and timestamp fields, so full-text search and range aggregation each use the structure built for them. Queries use a SQL-style syntax, so operators don't have to learn a new language. A typical example:
SELECT level, count(*)
FROM logs
WHERE service = 'order-service'
AND ts >= now() - 1h
AND message LIKE '%timeout%'
GROUP BY levelRegular expressions, fuzzy matching and histogram() time bucketing are all supported. When a query is slow, check the obvious thing first: a leading wildcard (%xxx) or a function call on an indexed field defeats the index and forces a full scan. Rewrite the condition and the same query usually drops back to milliseconds.
Once log volume grows, cost becomes the biggest concern. Observe tiers data by temperature: the last seven days live on SSD for sub-second queries, and older data is compressed and archived to cheaper storage — slower to query but still queryable. Retention rules are configurable, so you can keep error logs longer and retire DEBUG logs sooner.
There are three levers for cost: collect only what you need (ERROR/WARN or specific business logs), sample ultra-high-volume access logs, and set retention windows so data is cleaned up automatically. The value of logs is in traceability, not infinite retention. Spend the budget on data you'll actually query during an incident, not on terabytes nobody will ever open.