Slow, incomplete log search is usually an unstructured-data problem, not a slow engine. How field extraction, full-text search and query-based alerts turn log digging into a one-question answer.
Logs are the most basic debugging tool, yet also the easiest to lose control of: formats differ everywhere, field names are inconsistent, and a full-text scan can take ten seconds and still miss results. The root cause is usually not a slow engine but unstructured data — fields were never extracted, so you fall back to fuzzy matching, which is both slow and lossy.
Consider a common scenario: production starts failing and you want "all requests that returned 5xx from the payment service in the last hour". If status and service were never extracted as fields, you are stuck full-text searching 500 and filtering by hand, digging through hundreds of lines. Once field extraction is in place, that question is a two-line query.
Observe extracts fields at ingestion time rather than parsing them on every query. Three common approaches:
{"user":{"id":123}} becomes queryable as user.id.(?P<ip>\S+) .* \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<uri>\S+) .*" (?P<status>\d+) .* (?P<upstream_time>[\d.]+)$The resulting status, uri and upstream_time can then be filtered and aggregated directly.
responseTime, cost and latency to a single duration so cross-service queries do not have to remember three names.A word of caution on high-cardinality fields. Extracting everything is tempting, but fields like user.id or request_id have millions of distinct values and will bloat the index and slow queries down. Mark them as non-indexed attributes: you can still filter on them explicitly, but they never become a full inverted index. Reserve the inverted index for the low-cardinality fields you group by constantly — status, level, service, host.
Attach extraction rules as templates to the log source so one rule covers a whole batch of logs, instead of writing ad-hoc parsing per query. The earlier you extract fields, the easier every downstream search, alert and aggregation becomes.
Structured fields alone are not enough; logs always contain something that does not belong to any field. Full-text search covers that:
level:ERROR AND service:order AND "connection reset"Filter on structured conditions first, then match full text within the filtered set — an order of magnitude faster than pure full-text. The syntax supports AND/OR/NOT, ranges and wildcards, and you can jump from a trace_id straight into the corresponding trace.
A couple of syntax tips worth remembering. Quote a phrase to match it exactly — "connection reset" — rather than two separate terms. Be careful with a wildcard at the start of a term, which forces a full index scan and can be slow; prefer a suffix wildcard like timeout*. If a query feels sluggish, check whether a leading wildcard or a bare full-text term is doing a scan you could constrain with a structured filter.
Do not rewrite repeated queries every time. Save them as shared views so a new teammate can open "payment service errors" with one click. Better yet, attach a query to an alert: fire when the number of matching logs in a window exceeds a threshold. That turns an anomaly pattern in the logs — a sudden burst of a specific error — into an alert signal instead of an after-the-fact dig.
Saved views also give you a stable handle to link to. Paste a link to the "payment errors, last hour" view into your runbook and on-call handoff instead of describing the query in prose. When a view has an alert attached, the notification can deep-link straight into the filtered log list, so whoever is woken at 3 a.m. lands on the evidence rather than the home page.
The goal of log search is to get the answer on the first question — not to spend half an hour finding the one log line.