A Log Query Language: From grep to Structured Analysis

grep finds logs, not answers

The most common troubleshooting move is to type a keyword into the search box and scroll through dozens of pages. Keyword search answers "does it exist?", but not "how many, how often, which is slowest, what's the trend" — and those are exactly the questions that lead to a root cause. A keyword can confirm a suspicion; it cannot explain an incident. When an on-call engineer is staring at 300 lines that all contain the word "timeout", the keyword has done its job and left the real work untouched.

The gap is structural. Raw log search treats every line as a flat string, so the only operations available are "find lines containing X" and "show me more lines". Analysis needs structure instead: extract a latency value, count errors per endpoint, compare two time windows, spot the one slow outlier. That is the gap a query language fills.

Observe's query language does one simple thing: like a Unix pipeline, it splits a search into filter, extract, aggregate, sort and visualize stages, where each stage's output feeds the next. A single query both retrieves the data and computes the answer, so the person on call can go from "something is wrong" to "here is what is wrong" in one step.

The pipe syntax: five steps to a diagnosis

A query starts from a source and a filter, then narrows down step by step:

source="order-service"
| level != "debug"
| parse json
| stats count() by status, error_type
| sort -count

What this does: take order-service logs, drop debug lines, parse the JSON, group and count by status and error_type, then sort by count descending. In a few seconds you see which status code and which error type dominate, instead of eyeballing hundreds of lines.

The operator set is small — the ones you will actually use:

• parse — pull structured fields out of raw text
• stats — aggregate with count, sum, avg, p99, distinct and more
• where — conditional filtering
• sort / top — ordering and top-N
• timechart — bucket by time and draw the trend

If you know SQL, the mapping is straightforward: source and where play the role of FROM and WHERE, stats is GROUP BY with aggregates, and sort is ORDER BY. The difference is that it all operates on log and trace events rather than rows, and each stage forwards events instead of result sets.

timechart deserves its own mention because it is the fastest way to see when something changed:

source="order-service" level="error"
| timechart count() by 1m

This plots errors per minute. A flat line that suddenly spikes at 14:03 points you toward a deploy, a traffic surge or a config change — the kind of signal that stays invisible when you only page through individual lines.

Field extraction: turning text into data

Full-text search hits its ceiling at fields. The log line says "took 230ms" and "user_id=10023", but grep treats them as plain strings, so you cannot average them, sort by them or alert on them. Once parse turns them into fields, you can aggregate, compare and alert:

source="order-service" level="error"
| parse "took *ms" as latency
| parse "user_id=*" as user_id
| stats avg(latency) by user_id

Extracted fields join the platform's field catalog, so search, alert rules and dashboards can all reference them without re-parsing every time. Field extraction is the difference between a platform that stores text and a platform that understands your logs — and it is what makes the examples in the previous section possible at all.

Turn good queries into reusable assets

A good query should not be thrown away after one use. Observe lets you save a query as a template with parameter placeholders and share it across the team; you can also freeze it into an alert rule or a dashboard panel so the answer keeps updating on its own.

A few suggestions:

1. Freeze high-frequency troubleshooting scenarios (like "recent error distribution for a service") into templates so on-call gets an answer in one click
2. Put field extraction into your logging convention and agree on field names — avoid user_id and userId coexisting
3. Add a time range to aggregation queries so a full scan does not drag down the cluster

At its core, a query language turns debugging experience into reusable assets instead of starting from zero every time. The first time you write a good query it is a one-off; the moment you save it, it becomes part of how the team works.