← Back to blog
Product 4 min read 炬鲸团队

Query Logs with SQL: Inside OBSERVE's Search Engine

OBSERVE lets you search logs with plain SQL — SELECT, WHERE, GROUP BY — instead of a proprietary DSL. It joins logs, metrics, and traces in one query engine, cutting onboarding cost and shrinking triage time.

Why SQL for logs

Every log search tool ships its own query language. Lucene syntax, bespoke DSLs, custom expressions — new engineers burn half a day reading docs before their first useful query, and a single misplaced bracket sends them back to the start. OBSERVE takes a different route: it models logs as a wide table and lets you query them with plain SQL. SELECT, WHERE, GROUP BY, ORDER BY, and LIMIT all behave the way you expect. Anyone who can write SQL can search logs, which drives onboarding cost toward zero.

The payoff goes beyond onboarding. SQL forces the log store to behave like a real query engine — with an execution plan, indexes, and predictable cost — rather than a keyword box that occasionally surprises you with a timeout. That predictability is what on-call engineers actually need at 3 a.m., when the last thing they want is to learn a vendor's one-off syntax under pressure.

Basic queries

A typical investigation begins with a single SELECT. The example below pulls the last hour of ERROR logs from the order service:

SELECT * FROM logs
WHERE level = 'ERROR'
  AND service = 'order-service'
  AND timestamp >= now() - interval '1 hour'
ORDER BY timestamp DESC
LIMIT 100;

Fields span timestamp, level, service name, trace_id, span_id, hostname, container name, the raw message, and any JSON structured field inside the line. Operators include =, !=, LIKE, IN, BETWEEN, and REGEXP for regular expressions. To surface every slow-query log that mentions a timeout:

SELECT timestamp, host, message
FROM logs
WHERE message REGEXP '(timeout|timed out)'
  AND service = 'payment-service'
ORDER BY timestamp DESC;

Structured-field support is the quiet killer feature. Logs shipped as JSON — which is how most modern applications emit them — are queryable field by field, so you can filter on status_code = 500 or user_id = 42 without resorting to a regex. This matters because a structured-field filter can use an index, while a leading-wildcard regex forces a scan of everything in the window.

Aggregation and statistics

During an incident the real questions are "how many times did this happen, which nodes are affected, and when did it start". Aggregations answer these in one statement:

SELECT host, count(*) AS cnt
FROM logs
WHERE level = 'ERROR'
  AND timestamp >= now() - interval '5 minute'
GROUP BY host
ORDER BY cnt DESC;

Bucket by time to draw an error trend without leaving the tool:

SELECT time_bucket('1 minute', timestamp) AS t, count(*) AS cnt
FROM logs
WHERE level = 'ERROR'
GROUP BY t
ORDER BY t;

No more exporting to a spreadsheet to make a chart — the frontend renders the aggregation result directly, and the same query backs a dashboard panel when you save it as a view.

Cross-signal joins

Logs, metrics, and traces share one query engine, and this is where SQL pays off most. Given a single trace_id, one query pulls every log line and every span on that trace:

SELECT 'log' AS type, timestamp, message
FROM logs WHERE trace_id = 'a1b2c3'
UNION ALL
SELECT 'span' AS type, start_time, name
FROM spans WHERE trace_id = 'a1b2c3'
ORDER BY timestamp;

Before this, an engineer bounced between the logs page and the traces page, mentally reconstructing the timeline. Now the full context arrives in one result set, cutting fault-path triage from fifteen minutes to a couple.

When SQL beats full-text

Full-text search still has its place for exploratory grepping of unknown terms. But once you know what you are looking for — a specific error code, a host, a time range — SQL is faster to write, easier to read six months later, and cheaper to execute. It is also the only way to answer quantitative questions like "how many" or "which node", which a keyword search fundamentally cannot do.

Practical tips

  • Save frequently used queries as shared views so the team stops rewriting the same SQL every rotation.
  • Filter by trace_id or request_id first, then widen the time window — far more effective than paging through raw output.
  • Cap results with LIMIT; hand large aggregates to GROUP BY instead of rendering a hundred thousand rows in the browser.
  • Use REGEXP sparingly. On high-QPS services a leading-wildcard regex can scan far more data than a structured-field filter.

Under the hood, the engine maps SQL predicates onto a columnar store with time-partitioned indexes, which is why the same query stays fast whether it touches a thousand rows or a billion.