Querying Logs Like SQL: Inside Jujing OBSERVE's Log Search Engine

Why grep and regex no longer cut it

Traditional log triage relies on grep, regex, and shell pipes. That works when you have a few hundred megabytes of log files on a single box. Once traffic grows, logs reach terabytes per day, spread across hundreds of nodes, and grepping for a single keyword can take minutes — on top of which you still have to hand-craft sed and awk to filter fields. Worse, grep can only answer "contains / doesn't contain." It cannot group counts by status code, rank requests by latency to find the Top N, or correlate two conditions — and those are exactly the queries that pinpoint a problem. It also gives you no notion of context: a matched line tells you what happened, not what happened right before or after it. The log search engine in Jujing OBSERVE exists to give structured querying back to logs: let operators query logs the way they query a database, instead of wrestling with regex.

SQL-style search: SELECT ... FROM logs

Jujing's log search accepts SQL-like syntax over a virtual table called logs. To find requests from a service that returned HTTP 500 in the last hour, you write:

SELECT * FROM logs
WHERE service = 'order-api' AND status = 500
  AND timestamp > now() - 1h
ORDER BY timestamp DESC LIMIT 100;

Aggregation is just as direct: SELECT status, COUNT(*) AS cnt FROM logs GROUP BY status ORDER BY cnt DESC gives you the error distribution in one statement. You can also compute tail latency with SELECT approx_percentile(duration, 0.95) ..., which is genuinely useful for catching the slow outliers that averages hide. Fields don't need pre-built indexes; the engine picks an index path automatically based on the query shape, so even a first-time query returns in seconds. The trade-off is that very broad full-text scans over weeks of data are deliberately throttled — you're nudged toward narrowing the time window or adding a field filter, which is almost always the right thing to do anyway. For newcomers who don't know SQL, the UI ships a query builder that generates statements from point-and-click field and condition selections, so nobody has to memorize syntax.

Pipeline syntax and field filtering

Alongside SQL, Jujing supports a pipeline syntax that suits chained filtering and reads more like an operator's natural workflow:

service = 'payment' | status >= 400 | stats count() by uri | sort -count

This pipeline filters error requests from the payment service, counts them by URI, and sorts descending. Pipelines can be chained indefinitely, with a live preview at each step so you can narrow the scope incrementally — add a filter, watch the count drop, and keep going until the result set is small enough to eyeball. Field filtering covers exact match, range, prefix, regex, and exists/nonexists checks. Nested fields in JSON logs are addressed with dot notation (e.g. http.headers.user_agent), with automatic type inference so numeric fields can be compared directly and string fields can be full-text searched. Multiline stack traces are parsed and kept together rather than split into fragments, which matters when you're searching for an exception message.

Context navigation and slow-query diagnosis

Finding the target log is only the first step; the next is seeing its context. Jujing lets you page up and down around any single log line and automatically correlates all logs produced by the same request (via traceId / requestId), reconstructing the full call chain. That correlation is what turns a single error line into a story: the request came in, hit the cache, missed, called the downstream, and failed there. When a search is slow, the engine surfaces a slow-query analysis: matched rows, scanned shards, and time distribution, with suggestions to tighten the time range, narrow the service scope, or add filters. The pattern that works in production is: aggregate over a wide time window to find the anomalous period, narrow the window for precise search, then use context and trace correlation to isolate the root cause. That aggregate → refine → correlate path is an order of magnitude faster than grepping line by line, and it's the workflow the engine is optimized for.