A walkthrough of the SQL-like query syntax, inverted index, and hot/cold storage tiering in the Jujing OBSERVE log search engine, with ready-to-use query templates and tuning tips to turn raw logs into answerable questions.
A logging platform earns its keep not by how much it stores, but by how fast you can find an answer when something breaks. The old way — grepping files across a handful of machines — collapses as soon as volume grows. And plenty of teams that did centralize their logs still search by a single keyword and end up paging through hundreds of thousands of hits, or they filter only on the message text and miss everything that was packed into JSON. The Jujing OBSERVE search engine targets exactly that: treat logs as a table you can query at any time.
Searching depends on structure. At ingest time, the collector (Filebeat, Fluent Bit, or the built-in agent) parses each log line and extracts time, level, host, service, trace_id, and business fields. For unstructured logs, OBSERVE ships Grok and regex parsing rules that you can apply at ingest or retroactively when rebuilding the index.
Once logs are field-based, "searching" becomes "filtering fields." To find ERRORs from a specific host in the last five minutes:
level=ERROR AND host="pay-svc-03" AND @timestamp > now()-5m
This is far more precise than grepping the word "ERROR" — it won't match a user nickname that happens to contain it, and it won't miss logs packed into JSON objects that a text search would skip entirely.
Under the hood, extracted fields go through the same tokenizer as the message body, so a value like pay-svc-03 is indexed as a single term and matched exactly rather than split on hyphens. Field types matter too: duration and status_code are numeric, which is what makes p99(duration) and range filters like status_code >= 500 possible.
The query language is deliberately SQL-shaped to keep the learning curve flat. It supports AND/OR/NOT, parentheses, comparison operators, IN/LIKE fuzzy matching, and aggregations.
Count errors per service:
SELECT service, count(*) AS err_count
FROM logs
WHERE level="ERROR" AND @timestamp > now()-1h
GROUP BY service
ORDER BY err_count DESC
Find the slowest endpoints (assuming an extracted duration field):
SELECT api, p99(duration) AS p99_ms
FROM logs
WHERE kind="access" AND @timestamp > now()-1h
GROUP BY api
ORDER BY p99_ms DESC
You also get histogram aggregation for numeric fields, date_histogram for time-series counts, and correlation by trace_id — from a single error line you can drill straight into the full call chain.
One deliberate difference from a database: there are no JOINs and no subqueries. Logs are append-only and denormalized by design; instead of joining tables you filter on correlated fields (trace_id, order_id, request_id) that are already embedded in every relevant line. If you find yourself wanting a JOIN, the usual answer is to add one more field at ingest time, not to restructure the query.
Under the hood is an inverted index over tokenized terms. A query narrows on the index first and then filters by time range, so "the last hour" never scans the whole dataset. The time column uses columnar storage, so aggregations and sorts return in milliseconds to seconds.
Storage is tiered by temperature: the last seven days sit on a hot SSD tier with the lowest latency; days seven through ninety automatically move to a cold HDD tier that is still queryable but slower; anything older than ninety days is archived to object storage by policy and restored on demand. Cost stays bounded while query performance doesn't degrade linearly with data volume.
For genuinely high-cardinality searches — say, a specific trace_id among billions of lines — the engine keeps a separate per-field index for the fields you mark as "indexed", while high-cardinality-but-rarely-queried fields can be stored without indexing to save space.
Triage slow endpoints, then drill down:
SELECT api, p99(duration), count(*) FROM logs
WHERE kind="access" AND @timestamp > now()-30m
GROUP BY api ORDER BY p99(duration) DESC LIMIT 20
Find hosts with an error spike:
SELECT host, count() FROM logs
WHERE level IN ("ERROR","FATAL") AND @timestamp > now()-15m
GROUP BY host ORDER BY count() DESC
Cluster errors by keyword to see if the problem is concentrated:
SELECT error_keyword, count() FROM logs
WHERE level="ERROR" AND @timestamp > now()-1h
GROUP BY error_keyword ORDER BY count() DESC
Save your high-frequency queries and reuse them in alert rules, and agree on a field-naming convention across the team — that discipline is what turns a search box into a diagnostic tool. Searching is not the goal; finding the root cause quickly is.