An inside look at the three designs behind OBSERVE's fast log search—hot/cold tiered storage, inverted indexing, and field extraction—explaining how they cut query latency on large log volumes from minutes to seconds, plus concrete tuning advice.
Log data is huge, but the vast majority of queries target the last few hours or days. OBSERVE splits data into three tiers by default: a hot tier on SSD holding the last 24 hours with indexing done at write time, a warm tier on regular disks keeping 30 days, and a cold tier on object storage retaining 180+ days. Queries route automatically—hot-tier data returns in milliseconds, cold-tier data is thawed on demand.
Tiering isn't just "splitting directories by time." The boundaries adjust dynamically based on volume and tenant configuration. If one business peak produces 500 GB of logs in a single hour, the hot tier expands to 48 hours so that frequent peak-time queries don't fall back to cold storage and turn into slow queries. Conversely, cold data that nobody touches for a long time is demoted to an even cheaper archive tier, reducing object-storage cost.
Tiering also has a hidden payoff: compliance retention. Many industries mandate keeping logs for 180 days or longer. Storing all of that on SSD would be cost-prohibitive. Pushing more than 90% of history to object storage means cold queries are a few seconds slower, which is perfectly fine for forensics and audit use cases.
Full-text search relies on an inverted index. At write time the log body is tokenized and mapped as "term → document position," so queries match terms directly instead of scanning every line. The default tokenizer handles mixed Chinese and English, and supports custom delimiters and regex rules.
There's a common pitfall here: fields like traceId, requestId, and userId should never be tokenized—configure them as keyword. Otherwise requestId:abc-123-xyz gets split apart and queries can miss it. Another pitfall is treating a whole JSON log line as plain text; searching status:500 then returns nothing because status was tokenized. The fix is to flatten the JSON at the collector so status becomes an independent field.
More indexes aren't automatically better. Inverted indexes consume extra disk and memory, and writes have to update the index structures. A 100-field log with full-text indexing on every field will see noticeably lower write throughput. That's why OBSERVE indexes only the log body by default and lets you enable other fields on demand.
Full-text is fast, but structured queries are faster. OBSERVE can extract fields in the ingest pipeline with grok, regex, or JSON parsing—for example pulling status, upstream_response_time, and request_uri out of Nginx logs. Extracted fields can get a columnar index, so upstream_response_time > 3 scans columns directly instead of walking every document—an order-of-magnitude difference in latency.
Here's a concrete grok example. Take a single Nginx access line:
192.168.1.10 - - [06/Oct/2026:09:15:32 +0800] "GET /api/order?id=123 HTTP/1.1" 200 512 0.042The matching grok pattern:
%{IP:client_ip} - - \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATH:path}%{DATA:query} HTTP/%{NUMBER:version}" %{INT:status} %{INT:bytes} %{NUMBER:upstream_response_time}After extraction, status and upstream_response_time become independent, sortable, aggregatable fields. Finding slow requests is then a single upstream_response_time > 1 query instead of a regex scan across everything.
Don't index every field, though. The more indexes you have, the slower writes get, and queries can actually slow down too. For most teams, 10–20 high-frequency fields are enough; let full-text cover the rest. Review field usage monthly and retire fields nobody queries.
A quick way to verify the payoff: take a slow query that currently scans the cold tier, add field extraction for its filter column, and compare latency before and after. In practice, teams that extract 10–20 hot fields and leave the rest to full-text typically see query latency drop by 5–10x on their most-used dashboards, while write throughput stays within a few percent of baseline. The main discipline is resisting the urge to index everything "just in case"—every index you add is a write you pay for on every log line, forever.