Log Pipelines: Field Extraction and Enrichment Rules in Practice

Why a log needs to go through a pipeline first

Raw logs are usually a single long string, with the severity level, latency, user ID, and business status code all jammed into one line. If your only tool is full-text fuzzy matching, you pay for it three ways at once: the index bloats because every token gets indexed, searches slow down as the index grows, and aggregation is close to impossible—you cannot compute error rates, distributions, or averages on a field that does not actually exist because it is still welded into a sentence.

Jujing's log pipeline parses each record before it is written to storage, splitting the text into structured fields. Everything downstream—filtering, statistics, alerting, drill-down, even retention rules—then runs on those fields instead of regex-scanning full text. This is the difference between "searching logs" and "querying data," and it is the reason a pipeline is the first thing every mature logging setup gets right.

Built-in parsers and custom Grok

Jujing ships with parsers for the common formats—Nginx, Apache, JSON, Syslog—so the most frequent cases work with a single click and no config. JSON logs are flattened automatically: {"user":{"id":1001,"region":"cn"}} becomes the two directly filterable fields user.id and user.region, with no hand-written path expressions.

For private or legacy formats, you write extraction rules as Grok expressions. Take a typical Java application log:

%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} \[%{DATA:thread}\] %{DATA:logger} - %{GREEDYDATA:message}

With this rule applied, a raw line like:

2026-10-13 09:00:01 ERROR [http-nio-8080] c.j.order.OrderSvc - 库存不足, sku=1001

becomes five clean fields: ts, level, thread, logger, and message. You can then pin down records with a precise filter such as level:ERROR AND logger:OrderSvc, and aggregate error counts by level directly—no regex in your query at all.

Field enrichment: dictionary mapping and GeoIP

Extraction is only step one. Enrichment translates raw field values into readable, queryable information, and it removes join work from the query path entirely:

• Internal status codes like 0/1/2 map to "success/failure/timeout" through a dictionary, so dashboards show plain words instead of magic numbers.
• Source IPs go through a GeoIP database to add province and ISP, which lets you plot error distribution by region.
• userId joins a department table to attach business tags, so an alert already carries "which team, which service" without anyone digging through a directory.

Because enrichment happens once at write time, queries avoid joins, latency stays low, and dashboards render faster. You pay the cost once on ingest rather than on every search.

Advice and three pitfalls

Three pitfalls show up most often in production, and all three are cheap to avoid if you know to look:

1. Validate extraction rules on a small sample before enabling them globally. A single broken Grok pattern drives the parse failure rate up, and failed records often land in a dead-letter queue that is harder to read than the original mess.
2. Keep field names lowercase and agree on a naming convention. When both level and Level exist, aggregations fork, and alert conditions silently miss one branch.
3. High-cardinality fields—requestId, traceId, userId—should use exact-match indexing only, never tokenized. Otherwise index size and memory use spiral, and every query pays for it.

Keep pipeline config under version control: before changing a rule, replay a day of historical logs through a shadow channel and compare the parse success rate and field distribution. Only switch to production once the numbers check out. When something goes wrong, rolling back one rule config is enough—you never have to re-ingest history. That turns editing rules from a gamble into a routine change.

Two more things worth doing up front. Build Grok patterns in the built-in pattern tester before saving them: paste five or six real sample lines, iterate until each parses into the fields you expect, and watch the failure count. Patterns that handle the happy path but choke on a null field or an extra space are the ones that break a week later, so always include one malformed line in your sample set. And treat a parse failure rate above a fraction of a percent as a bug in the rule, not a cost of doing business—every failed parse is a log you cannot query later. A small investment in the pipeline now pays back on every search, dashboard, and alert you build on top of it.