Log Field Parsing and Index Configuration: Regex, JSON, and Multiline in Practice

Whether your logs are fast to search and accurate to query depends about half on the parsing rules you set at the collection end. Raw logs are just text; only after they are extracted into structured fields can you filter, aggregate, and alert on them with SQL-like syntax. This guide covers parsing configuration for the three most common log shapes—JSON, regex, and multiline stack traces—plus timestamp and timezone handling and how to choose index fields.

JSON logs: don't let them collapse into plain text

If your application already emits JSON (via Logback's LogstashEncoder, Go's zap JSON encoder, and so on), the first thing the collector should do is parse it as-is rather than flattening it into a single message field. Set format: json and the collector turns every key into a field automatically:

parser:
  format: json
  time_key: timestamp

Now level, service, and trace_id are directly searchable fields. The common mistake is wrapping JSON in another layer of plain text on the application side—writing the JSON object as a string into message. That throws away all the structure for nothing. The fix is cheap: switch the logging framework's output from PatternLayout to a JSON encoder and leave the rest of the code alone. If the JSON has nested objects (like request.headers), they are kept as sub-objects by default, so you search inner fields with a dotted path; add a flatten rule for the fields you need flat, or you won't find the inner content at all.

Regex parsing: adding fields to legacy logs

Legacy systems often emit fixed-format plain text, like this:

2026-08-24 10:11:12.345 [order] ERROR PaymentTimeout userId=10086 orderId=778899

Split it with named capture groups:

^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) \[(?<service>\w+)\] (?<level>\w+) (?<msg>.+)$

Pair the extracted fields with type conversion—time to a timestamp, userId to an integer—so that search and sort actually behave. The regex gotcha is performance: too many capture groups slow down ingestion. Use capture groups only for the fields you need and non-capturing groups for everything else. Before going live, replay a small batch of historical logs to confirm the match rate and the extracted output.

Multiline logs: keep a stack trace as one record

Java exception stacks and Python tracebacks span multiple lines. If you don't merge them, every line becomes a separate log record and the stack gets shredded across rows when you search. Configure a first-line pattern so continuation lines are folded into the same record:

parser:
  multiline:
    firstline: '^\d{4}-\d{2}-\d{2}'

Now an exception and its stack are stored as one unit, and opening it shows the full context instead of a single orphaned line.

Timestamps and timezones

The log timestamp is the basis for every search and sort. Three things to watch: first, point time_key at the real timestamp field, not the collection time—otherwise latency debugging gets distorted. Second, declare the timezone explicitly; multi-timezone services should store in UTC and convert to local time only at display. Third, hard-code the time format; auto-detection slows ingestion under load and guesses wrong more often than you'd like.

Field naming and a short pitfalls list

Standardize field names at the collection end: level should not be called severity in one service and log_level in another, or you'll be guessing field names in every search. Add rename rules in the parser to fold legacy field names into one convention. A short list of the most common pitfalls: nested JSON left unexpanded so inner fields are unfindable; regex without anchors causing false matches; multiline rules that don't cover every log source. Work through these and you'll avoid most "the logs are in but I can't find them" problems.

Choosing index fields

Not every field deserves an index. More indexed fields mean slower writes and bigger storage. Index only the fields you query at high frequency: level, service, trace_id, request_id, userId. Full-text fields like message go to an inverted index, while the rest stay readable but unindexed. With the indexing policy settled and SQL-like search on top, you can query millions of lines and still get answers in seconds.

As a final acceptance test: pick a real production incident and see whether you can locate it using only structured fields, with no raw-text searching. If you can, your parsing and indexing are in good shape. If you keep reaching for the free-text box, the parsing rules need another pass.