Structured Logging in Practice: Turn Messy Logs into Queryable Fields

Why structure logs in the first place

A raw log is just a string. Full-text search can find "payment failed", but it can't answer "which user, which order, what failure code, how many milliseconds" — because that information is buried inside the string where the search engine can't tell it apart. Structuring means splitting a log line into named fields (user_id=123, status=fail, cost_ms=450), which is what makes filtering, aggregation, alerting, and correlation possible.

OBSERVE's collection Agent runs an ingest pipeline that turns raw logs into structured fields before they land in storage. Configuring that pipeline is the point of this article.

Three parsing approaches

Automatic JSON parsing. If your application emits JSON logs (strongly recommended), you barely need rules. Add a json parser and the Agent flattens JSON fields into queryable fields:

parsers:
  - type: json
    field: message

The requirement is that the whole line is valid JSON. Nested objects flatten to a.b.c form and arrays expand to multiple rows. The limitation: it can't handle JSON mixed with non-JSON text — that needs regex.

Regex extraction (grok-style). For unstructured logs, use named capture groups to pull fields out. Take an Nginx access log:

log_format: '$remote_addr - $user [$time_local] "$request" $status $body_bytes_sent'

The matching rule:

parsers:
  - type: regex
    pattern: '^(?P<remote_addr>\S+) - (?P<user>\S+) \[(?P<time_local>[^\]]+)\] "(?P<request>[^"]*)" (?P<status>\d+) (?P<body_bytes_sent>\d+)$'

(?P<name>...) names the field; the matched text becomes its value. The trick to writing regex: start from 10-20 real samples, not from the log-format documentation. Common named patterns (IP, TIMESTAMP_ISO8601, UUID) can be saved to a pattern library and reused instead of hand-written every time. Watch regex performance too: avoid leading .* and nested quantifiers that trigger catastrophic backtracking — a single log line taking more than a few milliseconds will drag down the entire ingest pipeline.

Multiline merging. Java/Go exception stacks and Python tracebacks span multiple lines. Without merging, one exception becomes dozens of log lines and field extraction falls apart. Merge by the first-line pattern:

parsers:
  - type: multiline
    pattern: '^(\d{4}-\d{2}-\d{2}|\S+ \d+)'

A line starting with a date or timestamp begins a new record; everything else attaches to the previous one.

Types and timestamps

Extracted fields are strings by default, but aggregation needs correct types. Convert status to integer, cost_ms to float, and booleans to booleans so an alert like status >= 500 evaluates correctly. Settle on field naming too: lowercase with underscores (cost_ms, not CostMs), matching your code's naming so searches don't fight over case.

Timestamps matter even more: the timestamp inside the log can differ wildly from ingestion time (timezone drift, log buffering). Specify the time field and format explicitly to avoid scrambled log ordering:

time:
  field: time_local
  format: "02/Jan/2006:15:04:05 -0700"

Redaction and cost control

While you're structuring, do two more things. First, redact: mask phone numbers, ID numbers, bank cards, and tokens before they hit storage, so sensitive data never lands anywhere. Second, trim fields: drop useless long fields from debug logs to shrink index size and query cost.

Concretely: an order log containing both a phone number and a token becomes 138****1234 for the phone and a fully masked token. Redact at the collector, not at query time — once data lands in plaintext, taking it back is hard. Likewise, a debug log carrying a 4KB request body is rarely worth the index space; drop that field outright. Configure both before go-live — far cheaper than retrofitting.

Debugging a parsing rule

Parsing rules have a high error rate; don't expect to get them right on the first try. OBSERVE's parser has a test mode: paste a real log line and watch the fields after each step, then ship it. After go-live, spot-check: group by source and watch the parse-failure rate, then pull out the high-failure logs to see which format you missed. Parsing rules are alive — they change when the log format changes, so review parsing-rule changes in the same process as code changes. If a particular log format keeps defeating your parser, the better fix is usually to change the application to emit structured JSON — the best source of structured logs is always the application itself. Keep a few representative samples for every rule and re-run them in test mode whenever the format changes.