Collect Multiline Java Stack Traces with Filebeat into Jujing OBSERVE

Why multiline logs get lost

A Java exception usually looks like this: the first line reads Exception in thread ..., followed by dozens of stack trace lines. Filebeat reads line by line by default, so each line becomes its own event. Without multiline merging, every stack line ships as a separate log, alerts only match the first line, and you end up reassembling context with grep—unusable in production, where a stack trace is often the only clue you have during an outage.

Filebeat multiline configuration

The idea is to merge any line that does not start with a date into the previous event. With the filestream input:

filebeat.inputs:
- type: filestream
  paths:
    - /var/log/app/*.log
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

This means: when a line does not match the 2026-10-13 date prefix (negate: true), append it to the previous event (match: after). One full exception becomes a single event, cut off only when the next line starts with a date again.

The legacy log input uses the same syntax, just with multiline at the same level as paths. If your stack traces contain lines that also start with a date—for example a timestamp printed inside the trace—tighten the pattern to your framework's actual prefix, such as '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}'. A pattern that is too loose swallows two exceptions into one; too tight and the stack splits again. Test against a real sample file before shipping.

Two knobs are worth setting before you ship. multiline.max_lines caps how many lines one event can absorb (default 500), which protects you from a runaway thread dump that never terminates; and close_inactive tells the harvester how long to wait before finalizing a file, which matters when an exception is still being written as rotation happens. Both default to sensible values, but on a busy JVM that dumps ten-thousand-line stacks, raising max_lines is the difference between a complete trace and a silently truncated one.

Two ways to ship into Jujing

Option one: Filebeat posts directly over HTTP to Jujing's ingest endpoint. Simple to set up, good for moderate volume:

output.http:
  hosts: ["https://ob.jjhub.cn/api/ingest/logs"]
  headers:
    Authorization: "Bearer <token>"
  compression_level: 6

Option two: go through Kafka first, which is more robust for high volume or peak shaving, and lets you replay a topic if the downstream consumer hiccups:

output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092"]
  topic: app-logs
  partition.round_robin:
    reachable_only: true

Configure a consumer for the matching topic on the Jujing side. Both paths preserve the full multiline stack trace, so the choice is about volume and delivery guarantees, not data fidelity.

Verify and troubleshoot

Before you call it done, do three things:

1. Trigger a real exception and confirm the complete stack is searchable in Jujing, not just the first line.
2. Check multiline.timeout (default 5s). If an exception stops writing halfway, the timeout force-cuts the event—raise it for long-running thread dumps.
3. Monitor the shipper: Filebeat's filebeat.harvester metrics reveal whether files are being re-read, which points to duplicate ingestion and doubled storage.

One more note on timezones: if your timestamps carry no zone, the shipper and Jujing can drift eight hours apart, making stacks look like they "time-traveled." Normalize the zone on the shipper with a processors step. Remember the order: first check the event count, then the parsed fields, then whether the stack is complete. Pass all three and your multiline logs are solid—and the next incident will have the full picture waiting for you.

If the stack still looks broken after all three checks, the usual culprits are a multiline pattern that matches the wrong anchor, or a log framework that prints a thread ID instead of a timestamp on the first line. Point the pattern at the actual first-line prefix rather than assuming a date, and re-check against a freshly rotated file—rotation mid-exception is another classic cause of a stack cut in half. It is also worth adding host and service tags at the shipper via processors and fields, so that when the same exception appears across a cluster you can group it by host and see whether it is one node or many. That grouping is what turns a wall of stack traces into a single diagnosable incident, and it is the difference between staring at noise and finding the one failing node in minutes.