How the OBSERVE log ingestion pipeline sustains high throughput without dropping data at peak: the four-stage architecture, bounded in-memory queues plus disk buffering for backpressure, the tuning parameters that matter most, and a replay-based load test for capacity planning.
Log ingestion has two failures you can't afford: dropping logs at peak, and a collector that drags down the very app it's watching. OBSERVE's ingestion layer splits the pipeline into independent stages — tail, parse, enrich, deliver — joined by bounded queues and an on-disk buffer. This article explains the backpressure design, how to tune it, and how to plan capacity around it so you stop guessing and start measuring. If you run anything with spiky traffic, most of what follows is directly applicable to your setup.
A log line passes through four stages before it becomes searchable: tail read, regex/Grok parse, field enrichment, and batched delivery. The bottleneck is rarely the tail; it is almost always the parse. A multi-line Java stack trace or a large JSON object can cost several times what a single-line plain-text entry costs, and a slow parse stage backs up everything behind it, which then shows up as higher latency everywhere. Don't guess where the pressure is — read the collector's own stats: tail_files_open, parse_errors, buffer_queued, and dropped_events. These counters are the difference between "something is slow" and "the parser is the problem." If dropped_events keeps climbing, the downstream can't keep up and you should tune backpressure rather than blindly adding machines. One telltale signature: buffer_queued sits high and won't drain while delivery latency rises — that points at a slowing search cluster, not at the collector itself, so adding collectors won't fix it.
The collector uses two levels of backpressure: a bounded in-memory queue plus an on-disk buffer. The memory queue defaults to 4096 events; once it fills, new events spill into disk buffer files. Those files are split into 512MB chunks and retained for up to 48 hours — only past that ceiling are events dropped and counted. So when the backend stutters (say the search cluster is rebalancing shards), data lands on disk first and replays automatically once things recover, instead of being thrown away. This is the single most important property of the design: short outages become invisible to the data.
Two caveats matter. First, put the disk buffer on a dedicated data disk with enough IOPS, not the OS disk, or buffering will fight system I/O and make the problem worse. Second, alert the moment dropped_events becomes non-zero — don't wait for a user to file a ticket before you learn data was lost. Also remember the disk buffer isn't free: it trades disk space for reliability, and if the backend stays behind for too long the buffer fills up too. It solves short-lived stalls, not chronic under-capacity, and treating it as infinite storage will eventually bite you.
A collector config that has served well at a few hundred thousand events per second:
pipeline:
tail:
max_open_files: 512
read_buffer_bytes: 262144
parse:
workers: 8
multiline_max_lines: 500
buffer:
type: memory+disk
memory_queue_size: 8192
disk_chunk_bytes: 536870912
disk_retention_hours: 48
sink:
batch_size: 1000
flush_interval_ms: 1000
retry_backoff_ms: 500Rules of thumb: set workers to the number of CPU cores, then watch CPU saturation before adding more. Keep multiline_max_lines modest so one malformed entry can't hog a parse thread and stall the whole line. batch_size and flush_interval_ms control delivery granularity — bigger batches mean higher throughput but higher latency, so keep flush under one second when real-time alerting depends on this data. Use exponential backoff for retry_backoff_ms so a recovering backend isn't immediately buried by retries. The defaults are reasonable starting points, not targets; tune from your own stats, not from a blog post.
Before going live, run a replay test: take a million production lines and inject them at twice the peak rate, then watch whether buffer_queued drains back down and dropped_events stays at zero for the entire run. Size capacity as peak events-per-second times average bytes per line, then add 30% headroom for the next growth spurt. The conclusion is simple: decide which scenarios can never afford data loss, then work backward to size the queues and disk buffer. It's far cheaper than adding machines after the incident. When a run misses its target, the stats above will usually name the culprit directly — a saturated parse stage, a slow sink, or a buffer that keeps draining — so you adjust one parameter instead of changing everything blind. Keep the replay script around: it is the fastest way to prove that a tuning change actually helped, rather than just feeling faster.