How logs, traces, and metrics divide labor in a low-latency, strongly-consistent trading system, plus market-open monitoring and alerting, and one real incident.
Core trading systems have clear characteristics: latency-sensitive (millisecond scale), spiky concurrency (market open and close), long call chains (counter → order routing → matching → clearing), and strict consistency requirements. The traditional "dig through logs when something breaks" approach fails here—logging itself can slow the trading hot path, and nobody has time to read logs at market open.
Our approach splits observability into online and offline tracks. Online signals must be cheap and queryable in real time, for live alerting and fast triage; offline signals serve post-mortem analysis and compliance retention.
Raw dashboards don't change behavior; SLOs do. For the order path we track two: order acceptance success rate (target 99.99%) and end-to-end matching latency (P99 under 50ms at open). Each SLO maps to a burn-rate alert in OBSERVE—an error budget that depletes over a rolling window—so pages fire when the budget burns too fast, not when a single metric wiggles.
Every SLO page links directly to the traces and logs that consumed the budget, which keeps mean time to acknowledge short. On-call engineers stop guessing which service is at fault and start reading the trace.
Fifteen minutes before open, raise the sampling rate to 30% and drop it back to 5% after close, switched automatically by OBSERVE's dynamic sampling rules. Alert thresholds use relative baselines rather than fixed values: for CPU and queue depth, compare against the same time last week and only page when the value exceeds 2× the baseline, avoiding false alarms from the natural open spike.
Alert tiers: P0 (order path down, latency over threshold) pages by phone and SMS; P1 (degradation, queue backlog) sends SMS and channel notifications; P2 (resource levels) is logged only.
Five minutes into a trading day's open, the order-routing gateway's latency jumped from 3ms to 40ms. Once the alert fired, the on-call engineer sorted the trace view by latency and saw that 90% of slow requests were stuck in the risk-control service's "list check" step. Clicking into those spans showed the check cache's miss rate had gone from 1% to 90%—an upstream data-sync job had failed before open, so the cache never warmed up. Root cause was found in about 40 seconds; restoring the previous day's snapshot fixed it.
The workflow that mattered most was a fixed drill: a metric page points at a time window and a service; the trace view for that window, sorted by latency, names the slow span; the span's logs explain the state that caused the slowness. Keeping the three signals wired together—the same traceId in logs and spans, the same labels on metrics—is what makes the drill take seconds instead of an incident call. Whenever a new service joins the trading path, the first review checks whether its logs carry traceId and whether its metrics carry the same service and environment labels.