A real production incident shows how a securities trading platform used traces, metrics, and logs together to pinpoint a latency spike at market open to connection-pool exhaustion, plus instrumentation and alerting advice.
On a Monday at 9:30 sharp, a brokerage's core order-routing channel saw its order-placement P99 latency jump from a normal 60ms to 900ms, firing the latency alert, and shortly after some orders began timing out. Trading systems are millisecond-sensitive — every second of jitter risks client complaints and even compliance exposure. This case matters because it shows how a team without a dedicated observability platform gets knocked over repeatedly by an old "connection-pool exhaustion" bug, and how the right tools shrink the debugging path to a dozen minutes.
Trading paths differ from ordinary web services in three ways. First, load is extremely bursty — the minutes around market open and close carry dozens of times the normal throughput. Second, the chain is long: an order passes through the access gateway, risk control, fund validation, order routing, and the exchange receipt, one hop after another. Third, the debugging window is tiny — an issue that isn't found before the close might as well not be found. So instrumentation must cover every hop, and alert thresholds must be designed at minute granularity, not hour granularity.
Don't just watch CPU and memory. For trading, the metrics that matter are business ones: per-channel order throughput (orders per second), order latency including network round-trip, cancel-success rate, fill-report latency, and the latency distribution of each step in the order state machine. Slice them by channel, order type, and client type so that when something breaks you can see at a glance which channel and which order class is slow. Infrastructure metrics — CPU, GC, connection pools, disk IO — serve as supporting evidence to explain why a business metric went wrong.
The on-call engineer pulled three charts in OBSERVE: P99 latency for the order API, per-host CPU usage, and database connection count. The first two spiked together at 9:30, but CPU only reached 60% — nowhere near saturated. The suspicious one was the DB connection count, which had already hit the pool's ceiling before 9:30 and stayed there. That pointed to resource waiting, not computation — the most easily overlooked and most lethal class of problem in trading systems.
Taking the trace_id of a slow request and expanding its call chain revealed the pattern: the order service's call to the risk-control service was fast (8ms), but the "save order" step spent 720ms just acquiring a database connection. Combined with the metrics, this locked the hypothesis — the pool was exhausted, and threads were waiting for connections rather than executing slow queries. The value of tracing here is decomposing "the whole request is slow" into "this one step is slow," which cleared risk control and order routing of suspicion.
Filtering logs by trace_id surfaced a flood of Timeout waiting for idle connection errors, all clustered in the five seconds before the open. The cause was now clear: the pre-market quote warm-up script and a batch order-cancellation job started simultaneously and drained the connection pool, leaving real orders to queue for connections the moment the market opened. That's the loop the three pillars close together: metrics tell you which dimension is wrong, traces tell you which hop is slow, and logs tell you why.
There's no shortcut to observability for trading systems: define your business metrics clearly first, make traces and logs drillable, and design alerts by channel and tier. Once the tooling is in place, the rest is making instrumentation a habit.