Trading chains are long, multi-hop, and latency-sensitive, so incidents often get debugged by pulling several people onto a call to compare logs. This post shares an observability design — unified instrumentation, SLO alerting, trace-log correlation — plus a real incident review.
A single trade passes through more than a dozen stages from order placement to execution — gateway, risk control, market data, matching, clearing — and a hiccup in any one of them shows up in end-to-end latency. When something went wrong, our old routine was to pull every team onto a call and compare timestamps across each team's own logs, often taking half an hour or more to localize. We eventually rebuilt the whole approach.
Observability is only possible if the data can be correlated. We set one hard rule: every request entering the system must get a unique traceId at the gateway, propagated down the call chain. The correlation key across logs, metrics, and traces is unified as traceId + businessId (the business order number). That way, no matter which entry point you search from, you can pull the complete picture for the same trade.
We instrument with the OpenTelemetry standard to avoid vendor lock-in. Key spans are explicitly labeled — gateway entry, risk rule matching, market data subscription, matching engine, clearing persistence — and each span records latency and a result status code. Standardization matters here: without agreed span names, a trace is just a pile of anonymous spans nobody can read.
Traditional alerting sets arbitrary thresholds on CPU, memory, and QPS, producing both false positives and false negatives. We shifted to SLOs at the core: two targets — end-to-end trading latency P99 < 300ms and trade success rate ≥ 99.95% — with alerting based on multi-window burn rate.
Concretely: consuming 2% of the error budget within a 1-hour window triggers a warning, and 5% within 6 hours triggers critical, so a single transient spike doesn't page the whole team. Alerts carry the affected services, spans, and sample traceIds, so whoever receives one can click straight into the incident instead of asking "where do I even start looking?"
The biggest time sink in debugging is switching between three systems. We made the trace view the entry point: open a slow trade and you see the latency distribution across the entire call chain, with the slow span highlighted in red; click that span and it drills down into the logs on its node, with related metrics (GC, thread pool, connection pool) shown alongside. This collapses "which stage is slow" and "why it's slow" into a single step.
We also built a small set of team-level dashboards rather than one giant screen. Each team owns a dashboard scoped to its service with the top five latency contributors, error-rate trend, and the last few incident traces. When an on-call engineer is paged, the alert links directly to the dashboard for the affected service, so context is one click away instead of five searches.
Financial systems also carry hard retention and privacy requirements. We split storage by temperature: hot traces and logs for 7 days, warm for 90 days, and cold compliance archives for 3-7 years depending on the product. Trace data is sampled at the edge, but error traces and trades flagged by risk control are always kept in full. Personally identifiable information in logs is masked at ingestion, access to raw logs is restricted by role, and every lookup is written to an audit trail. Observability data becomes a liability if it leaks, so treat it like production data, not like a by-product.
One afternoon, the trade success-rate alert fired and P99 latency jumped from 120ms to 800ms. The old way would have meant lining up risk control and matching to compare timestamps. This time we clicked the traceId in the alert: the chain showed the risk-rule-matching span had grown from 5ms to 600ms, concentrated on a single rule; drilling into the logs showed that rule repeatedly triggering slow database queries. We found the rule had shipped that morning without an index, rolled it back, and recovered — about three minutes total.
The takeaway: correlation keys + SLO-based alerting + a unified view are what made those three minutes possible. Observability isn't a nice-to-have; for a trading system, it's the difference between how much money you lose during an incident. Every minute of unlocalized downtime is measured in real revenue.