Trading chains span a dozen systems, are latency-sensitive, and face strict regulation. This post shares a metrics-logs-traces approach to alerting and troubleshooting, plus a real incident resolved in minutes.
A securities trading chain spans a dozen systems: counters, order routing, risk control, clearing, market data, and accounts. A single order crosses seven or eight hops from the client to the exchange, and the whole path has to run in milliseconds. Three traits make observability especially difficult here. First, latency sensitivity: one system's GC pause can ripple into jitter across the entire chain, so a 'small' hiccup in one service shows up as a latency alarm in a completely different one. Second, cross-system spread: failure points are scattered, and looking at any one system's logs alone will not surface the root cause. Third, regulatory pressure: the CSRC and exchanges impose clear requirements on availability and audit trails, so every incident must be reconstructable and reviewable after the fact — which means you need data, not just alerts.
The three signals have distinct jobs. Metrics find the problem: error rate, P99 latency, and throughput are the aggregates that surface anomalies first. Logs explain it: once a metric alert fires, logs show which endpoint, which SQL, or which parameter went wrong. Traces locate it: they stitch the cross-system call graph of a single request and point straight at the slow system and span. The key move is to thread one trace_id through all three: from a metric alert you jump to the corresponding trace, and from a span you jump to the logs.
In practice, we require every system to propagate a trace_id (x-request-id) in HTTP headers. That way, even if a system has not adopted OpenTelemetry, as long as its logs carry that ID, you can still reconstruct the whole chain afterwards. This single convention is worth more than any one dashboard, because it makes every log line and every span addressable by the same key.
Alerts for a securities system need to be fast but not noisy. Three principles. First, tier them: core-chain error rate and latency during trading hours are P0 (instant phone or SMS), while post-market batch jobs are P2. Second, deduplicate: hundreds of alerts caused by one fault must collapse into one, aggregated by service before anyone is paged — otherwise the on-call engineer spends the first five minutes closing duplicates. Third, throttle: the notification channel itself needs suppression to avoid an alert storm knocking out the on-call channel. As a concrete threshold reference: fire a P0 alert when the core order endpoint's P99 latency exceeds 100ms for three consecutive sampling periods, or its 1-minute error rate exceeds 1%. Add a short automatic recovery window (a few minutes) so a transient blip does not page anyone twice.
Latency-sensitive trading systems benefit from defining SLOs per critical path — for example, 99.9% of order submissions complete in under 100ms over a rolling 30 days. Burn-rate alerts built on those SLOs are more stable than raw threshold alerts: they page early enough to act but late enough to ignore noise. Jujing OBSERVE tracks the error budget continuously, so the team sees at a glance how much of the month's budget a single incident consumed.
On the compliance side, the same trace_id that powers debugging also powers audit. Every order, every risk check, and every rejection can be reconstructed later by its trace_id, which satisfies the regulator's need for a complete, replayable record. Keep the retention policy aligned with the regulator's minimum — typically years, not days — and treat the trace archive as evidence, not just a debugging tool.
One trading day, shortly after the open, the order endpoint's P99 latency jumped from 30ms to 800ms and a P0 alert fired. The investigation order: first metrics, confirming the spike was confined to a single downstream called risk control; then traces, sampling a few slow requests and finding they stalled in the risk system's blacklist-query span at 700ms+; finally logs, searching by trace_id and discovering the blacklist query was doing a full table scan because an index was missing after that morning's hot update to risk rules. Root cause was found in about five minutes, and rebuilding the index restored latency. No system was restarted the whole time — metrics found the problem, traces narrowed the scope, logs confirmed the cause. The lesson is simple: the three signals are only as good as the single ID that connects them.