An observability design for securities firms: end-to-end tracing across order, matching, and confirmation, latency- and error-based alerting, and noise reduction that respects trading hours and market holidays.
Securities trading systems differ from ordinary business systems in a few ways. First, the path is short but extremely sensitive: order, matching, and confirmation complete within tens of milliseconds, and any jitter becomes a customer complaint. Second, load is strongly time-based - peaks around open and close, valleys during the lunch break and after close - so fixed thresholds drown you in false alarms during quiet periods. Third, compliance requires audit logs to be complete, tamper-evident, and retained long-term.
Observability here comes down to three things: full tracing, accurate alerting, and durable audit trails. Get any one wrong and you either can't see what happened, get paged for nothing, or fail an audit.
The order-to-confirmation path typically spans several systems: the brokerage counter, risk control, the matching engine, market data, and the confirmation channel. Instrument each with OpenTelemetry and thread one traceId through the whole chain, tagging spans with business attributes:
When an order misbehaves, entering its order_id reconstructs the full path from counter to matching to confirmation with per-hop latency. Quantitative trading relies on this heavily - isolating one bad order out of a batch depends on attribute filters plus trace reconstruction, and doing it by hand across five systems' logs is not realistic at batch sizes. Because every order matters, the trading path is typically sampled at 100%: the cost is justified by the ability to reconstruct any single customer's order on demand, which both regulators and support staff expect.
Average latency hides the long tail, so trading systems should not alert on means. A handful of slow orders buried under thousands of fast ones is exactly the kind of problem that averages make invisible and customers feel immediately. We monitor by order type:
Set these thresholds from measured baselines, not guesses - capture two weeks of P99 during normal trading, then set thresholds with headroom above the observed value so you alert on real regressions rather than normal variance. When an alert fires, attach the traceId and order_id to the notification so the on-call engineer opens the failing trace directly instead of hunting for it. The difference between "payment latency high" and "order 20241026... timed out at the matching engine" is ten minutes of investigation saved.
False alarms are the biggest pain in trading monitoring. We apply three layers of noise reduction:
This cut alert volume by about 70%, and on-call staff stopped being paged by lunch-break false alarms. The key insight is that alert rules should encode the market calendar, not just metric thresholds - the calendar is public and stable, so baking it into the rules removes a whole class of predictable noise.
Beyond tracing and alerting, watch the metrics that matter at peak load: order throughput per second, matching queue depth, and connection counts on the counter and market-data gateways. The open auction is a sharp, predictable spike; capacity planning should be validated against it specifically, and the dashboards you look at during that window should be built in advance, not assembled during the incident.
For audit logs, completeness, traceability, and tamper-proofing are the requirements. Keep audit logs separate from operational logs, enable tamper-proofing and long-term retention (no less than the regulator requires), and provide multi-dimensional search by time, account, and operation type with access control so only authorized staff can query. In a Xinchuang environment this must still satisfy classified-protection and industry compliance requirements, so treat audit capability as a hard selection criterion rather than an afterthought.