Trading systems demand millisecond latency and near-perfect uptime. We lay out an observability architecture for core trading paths, key metrics, alert thresholds, and a failure-drill plan, plus a real pre-market slow-query investigation.
Trading paths differ from ordinary web systems in three ways: peak load is irregular (call auctions, the final minutes before close), latency is critical (a single order taking a few hundred milliseconds is an incident), and the fault window is tiny (you cannot stop trading to debug mid-session). An observability system must therefore do three things: keep collection overhead low, make data queryable in real time, and let alerts carry different priorities per trading session.
We use layered collection. Inside the app, the OpenTelemetry SDK instruments three critical span types — order placement, cancellation, and execution report. The gateway captures every request, and databases and message brokers (plus the broker-counter interfaces from vendors like Hundsun and Kingsun) are covered via sidecar packet capture or log parsing. Everything lands in OBSERVE, and traces and logs share one trace_id for correlation.
The key design choice is sampling strategy: 5% of normal requests, but 100% of errors and any request slower than 200 ms — implemented with tail-based sampling so critical anomalies are never dropped.
Five metrics matter most during trading hours:
| Metric | Meaning | Alert threshold |
|---|---|---|
| Order latency P99 | entry to broker response | > 300ms |
| Execution report delay | broker report to client | > 500ms |
| Error rate | 5xx / business rejection | > 0.5% |
| Message backlog | queue consumer lag | > 1000 msgs |
| DB connections | pool utilization | > 80% |
Thresholds should vary by session: allow brief jitter around the call auction and the pre-close, but stay strict after the open. Severity tiers: P0 (trading impact) goes to phone + SMS, P1 (performance degradation) to IM, P2 (trend warnings) to email.
After a broker shipped a new build, pre-market load testing showed order latency P99 jumping from 80 ms to 400 ms. Expanding the trace by trace_id in OBSERVE pointed at the "fund verification" step, which now called a newly integrated risk-control service — and that service did a full table scan on every request. Digging into the logs showed the risk service's cache was not working: the new build changed a config item and the cache key was misspelled. After fixing the config, P99 fell back to 90 ms. Discovery to root cause took under 15 minutes, with no trading process restarted.
Every quarter we run a double-blind drill: inject a random fault (DB failover, queue backlog, one service pinned at full CPU) and have on-call engineers localize it using OBSERVE alone. The metric isn't "how fast was it fixed" but "how fast was the root cause found." Over three months, average localization time dropped from 40 minutes to 8.
The point of observability in finance is not more dashboards — it's turning root-cause localization into a measurable flow: metric alert → trace drill-down → log confirmation, all closing the loop in one screen. If you are still at "an alert fires and someone greps logs by hand," observability hasn't actually landed yet.