Securities trading systems demand low latency, high availability, and strict audit compliance. Using a core trading system as an example, this post covers end-to-end tracing across order, matching, and return, plus log retention compliance, alert severity tiers, and SLOs.
Observability for a securities core trading system is a different order of problem than for a typical web app.
Why trading systems are hard to observe
- Latency-sensitive: end-to-end order-to-return latency is measured in milliseconds; any overhead added by monitoring itself comes under scrutiny.
- Long, heterogeneous chains: one order flows through gateway, risk control, matching, clearing, and return—across multiple processes and sometimes multiple data centers.
- Strict compliance: logs must be auditable and retained (usually 5+ years), tampering must be detectable, and regulators must be able to pull the full context of any single order quickly.
All three mean this rework can’t stop at “just install a collection agent.”
Tracing a single order end to end
The first move is turning “one order” into one traceable trace:
- Unify trace context: generate a trace_id at the gateway and propagate it downstream through message/request headers, so risk control, matching, and clearing all hang off the same trace.
- Inject business fields into spans: write
order_id, account_id, and security_code as span attributes, so you can look up the whole chain by order number instead of guessing by timestamp. - Mark key points: add semantic span events at matching, risk rejection, and return push, recording latency and result codes.
After the rework, when support reports “an order didn’t fill,” an SRE can search the trace by order number in Observe and see—in seconds—where it stalled, how long it took, and the result code. Time to diagnose drops from hours to minutes.
Log retention and audit compliance
Trading log compliance goes well beyond “can we find it”:
- Integrity: logs must not be lost between generation and persistence. The collector needs local buffering and resume-from-breakpoint.
- Tamper evidence: enable hash chaining/signing on logs, so any modified line fails validation and audit gets the non-repudiation it requires.
- Tiered retention: trading detail logs retained 5+ years (hot/cold tiering—hot data queryable near-real-time, cold data archived to cheaper storage); ordinary runtime logs for 90 days.
Observe offers an “audit mode” that enables signing and tamper detection on designated log streams, with a full evidence chain on export.
Alert severity tiers and SLOs
Trading alerts can’t be one-size-fits-all. Define the SLO first, then hang alerts off it:
- Error rate: matching success ≥ 99.99%, with risk-control rejections excluded from the error count.
- Latency: order-to-return P99 ≤ 200ms.
- Availability: matching service monthly availability ≥ 99.99%.
The tiers we landed on:
| Tier | Trigger | Response |
| --- | --- | --- |
| P0 | matching availability below 99.99%, latency P99 at 2x threshold | incident within 5 minutes, call + SMS |
| P1 | single-service error rate > 1%, queue backlog over threshold | respond within 15 minutes |
| P2 | disk/memory over 80%, rising slow queries | handle within 30 minutes |
Each tier binds to explicit SLO metrics, with alert rules attached to the SLOs rather than scattered thresholds.
Layering observability collection
Trading observability data is collected in three layers so they don’t interfere:
- Infrastructure: host, network, and middleware metrics via a lightweight agent at second granularity, non-invasive to business processes.
- Application: traces and business logs reported directly by the SDK over OTLP, sharing the application’s lifecycle.
- Sideband: local log files from matching and other core processes are collected asynchronously by the agent, adding minimal overhead to the primary process.
The three layers are correlated in Observe by trace_id and timestamp, so you can drill down from “one order” to “CPU jitter on one machine.”
Lessons from the field
- Traces before metrics: get the chain clear first; many metric problems surface on their own once the chain is visible.
- Load test first: run a full-chain load test before and after the rework, compare latency overhead, and make sure observability itself isn’t the bottleneck.
- Compliance from day one: don’t add retention and tamper-proofing right before acceptance—configure them to audit standards from the start.
- Align with compliance early: confirm retention periods and searchable fields with the compliance team up front to avoid rework after launch.
At its core, trading-system observability is about making “the full lifecycle of one order” traceable, auditable, and reproducible. That’s a compliance requirement—and a hard operational need.