An observability design for payment and trading systems: tiered sampling, transaction-level baseline alerting, and an order-ID-to-trace loop that cuts slow-trade triage from hours to minutes.
The hard part of observability for financial trading systems isn't collection — it's the single slow trade that gets drowned out statistically. This article lays out a complete design: how to sample, how to alert, and how to go from an order ID to the root cause, while satisfying compliance along the way.
A trading chain spans five or six hops at minimum and a dozen at worst — a single payment crosses the gateway, risk control, accounting and settlement, and every hop has its own logs and database. The typical failure mode is knowing a trade was slow but not which hop slowed it down. A customer complains that "the payment took ten seconds to complete", and you're left holding an order ID, digging through logs across four systems. Error-rate alerting is almost useless against a single slow trade, because it drowns statistically in the flood of normal transactions. Worse, trading systems demand complete traces: drop one span and the whole chain stops assembling, leaving you with fragments instead of an answer.
A trading system can't afford to lose traces, but capturing everything is too expensive. Use tiered sampling: 100% on core transaction paths, 10% on non-core read endpoints, and drop health checks entirely. Three implementation notes. First, make the sampling decision once at the ingress gateway, so individual services don't sample independently and break the chain. Second, bind trace_id to the transaction order ID in both directions, so you can look up a trace from an order ID and recover the order ID from a trace. Third, make the sampling rate a dynamic config you can raise during a promotion without a code change. Agree on span attributes up front, too — every hop should carry the order ID, amount and channel, so alerts and investigations have real fields to key on. For slow trades, which are rare but critical, layer tail-based sampling on top: buffer traces briefly, keep the ones that exceed a latency threshold, and drop the rest — preserving the full picture of slow trades at a small cost.
Fixed thresholds either over-alert or miss incidents. Build a baseline per transaction type — P50/P95/P99 latency and success rate — and alert when a metric drifts three standard deviations from it. When a single trade exceeds its P99 threshold and its amount crosses a defined value, raise a high-severity alert and keep the full trace context for reconstruction. An example rule:
alert: trade_slow
expr: histogram_quantile(0.99, trade_duration) > baseline_p99 * 2
for: 1m
labels:
severity: warningThe for clause matters: it filters one-off blips so you alert on sustained degradation, not a single unlucky request. Build baselines per environment — production and staging have very different latency profiles. Tier the notifications too: P99 drift goes to the on-call channel, while alerts tied to high-value trades escalate to a phone call.
When an alert fires, use the order ID to find the trace_id, expand the call chain to locate the slow hop, then drill into that hop's logs and SQL — one continuous path with no switching between systems. With audit trails recording every query and export, you also satisfy the traceability that financial compliance demands. Don't forget retention rules: regulators typically require transaction-related logs to be kept for six months or more, so design your archival strategy up front rather than discovering the gap when an auditor asks. This loop takes slow-trade triage from hours down to minutes and leaves every postmortem with a paper trail you can hand to auditors.
Once this design is in place, run a drill every quarter: pick a few slow trades from past alerts and have the on-call engineer trace them from the order ID, timed, to see whether they can reach the root cause within ten minutes. The gaps the drill exposes are exactly what to fix next.