Locating Failures in Trading Systems: A Trace-plus-Log Correlation Playbook

Why trading systems are hard to observe

Trading systems are hard to troubleshoot for three reasons:

1. Long chains, many systems: one transaction spans gateway, risk control, account, clearing and core ledger — a dozen services or more, and a timeout in any one of them can fail the whole order.
2. Latency sensitivity: the instrumentation itself must not slow trading down. Instrumentation must be light, sampling controllable, and logging must never block business threads.
3. Compliant retention: trading logs are typically kept for five years or more and must be quickly searchable for evidence, so storage cost and query performance both need careful planning.

These three constraints mean a finance observability solution can't just copy the internet giants' "collect everything, analyze later" playbook. You have to make deliberate trade-offs. The goal isn't maximum telemetry volume; it's the shortest path from "a trade failed" to "here is exactly why," without slowing the trade itself or blowing the storage budget.

Full-chain tracing: from gateway to core ledger

The key is a single trace_id. Generate it at the gateway entry and propagate it through HTTP headers and message-queue headers so one transaction carries the same id across every system. Observe speaks OTLP; Java and Go services onboard via SDK or Agent automatically, while legacy systems get their trace_id parsed from logs by a shipper like Vector to complete the chain.

Propagation is the part that fails most often. A trace that stops at the gateway tells you nothing. Audit every hop that crosses a protocol boundary — HTTP to MQ, MQ to a batch job, a batch job back to a callback — and make sure each one forwards the trace context. This is exactly where money is lost in production, because the hops you forget to instrument are the ones that break first.

For latency-sensitive paths:

• Tiered sampling: 100% on the core chain, proportional sampling elsewhere
• Async reporting so failures never block the business path
• Track tail latency (P99) on hot spans, not just averages — a 20ms average hides the one-in-a-thousand 800ms outlier that actually loses trades

Finance has one more wrinkle: asynchronous settlement and reconciliation jobs don't inherit a live request context, so they need their own trace linkage. When a batch job picks up a queued trade, record that trade's trace_id as a span attribute so the async leg still attaches to the original transaction in the trace view.

A trace-plus-log troubleshooting runbook

When an incident hits, the standard play is three steps:

1. See a P99 latency spike on a service in the metrics view, drill into the error logs
2. Click "view trace" on a log line and locate which downstream call slowed down (say the clearing API went from 20ms to 800ms)
3. Compare successful and failed requests over the same window in the trace to tell whether it's an input problem or downstream capacity

The whole flow depends on correlation working in both directions: logs jump to traces, and spans pull up their logs. Lose either direction and you're back to hand-grepping. In a trading system, that difference is measured in money — every extra minute of manual investigation is a minute the venue is down or degraded. Practice this drill regularly against staging, not just during incidents: a playbook you've rehearsed is the one you'll actually execute at 2am.

Compliant retention and tiered alerting

With long retention windows, use hot/warm/cold tiers: the last 7 days stay hot for second-level search, 7 days to 2 years move to warm storage, and older data archives to object storage that can be restored for search. Snapshot the index before archiving so evidence can be reconstructed for regulators.

Tier alerts to avoid fatigue:

• P0: core ledger or clearing chain down — phone + SMS straight to on-call
• P1: core trade error rate over threshold — WeCom + webhook
• P2: non-core service anomalies — dashboard only, no paging

Attach alert rules to services, not machines, or they break after autoscaling. This playbook has been validated on trading lines at several brokers and payment providers — the core principle is locate fast, page sparingly, and keep the evidence.