An observability playbook for securities and banking trading systems: full-path tracing, tiered alerting, compliance retention, and rollout — cutting MTTR from hours to minutes.
Trading chains in securities and banking have two traits: long paths — one request crosses a dozen services from the app through gateway, risk control, order, matching, to settlement — and extreme sensitivity to latency and consistency, where a millisecond of jitter can cause reconciliation gaps. The old "check logs after it breaks" approach doesn't work here; you need full-path tracing and proactive alerting. The goal isn't more data, it's faster answers: when an order fails or a position doesn't reconcile, someone needs to know why within minutes, not after the market closes.
In OBSERVE, an order_id links the entire chain. For an order under investigation, enter the order number and see its latency distribution across services:
SELECT service, sum(duration_ms) AS total_ms, count(*) AS spans
FROM spans
WHERE order_id = '20261022001234'
GROUP BY service
ORDER BY total_ms DESCThe bottleneck service is obvious at a glance. Combine with a flame graph to find the exact method, then drill into that span's logs and slow SQL — resolution time drops from hours to minutes. For broader triage, flip the query: group by error status across all orders in the last hour to see whether a failure is isolated or systemic.
The worst thing in a financial system is alerts that "either never fire or never stop". Use three tiers:
Each tier maps to different channels and escalation: infrastructure goes to on-call ops, business goes to the trading team with risk control CC'd, and trace-tier alerts auto-mute overnight and only notify during trading hours. The key rule is separation of blast radius — a noisy infra alert must never drown out a business alert that actually moves money.
The moment a trade fails, two questions come at once: "what changed in the code" and "what broke in the infra". Keep both answerable in one place by tagging every span and log with a business key (order_id, account_id) and every infra metric with its host and deployment version. Then a single order_id query returns the spans, the logs, the deploy that went out, and the host CPU graph side by side. This cross-linking turns "it's slow, investigating" into "the matching-service deploy at 14:03 regressed P99 by 300ms, rollback in progress" — the difference between minutes and hours of outage.
Finance has hard requirements for log retention and auditing. OBSERVE supports tiered retention per regulation: trading logs kept 5 years, general run logs 1 year, alert records 3 years, with cold data auto-archived to object storage. Every query is audited — who queried what data, when, and what they exported is fully traceable, which also answers the auditors' recurring "who touched production data" question. Retention tiers are enforced per index, so a compliance change is a config edit, not a re-architecture.
Two phases: in the first two weeks, wire up traces and basic alerts for the core path (order, cancel, match); once stable, extend to accounts, settlement, and surrounding systems. In the first week, watch the false-positive rate — start with looser thresholds and tighten gradually rather than going high-sensitivity and triggering alert fatigue on day one. Before go-live, run a game day: inject a synthetic latency spike into the matching service and time how long it takes the alert to fire and the on-call to identify the span. If the full loop — spike to diagnosis — takes more than ten minutes, tighten the alerting before you trust it. Track one number above all: mean time to resolve (MTTR). If the new tooling doesn't move it, the instrumentation is decorative.