An observability playbook for financial core trading systems, built around end-to-end tracing for cross-system failures and alert noise reduction, with metric standards, SLO definitions, and on-call runbooks that also satisfy audit requirements.
The hard part of observability in financial systems isn't collection — it's two things: a transaction spans a dozen systems so failures are slow to locate, and there are so many alerts that on-call engineers have gone numb. This article lays out a playbook built around end-to-end tracing and alert noise reduction.
A single transfer flows through channels, risk control, accounting, and settlement — more than a dozen systems — and a timeout in any of them surfaces as "transaction failed." Single-system logs won't reveal the root cause; you need traces to stitch the whole path together. Key points:
traceparent across boundaries; don't invent your own header.txn.id) at the entry point, so support can look up the full trace from a ticket number.recordException with an ERROR status on failure.One real case: a 2% failure rate during peak traffic was located in five minutes — reverse lookup by txn.id plus sorting slow spans pointed to an exhausted connection pool in one settlement shard — where log-only investigation had previously taken almost two hours.
Financial systems generate alerts by nature: infrastructure, middleware, and business layers stack up, and one incident can fire hundreds of alerts. Reduce noise in three layers:
Layer one, suppression: the same metric sends only one alert within a convergence window; repeated triggers don't re-send. Layer two, grouping: alerts with the same transaction path and root cause are merged into one, with a list of affected services attached. Layer three, tiering: only SLO-related metrics get phone/SMS-level paging; everything else goes to message queues and dashboards.
Whether an alert should fire ultimately depends on SLOs. Define them per transaction type: core transfers target 99.99% availability, queries 99.9%. Track error budgets quarterly and pause non-urgent changes once the budget burns to 80%. Standardize metric definitions across the company — "failure rate" must mean "failed transactions / total transactions," not "failed requests / total requests," or numbers from different teams won't reconcile.
An SLO is only as good as its definition. Write it as success events over total valid events in a rolling window — "successful transfers / total transfer attempts, 30 days, target 99.99%." Exclude invalid attempts (rejected by risk control, wrong credentials) from the denominator, or your availability metric measures user behavior instead of system health.
Derive the error budget explicitly: at 99.99% you get roughly 4.3 minutes of downtime per month. That tiny number is the point — it forces prioritization. When the budget burns past 80%, freeze feature releases and redirect effort to reliability; when it's exhausted, stop everything but emergency changes. Wire the budget into the alerting tiers from the previous section so SLO breaches page a human while everything else stays on the dashboard.
Review SLOs quarterly. A 99.99% target with no one watching the budget is theater; a 99.9% target with a live burn-down chart and a weekly review is a working reliability practice. Pick whichever you can actually operate, not whichever looks impressive in a slide.
Regulators require log retention and traceable operations, so raw-log retention must meet the regulatory period, and every alert response is recorded: who acked, when, what actions were taken, and when it recovered — all landed in a ticket. Fix the on-call flow into a runbook: ack → locate via tracing → mitigate per the playbook → postmortem and update SLOs. Each step has a matching dashboard and query template, reducing reliance on individual experience.