Observability for Core Banking Systems: From Alert Storms to Precise Diagnosis

Pain Points in Financial Scenarios

Financial observability differs from internet companies in a few hard ways:

• Long transaction chains: one transaction crosses a dozen systems (gateway, risk, accounting, settlement), making diagnosis slow
• Noisy alerts: poorly tuned thresholds can flood hundreds of alerts from a single blip
• Regulatory requirements: logs must be retained, auditable, and masked
• Narrow change windows: core systems change only in off-hours, limiting what you can investigate

These constraints mean you can't copy the internet playbook of "collect everything, analyze later." Financial observability must emphasize alert quality, trace completeness, and data compliance.

Designing Monitoring and Alerting

Our core approach for a city commercial bank: alert tiering + noise reduction + on-call loop.

1. Tiering: P0 (money-related, e.g. reconciliation mismatch), P1 (availability, e.g. timeout spike), P2 (performance, e.g. P99 rise), P3 (capacity warning)
2. Noise reduction: aggregate same-source alerts within 5 minutes into one; use period-over-period baselines instead of fixed thresholds to cut false positives on holidays and releases
3. Loop closure: every alert links to a runbook with steps and last-handling notes

Key metrics: transaction success rate, P99 latency, reconciliation discrepancies, timeout rate — all drillable by transaction type, channel, and institution. Structure the main dashboard around that shape: the top row shows transaction success rate and any money-related P0 alerts, the middle shows time series per channel, and the bottom ranks reconciliation discrepancies by institution. The on-call engineer can then tell in one glance whether it's a global outage or a single-institution issue.

Beyond the dashboard, define SLOs for the two or three metrics that actually matter — transaction success rate and P99 latency are the usual candidates — and wire the alert thresholds to the SLO error budget rather than ad-hoc numbers. That keeps the alert count tied to real user impact instead of drifting with every release.

Tracing Implementation

The hardest part is passing the transaction ID across systems. Recommendations:

• Use trace_id to carry the global transaction ID, generated at the gateway and propagated downstream
• Use baggage for institution, channel, and business order numbers — don't put them in span attributes (sensitive-data leak risk)
• Sample by amount: 100% for large transactions, proportional for small ones
• Force-sample key nodes (risk checks, fund deduction) so any anomaly can be traced back to a full chain

One detail worth underlining: span attributes are shipped with the data and may be exported, so sensitive fields belong in baggage, which you can configure separately for cross-system propagation and masking. In practice this cut per-transaction diagnosis time from hours to minutes.

Putting It Together: A Worked Example

Here's how the pieces come together on a real failure. A reconciliation job reports a discrepancy at 02:13. The P0 alert fires with a runbook attached. The on-call engineer opens the alert, jumps to the trace view, and filters by transaction type and time window. Because large transactions are sampled at 100%, the exact failed transaction is there. The trace shows a four-second stall inside the risk-check service, and the overlaid metrics show that service's error rate spiked at the same moment.

From the trace, the engineer opens the correlated logs for that service and sees a connection-pool exhaustion message — the root cause, in under five minutes from the first alert. Without the trace-to-metric-to-log correlation, the same investigation would have meant grepping through three systems' logs by hand.

The point isn't the specific bug. It's that the three signals — metrics for "what changed," traces for "where," and logs for "why" — are wired together so each is one click from the next.

Compliance and Data Security

Three things are non-negotiable:

1. Masking: card numbers, IDs, and phone numbers are masked at the collection source; the platform stores only masked data
2. Retention: logs kept per regulatory requirements (usually 3+ years), cold data tiered to object storage to control cost
3. Auditing: who queried which log, when, and what was exported — all recorded for regulatory inspection

Data must stay in-domain. Deploy privately so logs, traces, and metrics all stay inside the bank's data center. Masking rules should be configurable and traceable — support both field allow-lists and regex — and the masking should happen on the collector side, not after data reaches the platform. The latter approach means sensitive data touches disk once before it's ever masked.