An observability blueprint for banks and payment providers: OpenTelemetry transaction tracing, tamper-resistant audit log retention, three-tier alerting, and a reference configuration for a mid-size city commercial bank.
Observability in finance is about more than fast troubleshooting. Regulators require audit logs retained for 3+ years and tamper-proof; transaction chains must be traceable end-to-end so a single order can be fully reconstructed from placement to settlement; and failures must be escalated by severity, with core transaction-chain outages detected within minutes. Those three requirements mean you can't just copy the internet-company "glance at a dashboard" playbook — the compliance and accountability constraints reshape the whole design.
A single payment request can pass through a dozen services — gateway, accounting, risk, clearing. OBSERVE uses OpenTelemetry to thread the whole chain with one trace_id carried through to the end. When a transaction times out, you reverse-look by transaction ID: is the risk-control interface slow, or is clearing piling up retries? The flame graph shows it directly.
Add business instrumentation — order_id, user_id, amount as Span attributes — and you get "trace by transaction ID, logs by trace," aligning business dimensions with technical ones. When reconciliation doesn't match, you can jump from the accounting system's logs straight to the original request. This is the part most monitoring tools skip: they show you latency percentiles, but not which specific order was affected.
Audit logs are written with a WORM (write-once-read-many) storage policy, retained for 3 years as regulators require, and archived rather than deleted on expiry. Every query and export is itself audited, and combined with access control (read-only roles, export approval) this satisfies the operation-audit requirements of graded protection and financial regulators. Even administrators cannot modify or delete audit logs, closing off the "change the logs after an incident" risk. Plan your storage tiers around this: audit data grows monotonically and never shrinks, so it should land on cheap object storage as early as your retention rules allow.
Alerts in financial systems can't be one-size-fits-all. We recommend three tiers:
Alert rules, on-call rotations, and escalation paths are all configuration-driven, with alert routing that supports multi-datacenter active-active setups. If a P1 alert goes unacknowledged for 5 minutes, it escalates to the next-level owner automatically, so nothing falls through the cracks. The goal is that a P1 is never waiting on a single human to notice a phone.
A typical mid-size city commercial bank runs a 3-node OBSERVE cluster plus a 2-node collection gateway and object-storage archiving, ingesting roughly 2 TB of logs per day with search P95 under 2 seconds. Core systems (core accounting, payments) get full trace coverage first; peripheral systems start with logs only, phased in over about two weeks to cover the core chains. After go-live, review alert false-positive rates weekly and keep tuning thresholds until P1 false positives are down to single digits. False positives are the fastest way to train on-call staff to ignore the very alerts you built the system to deliver.
Financial data isn't uniform, so don't treat it as one blob. We split it into three tiers. Hot tier: the last 3 days on SSD, for active incident response and real-time dashboards. Warm tier: 3 to 90 days on cheaper disks, for capacity planning and weekly review. Cold tier: 90 days to 3+ years on object storage, mostly audit logs that exist to satisfy retention rules and only get queried during an investigation or an inspection. The query layer spans all three transparently, so an auditor running a 2-year-old search gets the same SQL interface as an on-call engineer looking at the last 10 minutes — just slower. Set the tier boundaries by query frequency, not guesswork: pull your actual query patterns after a month and move the cutoffs to match. Most teams find that 95% of queries touch data under 7 days old, which is exactly why the hot tier earns its SSD cost and everything else should not.
Finally, don't skip the people side. Observability only pays off if the on-call rotation actually uses it during incidents. Run a monthly "game day" where you inject a fault into a staging transaction and have the on-call engineer find it end to end using only OBSERVE, with the clock running. That single exercise exposes gaps in tracing coverage and dashboards faster than any review meeting.