An observability plan for financial institutions: classified log retention and audit compliance, transaction tracing driven by a global trace ID, tiered alerting, and a three-phase rollout with per-phase acceptance criteria.
In financial systems, observability is not purely a technical concern — it also touches audit, security classification, and data retention. This article is for technical leads in financial institutions and lays out a plan that satisfies compliance while actually helping you resolve incidents.
Regulators set explicit requirements for retention periods, integrity, and tamper-evidence. The plan splits logs into three channels — business, operations, and security — so they never mix. Security logs are encrypted and stored separately with a hash chain for tamper evidence. Retention policies differ by category: security logs typically need six months or more, transaction logs longer, and operations logs can be shorter. The platform offers read-only, time-scoped archive queries, with auditors limited to read access, operators granted search, and developers isolated per project. Audit actions themselves are logged, so "who queried what, when" is always traceable.
Concretely, security and transaction logs often need to be kept for years and must survive a subpoena, so they go to encrypted, immutable storage with a retention lock; operations logs are usually fine on a rolling 30-to-90-day window. Splitting the three classes at ingestion time — not after — is what keeps long-term retention affordable, because the high-volume operational logs never touch the expensive tamper-evident storage. On top of the three channels, most institutions also apply a data-classification label — public, internal, confidential, or secret — to each log source. The label controls who can view the logs and whether the content is masked in the UI, which is a separate concern from retention but just as important for passing a security review.
The core requirement is that every transaction can be reconstructed. The plan uses a global transaction ID (global_trace_id) that propagates through gateway, trading, clearing, and accounting as a parent span in OpenTelemetry, with each system's local trace ID mapped to it. When something goes wrong, entering an order ID reconstructs every service the transaction touched, the time spent in each step, and the key fields along the way — turning cross-system reconciliation and dispute investigation from a hours-long job into a minutes-long one.
The practical step is to generate global_trace_id at the gateway for every inbound transaction and inject it as a span attribute, then pass it downstream through the W3C trace context so each downstream service records it in its own spans and logs.
Alerting in finance must be layered: P0 pages the on-call engineer by phone, P1 sends SMS plus a group message, and P2 creates a ticket. For critical metrics such as payment success rate, core transaction TPS, and fund consistency, set two thresholds — a soft threshold that warns and a hard threshold that alerts — to cut down false positives. For example, payment success rate: soft threshold at 99% triggers a warning for review, hard threshold at 97% pages P0; the gap absorbs normal fluctuation so you do not wake someone up over noise. Attach the correlated trace and log links directly to the alert, so the on-call engineer can keep digging the moment it fires instead of asking "where is the scene". A practical starting set of metrics is: request volume, error rate, latency percentiles, payment success rate, and a handful of fund-consistency counters. Instrument those first; the rest of the dashboards can grow from them.
Three phases: first get log collection and retention working to meet the compliance baseline; second add tracing across the main transaction path; third add alerting and automated response. Each phase has its own acceptance criteria — phase one is done when retention and audit queries pass review, phase two when any transaction can be reconstructed by its order ID, and phase three when a P0 alert carries a linked trace within seconds. Plan for the migration itself to take weeks, not days: the constraint is usually not the tooling but agreeing on the log classification scheme and retention policy across compliance, security, and operations before anything ships. This staging avoids the risk of a doomed big-bang migration.