An observability blueprint for banking, securities, and insurance: tamper-evident audit log retention, second-level alerting on trading paths, log-and-trace triage, and unified monitoring across data centers.
The hard part of observability in finance isn't the tech stack; it's the constraints. First, regulation: audit logs must be retained for years and must resist tampering. Second, trading paths are extremely latency-sensitive, where one slow query can mean direct monetary loss. Third, disaster recovery, with dual-active within a city and multi-center across regions as the norm, means monitoring itself has to span data centers. Here is a practical approach built around those three constraints.
The first requirement for audit logs isn't fast search; it's that they stay put, can't be altered, and reconcile end to end. OBSERVE routes audit logs through a dedicated pipeline: append-on-write, read-only retention, periodic hash chaining, and archival to object storage. Any modification is exposed by the hash chain.
audit:
retention_days: 1825 # five years
append_only: true
hash_chain: sha256
archive:
type: oss
bucket: audit-archiveCombined with tiered permissions, audit-log read access is limited to compliance and audit roles, so ordinary operations staff can't see them at all, cutting off insider tampering at the source. When a regulator asks for "all operations on customer accounts for the last three years," you export a consistent, timestamped set rather than scrambling across systems. Reconcile the hash chain against the archive daily, so tampering is detected quickly instead of discovered during an audit.
Trading paths need second-level alert latency. Watching dashboards by hand doesn't scale, so the right move is to define golden signals, latency, error rate, throughput, for the critical paths and detect breaches continuously with sliding windows. For example, treat "order API P99 latency above 200ms for five consecutive seconds" as a P0 that goes straight to phone, rather than waiting out a five-minute aggregation window.
sum(rate(order_api_latency_seconds_bucket{le="0.2"}[30s]))
/ sum(rate(order_api_latency_seconds_count[30s])) < 0.99This PromQL reads as "99% of requests over the last 30 seconds completed within 200ms." A value below 0.99 means requests are slowing down, and with silence periods and grouping in the alert engine, you avoid an instant wall of duplicates. The point of a P0 on latency rather than on errors is timing: by the time an error rate spikes, money has already been lost. Latency warns you before the failure, which is exactly when you can still do something.
The worst kind of trading incident is one where the alert fires but nobody knows which layer broke. OBSERVE correlates traces and logs through traceId, so from a triggered alert you can walk the trace hop by hop, gateway, trading service, ledger, then open the logs attached to the failing span and see the SQL and stack trace directly. A fault that used to take three teams half an hour to localize usually shrinks to under ten minutes.
One detail that pays off in finance is tagging every span with a business key such as order ID, account ID, or transaction ID in addition to the technical fields. Then a customer complaint like "my order never completed" becomes a one-line trace search instead of a multi-team hunt.
With dual-active and multi-region topologies, a monitoring platform deployed in only one center goes dark the moment you cut over. Deploy a collection node in every center, forward data to the primary while keeping a local buffer, and let a secondary center take over queries and alerting when the primary fails. The local buffer also covers data loss during cross-region line jitter.
Finally, roll out in phases that respect the trading calendar. Instrument one non-critical service first, watch the overhead, then expand. Keep sampling off for the critical paths and on for the noisy ones. Run any load test outside trading hours, and rehearse the DR cutover during a maintenance window, so the first real failover is not also the first time anyone has done it. Throughout, keep an eye on collector overhead on the instrumented hosts; on modern hardware it should stay well under a few percent of CPU, and if it does not, revisit your sampling and batch settings before the rollup continues.