Banks span decades of tech stacks where failures cascade across systems. A layered collection approach, a real incident showing 10-minute root-cause isolation, and rollout takeaways on alert tiering and compliance.
A bank's distributed architecture looks nothing like an internet company's. Core systems run on CICS or mainframes, channel systems on x86 clusters, plus a pile of outsourced systems — the tech stack spans decades, and failures cascade across systems. A slow transfer could be the core transaction, or the front gateway, but classic monitoring only shows "CPU is high on some system", not which link is the culprit.
Worse, each bank system has its own log format, time base and monitoring tool. The core has its logs, the channels have theirs, and an incident means every team digs separately and later lines up timelines by hand just to see the whole picture. OBSERVE's deployments across several banks produced a repeatable playbook.
The first layer is the transaction path. A lightweight probe on core CICS transactions collects transaction ID, transaction code and latency, reported as spans that join up with the OpenTelemetry traces from peripheral systems into one complete path from the counter or mobile app to the core. The key is carrying the transaction ID as the unified trace context end to end.
The second layer is infrastructure. Mainframes, Oracle and storage report metrics via exporters into a unified alerting system. Many bank mainframes are still SNMP or out-of-band managed, so they need dedicated collectors — you can't expect them to natively speak the Prometheus protocol.
The third is business logs. Core transaction records and channel message logs feed a single search index, so you can pull up every log line for a transaction by its transaction ID. Once logs are unified, cross-system log comparison goes from "flip through files by hand" to "one SQL query".
After a city commercial bank went live, mobile transfers jumped from 800ms to 3 seconds. The traditional way — each team digging through its own systems — went nowhere for two days.
With OBSERVE, searching by transaction ID in the tracing view found the root cause in ten minutes: the core system's account-query response had slowed, and drilling down showed a shard's connection pool was exhausted, with connection wait eating 1.8 seconds. The trigger was a batch repayment job that hadn't released its connections. Restarting that batch job brought latency straight back down. The trace made the causality visible in a way no per-system dashboard could — one timeline, one blame line.
The point of this case isn't "tracing is great" — it's that once the transaction ID flows through, investigation shifts from "which system is it" to "which step is slow".
Three practical conclusions for tech leads:
Don't try to land all of this at once. The steady order is: tracing plus the transaction-ID link first (fastest payoff), then unified log search once that's stable, and infrastructure metrics with alert tiering last. Each phase delivers value on its own, and the gradual pace gives you room to align the masking policy with compliance and security along the way, rather than bolting it on after the data is already flowing.
Three months after rollout, mean time to detect (MTTD) dropped from two hours to fifteen minutes, and cross-system issues shrank from 30% of incidents to 5%. Those numbers came from production telemetry, not a slide deck.