A field guide for banks and brokers: deriving collection boundaries and masking from audit requirements, tracing across legacy monoliths and new microservices, data-driven incident drills, and a four-phase rollout plan.
The hard part of observability in financial systems isn't tooling—it's the "both at once" problem: you need to find production issues fast while satisfying audit and compliance, and you need enough telemetry without ever touching sensitive data. Here's an approach we've seen work at banks and brokerages.
Financial logging has a simple rule: fully traceable, but sensitive data invisible. Define your collection boundary first. Mask transaction records, account numbers, passwords, and ID numbers on the collection side—replace them with masks or hashes so raw values never reach the observability platform. The fields you keep should answer two questions: "who initiated this transaction, when, and through which channel," and "where in the system was it processed, and with what status."
Masking itself deserves care. Regex substitution works for fixed-format values like card numbers and phone numbers; a salted hash is right for identifiers you need to correlate across systems without revealing (account IDs, order numbers); and tokenization is the choice for anything that must round-trip back to the source system for remediation. Decide per field—over-masking destroys the signal you're trying to capture, while under-masking is a compliance finding waiting to happen at the next audit.
Store audit logs and business logs on separate pipelines. They have different retention and different access rules: audit retention runs to five years or more per regulation, while business logs can rotate after 30 days. Keeping them on separate pipelines also means a developer debugging a slow query doesn't accidentally gain read access to the compliance archive—a separation that auditors will explicitly look for.
On retention, be concrete in the contract: audit pipeline at 5+ years, business logs at 30 days, traces at 7 days, metrics at 13 months of raw plus 3 years of downsampled. Storing everything forever is both expensive and a compliance liability—retention is a feature, not a default.
Core financial systems are often a mix of legacy monoliths and new microservices, so full tracing can't happen overnight. Prioritize by transaction path: instrument the customer-facing gateway, core trading, accounting, and payment chains first, then expand outward. For legacy systems you can't convert to OTel yet, use a Collector at the boundary for HTTP/gRPC sidecar capture, pass the upstream trace_id through, and stitch together a semi-automatic service map.
Once the map is live, watch two metrics. The first is cross-system trace completeness—the fraction of core transactions where you can see the full path from gateway to settlement. The second is the P99 latency distribution of those chains, so you know where the tail actually lives instead of guessing. In our experience the bottleneck is rarely where the incident was first reported.
A useful intermediate step is to track trace completeness as a dashboard metric per system. If a given monolith sits at 40% completeness, you know exactly which integration to tackle next, and you can measure progress objectively instead of arguing about whether tracing is "done."
Incident response should run on data, not intuition. In every postmortem, split "alert to root cause" into three segments: detection (alert latency), localization (search and trace drill-down time), and remediation (change and rollback time). Measure all three with your observability data and attack the biggest one—you can't improve what you don't time.
Run a real fault-injection drill monthly. The goal is to verify alerts actually reach people and on-call engineers can localize root cause through search and traces, not just that the dashboard looks good in a demo. Inject a latency spike into a mid-tier service, then watch the clock: how long until someone has the offending span on screen?
Log the three segment times after each drill and chart them over months. Teams that do this consistently see localization time drop sharply in the first quarter, because the data exposes exactly which step—alert routing, search ergonomics, or on-call access—is the real blocker.
Phase one (2 weeks): get log collection and masking working to meet the audit baseline. Phase two (4 weeks): add tracing and golden metrics for core transaction chains. Phase three (2 weeks): wire up alerting and on-call escalation. Phase four (ongoing): use the data for capacity planning and postmortems. In short—first make things visible, then make them searchable fast, and only then chase prediction.