Observability in finance must serve triage, audit, and transaction consistency at once. This post covers tracing a payment chain with trace_id, designing key metrics and alerts, and handling masking and audit logs.
Observability in a typical product mainly serves incident response. Finance adds two more layers on top: audit compliance and transaction consistency. A payment crosses gateway, risk control, accounting, and settlement — a dozen services or more — and when something goes wrong it isn't just "slow," it may be "the money doesn't add up." So monitoring can't stop at technical metrics; it has to watch business metrics like success rate, amount consistency, and reconciliation gaps, and every step must leave a trace.
The approach is straightforward: the gateway generates a globally unique trade_id for each payment request and lets it double as the trace_id propagated to every downstream service. Every span, from authentication to ledger entry, then threads onto one trace, and any single transaction can be reconstructed with one click.
Two things matter. First, propagate trade_id explicitly through message queues and async tasks — don't rely on thread context. Second, attach key business fields as span attributes, such as order.amount, order.channel, and order.status; filtering by amount or channel is far faster than filtering by latency when you're hunting a specific transaction.
Beyond technical metrics, build at least these business metrics:
Alerts should be tiered. A success rate below threshold is a P1 that pages the on-call engineer; a reconciliation mismatch is a P0 that escalates by phone. Put the trade_id and a trace link in every alert, so one click shows exactly which service and which span caused the problem — no more digging through logs.
Financial logs are full of sensitive data: card numbers, phone numbers, ID numbers. Mask it on the ingestion side before it reaches OBSERVE — regex out the middle digits of card numbers, keep only the first three and last four digits of phone numbers. Masking must happen before data lands in storage, not just as a display filter at query time; once unmasked data is persisted, you're already out of compliance.
Keep audit logs separate from operational logs: who queried which transaction at what time, and what configuration changed, must be recorded and tamper-proof. Audit logs follow regulatory retention requirements and should not share the expiry policy of your observability data.