Observability for Multi-Active Financial Architectures: Unified Logging, Tracing, and Alerting Across Data Centers

When a financial core system moves to same-city dual-active or two-city three-center topologies, the hard part isn't the application refactor — it's answering "which data center, which call path broke" when something fails. Split logging across sites, traces that die at the gateway, and alert storms are the three recurring pains. Here's a unified approach. The goal isn't more dashboards — it's one place where an on-call engineer can trace a single transaction end to end across every data center in under a minute.

Three core pain points

1. Split logs: one ELK per data center means three consoles to check, and cross-site call logs never line up.
2. Broken traces: the trace context gets dropped at the cross-site gateway or message queue, so a transaction has no end-to-end view.
3. Alert storms: a failure in site A triggers cascading alerts in site B, burying the on-call engineer and hiding the root cause.

All three share one root cause: observability was built per-site, not as one system — so the fix is architectural, not a matter of adding another panel.

A unified ingestion architecture

Deploy a set of collectors (OTel Collector with local buffering) in each data center, all reporting to a central OBSERVE. The collectors carry a local disk queue, so data isn't lost during network blips and backfills automatically once the link recovers:

exporters:
  otlphttp:
    endpoint: https://ob.jjhub.cn/otel
    sending_queue:
      storage: file_storage
      queue_size: 10000
extensions:
  file_storage:
    directory: /var/lib/otelcol

For tracing, require every service to use the same SDK, and propagate the trace context explicitly at the gateway and the message queue — don't carry it only in HTTP headers. The MQ message header must also carry traceparent, otherwise the transaction breaks the moment it crosses onto the queue.

Keeping traces unbroken across the gateway

The most common break point is the message queue, because trace context doesn't travel in the payload — it has to ride in the message header. Concretely, the producer injects traceparent into the MQ header, and the consumer's SDK extracts it and starts a child span. On the producer side:

Map<String, String> headers = new HashMap<>();
GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
    .inject(Context.current(), headers, (map, k, v) -> map.put(k, v));
producer.send(new Message(payload, headers));

On the consumer side, extract before processing:

Context ctx = GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
    .extract(Context.current(), msg.getHeaders(), (map, k) -> map.get(k));
try (Scope s = ctx.makeCurrent()) { process(msg); }

Use the W3C traceparent format everywhere and forbid bespoke header names — mixed formats are how traces silently split. Gateways are the second break point: if your gateway rewrites headers, verify it whitelists the trace headers rather than dropping them.

Alert tiering and noise reduction

• Tag everything by data center and business domain. Write alert rules as "isolate by site first, aggregate by business second."
• Define dependencies so a site-A failure automatically suppresses site-B's cascading alerts, leaving only the root cause.
• For key metrics (transaction success rate, P99 latency), use cross-site comparison alerts: page only when one site deviates from the others beyond a threshold, so global throttling doesn't produce false positives. A practical rule: trigger when abs(rate_A - rate_B) / rate_B > 0.1 holds for five minutes.

Capacity and retention

In a three-center setup, plan storage around the busiest site plus headroom, not the sum of all sites — traffic shifts during a failover, and the surviving sites suddenly carry the load. Set retention per data class: raw logs for a few days of active triage, aggregated metrics longer, and sampled traces longest for capacity analysis. Review the sampling policy quarterly; tail-based sampling in particular can drift as traffic patterns change, and a policy tuned last quarter may now be dropping traces you need.

Compliance and data governance

Finance has hard requirements on retention and audit: encrypted log storage, automatic archiving by retention period, and query audit trails. OBSERVE supports field-level masking (card numbers and ID numbers masked automatically) and query auditing, satisfying dengbao and data-security requirements. Plan these during design, not before acceptance — reviewers will ask you to justify cross-site data flows, audit scope, and the masking field list. In a multi-site setup, decide early which fields are masked at the edge (before they leave a data center) versus centrally, because that choice affects both compliance posture and troubleshooting ergonomics.