Observability for Financial Transaction Systems: Golden Signals and SLOs

Why trading systems need different observability

For a trading system, reliability isn't "good enough" — every millisecond has a price. A payment that's 500ms late may already have lost the customer; an order that stalls at one node can cause reconciliation mismatches. Observability here isn't a nice-to-have; it's part of risk control and compliance.

The nature of these systems dictates the focus: traffic has sharp peaks and valleys, latency must be read from the tail rather than the average, errors must distinguish business failures from system faults, and every step must leave an auditable trace. Let's work through metrics, SLOs and sampling.

Golden signals: watch four numbers

Google SRE's golden signals fit trading systems well, with adjustments:

• Latency: skip the average; watch P95 and P99. Average latency can look great while 1% of slow requests drive away key customers. For trading, watch P99 directly and define "slow" explicitly, e.g. core payment P99 < 300ms
• Traffic: not just QPS — split by transaction type. Queries, orders, payments and refunds have very different volumes; mixing them hides anomalies in any single type
• Errors: separate error types. HTTP 5xx is a system fault, while business return codes (insufficient balance, risk rejection) are normal business outcomes. Lumping both into "error rate" produces constant false alarms
• Saturation: connection pools, thread pools, message-queue backlog. In trading systems, saturation usually degrades before latency does, making it a strong early-warning signal

In Observe, build a metric card set per core interface showing P99 latency, QPS, error rate and queue backlog side by side. But a dashboard isn't the destination — it only tells you "is it healthy right now". SLOs tell you "how many more changes can we survive this month".

Quantify reliability with SLOs

SLOs turn "reliable" from an adjective into a number. For a trading system, start with these three SLIs:

• Availability: successful requests / total requests, target 99.9%
• Core-interface P99 latency: target 300ms
• Transaction success rate: successfully settled / accepted, target 99.95%

An SLI must be defined precisely enough to code against. Does "successful request" mean HTTP 2xx, or "business return code is success"? Which table in which system defines "successfully settled"? A vague definition makes the SLO meaningless.

Once SLOs are set, the error budget follows. With a 99.9% availability target and a 30-day month, the budget is 0.1% × 30 days ≈ 43 minutes. The team can spend those 43 minutes on releases, canaries and experiments — and should stop when the budget runs low. That's the real point of an SLO: not punishment, but a quantified space for change, turning "can we ship this?" from gut feeling into arithmetic.

Let alerts follow the SLO

With an error budget, the alerting strategy should shift too:

• Burn rate: when the error budget is consumed unusually fast (say half of it in one day), there's a persistent problem that needs immediate attention
• Multiple windows: watch a 30-day window and a 1-day window together — the former tracks long-term trend, the latter catches sudden deterioration
• Fewer but better: alert on "budget nearly exhausted" rather than "error rate exceeded threshold for one minute", reducing false alarms at the source

This turns alerting from watching instantaneous metrics into watching your reliability balance — a natural complement to the dynamic baseline described in our earlier article.

Balancing sampling and compliance trails

Trading traffic is heavy, and full tracing is expensive to store, so sampling is required. But sampling can't be one-size-fits-all:

• Always sample errors: any trace with an error or timeout is kept unconditionally
• Always sample key transactions: payments, refunds and large transfers are traced at 100%
• Down-sample ordinary queries: healthy requests can be sampled at 10% or less, enough for trend analysis

The sampling policy itself must be auditable — you need to explain which traces were sampled and why. During a compliance audit, an enterprise customer will demand "the full trace of this specific transaction at this specific time"; if your sampling policy can't be explained, you effectively have no evidence. Observe supports rule-based sampling and writes sampling decisions into the log so they can be traced back.

Suggested rollout order: define SLIs and SLOs for core interfaces first, then build golden-signal dashboards, then tune sampling. Observability stacks layer by layer — don't chase "100% coverage with no blind spots" on day one; it's expensive and unnecessary.