Observability for Securities Trading Microservices: A Practice Report

Background and pain points

After a broker split its core trading monolith into a dozen-plus microservices—gateway, order, matching, clearing—mean time to locate jumped from minutes to hours. A failed order could be a gateway timeout, order-service rate limiting, or a matching rejection, and nobody could tell by grepping logs. Worse, the 30 minutes before market open is the day's critical window, where any jitter balloons into an incident.

A unified traceId across the whole chain

The first step was threading a traceId through every service. The gateway generates a traceId at the entry point and propagates it via HTTP headers; every service prints traceId and spanId in its logs. Any order can then be traced from its order number back to a traceId, then down to the specific service and call segment.

OBSERVE correlates traces with business fields like orderNo: you search logs by orderNo to get the traceId, then drill into the full trace. After rollout, mean time to locate dropped from 40 minutes to under 5—most issues show up in the first span as either a timeout or a rejection.

Alert governance during the market-open peak

Order volume at the open is 10x normal, so static thresholds would spam false alarms. We made three changes:

1. Alert baselines are computed dynamically from "same time yesterday + same time last week" instead of hard-coded thresholds.
2. Added a silence window: during the 9:15–9:25 call auction, alert sensitivity is lowered so matching jitter doesn't flood the channel.
3. Separated "single order failure" from "success-rate drop"—only a success rate below 99.9% for two consecutive minutes triggers P1; single failures go to a queue without paging anyone.

Alert tiering matters just as much: P1 pages via phone + SMS, P2 goes to WeCom, P3 lands in email. Without it, the first alert storm at market open makes the on-call engineer mute the channel—and the genuinely serious alerts get buried.

Designing SLOs

We defined three SLOs for the trading path: order-API P99 latency ≤ 200ms, order success rate ≥ 99.99%, and matching confirmation latency ≤ 1s. SLOs use a 28-day rolling window; when the error budget runs low, we pause non-urgent releases instead of waiting for problems to erupt. This gave dev and ops a single shared ruler, and let us predict before a release whether it would burn the budget.

SLOs also need a companion action: burn-rate alerting. When the error budget is consumed quickly—say 10% burned in one hour—that surfaces problems much earlier than watching the success rate drop after the fact, letting you intervene before customers notice.

Lessons learned

• Get traceId working before you talk about alerts—alerts on a broken trace are just noise.
• Peak-time alerts must be dynamic; static thresholds are nearly useless in finance.
• Attach SLOs to business metrics (success rate, latency), not infrastructure metrics like CPU or memory.
• Observability isn't a one-time project: sampling rates, alert baselines, and SLO thresholds all need ongoing tuning as the business changes.

Sampling at peak load

Full trace collection during the open is neither possible nor necessary. We sample normal requests at 5–10% but always keep slow requests (P99+ candidates) and all failed orders at 100%, using the Collector's tail-sampling policy keyed on latency and status code. Failed orders are the highest-value signal in trading—a dropped trace on a failed order means an unreproducible incident—so they are never sampled away.

We also feed the SLO dashboards from the unsampled metrics stream rather than the sampled traces. SLO math needs accurate counts; deriving a success rate from a 10% trace sample would inflate the error-budget uncertainty and make burn-rate alerts unreliable. Metrics are cheap to keep whole, so keep them whole.

On the operational side, every release is gated on three numbers visible in a single dashboard: P99 order latency, order success rate, and the remaining error budget for the current 28-day window. If a canary release starts burning budget faster than normal, we roll back before the full deploy. That one screen replaced a dozen scattered monitoring views and noticeably cut the post-release incident rate.

The key takeaway: in trading, observability is measured by how fast you can answer "is this order stuck, and where," not by how many dashboards you have. Everything we built—traceId propagation, dynamic baselines, SLOs—serves that one question.