Observability for Securities: Millisecond Alerts on the Trading Path and Audit Compliance

Observability for securities firms has two unusual requirements. First, trading paths are extremely latency-sensitive — a few dozen milliseconds of jitter can trip risk controls or disrupt matching. Second, regulators require key logs to be traceable, tamper-evident, and retained for years. Neither constraint maps cleanly onto the playbooks used by consumer internet companies.

Instrumentation and baselines on the trading path

A typical securities chain looks like: market-data gateway → order entry → the OMS (centralized trading) → matching → confirmation. Instrument every hop, and always attach two identifiers: order_id and session_id. Without them you can't stitch a single order's context back together when something goes wrong.

Use manual OpenTelemetry spans and instrument only the critical boundaries — don't blanket-instrument the whole path, because every extra span adds a few milliseconds on a path where milliseconds matter:

ctx, span := tracer.Start(ctx, "oms.match",
    trace.WithAttributes(
        attribute.String("order_id", orderID),
        attribute.Int64("qty", qty),
    ))
defer span.End()

Don't alert on fixed latency thresholds. Use a baseline-plus-deviation model: compute each endpoint's P95 over the trailing 14 days, then fire when the current P95 exceeds 2x the baseline for a full minute. Fixed thresholds (like "alert over 200ms") false-alarm during market spikes and are uselessly loose off-hours — you'll never tune them right.

Alert tiering and escalation

Securities alerts must be tiered, or a market-data hiccup drowns out a real outage:

• P0: trading path down, matching unavailable — phone + SMS, respond within 1 minute;
• P1: latency over baseline, degradation active — SMS + WeChat Work;
• P2: single-node anomaly, resource-level alerts — WeChat Work only.

Spell out the escalation rules: a P1 unacknowledged for 5 minutes auto-escalates to P0; P2 alerts are silenced outside trading hours. Configure this once in JUJING OBSERVE's tier-plus-escalation policy rather than relying on people manually forwarding alerts.

Audit and compliance retention

Regulators typically impose two hard requirements: log retention of at least 3 years (5 for some business lines), and tamper-evident storage for key operational logs. The practical approach is a two-tier split:

• Hot data (last 30 days) sits in the search tier for day-to-day troubleshooting;
• Cold data archives to object storage, with daily hash digests kept offline in a chain so post-hoc tampering is detectable.

Account operations (login, permission changes, data exports) get a separate audit log with operator, timestamp, source IP, target object, and before/after values, stored apart from business logs. Rehearse audit-log recovery regularly so you don't end up with data you can't actually retrieve.

Sequencing the rollout

Don't rebuild the entire chain at once. Pick one core path — ordinary A-share order placement is a good start — instrument it deeply, get alerts running, and let it run for a full trading week before expanding sideways. The value of observability shows up during incidents, but the cost is paid up front in instrumentation, so doing it in batches is far less risky than a big-bang cutover.