An observability blueprint for securities trading and market-data systems, covering low-latency tracing, aggregated quote monitoring, trade-log retention for compliance, and alert-threshold design.
Trading paths are unforgiving: every hop in an order, fill, or return is measured in milliseconds, and a timeout can mean a mis-executed order. Regulators also require complete trade-log retention so any issue can be traced back to a single order. Together these mean trading observability must answer not just "is there an alert" but "can we pinpoint one specific order".
Generic monitoring asks "is the system healthy overall". A trading system has to answer "at which hop did this particular order lose 30ms". Different goal, different design — and the difference shows up in every layer of the solution, from how you sample to how you store logs. A general-purpose observability stack dropped onto a trading system will almost always be either too noisy, too slow, or too coarse to be useful. The difference is not academic: a generic stack that reports 10-second rollups is useless when a fill path runs in 3 milliseconds, and a stack that samples 1% of requests will miss the one order that actually failed. Both failures stay invisible until they cost money.
Low latency and high sampling are fundamentally at odds, so trading systems should sample by business priority: 100% sampling for core order and fill paths, ratio-based sampling for market-data pushes and query endpoints. Use parentbased_traceidratio with a custom sampler that distinguishes strategies by interface prefix.
The key move is to inject business identifiers — order number, security code, client ID — into span attributes, so the trace view can retrieve one order's full path by its order number directly. Keep the instrumentation itself lightweight: add only the attributes you actually query, so you don't pile extra serialization cost onto the trading thread. The overhead of a well-tuned agent on the critical path should stay well under a millisecond; if it doesn't, trim your span attribute count rather than disabling tracing outright.
Market data is far too high-volume to report every tick. Instead, the collector aggregates locally and reports per-minute push latency, packet-loss rate, and snapshot-mismatch counts per security, which the platform then aggregates by market and sector. Anomalies typically show up as a sudden latency spike on one specific security code — a much earlier and more precise signal than the overall average, and it points straight at the offending feed source rather than a vague "the market is slow". Collectors can also keep raw tick data for a short rolling window of a few hours, so that when the aggregated metric flags a problem, you can drill into the exact ticks without having to store everything forever.
Use multi-level thresholds rather than a single red line on the trading path:
The point of tiering is that warnings can stay in the team chat while only critical events escalate to on-call, so nobody gets paged at 3 a.m. by low-severity noise. Attaching trace samples to critical alerts is what turns a vague "latency is high" page into a specific "here is the slow order and the slow hop". Measure your own baselines before setting these numbers — a 50ms P99 threshold that is right for one venue is wrong for another.
Trade logs must be retained for 15+ years per regulation, so use hot/cold tiered storage: 30 days of hot data for real-time search, with cold data archived to object storage and thawed on demand. Archived logs keep order number and trace_id, so they remain traceable years later. Apply the tiering labels at the collector — otherwise an archived log can't be retrieved by order number at all, and a regulator's request becomes a multi-day archaeology project. Retention is not just storage, either: your archive schema and searchable fields must survive 15 years of tool upgrades, so keep the archived fields simple and stable — order number, trace_id, timestamps, and the raw log line — rather than depending on a query DSL that may change.