Fixed thresholds misfire on metrics that naturally fluctuate, causing alert fatigue. Observe learns each metric's normal band and alerts only on clear deviation — with guidance on when fixed thresholds still make sense.
Setting alert thresholds is the most awkward job in operations. Set them too high and you miss real incidents; set them too low and you get paged at 3 a.m. Worse, most business metrics naturally fluctuate — QPS is high during the day and low at night, higher on weekdays than weekends, and spikes several-fold during a promotion. A single fixed threshold can only ever cover a small slice of the real curve.
Take a real example: an order service runs at about 500 QPS on a normal day but hits 3000 during a flash sale. Set the threshold at 800 and you're fine on ordinary days, but the moment the sale starts your phone won't stop buzzing; set it at 5000 and the sale is quiet, but you'll get no alert when the service loses half its capacity on a Tuesday. That's the structural flaw of fixed thresholds: they assume the metric is flat, and business metrics are never flat.
The result is usually one of two things: alert fatigue, where the team mutes the alert channel and misses real failures, or no alerts at all, where you learn about problems from user complaints.
Observe replaces fixed thresholds with a dynamic baseline: the system learns the "normal band" from the historical distribution of a metric and only fires when the current value clearly departs from it.
A dynamic baseline is not a simple moving average — smoothing alone can't learn what "normal" means. We decompose a metric into three layers:
Together they form a band that shifts with time and day of week, not a fixed line. Back to the order service: on the sale day, the system references last year's promotion and the recent traffic ramp to lift the band's upper bound, so 3000 QPS counts as normal and nothing fires. But if QPS suddenly triples at 3 a.m. — even to a modest absolute value — the alert fires, because "this busy at this hour" is itself an anomaly.
An alert only triggers when the value stays out of bounds for N consecutive samples — N is configurable and filters out momentary spikes.
alert:
name: "order-service QPS anomaly"
metric: order_service_qps
type: dynamic_baseline
sensitivity: medium # low/medium/high
consecutive: 3 # consecutive out-of-band samples
cooldown: 30m # cooldown to prevent alert storms
notify: ["email", "wecom"]Sensitivity controls how tight the band is: high means a narrower band that reacts to small deviations, good for low-noise metrics like latency; low is more forgiving, good for inherently jittery metrics like QPS. Unsure? Start at medium and watch for a week.
A dynamic baseline isn't a silver bullet. Keep fixed thresholds for these cases:
Our recommendation: fixed limits for capacity metrics, dynamic baselines for metrics that "breathe" — traffic, latency, error rate. The two coexist on the same service: "alert immediately when connections exceed 2000" runs alongside "alert when QPS deviates from baseline for 3 samples", each doing its own job.
Start with the metrics that misbehave most. In our experience, RPC latency, message-queue backlog and GC count are the worst offenders for fixed-threshold false alarms; switching just those to dynamic baselines typically cuts alert volume by more than half.
After the switch, keep a one-week observation period. If a day produces noticeably more alerts, first check whether something really happened (e.g. a canary release) before reaching for the sensitivity dial. Once the team trusts that the platform understands its business curve, migrate more metrics gradually.
The value of a dynamic baseline isn't fewer alerts — it's that every remaining alert is worth clicking. When the team stops being woken by "wolf" at night, the real daytime failures stop drowning in mute.