Inside OBSERVE's Alerting Engine: From Thresholds to Storm Control

Alerting isn't “fire when a number crosses a line”

Static thresholds are the most common source of alerts — and of false alarms. CPU hits 90% and it pages; the nightly backup job runs and it pages. After three months on call, people mute those alerts, and the moment alerts lose credibility the whole monitoring setup is worthless.

The core job of OBSERVE's alerting engine is to translate an abnormal signal into “notify the right person, at the right time, through the right channel” without drowning them. That decomposes into three layers: how a rule is expressed, how duplicate noise is suppressed, and how a notification escalates. Get any one of the three wrong and you either wake someone up at 3am for nothing, or bury a real incident under a wall of pings.

The rule model: multi-condition + duration + relative baseline

A rule is far more than “metric > threshold”. A full expression supports four capabilities:

• Multi-condition: error_rate > 0.02 AND qps < 100, with arbitrary AND/OR nesting. You can require, for example, that an error spike only counts when traffic is below a floor — otherwise a retry storm from a single client looks identical to a real outage.
• Duration: for: 3m — the metric must stay out of bounds for three minutes before firing, which filters out momentary spikes from a cache flush or a deploy restart.
• Relative baseline: compare against the same period last week, or the previous five minutes, instead of a hard-coded absolute value.
• Cross-metric: watch error rate and latency together, and only count it as abnormal when both move. A slow-but-healthy endpoint and a fast-but-erroring one each get caught, but the noisy middle doesn't page.

A typical rule looks like this:

rules:
  - name: order-api-degraded
    expr: error_rate{service="order"} > 0.02 AND p99{api="/order/create"} > 800ms
    for: 3m
    labels: { severity: P1, team: order, runbook: "runbook/order-degraded" }

for is the first gate against false alarms. In production, use 1-2 minutes for P0 and 3-5 minutes for P1; too short and spikes trip it, too long and response is delayed. Tie the duration to how fast a human can actually respond — there is no point alerting in 30 seconds if the on-call needs five minutes just to page in.

Noise reduction: grouping, suppression, silence

Alert storms are the number one on-call killer. One upstream service dies, dozens of downstream services alert, and the phone blows up. The engine has three built-in tools:

• Grouping: alerts with the same labels (service, cluster) collapse into one notification, sent once. Ten failing replicas of the same service become a single message with a count, not ten messages.
• Suppression: when a P0 fires, it auto-suppresses P1/P2 from the same source — “gateway down” suppresses “gateway latency high”, because the latency alert adds nothing once the outage is known.
• Silence: pre-schedule quiet periods for release windows or planned maintenance; alerts in that window aren't notified but are still recorded in the audit log so nothing is invisible later.

Stack all three and a real incident typically produces one P0 notification plus a summary, not fifty messages. The discipline behind it is worth stating: suppression is not “hide the problem”, it is “don't spam me with consequences of a problem I already know about”. The suppressed alerts are still queryable and still counted, they just don't page.

Routing and escalation

Channels cover SMS, phone, WeCom, DingTalk, and email. The key is the escalation chain: a P0 pages the on-call engineer, and if it isn't acknowledged within five minutes it escalates to the second line, then to the team lead, and so on up. On-call rotations are weekly and the engine resolves the current on-call automatically — nobody edits contact lists by hand. A hand-edited contact list is exactly what goes stale the week someone changes teams.

Notifications carry the alert graph, the linked traces, and the runbook URL. The on-call engineer can drill down straight from the message instead of asking “where's the outage and what do I do”. That single change — putting the evidence and the procedure inside the page — cuts more MTTR than any dashboard redesign ever will.

Practical advice

• Every rule must carry a runbook link; don't ship an alert without a procedure.
• Prefer relative baselines for thresholds; reserve absolute values for metrics with an explicit SLO.
• Review alerts monthly and delete any rule that hasn't fired in a month — an unused alert is noise, not a safety net. A monitoring system people trust is one where every page means “act now”.