Inside the Observe Alert Engine: Thresholds, Silences and Storm Suppression

Why alerts get treated as noise

The on-call engineer's worst enemy isn't a missing alert — it's too many of them. When one service hiccups and dozens of instances time out at once, an alert per instance floods the phone. Worse, once an alert is proven to be a false positive, people instinctively ignore the next real one. This is alert fatigue, and it compounds: every noisy page makes the whole channel less trustworthy, until a genuine outage scrolls past unnoticed.

The value of an alerting system isn't "how many it can send", but "whether the right one fires and actually gets delivered". Observe's alert engine is built around exactly that: configurable rules, evidence-based triggering, and converged delivery.

The three-layer rule structure

An alert rule has three parts:

1. Source: logs, metrics, or traces. Logs match by keyword or error rate, metrics by threshold, traces by duration or error-span ratio.
2. Condition: a comparison operator (>, <, =, percentage) plus a threshold and a time window — for example, "5xx error rate above 1% over 5 minutes".
3. Action: notification channels (email, webhook, WeCom) plus severity (P1/P2/P3) and optional auto-escalation.

A rule in YAML looks like this:

alert:
  name: order-service-5xx
  source: metrics
  query: 'sum(rate(http_5xx{service="order-service"}[5m])) / sum(rate(http_total{service="order-service"}[5m]))'
  condition: "> 0.01"
  for: 5m
  labels: { severity: P2, team: order }
  notify: [wecom, webhook]

for: 5m is the part that matters: the condition must hold for five minutes before the alert fires, so brief spikes are filtered out automatically instead of becoming false alarms.

Silence windows and deduplication

Two easy-to-miss but highly useful settings:

• Silence: mute related alerts during release windows or planned maintenance, then resume automatically. No more "alerts flood the channel during deploys and nobody reads them".
• Deduplication: a given alert isn't re-sent until it recovers. You get notified on state change — once on trigger, once on recovery — rather than every five minutes.

Picking thresholds: absolute, percentile, or error budget

The threshold you choose shapes how noisy the alert is. Three common styles:

• Absolute count ("more than 50 errors in 5 minutes"): simple, but breaks as traffic grows — a doubling of traffic doubles errors and starts paging for no reason.
• Percentile latency ("P99 > 800ms"): tracks user experience directly, but needs a baseline and some care when traffic is low.
• Error budget ("error rate burns through 20% of the monthly budget in an hour"): the SRE approach; it ties alerts to what actually matters and avoids pages for trivial dips.

For a service with steady traffic, a ratio (error rate) is usually the right default. Reserve absolute thresholds for things where any occurrence is bad — a failed settlement job, for example.

How the engine evaluates rules

Under the hood, each rule is evaluated on a fixed cadence (default every 15 seconds) against its recent time window. A rule moves through three states: pending while the condition is true but the for window hasn't elapsed, firing once it crosses the window, and resolved when the condition clears. State changes — not repeated evaluations — are what trigger notifications, which is how deduplication falls out naturally rather than as a bolt-on feature.

Escalation is stateful too. A P2 that stays open for 30 minutes can automatically page a second channel or bump to P1, with no extra glue code. Most teams configure escalation once and then forget about it, because the rule keeps working through the night while nobody is watching the dashboards. That is the point of a good alerting system: it should hold the line while you sleep.

Storm suppression

This is where the engine earns its keep. When one failure hits 50 instances at once, the engine:

1. Groups alerts by labels like team/service/region.
2. Converges them into a single summary alert that lists the affected-instance count and the top 3 instances, instead of 50 separate ones.
3. Correlates same-root-cause alerts in the same window into a primary + secondary pair, so resolving the primary auto-closes the rest.

The result: one actionable page instead of fifty identical ones.

Practical tips

• Don't set thresholds too aggressively at first. Watch two weeks of normal fluctuation, establish a baseline, then set thresholds above it.
• Give every rule an owner and a runbook link, so whoever gets paged knows whom to contact and what to do.
• Review alert history periodically: rules that fire often but are never acted on should be deleted or downgraded, not left to pile up as noise. A rule nobody responds to is just a tax on the on-call rotation.