Beating Alert Fatigue: Tiered Alerts, Suppression and On-Call Rotation in Observe

Why alerts cause fatigue

The most common way an alerting system fails is not staying silent — it is going off too much. A momentary CPU spike, or a network timeout that recovers on its own, can trigger an alert and wake someone at 3 a.m. Over time the person on call becomes numb to alerts, mutes the notifications, then mutes the whole system, and the real incident drowns in a sea of noise.

The fix is to make every alert actionable: when it fires, the person on call should know exactly what to do, not wonder whether it is a false positive.

Tiering: three levels of severity

Observe ships three alert levels out of the box:

• P0 critical: directly affects availability — error rate above 5%, or P99 latency of a core endpoint doubling. P0 goes out over phone, SMS and WeCom at the same time.
• P1 warning: a degrading trend, such as disk usage exceeding 85%. Delivered over WeCom and email.
• P2 info: informational, like a slow log from a nightly job. Visible only in the console, never interrupting anyone.

A threshold rule looks like this:

alert:
  name: order-service-error-rate
  level: P0
  metric: http_error_rate
  condition: '> 0.05'
  duration: 5m
  notify: [phone, sms, wecom]

The duration: 5m matters a lot: the metric has to stay above the threshold for five consecutive minutes before the alert fires, so a transient spike never pages you.

Consider a real case. An e-commerce team sets a P0 rule on the order endpoint with a 5% threshold and a 5-minute duration. At 2 a.m. a burst of payment-callback retries pushes the error rate to 6%, but it falls back within three minutes — the duration window keeps that blip from paging anyone. The alert that actually fires comes the next morning, when the error rate sits at 8%, and the on-call engineer jumps in before it turns into daytime customer complaints.

Suppression and silence: shutting down the noise

Two mechanisms filter out most of the noise:

1. Alert suppression: when a service goes down, the dozens of alerts from its downstream dependencies are suppressed, leaving only the root cause. One incident should never produce a hundred notifications.
2. Silence windows: set a quiet period for planned maintenance or releases. Alerts inside the window are automatically downgraded to P2, so a deploy does not page the whole team.

On-call rotation and escalation

• Schedules: weekly rotation with automatic handoff, and a roster visible to the whole team.
• Escalation: a P0 that is not acknowledged within 10 minutes escalates to a secondary on-call, so nothing slips through.
• Acknowledgement: the on-call confirms or resolves the alert, closing the loop.

From alert to recovery

A good alert does not just fire — it walks you to recovery. Observe ties every alert to its context: the metric curve at trigger time, the related error logs, and the traces from the same window. When the on-call engineer opens an alert, there is no hopping between three dashboards hunting for clues.

Together with acknowledgement, reassignment, comments and resolution states, the full lifecycle of an incident is recorded from detection to closure. In a postmortem you can replay by alert: how many fired, how long each took to acknowledge, and what the root cause turned out to be.

Choosing thresholds that don't lie

A flat "CPU above 80%" rule fires on every healthy traffic peak. Two adjustments help. First, use duration windows so only sustained conditions page. Second, prefer ratios and error rates over raw counters — a 5% error rate means something whether you serve ten requests a second or ten thousand. For metrics with daily or weekly seasonality such as batch jobs or payday traffic, switch to anomaly detection that compares against the same time last week instead of a fixed number.

Practical recommendations

1. Start with P0 rules on your core services and run them for a week, watching the false-positive rate.
2. Treat alert volume itself as a metric: healthy teams sit around a 5:1 to 10:1 alert-to-incident ratio. Higher means your rules are too sensitive.
3. Review alerts regularly, and delete or downgrade anything nobody acts on.

The goal of alerting is not that everything fires — it is that everything that fires deserves a look.