From Thresholds to Convergence: Inside the Observe Alerting Engine

Solve “nobody reads alerts” first

The most common complaint about monitoring isn't “it misses problems” — it's “it texted me forty times at 3 a.m.” Thresholds set loose enough to avoid noise will miss incidents; set tight enough to catch them and they page constantly. Operators end up either muting everything or reflexively clicking “ack”, and real failures drown in the noise. Consider a service whose error rate sits at a healthy 0.1% during business hours but times out sporadically in the low-traffic hours before dawn: a threshold calibrated to the daily average will never fire for those periods, and the on-call engineer sleeps through a real degradation. Observe's alerting engine is built around three goals — controllable false negatives, converging false positives, and traceable handling.

Four alert types cover most cases

The first is threshold alerts: CPU, memory, error rate, or P99 latency crossing a configured value, which suits resource problems. The second is keyword alerts: a specific pattern appearing in the log stream — an OutOfMemoryError, a connection pool exhausted, a payment callback that keeps failing — triggers an alert, which is useful for business anomalies you can't express as a single metric. The third is period-over-period alerts: comparing against the same period last week or the previous window and firing only when the deviation exceeds a margin, which catches “not past the threshold but clearly wrong” failures such as a weekend traffic curve that has quietly shifted. The fourth is composite alerts: several conditions combined with AND/OR, such as “error rate above 5% AND QPS not dropping”, which filters out the noise of a single jittery metric. All four types accept a severity level — info, warning, critical, fatal — so different levels can route to different channels and different on-call rotations.

Suppression, deduplication, escalation: quiet the storm

The real problem with alerting is the storm — when the database dies, dozens of downstream services all report “connection timeout” at once and bury the operator. Observe converges this with three mechanisms:

• Suppression: once you define a suppression chain, a root-cause alert (like mysql-down) silences all downstream alerts, leaving only the one that matters
• Deduplication: the same alert notifies once per window; repeat triggers just increment a counter instead of re-paging
• Escalation: if an alert stays unacknowledged for N minutes, it escalates to a higher priority or a phone channel so it doesn't silently expire

Scheduled releases and maintenance windows can be skipped entirely with silence rules, so you don't page yourself during a deploy you already know about. A single rule reads like this:

alerts:
  - name: order-service-error-rate
    metric: http_error_rate
    condition: "> 0.05"
    window: 5m
    dedup: 10m
    suppress_by: [mysql-down]
    escalate_after: 15m
    notify: [dingtalk, webhook]

A closed loop, not a black hole

Alerts go to WeCom, DingTalk, Feishu, email, webhooks, or SMS. Every alert carries a trace id and a log snapshot; clicking the link jumps straight to the relevant trace, so no one has to grep logs by hand at 3 a.m. Who handled it, how long it took, and the outcome are all recorded, forming a traceable handling loop and feeding data for post-incident reviews. Tie alerts to your SLOs and you can watch the error budget burn down in real time, which turns the “should I wake this person up?” decision from a gut call into a data point. This is what separates a pageable alert from background noise: the receiver can act on it without leaving the platform.

Three common mistakes

• Alerting on every metric: that just produces alert fatigue. Only alert on things that require a human action.
• Watching averages: average latency hides the long tail. Base alerts on percentiles (P95/P99) instead.
• Sending without tracking: with no acknowledgement, no escalation, and no post-mortem, an alert might as well not exist.

Getting started

Don't rush to configure a wall of thresholds. Start with just two: user-visible error rate and core-dependency availability. Run for a week, watch the false-positive rate, then tighten gradually. Once that's stable, layer on alerts for the few SLOs you actually track — the ones whose breach means a customer noticed something wrong. An alert's value isn't in its volume — it's in whether each one makes someone take an action.