An observability playbook for e-commerce peaks: traffic forecasting, full-chain load testing, real-time alerting, and fault drills across warm-up, flash sale, order and payment — with a rollout timeline and the metrics that matter.
A big promotion differs from a normal day not in "more traffic" but in "different traffic shape." During a flash sale, traffic spikes to dozens of times normal within seconds of the drop, then falls off fast; the warm-up, purchase, order, and payment paths take turns bearing load. Traditional monitoring with fixed thresholds either drowns the on-call channel during the peak or fails to react to the more dangerous signal — traffic suddenly dropping to zero. The goal is to know before the peak whether the system will hold, to detect anomalies within 30 seconds during it, and to reconstruct every order afterwards.
Two weeks out, get the projected peak from the business side — usually 3–5x the normal peak — then apply a 1.2x safety factor. Use that as the load-testing target. Full-chain testing doesn't hit one endpoint; it simulates real users walking "homepage → browse → cart → order → payment," with the load tool mixing traffic across stages in proportion. Watch three things: per-endpoint P99 latency within SLO, error rate under 0.1%, and the saturation of downstream dependencies (database, cache, message queue). Use isolated, dyed traffic tagged loadtest=true in the observability platform so you can filter and analyze it separately from real traffic.
Write your SLOs down before you test — "P99 under 300ms for the order endpoint" — so the load test has something to pass or fail against. And don't stop at the first failure: the point of load testing is to find the ceiling, so keep ramping until something breaks, fix it, and re-test.
Peak-time alerting should watch both ends: catch metrics spiking high (CPU, latency, error rate over threshold) and metrics collapsing — order and payment volume suddenly dropping to zero or halving is often more dangerous than a spike. Dynamic baselines shine here because peak traffic shape is nothing like a normal day, so fixed thresholds are guaranteed to blow up. Build a separate set of peak alert rules for the core paths (order, payment), route P1 alerts to the on-call lead via phone plus SMS, and target 1-minute acknowledgement and 5-minute localization. Put key metrics on a live dashboard so the war room watches real-time curves rather than scrolling through alerts. Rotate someone to watch the board and nothing else during peak hours — a dedicated pair of eyes beats a hundred alert rules.
Drill a week before the peak, not during it. The checklist should include at least: database failover, cache breakdown, message-queue backlog, and rate-limit degradation. For each, inject a real fault in the observability platform (e.g. chaos engineering to add latency) and watch whether alerts fire on time, whether the on-call engineer responds per plan, and whether the degradation switch actually works. The output of a drill isn't "we drilled" — it's an updated runbook mapping each failure to who owns it, what action, and which switch. Degradation switches deserve special attention: a switch you haven't tested is a switch that won't work when you need it.
Suggested cadence: T-14 finish forecasting and load-test prep, T-7 run full-chain testing and fix exposed bottlenecks, T-3 complete drills and update runbooks, T-1 freeze changes and put everyone on standby, peak day watch live. Among deployed customers, peak MTTR dropped from hours to minutes and flash-sale error rates stayed under 0.05%. Observability here isn't "find problems after the fact" — it's knowing beforehand whether you'll hold, reacting within 30 seconds during, and having the evidence to review after.