An Observability Playbook for E-commerce Flash Sales

The observability challenge of a flash sale

Flash-sale traffic is three to ten times normal volume, and it arrives in a burst of a few minutes. Code that behaves perfectly in normal conditions will expose connection-pool exhaustion, lock contention and GC pauses under a flood. The job of an observability system is not to let you read logs afterwards — it is to let the team see anomalies in seconds and localize the root cause in minutes. Everything below is what we have standardized across e-commerce customers; pick the pieces that match your architecture.

Before the sale: calibrate capacity and thresholds with load tests

Start load testing three weeks before the sale and feed the results back into two places:

1. Capacity baseline: from the peak QPS your tests measured, confirm the limits of database connections, thread pools and gateway concurrency, and leave 30 percent headroom. If the test shows a connection-pool ceiling at 1,200, do not run production at 1,150 during the sale.
2. Alert thresholds: switch thresholds from "daily values" to "sale values". If 70 percent CPU alerts you on a normal day, 85 percent is the abnormal bar during the sale — otherwise the alerts will drown you before the incident does.

Thresholds are not guesses; they are computed from load-test data. If your tests show P99 latency degrading at 90 percent CPU, set the CPU alert at 85 percent and the latency alert at the point where P99 starts to climb.

Before the sale: silence windows, on-call and a single dashboard

Configure silence windows for every release, warm-up and load-test period in advance, so false alarms do not drain the team's energy. Design the on-call rotation around a "golden five minutes": if the first-line engineer cannot localize the issue in five minutes, escalate immediately to a second-line architect. Agree on a "stop the bleeding first" principle — roll back, rate-limit or scale out to stop the loss, and only then dig into the root cause. Post this principle where everyone on call can see it, because under pressure it is the first thing people forget.

Also build one "sale-day dashboard" in advance: global error rate, the top five slowest services, queue depth and the current alert count, on a single screen. The on-call engineer opens this first; individual service pages come second.

During the sale: localize fast with tracing

During a sale, the worst situation is "one endpoint is slow, and dozens of services all have slow logs". Use distributed tracing to connect a full call chain by trace id and look at the time distribution directly: is it the database, a downstream call, or a GC pause? Observe aggregates slow requests into a Top-N of the most expensive spans — click one and you see exactly which method or which SQL statement dragged down the whole chain.

Fix the troubleshooting sequence in advance: check the overall error rate, find the service that spiked, pull one slow trace, locate the expensive span, then fix or degrade that path. Once the team has drilled this sequence, minute-level localization is achievable — and that is the difference between a five-minute blip and a fifteen-minute outage on a peak day.

After the sale: retrospective and reuse

Run a retrospective within 48 hours after the sale ends. Record which alerts were useful, which were noise and which thresholds need adjusting, and write them into a runbook for the next sale. Treat the observability configuration like code — version it, and iterate on it for each sale instead of starting from scratch every year. A runbook that survived one sale intact is worth more than any generic best-practices document.

The goal is simple: on a peak day, the team should spend its time shipping, not firefighting. A runbook that was rehearsed before the sale is how you get there.