A promotion is the hardest test of e-commerce observability. This covers load-test baselines, a live dashboard, alert noise reduction, degradation plans, full-chain load testing and auto-scaling, and how tracing informs degradation decisions.
A promotion is the single hardest test of an e-commerce observability setup. A system that runs fine day to day will expose its hidden bottlenecks all at once when traffic multiplies. This piece walks through the jobs to do before and during a promotion—load-test baselines, a live dashboard, alert noise reduction, degradation plans, plus full-chain load testing and auto-scaling—and how each one is actually done.
The first thing before a promotion is not adding machines—it is load testing. Use the observability platform to record metrics during the test and build a capacity baseline:
Once the test data becomes a baseline, you can compare live numbers against it on promotion day: the moment an endpoint's QPS approaches 80% of its baseline, you get an early warning instead of waiting for it to collapse. Make the test traffic look like real traffic—a read-only endpoint pushed to its limit tells you nothing about the write amplification inside the checkout chain.
Single-endpoint testing only exposes single-point bottlenecks. The real problem in a promotion is chain-level amplification—a checkout drags inventory, coupons, risk control, and payment down with it. Full-chain testing pushes real business traffic (or replayed production traffic) through the whole chain and watches the QPS and latency of every downstream, to find the shortest stave in the barrel. Replayed traffic must be redacted so real user data never reaches the test environment.
During the on-call shift, more charts on screen is not better. Watch five numbers: core checkout success rate, payment success rate, core endpoint p99, inventory and coupon service health, and error-rate spike alerts. Everything else you drill into on demand. This keeps noise from drowning the signal that matters. Build and rehearse the dashboard ahead of time—you do not want to discover on the day itself that a panel's data source was never connected.
When traffic doubles, alerts double too—and many are false alarms: CPU is high because of legitimate scaling, slow queries because of cache warm-up. Group alert rules by scenario in advance, and enable dedicated silence windows and aggregation policies for the promotion window. Keep only the alerts that affect transactions, so the person on call can see the real fire instead of a wall of notifications.
Once you have a baseline, scaling stops being a guess. Feed the core endpoints' QPS baseline into the scaling policy: scale out as QPS approaches 80%, scale back in when it falls below 40%. That absorbs the peak without burning money the rest of the time. Scaling must arrive before the traffic does, because new instances pay cold-start and connection-pool costs—don't wait until the service is already down.
The worst case in a promotion is a cascading failure. Tracing's value here is that when a downstream can't take any more load, the span tree shows exactly how many entry points "cutting this hop" would affect. Degradation decisions need data behind them: cut non-core side paths like recommendations and loyalty points first, and protect the checkout and payment main path. Degradation switches must be rehearsed ahead of time—you don't want to be hunting for the switch on the day itself.
The end of the promotion is not the finish line. Record the day's real peak, the bottlenecks, and the degradation decisions back into the baseline, and the next promotion's load-test targets and scaling thresholds become more accurate. The observability platform keeps this data, so during the review you can replay the metrics and traces of every critical moment on a timeline—far more reliable than memory. Promotion stability is built one round of this at a time.