An observability plan for e-commerce peak events: load-test-derived baselines, full tracing on critical paths, second-level aggregation alerting, capacity autoscaling hooks, and a before/during/after checklist.
Peak traffic is several to dozens of times normal, and it arrives in second-scale bursts. The problems you can find by paging through logs and staring at dashboards on a normal day — there's no time for that at peak. By the time you open the panel, the spike is over, or the money is already lost. The goal of peak-day observability is to turn “dig through logs afterward” into “know in the next second that something's wrong”.
The most important pre-peak action is load testing, but a load test isn't just “can it survive” — it's a baseline factory. Push the target TPS to 1.2x the expected peak and record each core endpoint's P99 latency, error rate, database connection count, and cache hit rate at that load. Those numbers are where the alert thresholds come from on the day — not a 90% CPU number someone pulled from the air, but water levels measured under real load.
Import the load-test data into OBSERVE and diff it against live traffic: if an endpoint's latency drifts 30% off its load-test baseline, that's the anomaly signal, even if the absolute number still looks “fine”.
A 5% trace sample rate is fine the rest of the year; during peak, switch the core paths (order placement, payment, inventory) to 100%. Yes, the cost goes up, but peak lasts a few days and buys you the ability to reconstruct any single order end to end. Pair it with tail sampling: slow and failing requests are kept at 100%, ordinary requests sampled proportionally, so you get both cost control and investigation power in one pipeline.
Peak-day alerting can't run at minute granularity. At 3,000 orders a minute, latency goes from 50ms to 500ms in thirty seconds — minute-level aggregation has already missed it. OBSERVE supports second-level aggregation: roll the three golden metrics — order success rate, payment latency, inventory deduction — on a 5-second window, threshold them against the load-test baseline, and page by phone the moment one breaks.
Alert tiers stay the same: P0 (order/payment path down, success rate below threshold) goes straight to phone; P1 (degradation, queue backlog) goes to SMS and group chat; P2 is logged. But during peak, the P0 response bar is seconds, so the page itself carries the high-latency trace and the log snapshot at the moment of the alert. The on-call engineer answers the phone already looking at the scene.
The observability system also has to answer a capacity question: when do we add machines. Turn cluster CPU, connection pools, and queue depth into trend dashboards, and set expansion thresholds ahead of time — for example, if gateway CPU stays above 70% for five minutes, trigger autoscaling. Again, the threshold comes from the load test, not from experience. The same hook that pages a human can call the autoscaler, so the system expands before a human even picks up.
An alert that isn't wired to a mitigation path is just noise with a nicer dashboard. For each P0 path, write down the two or three mitigations you're willing to take automatically or near-automatically: shed non-core traffic, degrade the recommendation feed, fail over to a read replica. During peak, the fastest correct action is usually “degrade a non-critical dependency and keep the core path up”, so decide those degradation points ahead of time and rehearse them in a chaos drill, not on the day. The alert carries the runbook link and the trace; the runbook should say what to degrade, not ask the on-call engineer to invent a plan at 3am under a traffic spike.
A related discipline: settle the “is it us, or is it the network / DB / partner” question before peak. Pre-define health checks for every external dependency — payment gateway, CDN, third-party inventory feed — and surface them on the same board as your own services, so a partner outage doesn't get misdiagnosed as an internal fault for twenty minutes. That twenty minutes is exactly the window where peak-day revenue leaks out.