Peak traffic is several to dozens of times normal, and every minute of failure costs money. This article shows how to find slow points with traces, locate limits with load testing, and tier alerts from SLOs.
The most common mistake in peak-season readiness is focusing only on capacity: add machines, add cache, add bandwidth. But what actually keeps people up on a big-sale night is not a lack of capacity — it is not knowing where the bottleneck is, when things start to slow down, or what to roll back first when something breaks. Those three are all observability problems.
Capacity can be estimated in advance through load testing, but production hotspots, slowdowns and dependency flapping cannot be simulated — they can only be caught by live traces and metrics. The observability platform's role in a sale is not about how many machines you can add; it is about how fast you can detect, locate and recover. A team that can add capacity in minutes but needs two hours to find a fault is still going to lose money on a sale night.
Before the sale, simulate peak traffic with full-chain load testing. Once the test data is in the platform, look not just at overall throughput but at each service's P99 latency and its dependencies. A typical finding: overall TPS hits target, but one downstream service's P99 is already at the timeout threshold — it will be the short board when real traffic arrives.
order → stock → promotion → payment → order persistence
↑ P99 1.8s (threshold 500ms)Once you have found the slow point, decide whether to add cache, add instances or change code — rather than scaling everything on instinct. The trace is what turns a vague "we should be faster" into a concrete "the stock service's database query is the problem, fix that". Without it, capacity work is guesswork distributed evenly across every service, and most of it is wasted.
During the sale, alerts need finer gradation than usual. Start by defining SLOs for the core path — say, order success rate ≥ 99.9% and P99 ≤ 500ms — then tier alerts by impact:
The key is to tighten thresholds a day ahead, so you are not waiting for real traffic to trip the alert. At the same time, silence irrelevant low-priority alerts so on-call only sees what actually needs handling. The goal is a single, short list of things that require a human — not a scrolling wall of notifications that everyone has learned to ignore.
SLOs also give the team a shared language for trade-offs. "Do we ship this feature now or wait until after the sale?" becomes a question about remaining error budget rather than a matter of opinion, which is exactly the kind of decision that should not be made on instinct at 10 p.m. the night before.
Fast recovery depends on playbooks, playbooks need rehearsals, and rehearsals need observability. Three common moves:
Build the metrics for these moves into a dashboard ahead of time, so you are not assembling data at 2 a.m. on sale night. A playbook that is not observable is a playbook you will not trust when it matters.