Promotion traffic is dozens of times normal and the window to respond is minutes. A practical checklist for making logs, traces and alerts work together at peak.
The difference between promotion day and a normal day is not the stack — it is the fault-tolerance window. On a normal day, if an endpoint is slow, users just retry and you can investigate tomorrow. During a promotion, traffic is dozens of times normal: one slow SQL can take down the whole order flow, and a few minutes means millions in losses.
So before changing anything, answer three questions. One: if the order flow stalls, can I pinpoint the offending service and method within five minutes? Two: if the database connection pool saturates, will I get an alert before customers complain? Three: during load testing, can I tell at a glance whether errors are concentrated in a few endpoints or systemic? Whatever you cannot answer is your gap this round — and the gap will not close itself at 11:59 p.m. on the night before.
The first rule of log work is unified fields. At minimum, every request log should carry four fields: trace_id, service, method and status. With those four, correlating logs across services is one SQL query instead of hand-grepping a dozen files.
SELECT * FROM logs WHERE trace_id = 'a1b2c3' ORDER BY ts ASCThe second rule is leveled logging. Reserve ERROR for things that need a human; expected business failures (like out of stock) are WARN. Do not let the log stream become mush where everything is red.
The third rule is structured logging. Put latency, downstream latency and return codes into key-value fields rather than burying them in a free-text message you have to parse with regexes later. A structured field is queryable; a sentence is not.
Run a full end-to-end load test before the promotion and watch two things. First, any node where P99 latency suddenly spikes — usually an unindexed query or a downstream call without a timeout. Second, the fan-out of each dependency: an endpoint that serially calls ten downstream services will inevitably queue at peak, even if each individual call is fast.
Grouping latency by service in the trace view is far faster than reading code to find the bottleneck. And during load testing, temporarily raise the sampling rate to 100% — sampling is exactly the thing that drops the slow requests you most want to see. A trace that was sampled away is a bottleneck that gets promoted to a production incident.
Machine metrics like CPU and memory are actually insensitive during a promotion — traffic can grow tenfold while they stay within limits, even as a downstream dependency times out. Attach alerts to business metrics instead:
The logic is simple: a machine can be fine while the business is bleeding. Alert on what the customer experiences, not on what the kernel reports.
Pair this with degradation switches: when an alert fires, degrade non-core features (recommendations, "you might like") with one click and give the resources back to the core flow. A switch you can flip in seconds is worth more than a perfect root-cause analysis that takes twenty minutes — during a promotion, seconds are the only currency that matters.
The night before is not the time to learn you sized for double the traffic and got ten times. Use the previous promotion's archived numbers as your baseline: peak QPS per service, P99 latency at that peak, and error rate. Size your pools, queues and downstream timeouts against those numbers, then load-test at two or three times the projected peak. The observability data you already have is the cheapest capacity-planning tool available — read it before you start buying.
After the promotion, archive peak QPS, per-service P99 and error rates as the baseline for next time. Calibrate alert thresholds and load-test targets against it. Observability is not a one-off project — it is an asset that compounds with every promotion, if you treat each run's numbers as the starting point for the next one.