Observability Readiness for Peak-Sale Events: Capacity, Tracing and Alerting

Every minute counts during a peak sale

On the day of a big promotion, traffic is three to ten times normal, and any component that buckles costs money by the minute. The value of an observability system isn't in the quiet weeks — it's in telling you exactly where things are stuck when the flood hits. This checklist spans four phases: capacity planning, load testing with traces, alerting strategy, and day-of response.

Capacity planning: work out the peak first

Start by doing the math. Estimate peak QPS and the volume of logs, metrics and traces it produces, then work backwards to what the observability system itself must absorb:

• Collection side: peak QPS × average span count × sampling ratio, to estimate spans per second written
• Storage side: estimate disk from data volume, and CPU/memory from query complexity
• Index side: estimate the size of individual indexes and adjust shard counts ahead of time

The observability platform is the bottleneck everyone forgets — you scale the business systems and the monitoring itself gets crushed first. Capacity planning must include the observability stack.

Load testing with traces attached

Load testing isn't just about making the business flow pass; it has to carry traces along. During a full-chain load test, collect traces at 100% and watch two things: the latency distribution across each trace, which pinpoints the real bottleneck service, and the observability system's own write latency and span-drop rate.

After the test, diff the trace data: comparing the same trace's latency at baseline versus under load shows at a glance which services degrade fastest under pressure, so you optimize them first.

Sampling and alerting strategy

Data volume spikes during a promotion, so set sampling policy ahead of time:

• Sample normal requests at a low ratio (e.g. 1%) and keep 100% of abnormal ones
• Raise the sampling ratio on core transaction paths (order placement, payment) so critical traces stay complete
• Alert on P99 rather than averages, which a flood of normal requests will dilute

On the alerting side, run a fire drill before the event: inject a fault on purpose and confirm the alert fires, reaches a human, and that the handler can locate it fast. Sweep your silence windows beforehand so a real alert doesn't get silently swallowed.

Day-of incident response

On promotion day, the observability platform is the war room. We recommend:

• Put the core dashboard on a big screen, watching end-to-end P99 latency, error rate, and per-service CPU/memory
• Route alerts by severity: fatal alerts go straight to phone, warnings go to a channel, so the critical signal isn't drowned
• Fix the triage flow: alert → trace → correlated logs → locate service → remediate, with a tool for each step

Post-mortems matter just as much. After the event, use the trace data for a full review: find the ten slowest service calls and the highest-error endpoints, and turn them into the checklist for the next round.