SLOs and Error Budgets: Turning Service Reliability into a Measurable Number

Why “99.9% availability” gets ignored

“Our SLA is 99.9%.” It’s written into the contract, but almost nobody actually watches it. The reason is simple: 99.9% is a monthly target. By the time you calculate at month-end that you missed it, the incident happened twenty days ago. Assigning blame is pointless, and improvement never starts.

SLOs take a different approach. They break availability down into indicators you can watch in real time, so a team knows every day — every hour — how far it is from the line. An SLO isn’t a report for executives; it’s a dashboard for the on-call engineer. The moment you stop defining reliability as a feel and start defining it as a number, the whole conversation changes.

SLI, SLO, error budget: three words, three jobs

These three terms get thrown around loosely, so let’s pin them down with one running example — a checkout service.

• SLI (Service Level Indicator) is a measurable property of your service, like “the share of requests whose P99 latency stays under 300 ms over 5 minutes.” It has to be computable, not an adjective. “Fast” is not an SLI; “99.5% of requests under 300 ms” is.
• SLO (Service Level Objective) is the target for an SLI, like “that share stays at or above 99.5% over a month.” Once an SLO is set, the team has an explicit line: below it, reliability work is urgent; above it, it’s optional.
• Error budget is what’s left after you subtract the SLO from 100% — in this case 0.5%. It’s your allowance for failure for the month. While the budget lasts, you can ship and experiment; when it’s nearly spent, you freeze changes and fix reliability.

String these three together and reliability stops being a slogan and becomes something you can keep a ledger of.

Defining an SLO in OBSERVE

In OBSERVE you don’t hand-write PromQL to create an SLO. You pick a metric, a window, and a target, and the platform wires up the rest. Here’s an SLO definition based on request latency:

slo:
  name: checkout-api-availability
  sli:
    source: prometheus
    query: >
      sum(rate(http_request_duration_seconds_bucket{le='0.3',service='checkout'}[5m]))
      /
      sum(rate(http_request_duration_seconds_count{service='checkout'}[5m]))
  objective: 99.5
  window: 30d
  budget:
    fastBurnThreshold: 14.4   # exhausts the monthly budget in 1 hour
    slowBurnThreshold: 3.0    # exhausts it in 6 hours
  labels:
    team: checkout
    severity: page

Once defined, OBSERVE continuously computes the current SLI attainment and remaining error budget, and draws the budget-burn curve on a dashboard. An on-call engineer can see at a glance how much “mistake allowance” is left this month — and whether the team can afford to ship that risky refactor tomorrow.

Burn-rate alerts: don’t wait for the budget to run dry

Watching only the remaining budget has a flaw. The budget bottoms out at month-end, so a mid-month spike in burn speed is easy to miss until it’s too late. OBSERVE solves this with burn rate — the actual burn speed divided by the allowed average burn speed.

With a monthly budget of 0.5%, you can burn roughly 0.017% per day on average. If you burn 0.17% in a single day, the burn rate is 10x. OBSERVE alerts in two tiers:

• Fast burn — the budget will be exhausted within an hour, triggering a P1 page straight to the on-call engineer.
• Slow burn — the budget will be exhausted within six hours, opening a ticket to be handled during business hours.

The payoff is an alert within minutes of a failure, not a report at month-end. For a team that has been burned by silent outages, this is the difference between catching a regression on Tuesday and explaining it to customers on Friday.

Getting started

Three pieces of advice for teams new to SLOs:

1. Start with one or two, not ten. Pick your two or three most critical services, get SLOs running smoothly, then roll out. A half-finished SLO rollout across fifty services teaches nobody anything.
2. Align SLOs with the business. A beautiful indicator nobody feels is worthless. Prefer SLIs that map directly to user experience — latency and error rate beat CPU and disk.
3. Actually spend the error budget. While it lasts, let teams ship releases and run canary experiments on purpose. That’s the real value of an SLO — it gives you room to fail, not a list of people to blame.