A product team wants to release a new checkout feature.
The service team is concerned because production has experienced several failures during the last week.
The product team says:
The release is important.
The service team says:
The system is unstable.
Both statements may be true.
Neither provides a shared decision boundary.
How unstable is the service?
How much reliability has already been lost?
Which risks remain acceptable?
At what point should feature delivery pause?
Without an agreed model, the conversation can become a contest between urgency and caution.
An error budget gives both sides a common piece of evidence.
Suppose the checkout service has this objective:
Over a rolling 28-day period, at least 99.9% of eligible checkout attempts must complete successfully within four seconds.
The objective permits 0.1% of eligible events to be bad.
That 0.1% is the error budget.
SERVICE-LEVEL OBJECTIVE
99.9% good events
ERROR BUDGET
0.1% bad events
If the service handles 1,000,000 eligible checkout attempts during the window, the budget permits:
1,000 bad events
The team can now ask:
How much of that allowance
has already been consumed?
That question turns reliability into something that can influence engineering work.
An error budget makes reliability negotiable by defining how much failure the service can tolerate before the balance between delivery and reliability must change.
Negotiable does not mean unimportant.
It means the organisation has stopped pretending that the only acceptable number of failures is zero while continuing to make changes that carry risk.
Reliability already contains a trade-off
Every system could theoretically be made more reliable.
The organisation might add:
- more redundancy
- additional regions
- larger capacity margins
- slower deployment processes
- more testing
- stronger isolation
- more operational staff
Each improvement has a cost.
It may consume:
- engineering time
- infrastructure spend
- delivery speed
- architectural complexity
- operational attention
A 100% reliability target leaves no formal room for failure.
TARGET
100%
PERMITTED FAILURE
0%
That sounds ideal until the team tries to operate a distributed system beneath it.
Networks fail.
Dependencies become unavailable.
Deployments introduce defects.
People make mistakes.
Capacity assumptions prove wrong.
The organisation still wants to change the service.
The practical question is therefore not:
Can we eliminate every possible failure?
It is:
Which level of failure can users tolerate,
and which investment is justified
to remain within that boundary?
The SLO defines the boundary.
The error budget expresses the room inside it.
The budget is derived from the promise
The error budget should not be selected independently.
It follows from the SLO.
For a 99% objective:
error budget:
1%
For a 99.9% objective:
error budget:
0.1%
For a 99.99% objective:
error budget:
0.01%
Each additional nine sharply reduces the permitted failure.
For a 30-day time-based availability objective:
99%
about 7 hours 12 minutes
99.9%
about 43 minutes 12 seconds
99.99%
about 4 minutes 19 seconds
The difference is not decorative precision.
A team operating at 99.99% has far less room for:
- failed deployments
- dependency outages
- recovery delay
- maintenance impact
- capacity mistakes
A stricter SLO should therefore follow a real user or business requirement.
Otherwise the team may spend heavily defending a promise nobody needs.
The budget is not permission to schedule failure
An error budget can be misunderstood as:
We are allowed to break production
for this amount of time.
That is not the purpose.
The budget represents the amount of user-visible failure the service can experience while still satisfying its objective.
Failure may come from:
- software defects
- infrastructure incidents
- deployment problems
- dependency failure
- slow requests
- operational mistakes
- capacity exhaustion
The team should not attempt to spend every remaining unit.
ERROR BUDGET
is a risk allowance
not
an outage reservation
Unused budget creates confidence that the current delivery approach is compatible with the service promise.
Rapidly consumed budget indicates that the system or change process is taking more risk than the objective permits.
Healthy budget supports continued change
Suppose checkout has consumed only 20% of its error budget halfway through the 28-day window.
The service is performing comfortably within its objective.
BUDGET USED
20%
BUDGET REMAINING
80%
That does not guarantee the next release is safe.
It does suggest that the team has room to continue ordinary delivery.
The organisation may allow:
- routine releases
- controlled experiments
- dependency upgrades
- infrastructure changes
The budget creates room for innovation because the reliability promise is still being met.
This is one of the most useful aspects of the model.
Reliability is no longer used only to stop change.
A healthy budget can support change.
RELIABILITY HEALTHY
│
▼
CONTROLLED DELIVERY CONTINUES
The budget protects both stability and progress.
Budget consumption should change behaviour
An error budget becomes useful only when its condition affects decisions.
A simple policy might define three states.
Healthy
budget remaining:
more than 50%
The team continues normal delivery.
At risk
budget remaining:
between 10% and 50%
or
burn rate elevated
The team may:
- investigate recurring failures
- increase release scrutiny
- reduce deployment size
- prioritise known reliability defects
- postpone unusually risky changes
Exhausted
budget remaining:
0%
The team may:
- pause non-essential risky releases
- prioritise service recovery
- address the largest failure sources
- require explicit approval for exceptions
HEALTHY
│
▼
DELIVER NORMALLY
AT RISK
│
▼
REDUCE RISK
EXHAUSTED
│
▼
RESTORE RELIABILITY
The exact thresholds depend on the service and organisation.
The important property is that the response is agreed before the budget disappears.
Current compliance can hide rapid deterioration
Suppose the rolling SLO currently reports:
99.94%
The target is:
99.9%
The service remains compliant.
But during the last hour, 20% of checkout requests have failed.
The rolling percentage may not fall below the target immediately because it includes several weeks of healthy traffic.
LONG WINDOW
still appears healthy
CURRENT FAILURE
severe
This is why error-budget burn rate matters.
Burn rate describes how quickly the budget is being consumed relative to the sustainable rate.
burn rate = 1
means the budget is being consumed at exactly the rate permitted by the objective.
burn rate = 5
means it is being consumed five times too quickly.
A high burn rate can warn the team before the complete window falls out of compliance.
Fast burns and slow burns need different responses
A total outage may consume budget very quickly.
FAST BURN
large failure rate
short duration
For example:
80% of checkout attempts failing
for ten minutes
This may require immediate incident response.
A slow defect may consume the same budget over several days.
SLOW BURN
small failure rate
long duration
For example:
1% of checkout attempts failing
for one week
The service may appear mostly healthy during any individual hour.
The cumulative impact still threatens the SLO.
A useful policy considers both:
SHORT WINDOW
detect severe sudden failure
LONG WINDOW
detect sustained degradation
The error budget provides one model for comparing these different failure shapes.
Both consume the same finite reliability allowance.
Product and reliability teams gain shared language
Without an error budget, reliability conversations can become personal.
PRODUCT
Operations always blocks delivery.
OPERATIONS
Product always ignores risk.
An error budget reframes the discussion:
The checkout service has consumed
85% of its 28-day failure allowance.
The current burn rate indicates
the remaining budget may disappear
within two days.
That evidence does not dictate every decision.
It makes the trade-off visible.
The product team can see that continued risky delivery threatens a user-facing promise.
The service team can no longer use vague anxiety as a permanent veto when the budget remains healthy.
ERROR BUDGET
creates a shared boundary
between
delivery urgency
and
reliability risk
The conversation becomes less about which team has greater organisational gravity.
It becomes about the current state of the service promise.
Reliability work should target budget consumption
An exhausted budget does not mean:
Do miscellaneous reliability work.
The team should identify which failures consumed it.
For checkout, the bad events might break down as:
payment-provider timeouts:
420
inventory deadline failures:
270
database connection exhaustion:
190
deployment regression:
120
The largest source is payment-provider timeouts.
The team might consider:
- provider failover
- timeout redesign
- safer outcome reconciliation
- circuit breaking
- capacity changes
- a different provider contract
The budget gives urgency.
Telemetry gives direction.
ERROR BUDGET
shows the promise is threatened
LOGS, METRICS AND TRACES
show why
Reliability work should reduce the most important sources of user-visible failure rather than polish whichever subsystem is most technically fashionable that month.
Not every failure should count equally
The SLO defines which events are eligible and which are good.
Suppose checkout returns:
InvalidArgument
because the request is malformed.
The service may have behaved correctly.
That event may not consume budget.
A payment-provider timeout that prevents a valid customer from completing checkout probably should.
The classification needs clear rules.
VALID CUSTOMER REQUEST
│
├── completes correctly ──► good
└── internal failure ─────► bad
INVALID REQUEST
│
└── rejected correctly ───► excluded
The rules must not become a way to protect the metric from reality.
A team should not repeatedly reclassify failures as excluded because the budget is running low.
The budget remains credible only when:
- eligibility is documented
- classifications are stable
- exclusions are reviewed
- measurements can be audited
An unreliable denominator creates a wonderfully reliable fiction.
Error budgets can apply beyond customer APIs
The consumer of a service may be another engineering team.
Suppose the bfstore platform offers a service-creation path.
Its SLO might say:
99% of valid service requests should reach a healthy development deployment within fifteen minutes.
The error budget includes requests that:
- fail provisioning
- remain stuck
- create unusable resources
- require undocumented platform intervention
- exceed the time threshold
PLATFORM USER JOURNEY
declare service
│
▼
platform provisions capability
│
▼
healthy deployment
The platform team can use the budget to balance:
- new platform features
- reliability of existing paths
- controller upgrades
- migration work
- operational defects
The same principle applies.
A platform product needs a reliability promise and an agreed response when it stops meeting it.
Dependencies spend the service’s budget
A service may fail because an external dependency fails.
From the user’s perspective, the distinction may not matter.
CHECKOUT FAILED
because
payment provider unavailable
The checkout error budget is still consumed.
This can feel unfair to the service team.
It is also architecturally revealing.
The service promise depends on the provider.
If the dependency cannot support the objective, the service may need:
- a stronger provider
- a fallback provider
- graceful degradation
- asynchronous recovery
- a revised SLO
- a changed product experience
DEPENDENCY FAILURE
│
▼
USER FAILURE
│
▼
SERVICE BUDGET CONSUMED
The budget makes dependency risk visible rather than allowing every team to declare its own component healthy while the customer journey remains broken.
Shared journeys need shared ownership
Checkout crosses several bfstore services:
basket
inventory
payment
order
A checkout failure may originate in any of them.
The end-to-end error budget belongs to the journey.
Component teams may also maintain local SLOs.
CHECKOUT JOURNEY SLO
│
├── basket SLO
├── inventory SLO
├── payment SLO
└── order SLO
The local objectives should support the customer-facing target.
Otherwise each team can meet its own number while the combined journey misses its objective.
Shared ownership requires:
- one coordinating journey owner
- clear component responsibilities
- common telemetry
- agreed incident escalation
- visibility into budget contribution
An error budget should not become a parcel passed between teams until somebody reluctantly signs for it.
Policy needs room for exceptions
An exhausted budget may normally pause risky releases.
A critical security fix may still need deployment.
A regulatory change may have a fixed deadline.
A feature may itself reduce the failure rate.
The policy should allow explicit exceptions.
An exception might require:
- the reason
- expected risk
- mitigation plan
- rollback strategy
- named approver
- post-change verification
BUDGET EXHAUSTED
│
▼
CHANGE REQUEST
│
├── ordinary feature ──► postpone
└── critical change ───► explicit exception
The exception should remain exceptional.
If every release is urgent enough to bypass the budget policy, the organisation does not have a reliability policy.
It has a decorative percentage.
The budget should not become a performance score
Teams may behave badly when an SLO is used to rank or punish them.
They may:
- exclude more events
- avoid reporting failures
- resist necessary changes
- lower the target
- manipulate alert thresholds
- hide dependency problems
The purpose of the error budget is to improve decisions.
It is not to declare one team good and another team careless.
HEALTHY USE
understand risk
prioritise work
improve service
UNHEALTHY USE
rank teams
assign blame
reward green dashboards
A missed SLO should create learning and corrective action.
It should not create an incentive to make the measurement less honest.
Budget reset does not reset the system
A calendar-based budget may reset at the beginning of the month.
31 OCTOBER
budget exhausted
1 NOVEMBER
new budget
The underlying reliability problem may still exist.
A reset should not automatically authorise immediate risky delivery.
The team should consider:
- current burn rate
- unresolved incidents
- known failure causes
- recovery progress
- confidence in recent fixes
Rolling windows reduce the sharp reset effect because older failures leave the window gradually.
Even then, historical failures disappearing from the calculation do not prove the system improved.
FAILURE LEFT WINDOW
does not necessarily mean
FAILURE CAUSE REMOVED
Budget accounting needs engineering judgement.
A bfstore example
Suppose the checkout SLO is:
Over a rolling 28-day period, at least 99.9% of eligible checkout attempts must produce a confirmed order within four seconds.
During the current window:
eligible checkout attempts:
500,000
permitted bad events:
500
observed bad events:
410
The service has consumed:
82% of its error budget
Most bad events came from:
payment timeouts:
250
inventory failures:
90
slow order persistence:
50
deployment regression:
20
The current burn rate is increasing because payment-provider latency remains elevated.
The policy classifies the service as at risk.
The team responds by:
- postponing a high-risk checkout refactor
- continuing a low-risk security patch
- investigating payment-provider behaviour
- improving reconciliation for uncertain payment outcomes
- validating provider fallback options
- reviewing timeout allocation
- monitoring burn rate after each change
The budget did not make the decisions automatically.
It changed the evidence available for making them.
The product team can see why the refactor is being postponed.
The service team can see which reliability work matters most.
The security fix can continue through a documented exception to the ordinary restriction.
Reliability has become negotiable without becoming optional.
The questions I now ask
Promise
Which SLO creates this error budget?
Consumer
Whose experience is being protected?
Allowance
How much failure does the objective permit?
Consumption
How much of that allowance has been used?
Burn
How quickly is the remaining budget disappearing?
Cause
Which failures consumed the budget?
Policy
What changes when the budget is healthy,
at risk or exhausted?
Exceptions
Which critical changes may still proceed,
and who approves them?
Ownership
Who coordinates the response
across contributing services?
Integrity
Can event eligibility or exclusions
be manipulated easily?
Review
Did the policy improve decisions
and user outcomes?
An error budget is useful only when these questions have operational answers.
The mental model I am keeping
My earlier model was:
ERROR BUDGET
the amount of downtime
a service is allowed
The stronger model is:
USER PROMISE
│
▼
SLO
│
▼
ERROR BUDGET
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
HEALTHY AT RISK EXHAUSTED
│ │ │
▼ ▼ ▼
DELIVER NORMALLY REDUCE RISK RESTORE RELIABILITY
│ │ │
└────────────────┼────────────────┘
▼
REVIEW AND LEARN
The SLO answers:
Which level of service
should users depend on?
The error budget answers:
How much failure fits
inside that promise?
Budget consumption answers:
How much reliability allowance
has already been used?
Burn rate answers:
How quickly is the allowance
being consumed now?
Telemetry answers:
Which failures are responsible?
The policy answers:
How should current reliability
change delivery behaviour?
Exceptions answer:
Which important changes still justify
taking additional risk?
Ownership answers:
Who coordinates recovery
and protects the promise?
Review answers:
Did the model lead to better
engineering decisions?
Error budgets make reliability negotiable because they replace two impossible positions:
Never accept failure.
and:
Keep releasing regardless of failure.
with a measurable agreement:
This is the level of service
our users should receive.
This is the amount of failure
we can tolerate.
This is what we will do
when that tolerance is consumed.
The budget creates room for change while the service is healthy.
It creates pressure to reduce risk when reliability declines.
It creates a reason to prioritise corrective work when the promise is no longer being met.
An error budget does not decide whether reliability or delivery matters more. It defines the point at which the service has consumed enough reliability that the organisation must renegotiate what it does next.