I first encountered reconciliation as a Kubernetes idea.
A Deployment declares:
spec:
replicas: 3
The cluster currently has:
2 available Pods
A controller notices the difference and creates another Pod.
EXPECTATION
3 replicas
OBSERVATION
2 replicas
CORRECTION
create 1 replica
That is a reconciliation loop.
But once I understood the shape, I began seeing related loops everywhere.
A GitOps controller compares an approved declaration with the cluster.
An autoscaler compares capacity with measured demand.
A load balancer compares registered targets with health results.
A certificate controller compares current validity with renewal policy.
A payment recovery worker compares an uncertain local outcome with the provider’s final record.
The technologies and expectations differ.
The broad pattern remains:
EXPECTATION
│
▼
OBSERVE OWNED STATE
│
▼
COMPARE
│
▼
BOUNDED CORRECTION
│
▼
OBSERVE AGAIN
That has changed how I think about automation.
A reconciliation loop repeatedly asks whether observed reality satisfies an expectation and takes bounded, retry-safe action within its authority when it does not.
The loop matters because the world does not remain fixed after the first successful command.
Processes crash.
People make manual changes.
Networks partition.
Credentials expire.
Traffic grows.
External systems complete work after a timeout.
A one-time action can establish a condition.
A reconciliation loop accepts continuing responsibility for checking whether that condition remains true.
One-shot automation stops after the action
A one-shot script may say:
create three workers
It runs successfully.
Three workers now exist.
Later, one worker crashes.
AFTER THE COMMAND
3 workers
LATER
2 workers
The completed command does not naturally return to ask what happened.
A controller may use scripts or API calls internally, but its operating model is different:
Keep three workers available.
It observes two, starts a replacement and checks again.
The distinction is not merely imperative syntax versus declarative syntax.
It is also about time and responsibility.
ONE-SHOT AUTOMATION
Did this action complete?
CONTINUING CONTROL LOOP
Does the expected condition
still remain true?
A declarative file without an active observer is still only a statement.
The loop gives that statement operational force.
Related loops maintain different kinds of expectation
Not every control loop maintains one explicit snapshot.
Several related patterns fit the same family.
| Pattern | Form of expectation | Example |
|---|---|---|
| State reconciliation | An explicitly declared condition | A Deployment should have three available replicas |
| Policy control | A target derived from current measurements | Capacity should remain suitable for measured demand |
| Record reconciliation | Agreement or resolved status across systems | An uncertain payment must reach a known outcome |
These mechanisms are not identical.
A GitOps controller works from versioned declarations.
An autoscaler calculates a changing target from metrics and policy.
A payment reconciler resolves disagreement between records.
What they share is continuing responsibility. None assumes that one completed action keeps the world correct forever.
The expectation may be an exact value:
3 replicas should be available
a range:
between 3 and 10 replicas
an invariant:
at least one healthy endpoint
in each availability zone
or a terminal business result:
every unknown payment outcome
must eventually be resolved
Desired state is often an acceptable region, not one perfect photograph of the system.
A loop needs an observable expectation
A reconciler cannot compare reality with nothing.
It needs both:
an expectation
and
evidence about the current world
Suppose a controller is responsible for three replicas, but its observation is stale.
REALITY
2 replicas
CONTROLLER CACHE
3 replicas
Its comparison logic may be correct.
Its evidence is wrong.
Observation can fail because:
- an API is unavailable
- cached state is stale
- telemetry is delayed
- an object has not appeared yet
- clocks disagree
- a request succeeded but its response was lost
- an external provider exposes incomplete status
The loop therefore needs to distinguish:
not observed
from
observed as absent
and:
still progressing
from
permanently failed
The quality of the decision depends on the quality of the evidence.
A loop is not clever merely because it repeats itself.
Correction must be safe under repetition and uncertainty
A reconciler may perform the same comparison many times.
It may also retry when it does not know whether an earlier action succeeded.
Suppose it asks an external system to create a resource, but the response is lost.
REQUEST SENT
resource created
RESPONSE
lost
On the next pass, blindly issuing another create request may produce a duplicate.
Stable identity helps:
desired resource identity:
bfstore-catalog-runtime
The loop can ask whether that logical resource already exists before acting again.
Idempotent operations are especially useful, but literal idempotency is not the only protection. Retry safety can also come from:
- idempotency keys
- stable operation identifiers
- precondition checks
- deduplication
- recorded progress
- compensating actions
The wider requirement is that uncertainty or repetition must not create duplicate or conflicting effects.
A good correction should also reduce the observed difference.
REPEATABLE ACTION
may be safe to repeat
but make no progress
CONVERGENT ACTION
moves reality towards
an accepted condition
The loop needs both retry safety and direction.
Progress takes time
Distributed systems rarely change instantly.
A controller creates a Pod.
The object is accepted, but the desired result still requires several steps:
OBJECT ACCEPTED
│
▼
SCHEDULED
│
▼
IMAGE PULLED
│
▼
PROCESS STARTED
│
▼
READY
Checking too quickly can make normal progress look like failure.
Acting again immediately can create overcorrection.
Waiting does not necessarily mean blocking one process until the consequence appears. A controller may record progress, requeue the object and observe it again after an appropriate delay.
Useful mechanisms include:
- status conditions
- observed generations
- retry delays
- exponential backoff
- timeouts
- progress deadlines
A loop that reacts too slowly leaves failure in place.
A loop that reacts too quickly can manufacture instability.
Ownership prevents controllers from fighting
Suppose Git declares:
replicas:
3
A horizontal autoscaler calculates:
replicas:
8
If both controllers believe they own the live replica count, the environment can oscillate.
GITOPS CONTROLLER
restore 3
AUTOSCALER
set 8
GITOPS CONTROLLER
restore 3
Each controller is locally correct according to its own expectation.
The combined system is wrong.
A better design lets Git own the scaling policy:
minimum replicas:
3
maximum replicas:
10
target utilisation:
65%
The autoscaler owns the current replica count within those limits.
GIT
owns scaling policy
AUTOSCALER
owns live capacity
A controller should observe and modify only the resources, fields or transitions assigned to its authority.
Two sensible loops can create nonsense when their boundaries overlap.
Infrastructure contains nested control loops
Kubernetes becomes easier to understand when I stop imagining one central intelligence.
It contains specialised controllers with bounded responsibilities.
GITOPS RECONCILER
│
▼
DEPLOYMENT CONTROLLER
│
▼
REPLICASET CONTROLLER
│
▼
PODS
│
├── scheduler assigns nodes
└── kubelet maintains containers
A GitOps reconciler updates a Deployment.
The Deployment controller manages rollout state.
The ReplicaSet controller maintains the required Pods.
The scheduler assigns unscheduled Pods.
The kubelet works to keep each assigned Pod running as specified.
Service and EndpointSlice-related controllers maintain the relationship between Services and ready endpoints.
One merge can therefore activate several nested loops.
This explains why a successful Git merge is not the same as a healthy deployment.
The upstream loop accepted new intent.
The downstream loops still need to realise it.
Autoscaling and health management add other expectations.
An autoscaler compares measured demand with scaling policy:
MEASURED UTILISATION
82%
TARGET
65%
ACTION
increase capacity
A health-management loop compares endpoint observations with routing policy:
healthy endpoint
route traffic
unhealthy endpoint
remove from service
The health check does not repair the application. It changes how the platform treats the instance.
Different loops can respond to the same failure at different layers:
READINESS
stop sending traffic
KUBELET
restart a failed container
DEPLOYMENT
replace an unavailable replica
AUTOSCALER
adjust capacity
Understanding ownership makes those responses less mysterious.
Business reconciliation follows valid transitions
Reconciliation is not limited to infrastructure.
Suppose the bfstore payment service sends an authorisation request.
The provider completes the charge, but the response is lost.
Locally:
payment outcome:
unknown
At the provider:
payment outcome:
authorised
A recovery worker compares the records and moves the local workflow towards a valid terminal result.
LOCAL PAYMENT RECORD
│
▼
QUERY PROVIDER
│
▼
COMPARE OUTCOMES
│
▼
UPDATE, COMPENSATE
OR ESCALATE
The expected condition is not:
repeat the payment
It is:
resolve the uncertainty around
this logical payment operation
That distinction prevents recovery from becoming duplicate side effects wearing an operations badge.
The workflow still needs a state machine.
OBSERVE CURRENT STATE
│
▼
CHECK VALID TRANSITION
│
▼
PERFORM NEXT ACTION
│
▼
RECORD RESULT
The desired outcome supplies direction.
The state machine constrains the route.
A loop without transition rules may skip required work or repeat dangerous actions.
A state machine without reconciliation may become stranded after a transient failure.
Some loops end; others continue indefinitely
Not every responsibility needs a permanent controller.
A useful distinction is:
BOUNDED RECONCILIATION
maintain responsibility until
one terminal outcome is reached
CONTINUOUS RECONCILIATION
maintain an accepted condition
for as long as the system exists
A Kubernetes Job, database migration or report-generation workflow may be reconciled until it succeeds, fails terminally or requires intervention.
A certificate controller maintains an ongoing condition:
a sufficiently valid certificate
should continue to exist
Both are different from a fire-and-forget command.
The important question is how long responsibility must remain active, not whether the implementation happens to be called a script or controller.
Building a permanent controller around every small task can still turn a bounded job into a lifelong administrative pet.
Failure and progress need visible states
Some expectations cannot currently be satisfied.
Examples include:
- a quota prevents creation
- an image does not exist
- policy rejects the resource
- a required dependency is missing
- the specification is contradictory
- an external outcome remains unavailable
A naive loop can retry forever without telling anyone why.
A stronger loop distinguishes several situations:
| Situation | Appropriate response |
|---|---|
| Transient failure | Retry with delay and bounded backoff |
| Progressing | Observe again without duplicating the action |
| Unknown outcome | Re-observe or reconcile with the external authority |
| Terminal failure | Record the reason and stop unsafe retries |
| Human decision required | Escalate with enough evidence to act |
Useful status should expose:
- current expectation
- last successful observation
- last attempted correction
- error reason
- retry count
- time since progress
- whether intervention is required
The loop may continue checking while declaring:
ProgressBlocked
or:
ManualReviewRequired
The loop itself also needs observability.
Metrics can show reconciliation attempts, duration, failures, queue depth and unresolved-drift age.
Logs can preserve detailed decisions.
Traces can follow one reconciliation across several APIs.
Status fields can expose the condition of the managed resource.
A control system should not hide impossible desire behind infinite enthusiasm.
A reconciliation-loop review
| Area | Review question |
|---|---|
| Expectation | Which state, range, invariant or terminal result should be maintained? |
| Observation | Which evidence represents reality, and how can it be stale or incomplete? |
| Ownership | Which resources, fields or transitions may the loop control? |
| Correction | Which action reduces the observed difference? |
| Retry safety | Can uncertainty or repetition create duplicate effects? |
| Timing | How are progress, delay, backoff and deadlines handled? |
| Coordination | Can another controller legitimately change the same state? |
| Failure | When should the loop retry, stop or escalate? |
| Visibility | Can operators see expectation, evidence, action and progress? |
Those questions reveal more than the name of the tool performing the automation.
The mental model I am keeping
My earlier model was:
RECONCILIATION
a Kubernetes controller
fixing missing resources
The stronger model is:
EXPECTATION
│
▼
OBSERVE OWNED STATE
│
▼
COMPARE
│
┌────────────┴────────────┐
│ │
▼ ▼
SATISFIED DIFFERENCE
│ │
▼ ▼
WAIT RETRY-SAFE CORRECTION
│ │
└────────────┬────────────┘
▼
OBSERVE AGAIN
Once I noticed this pattern, controllers stopped looking like Kubernetes-specific machinery.
They became a general way to express continuing responsibility.
GitOps reconciles declarations with clusters.
Autoscalers reconcile demand with capacity.
Health systems reconcile routing with endpoint readiness.
Certificate controllers reconcile current validity with renewal policy.
Recovery workers reconcile uncertain local records with external outcomes.
The nouns change.
The loop remains.
That does not mean every system should become declarative or every operation should run forever.
It means that whenever a condition must survive failure, drift and time, a one-off command may be only the beginning.
The deeper automation question is no longer:
How do I perform this action?
It is:
What should remain true, how will the system notice when it is no longer true, and which safe authority will keep trying to restore it?
References and further reading
Kubernetes controllers Explains the controller pattern and how Kubernetes control loops move current state towards desired state.
Kubernetes Jobs Documents bounded workloads that run to completion.
Horizontal Pod Autoscaling Explains how Kubernetes adjusts workload scale using observed metrics and configurable behaviour.
OpenGitOps principles Describes declarative, versioned, automatically pulled and continuously reconciled system state.
Kubernetes operator pattern Introduces application-specific controllers that extend Kubernetes through custom resources and control loops.