My earliest automation scripts looked like instructions left for a very obedient colleague:
create the directory
copy the configuration
start the process
add the firewall rule
restart the service
The script began at the top, performed each step and stopped at the bottom.
START
│
▼
STEP 1
│
▼
STEP 2
│
▼
STEP 3
│
▼
DONE
That model is useful.
Many jobs really are sequences. A database migration must execute statements in an intentional order. A release may need to build an image before publishing it. A backup must be created before it can be verified.
But the sequence model left me with an awkward question:
Who checks tomorrow that the result is still true?
A script might create three service instances today. One instance might disappear tonight.
The script completed successfully. The intended system no longer exists.
SCRIPT RESULT AT 14:00
three instances running
ACTUAL STATE AT 03:00
two instances running
The automation performed its steps. It was no longer present to defend their outcome.
Kubernetes introduced me to a different model:
desired replicas:
3
That statement is not merely an instruction to create three Pods once. It becomes a condition that the control plane continues trying to maintain.
DESIRED STATE
three replicas
│
▼
OBSERVE ACTUAL STATE
│
├── three exist ──► no corrective action
│
├── two exist ────► create one
│
└── four exist ───► remove one
The declaration survives the command that submitted it.
That changed how I think about automation:
Automation is not only the execution of steps. It can also be the continuous defence of an intended condition.
Imperative automation describes actions
An imperative instruction says what to do.
mkdir -p /srv/catalog
cp catalog.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable catalog
systemctl start catalog
The sequence contains operational knowledge:
create directory
install unit file
reload service manager
enable service
start service
This can be clear and appropriate. It also assumes something about the starting state.
What happens when the directory already exists, the file has changed, the service is already running, or the command fails halfway through? What happens when another process changes the configuration later?
A safer script starts checking before acting:
if directory missing:
create it
if file differs:
replace it
if service not enabled:
enable it
if service not running:
start it
As the script becomes safer to repeat, it begins to inspect current state and choose only the actions that are still required.
It starts moving towards reconciliation.
Desired-state automation describes an outcome
A desired-state declaration says what should be true.
Conceptually:
service:
name: catalog
enabled: true
state: running
The automation compares that declaration with reality.
DESIRED
catalog enabled and running
ACTUAL
catalog disabled and stopped
ACTIONS
enable catalog
start catalog
If the service is already enabled and running, no action is required.
The desired state remains stable. The corrective action depends on the observed state.
DESIRED STATE
stable intention
CORRECTIVE ACTION
context-dependent step
The declaration does not need to prescribe every route from every possible starting point. A reconciler decides which action is necessary now.
Reconciliation turns intention into behaviour
A desired state becomes useful when something repeatedly compares it with actual state.
while true:
observe
compare
act
wait
A simplified reconciliation loop looks like this:
DESIRED STATE
│
▼
COMPARE WITH ACTUAL STATE
│
├── no difference ──► wait and observe again
│
└── difference
│
▼
corrective action
│
▼
observe again
The loop matters because drift can appear after the original action and because corrective actions can fail.
Suppose a controller needs to create a Pod. The first attempt may fail because a dependency is temporarily unavailable. The desired state remains recorded throughout, so the controller can try again without requiring the user to resubmit the declaration.
attempt 1:
failed
attempt 2:
failed
attempt 3:
succeeded
Drift is simply the difference between what should exist and what currently exists.
DESIRED
three catalog replicas
ACTUAL
two catalog replicas
DRIFT
one missing replica
A one-shot script may create the desired state. A control loop treats later drift as an ordinary condition to observe and correct.
That is a stronger operational promise. It is also a larger responsibility.
Idempotency and convergence make reconciliation safe
A reconciliation loop may attempt the same corrective action several times.
Suppose it requests:
create network bfstore
The first request succeeds, but the response is lost. The controller cannot tell whether the network was created, so it tries again.
If each attempt blindly creates a new network, retries multiply resources:
attempt 1:
bfstore-network-1
attempt 2:
bfstore-network-2
A safer operation uses a stable identity:
network name:
bfstore
Repeated attempts converge on one logical object.
attempt 1 ──┐
├──► network bfstore
attempt 2 ──┘
This is why idempotency appears so often in controllers and infrastructure tooling. A reconciler must be able to request the same condition repeatedly without producing unlimited side effects.
Desired-state automation depends on repeatable observation and idempotent correction.
The other important property is convergence. Each reconciliation should move the system towards a stable result.
If the actual replica count is zero and the desired count is three, a controller might create one replica per pass:
pass 1:
0 → 1
pass 2:
1 → 2
pass 3:
2 → 3
A successful API call is not enough. The series of actions must approach the intended condition rather than oscillate or create new drift.
A bfstore example
Suppose bfstore’s catalog Deployment declares:
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog
spec:
replicas: 3
selector:
matchLabels:
app: catalog
template:
metadata:
labels:
app: catalog
spec:
containers:
- name: catalog
image: bfstore/catalog-service:1.0.0
The desired state is:
three Pods
using catalog-service:1.0.0
matching app=catalog
Initial state
actual Pods:
0
The Deployment and ReplicaSet controllers create the required resources. The scheduler assigns nodes. The kubelets start the containers.
Healthy state
actual ready Pods:
3
No replica correction is required, but the control loops continue observing.
One Pod terminates
actual Pods:
2
The ReplicaSet creates a replacement.
The desired state did not change. Reality drifted.
The image is updated
image: bfstore/catalog-service:1.1.0
The desired state changes. The Deployment controller coordinates replacement of the older Pods according to its rollout strategy.
The new image is broken
desired:
three version 1.1.0 Pods
actual:
new Pods repeatedly fail
The controller cannot invent a working binary. It continues attempting the declared rollout while status, conditions and events expose the failed progress.
The declaration must be corrected or rolled back.
This one example contains the core model:
desired state persists
controllers observe drift
actions can be retried
status reports progress
impossible desires require human change
Specification and status answer different questions
Kubernetes resources helped clarify the distinction between intention and observation.
SPEC
What should be true?
STATUS
What does the system currently know to be true?
For example:
spec:
replicas: 3
may coexist with:
status:
availableReplicas: 2
The resource is not contradictory. It is reporting incomplete convergence.
desired:
3
available:
2
condition:
progressing
This is more useful than a single field called replicas: 3 with no indication of whether three is a request, an observation or an aspiration written during a particularly optimistic lunch.
Separating specification from status makes progress, drift and failure visible.
Acceptance, convergence and failure are different outcomes
When an API accepts a desired-state resource, it confirms that the intention has been recorded.
It does not necessarily confirm that reality already matches it.
kubectl apply
│
▼
resource accepted
│
▼
controller observes resource
│
▼
work begins
│
▼
status changes over time
This resembles other asynchronous operations. A command can be accepted before its final outcome is known.
API ACCEPTANCE
The system accepted responsibility
for the desired state.
CONVERGENCE
The system successfully made
the desired state real.
A pipeline that stops after kubectl apply has proved submission, not rollout success.
A useful desired-state system therefore reports progress and failure. If a Pod cannot be scheduled because no node has enough memory, the Pod may remain pending while its conditions and events explain the scheduling failure.
The controller may continue retrying because capacity could become available. But it should not fail forever in silence.
Useful status can include:
- current phase
- observed generation
- conditions
- failure reason
- last attempted action
- responsible component
- retry time
- timestamps
Status is the evidence trail for what the control loop has actually achieved.
Some failures cannot be repaired by retrying. If the desired image does not exist:
image:
bfstore/catalog-service:9.9.9
no amount of determination can pull it successfully.
The desired state itself must change.
DESIRED STATE
references missing image
ACTUAL STATE
cannot converge
REQUIRED ACTION
correct the declaration
Reconciliation can repair temporary drift. It cannot make an impossible specification possible.
Authority determines which intention wins
Two systems cannot safely maintain conflicting intentions without rules about precedence.
Suppose one tool declares:
catalog replicas:
3
while another declares:
catalog replicas:
5
Each may continually undo the other.
CONTROLLER A
sets replicas to 3
CONTROLLER B
sets replicas to 5
CONTROLLER A
sets replicas to 3 again
The system oscillates.
Desired-state automation therefore needs explicit ownership:
Which source is authoritative?
Which fields may each controller manage?
Who may change the declaration?
How are competing intentions prevented?
A Git repository, Kubernetes API object or infrastructure state file may form part of that authority. The important property is not the storage format. It is that the organisation knows which intention should win.
This becomes especially important during manual intervention.
Suppose a user changes a Deployment from three replicas to five.
If the cluster API is authoritative, five becomes the new desired state. If Git is authoritative, a GitOps controller may restore three. A platform might also allow a temporary operational override and record it separately.
Drift correction is not automatically virtuous. Sometimes the drift is an emergency repair. Sometimes it is an unauthorised change.
A controller can also fight an operator while behaving exactly as designed.
OPERATOR
deletes broken object
CONTROLLER
desired state says object must exist
RESULT
broken object recreated
Safe intervention may require changing the desired state first, pausing reconciliation, applying an approved override or transferring ownership temporarily.
Relentless automation is useful only while the desired state remains trustworthy.
Controllers need bounded responsibility
A controller should own a clear set of fields and outcomes.
Suppose one controller manages replica count while another manages image version. This can work when ownership is explicit.
It becomes dangerous when both rewrite the complete resource:
CONTROLLER A
sets replicas
CONTROLLER B
sets image,
accidentally resets replicas
Good control-plane design asks:
Which fields does this controller own?
Which external objects may it create?
Which changes may it reverse?
Which states require human approval?
Broad, overlapping ownership creates conflict. Smaller controllers can be easier to reason about, provided their interactions remain visible.
Reliable reconciliation must account for uncertainty
A controller acts on what it can observe.
If that observation is stale or incomplete, the chosen action may be wrong.
Suppose a controller observes two replicas even though a third has already been created but has not yet appeared in its view. It may create another and temporarily produce four.
Distributed desired-state systems therefore need to account for:
- delayed observations
- eventual consistency
- cached state
- duplicate events
- concurrent updates
- partial failure
- uncertain outcomes
They need stable identities, versions and conflict handling rather than an assumption that every read is perfectly current.
Uncertain outcomes are particularly dangerous when a controller manages an external system.
CONTROLLER CLOUD API
create database
────────────────────────►
database created
response lost
outcome unknown
The controller should retry using a stable identity, such as bfstore-catalog-db, or query the external system before acting again. Otherwise each uncertain attempt could create another database.
A control plane is not above distributed systems. It is one.
Events, periodic checks and restarts
A controller could poll continuously:
observe every five seconds
That is simple but potentially wasteful or slow.
It can also react to change notifications:
resource changed
│
▼
enqueue reconciliation
Events provide prompt reaction. Periodic reconciliation provides eventual correction when an event is missed or when state changes outside the normal event path.
The event is a hint that work may be required. Observed state remains the authority for deciding what to do.
A controller must also survive its own restart.
If it stops halfway through an action, it should reconstruct progress from durable desired state and externally observable reality rather than depending on lost process memory.
CONTROLLER MEMORY
gone
DESIRED RESOURCE
still stored
EXTERNAL RESOURCE
perhaps partly created
STATUS
records last known observation
The process is disposable. The intention outlives the worker.
Desired state should express stable meaning
A declaration works best when it describes durable intent rather than incidental execution detail.
maintain three catalog replicas
is stable.
This is more fragile:
start replica A on node 1
start replica B on node 2
start replica C on node 3
unless those exact placements are part of a real requirement.
The more execution detail the user hard-codes, the less freedom the controller has to recover.
OUTCOME-FOCUSED INTENT
run three replicas across suitable nodes
STEP-FOCUSED INTENT
run this exact process on this exact machine
The second may still be appropriate for stateful or specialised workloads. The point is to declare constraints because they matter, not because they happened to be true during the first deployment.
Compact declarations can hide destructive action
A small desired-state change may produce a large runtime effect.
replicas:
5 → 0
The configuration diff is one number. The effect is deletion of every replica.
Removing a resource from an infrastructure declaration may similarly tell a tool to destroy it.
DECLARATION CHANGE
resource absent
PLANNED ACTION
delete real infrastructure
Declarative syntax can make dangerous actions look tranquil.
Desired-state systems need controls such as plans, previews, policy checks, approvals, deletion protection, lifecycle rules, backups and audit trails.
The declaration describes the destination. Operators still need to understand the road being taken.
Deletion is particularly important for resources that carry valuable history. Removing a database declaration might mean:
delete the database immediately
or it might require:
retain the data
create a final backup
require explicit confirmation
block deletion while dependants exist
Presence and absence are not always enough to describe a safe lifecycle.
A declaration does nothing without a reconciler
A YAML file can describe an intention. It does not enforce anything by itself.
replicas: 3
It becomes active only when a system reads the declaration, understands its meaning, observes the target environment, calculates the difference and performs corrective action.
FILE
│
▼
RECONCILER
│
▼
REAL SYSTEM
Without a reconciler, the file is documentation. Possibly excellent documentation, but still documentation.
This helps explain why Kubernetes resources depend on controllers. A Deployment gains behaviour because a controller understands its specification.
A custom resource whose definition is installed, but which has no corresponding controller, may be stored successfully and then sit there like an exquisitely labelled parcel nobody has been assigned to deliver.
Declarative does not mean that actions disappear. Some component still creates Pods, deletes resources, attaches volumes and starts processes.
The difference is who decides which action is required.
IMPERATIVE USER
chooses and issues each action
DECLARATIVE USER
declares intended result
CONTROLLER
observes state and chooses actions
Declarative systems move action selection into a reconciler. The underlying world remains full of commands.
Related tools use different reconciliation models
Desired-state thinking appears outside Kubernetes, but not every tool reconciles in the same way.
| Model | Where desire is recorded | Reconciliation cadence |
|---|---|---|
| Kubernetes controller | API object | Continuous |
| Terraform | Configuration and state | During a run |
| Configuration management | Machine configuration | Scheduled or invoked |
| GitOps | Git repository | Continuous or frequent |
Terraform compares configuration with known remote objects, constructs a plan, applies changes and exits. It uses desired state, but a typical CLI run does not remain active as a permanent controller.
Configuration-management tools inspect a machine and apply only the changes needed to make packages, files and services match their declarations. Repeated runs should converge.
GitOps separates the source of desire from the component that enforces it:
GIT REPOSITORY
declared cluster state
│
▼
GITOPS CONTROLLER
│
▼
KUBERNETES API
│
▼
CLUSTER
Git provides history, review, authorship and rollback material. The controller provides reconciliation. The repository alone does not update the cluster.
This distinction taught me not to treat declarative and continuous as the same property.
Desired state does not remove sequence
Some changes require ordered transitions.
A database cannot always move directly from version 1 to version 5. A safe application rollout may need to deploy compatible code, run a migration, verify readers, enable new behaviour and remove old fields later.
Desired state describes the target. A controller may still need a state machine or workflow to reach it safely.
TARGET
schema version 5
SAFE PATH
several controlled transitions
Declarative automation does not abolish procedural knowledge. It places procedure behind a stable outcome.
Not every task needs a permanent controller either.
Some jobs are naturally finite:
- generate a report
- export a dataset
- run a migration
- restore a backup
- rotate a certificate once
- perform a controlled repair
A Job can express:
this work should complete successfully
rather than:
this condition should remain true forever
Ongoing state and finite outcomes need different lifecycle models.
Desired state changes the operator’s role
With imperative automation, the operator often thinks in steps:
Which command should I run next?
With desired-state automation, the operator thinks in conditions:
What should be true?
What is currently true?
Why have they not converged?
Which controller owns the difference?
What evidence shows progress?
Troubleshooting moves through layers:
declaration
API acceptance
controller observation
planned action
external execution
reported status
The operator does not merely rerun commands. They inspect the control loop.
A desired-state checklist
When designing a desired-state system, I now ask:
- Intention: Which condition should remain true?
- Authority: Where is that intention recorded, and which source is allowed to change it?
- Observation: How is actual state discovered, and can that view be stale?
- Identity: How does the reconciler recognise the same logical resource across attempts?
- Correction: Can the chosen action be attempted again safely?
- Convergence: Does repeated reconciliation approach a stable result?
- Ownership: Which controller owns each field, object and external effect?
- Status: How are progress, failure and impossible requests reported?
- Deletion: What real-world effects follow when the declaration removes a resource?
- Intervention: How can an operator pause, override or redirect reconciliation safely?
- Recovery: Can work resume after the controller restarts?
- Audit: Can we see who changed the desired state and what happened afterwards?
A desired-state system is not complete merely because it contains YAML and a loop.
The loop needs trustworthy intention, observation, ownership and evidence.
The mental model I am keeping
My original model was:
AUTOMATION
replace manual commands
with scripted commands
The stronger model is:
DESIRED STATE
│
▼
OBSERVATION
│
▼
COMPARISON
│
┌────────────┴────────────┐
│ │
▼ ▼
STATE MATCHES DRIFT EXISTS
│ │
▼ ▼
wait corrective action
│ │
└────────────┬────────────┘
▼
observe again
Imperative automation asks:
Which steps should run?
Desired-state automation asks:
Which condition should remain true?
Idempotency asks:
Can correction be attempted again safely?
Status asks:
What has the system actually achieved?
Authority asks:
Which declaration wins when intentions conflict?
And reconciliation asks:
Who keeps checking after the original command ends?
The answer does not always need to be Kubernetes. It might be a configuration-management agent, an infrastructure-as-code run, a GitOps controller, a custom operator, a scheduled reconciliation job or an application workflow.
The important shift is in the design.
A script can perform work.
A control loop can preserve intent.
That changed automation for me from a recorded sequence of keystrokes into an ongoing conversation between what should be true and what the world has managed to become.
References and further reading
Kubernetes controllers
Explains control loops that observe cluster state and act to move it towards declared state.
Kubernetes API conventions
Describes API conventions including the distinction between desired specification and observed status.
Kubernetes object management
Introduces imperative commands, imperative object configuration and declarative object configuration.
Terraform: Core workflow
Describes configuration, planning and application of infrastructure changes.
The Operator Pattern
Explains extending Kubernetes with custom resources and controllers that encode operational knowledge.
GitOps principles
Describes declarative configuration, versioned desired state, automated pulling and continuous reconciliation.