A production service fails.
The first diagram shows three clear boundaries:
AWS account
Kubernetes cluster
private network
The architecture appears well isolated.
Then the investigation begins.
The service shares a cluster with several other workloads.
The cluster uses one central DNS service.
Its deployment controller holds permissions across multiple Namespaces.
The account depends on organisation-wide identity and policy.
The private network connects to shared observability, DNS and egress systems.
The boundaries are real.
They are not complete.
A cloud account can contain some administrative mistakes.
It does not automatically contain a cluster-wide controller failure.
A cluster can separate workloads from another cluster.
It does not automatically prevent network access to shared databases.
A network can restrict communication.
It does not remove an identity’s permission to change resources through an API.
Accounts, clusters and networks are all failure boundaries, but each contains a different kind of failure.
I use failure boundary broadly here to mean a boundary intended to contain the effects of faults, compromise or operational mistakes.
The mechanism differs at each layer.
The useful question is not:
Which boundary is strongest?
It is:
Which failure are we trying to stop,
and which boundary is capable
of stopping it?
A compact model is:
ACCOUNT
who may administer
CLUSTER
what shares one Kubernetes control plane
NETWORK
what may communicate
Resilient architecture is rarely built from one perfect wall.
It is built from several imperfect boundaries that overlap.
Accounts contain administrative reach
A cloud account groups resources beneath an administrative and billing boundary.
It can separate identities, policies, workloads, costs, account-scoped limits, logs and encryption keys.
For bfstore, development and production may live in separate accounts:
bfstore-dev
bfstore-prod
A developer role assigned only to the development account cannot automatically administer production.
A mistaken infrastructure deployment aimed at development should not modify production resources.
A runaway development workload incurs cost and consumes account-scoped limits in development rather than directly exhausting equivalent production allocations. The account boundary reduces administrative blast radius.
It is particularly useful for containing:
- mistaken API calls
- broad account-local permissions
- cost spikes
- resource deletion
- compromised account-local identities
- some forms of quota exhaustion
It gives the organisation a clearer unit for access, policy, cost and investigation.
Central authority can cross the boundary
An account boundary can be crossed by identities and controls deliberately created to cross it.
Examples include organisation administrators, delegated security roles, central deployment systems, cross-account roles and break-glass access.
CENTRAL AUTOMATION
│
├── dev account
├── staging account
└── prod account
If the central automation identity is compromised, account separation may offer less protection than the diagram suggests.
Likewise, an organisation-wide policy mistake can affect several accounts at once.
The account is still a boundary.
The central control has been given authority to cross it.
That authority needs its own containment:
- separate roles by environment
- scoped account assignments
- staged policy rollout
- temporary privileged access
- policy-test accounts
- independent audit trails
Account isolation is strongest when cross-account authority is narrow, deliberate and observable.
Separate accounts can protect evidence and recovery
Accounts become particularly useful when operational evidence and recovery material are kept outside workload accounts.
A production workload account may send logs to:
bfstore-log-archive
and backups to:
bfstore-backup
PRODUCTION ACCOUNT
│
├── logs ─────► LOG ARCHIVE ACCOUNT
│
└── backups ──► BACKUP ACCOUNT
The separation only protects evidence when production identities and cross-account administrators cannot modify retention controls or delete protected copies.
The important property is not merely another account.
It is independent authority and restricted deletion.
PRODUCTION
may write evidence
PRODUCTION
may not erase evidence
The boundary protects more than running resources.
It protects the organisation’s ability to understand and recover from failure.
Clusters contain Kubernetes fault domains
A Kubernetes cluster groups workloads beneath one control plane.
The cluster contains a shared API server, scheduler, controllers, DNS, networking, node infrastructure and cluster-scoped extensions.
Workloads in different clusters do not share those components. A control-plane failure in cluster A does not directly stop the Kubernetes API in cluster B.
A broken cluster-scoped controller in cluster A cannot alter objects in cluster B unless it also holds external authority there.
A cluster boundary is therefore useful for containing:
- control-plane failures
- cluster upgrades
- faulty admission webhooks
- cluster-scoped controllers
- cluster-wide capacity exhaustion
- failures in shared or critical node pools
- CNI failures
- cluster DNS failures
It defines a meaningful operational fault domain.
Namespaces are not smaller clusters
Namespaces provide useful separation inside a cluster.
They can scope:
- names
- role bindings
- quotas
- network policies
- ownership
For example:
catalog Namespace
payment Namespace
platform Namespace
But all Namespaces still depend on the same cluster control plane.
A broken API server affects all of them.
A cluster-wide DNS failure affects all of them.
A badly configured admission webhook may block deployments everywhere.
A privileged cluster administrator can cross Namespace boundaries.
Namespaces can constrain some workload access, naming and resource consumption.
They do not create a separate Kubernetes fault domain.
NAMESPACE
logical and policy boundary
CLUSTER
control-plane and infrastructure boundary
Namespaces are valuable.
They should not be treated as lightweight substitutes for independent clusters.
Shared clusters create shared operational risk
Running many workloads in one cluster can improve:
- utilisation
- operational consistency
- upgrade efficiency
- cost
- central policy
It also creates common failure paths.
Suppose bfstore places application workloads and platform tooling in one cluster:
SHARED CLUSTER
│
├── application services
├── Crossplane
├── Backstage
├── infrastructure runners
└── observability components
A capacity incident may affect both applications and the tools required to investigate or recover them.
A faulty cluster policy may block every team.
Separating platform control-plane tooling from application workloads can reduce that radius:
PLATFORM CLUSTER
Crossplane
Backstage
automation
shared controllers
WORKLOAD CLUSTER
catalog
basket
order
payment
The separation becomes more compelling when failure of the workload cluster could also remove the tools, credentials or controllers required to restore it.
It also costs more and introduces another cluster to patch, observe and recover.
The decision depends on the consequences of sharing the fault domain and the organisation’s ability to operate the additional boundary reliably.
Networks contain communication reach
A network boundary controls which systems can communicate.
It may be implemented through VPCs, routing, security groups, firewalls, private endpoints, Kubernetes NetworkPolicies and service meshes.
A workload may exist in the same account and cluster as another workload while being unable to reach it over the network.
CATALOG SERVICE
│
├── may reach catalog database
├── may reach telemetry endpoint
└── may not reach payment database
This can contain:
- accidental communication
- lateral movement from a compromised workload
- exposure of private services
- dependency mistakes
- broad internal reach
The network boundary answers:
Which systems can exchange traffic?
That differs from the account question:
Which identities can administer resources?
and the cluster question:
Which workloads share one control plane?
Network isolation does not replace least privilege
Suppose a platform controller cannot reach a database endpoint directly.
If it can still reach the cloud control-plane API and has permission to delete the database, blocking the database’s data-plane endpoint does not prevent that action. Likewise, a workload inside a private subnet may possess permission to read a secret through an accessible cloud API.
The network is private.
The identity is still powerful.
A strong design needs both:
IDENTITY
controls which actions are authorised
NETWORK
controls which communication paths exist
One limits authority.
The other limits reach.
They overlap, but they do not solve the same problem.
Shared networking reconnects isolated accounts
Several accounts may connect through a central network.
For bfstore, a network account might own IP address management, transit routing, central egress, private DNS and shared inspection.
DEV ACCOUNT ───────┐
│
STAGING ACCOUNT ───┼──► CENTRAL NETWORK
│
PROD ACCOUNT ──────┘
This provides consistent control and reduces duplicated infrastructure.
It also creates shared dependencies.
A routing mistake may affect several workload accounts.
A central firewall policy may block legitimate production traffic.
The administrative accounts remain separate.
The communication fabric is shared. Central networking can improve governance.
Its wide blast radius means changes need stronger testing and staged rollout.
DNS is a hidden shared boundary
Services may run in separate accounts and clusters while relying on one DNS capability. If DNS fails, the underlying services may remain healthy but unreachable by name.
A private hosted-zone error can direct several services to the wrong endpoint.
A resolver rule can affect multiple networks.
The same pattern appears with certificate authorities, identity providers, time synchronisation, artefact registries and central proxies.
Small shared dependencies can reconnect systems that otherwise appear isolated.
Boundaries are nested and interconnected
A production application may sit inside several layers:
ORGANISATION
│
▼
PRODUCTION ACCOUNT
│
▼
PRODUCTION VPC
│
▼
KUBERNETES CLUSTER
│
▼
PAYMENT NAMESPACE
│
▼
PAYMENT POD
Each boundary contains different failures.
A process crash may stop at the Pod.
A runaway workload may be constrained by requests, limits and quotas.
A cluster DNS failure may reach every workload in the cluster.
A routing failure may reach the VPC or shared network.
A central identity compromise may reach several accounts.
The system does not have one blast radius.
It has several possible radii depending on the initiating event.
A useful design tries to stop common failures at the nearest practical boundary:
APPLICATION PROCESS CRASH
contained by workload runtime
COMPROMISED WORKLOAD
contained by workload identity
and network policy
FAILED CLUSTER UPGRADE
contained to one cluster
or environment
BAD DEVELOPMENT DEPLOYMENT
contained to the development account
PRODUCTION ACCOUNT COMPROMISE
unable to erase protected logs
and backups elsewhere
The further a failure travels before meeting resistance, the more systems must participate in recovery.
Isolation must match criticality and operating capacity
It is tempting to conclude that every service should have:
- its own account
- its own VPC
- its own cluster
- its own database
- its own observability stack
This creates smaller individual fault domains.
It also creates a larger operational estate. A neglected cluster is not safer because it is dedicated.
An untested backup account is not recoverable because it is separate.
A VPC with inconsistent routing policy may create more incidents than it contains.
The goal is not maximum fragmentation.
It is deliberate containment that the team can operate reliably.
Not every bfstore capability requires identical isolation.
A recommendation service may tolerate shared cluster placement, graceful degradation and lower recovery priority.
A payment service may justify stricter workload identity, tighter network reach, a dedicated database, stronger deployment isolation and higher operational priority. A useful question is:
What is the cost of sharing
this failure domain?
If the consequence is small, shared infrastructure may be sensible.
If it includes payment loss, data corruption or blocked recovery, stronger isolation may be justified.
The second question is:
Can we operate the additional
boundary reliably?
Isolation that cannot be patched, observed, tested and recovered is only reassuring on a diagram.
Cross-boundary dependencies need explicit ownership
Suppose the production workload account depends on:
- central DNS from the network account
- identity from the identity account
- telemetry from the observability account
- images from shared services
- backups in the backup account
PRODUCTION
│
├── network
├── identity
├── shared services
├── observability
└── backup
The account boundary reduces some risks.
The service still depends on other accounts.
Those relationships should be visible.
The architecture should answer:
- who owns each dependency?
- what happens when it is unavailable?
- can production continue temporarily?
- how is failure detected?
- which recovery path exists?
- which service-level objective applies?
A boundary does not remove dependency.
It makes the dependency cross an explicit interface.
That interface needs ownership.
Testing should prove containment
A diagram may show separate accounts, clusters and networks.
Testing should verify that the expected boundaries work.
Account test
Can the development automation role
modify production infrastructure?
Expected:
denied
Cluster test
Does failure of the development cluster
affect the production Kubernetes API?
Expected:
no
Namespace test
Can the catalog workload
consume all cluster CPU?
Expected:
constrained by requests, limits
and quotas
Network test
Can the review service
connect to the payment database?
Expected:
denied
Evidence test
Can a production workload identity
delete central audit logs?
Expected:
denied
Recovery test
Can the production account be rebuilt
using backups and evidence stored elsewhere?
Expected:
yes, through a tested procedure
Boundaries are architectural claims.
Tests turn them into evidence.
The questions I now ask
Failure
Which failure are we trying to contain?
Boundary
Which layer is capable
of stopping it?
Authority
Which roles, controllers or policies
can cross that boundary?
Dependency
Which shared systems reconnect
otherwise isolated environments?
Criticality
What is the cost of sharing
this failure domain?
Operation
Can the team patch, observe,
test and recover the boundary?
Proof
Have we demonstrated that it
stops the intended failure?
These questions are more useful than a diagram containing several reassuring rectangles.
The mental model I am keeping
FAILURE EVENT
│
▼
WHICH LAYER IS AFFECTED?
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
ADMINISTRATION CONTROL PLANE COMMUNICATION
│ │ │
▼ ▼ ▼
ACCOUNT CLUSTER NETWORK
│ │ │
└──────────────────┼──────────────────┘
▼
DOES THE BOUNDARY HOLD?
│
┌─────────────┴─────────────┐
│ │
▼ ▼
CONTAINED PROPAGATES
│ │
▼ ▼
LOCAL RECOVERY NEXT BOUNDARY TESTED
An account cannot contain every Kubernetes failure.
A cluster cannot contain every identity compromise.
A network cannot contain every cloud API action.
Each boundary has a job.
Resilient architecture comes from understanding those jobs, combining the boundaries deliberately and remaining suspicious of any single box labelled:
isolated
Accounts contain administrative reach, clusters contain shared control-plane failure and networks contain communication. The architecture becomes resilient when those boundaries overlap without being mistaken for one another.
The goal is not to draw more walls.
It is to know which wall should still be standing when a particular part of the system catches fire.