What I now expect from a reproducible cloud environment

At the beginning of the year, I would have described a reproducible cloud environment like this:

Delete the environment.

Run Terraform again.

Receive another environment.

That is part of reproducibility.

It is no longer enough for me.

Terraform may recreate the network, cluster and database while several important questions remain unanswered:

• Did it use the same module and provider versions?
• Was the same application artefact deployed?
• Did the new environment receive the intended identities and policies?
• Was its configuration restored?
• Are logs, metrics and traces reaching the observability platform?
• Can its data be recovered?
• Does it satisfy the same security controls?
• Can anyone prove that its required capabilities work?

An environment can be recreated structurally and still be operationally different.

ENVIRONMENT A

network exists
cluster exists
database exists
application runs


ENVIRONMENT B

network exists
cluster exists
database exists
application runs

The resource lists match.

Their behaviour, authority and recoverability may not.

This has changed what I mean by reproducibility.

An operationally reproducible cloud environment can be recreated from controlled declarations and dependencies, restored to an understood state, and verified against its intended capabilities.

Reproducibility and recovery are related but different.

REPRODUCIBILITY

Can the intended capabilities be recreated
from controlled inputs?


RECOVERABILITY

Can valuable data, management continuity
and operational history be restored?

A production environment needs both.

Creation is not the same as confidence.

Reproducibility means equivalent capability

Cloud environments contain values that may change between creations:

• resource identifiers
• IP addresses
• availability-zone placement
• timestamps
• generated credentials
• service endpoints
• underlying physical hosts

Two environments do not need identical resource IDs to represent the same design.

ENVIRONMENT A

network ID:
    net-71f2

database endpoint:
    catalog-a.internal


ENVIRONMENT B

network ID:
    net-93c8

database endpoint:
    catalog-b.internal

The important question is whether both satisfy the same intended properties:

three availability zones

private application subnets

private database subnets

approved ingress paths

encrypted storage

workload identity

central logging

tested backup policy

Reproducibility is closer to equivalent capability under controlled inputs than byte-for-byte sameness.

The environment may receive new cloud-generated identities.

It should preserve the same architectural promises.

Declarations and controlled inputs preserve intent

A reproducible environment needs a durable description of what should exist.

That description may include:

• Terraform root and child modules
• Kubernetes manifests or Helm values
• IAM policies
• policy-as-code rules
• CI/CD workflow definitions
• configuration schemas
• operational documentation

The declarations should preserve relationships, not only resource presence.

NETWORK
   │
   ├──► APPLICATION SUBNETS
   ├──► DATABASE SUBNETS
   └──► APPROVED EGRESS

CLUSTER
   │
   ├──► WORKLOAD IDENTITIES
   ├──► OBSERVABILITY EXPORT
   └──► APPLICATION RELEASES

A collection of console screenshots may help explain what once existed.

It is not a dependable reconstruction mechanism.

The intended architecture needs to be machine-readable enough to execute and human-readable enough to review.

Infrastructure code is only one input. The result can also depend on:

• Terraform CLI version
• provider versions
• module revisions
• variable values
• cloud account and region
• organisation policy
• external data sources
• service quotas
• container images
• deployment configuration

A module source such as:

module "network" {
  source = "../../modules/network"
}

is controlled only as precisely as the repository revision containing it.

A remote module should use an immutable or versioned source.

Terraform itself should run through a controlled toolchain. Provider selections should be recorded in the dependency lock file. Module revisions should be identifiable.

Application deployments need the same precision.

WEAK RELEASE INPUT

catalog-service:latest


STRONGER RELEASE INPUT

catalog-service@sha256:7b91...

Reusable modules should preserve architectural decisions rather than historical accidents.

ACCIDENT

one route table happened to be named rt-3


DECISION

each private subnet receives
the approved egress path

The caller supplies meaningful environmental choices.

The module supplies the approved implementation.

Documentation explains what code cannot: why a CIDR was selected, why a repository is trusted, how access is revoked, which break-glass procedure exists and who owns future changes.

Code answers:

What will automation create?

Documentation answers:

Why is this the environment
we intended to create?

Authority and drift must remain explicit

A reproducible environment rarely has one universal source of truth.

Different systems are authoritative for different domains.

GIT

declared infrastructure and deployment intent


TERRAFORM STATE

mapping between declarations and remote resources


CLOUD PLATFORM

resources that currently exist


CONTAINER REGISTRY

immutable application artefacts


SECRET MANAGER

current sensitive configuration

If Git declares three replicas but an operator changes production to five, which value should survive the next reconciliation?

If a database exists in the cloud but is absent from Terraform state, who owns it?

If the production tag moves, which digest should the cluster run?

Ambiguous ownership turns reconstruction into guesswork.

Drift makes the problem worse over time.

Manual changes may alter:

• firewall rules
• replica counts
• IAM policies
• route tables
• tags
• backup retention
• deployment configuration

Some changes are mistakes.

Some are emergency interventions.

Some are necessary experiments.

The lasting problem is unmanaged ambiguity.

DETECT CHANGE
      │
      ▼
IDENTIFY OWNER AND REASON
      │
      ├── codify
      ├── revert
      └── retain as approved exception

The goal is not to forbid humans from touching production under every circumstance.

It is to ensure that the next reconstruction does not quietly omit an undocumented change on which production now depends.

State preserves management continuity

Terraform state does not contain the cloud environment itself.

It contains Terraform’s record of which real objects satisfy the configuration.

CONFIGURATION ADDRESS

module.network.aws_vpc.this
        │
        ▼
STATE
        │
        ▼
REMOTE RESOURCE

vpc-071f...

If state is lost, the network may continue functioning.

Terraform may no longer manage it safely.

Recovering an existing environment may require restoring or reconstructing its previous state mappings.

Creating a genuinely new replacement environment normally produces new state containing new remote identities.

In both cases, the management record must accurately identify the resources automation now owns.

That requires:

• remote state storage
• restricted access
• encryption
• locking
• version history
• backups
• restore testing
• deliberate state boundaries

Restoring infrastructure without restoring management continuity creates an environment that runs but cannot be changed confidently.

That is not complete recovery.

Identity and policy are part of the environment

Recreating compute without recreating its access model can produce either a useless environment or a dangerous one.

The environment needs identities for:

• human operators
• CI/CD workflows
• Terraform automation
• Kubernetes workloads
• backup processes
• observability ingestion
• security inspection

Each identity needs two clear boundaries:

TRUST

Who may obtain this authority?


PERMISSIONS

What may that authority do?

A rebuilt production environment should not depend on copying permanent administrator credentials from an old pipeline.

It should re-establish controlled trust:

HUMAN

identity provider
      │
      ▼
temporary account role


CI WORKFLOW

OIDC token
      │
      ▼
temporary deployment role


KUBERNETES WORKLOAD

service account
      │
      ▼
temporary cloud role

Policies are also part of the architecture.

A recreated resource can have the correct name and the wrong security posture.

The intended environment may depend on:

• approved regions
• mandatory encryption
• restricted public access
• central audit logging
• protected backups
• least-privilege roles
• required tags
• prohibited privileged workloads

Different controls belong at different layers.

MODULE GUARANTEE

storage encryption enabled


PIPELINE POLICY

production plan contains no public database


ORGANISATION GUARDRAIL

central audit logging cannot be disabled


ADMISSION POLICY

privileged containers rejected

Reproducibility includes reproducing the constraints, not only the resources constrained by them.

Software and configuration need traceable identities

A recreated environment should deploy an identified application artefact.

It should not compile a fresh production binary merely because the infrastructure was rebuilt.

SOURCE REVISION
      │
      ▼
CONTROLLED BUILD PROCESS
      │
      ▼
IMMUTABLE IMAGE DIGEST
      │
      ├──► old environment
      └──► recreated environment

This preserves the distinction between:

REBUILD INFRASTRUCTURE

and

REBUILD APPLICATION SOFTWARE

The image digest identifies the executable content.

The complete deployed release also needs the identity of the declaration and configuration under which that content runs.

application artefact:
    sha256:7b91...

deployment declaration:
    commit a42d6e1

environment configuration:
    version prod-184

Environment configuration may live in:

• ConfigMaps
• parameter stores
• secret managers
• encrypted configuration repositories
• deployment manifests
• managed database records

A reconstruction process needs to know:

Which configuration is declarative?

Which values are generated?

Which secrets must be restored?

Which values can be recreated safely?

Which values must remain stable?

A generated database password can be replaced if every authorised consumer receives the new value.

An external provider credential may need to be restored or rotated through a separate process.

A signing key may have consequences extending far beyond one environment.

Treating all secrets as interchangeable strings hides their different lifecycles.

Stateful systems require tested recovery

Stateless compute is usually easy to recreate.

Data is not.

APPLICATION POD

replace from image


DATABASE CONTENT

restore from durable evidence

A new empty database may satisfy the Terraform configuration while failing the business completely.

Stateful recovery needs:

• backups
• retention policy
• replication where appropriate
• restore procedures
• integrity checks
• recovery objectives
• application reconciliation

A backup is not proven merely because a scheduled job reports success.

BACKUP
   │
   ▼
RESTORE TEST
   │
   ▼
APPLICATION VERIFICATION

For bfstore, verification might include proving that:

• catalog records are present
• basket and order schemas remain compatible
• Kafka consumers resume safely
• outbox records are not republished incorrectly
• payment reconciliation can continue
• application identities can access restored data

Infrastructure recreation and data recovery are related procedures.

They are not the same procedure.

Verification must prove operational capability

Terraform may report:

Apply complete.

That proves Terraform completed the planned operations according to the provider’s reported results.

It does not prove that:

• routes carry the intended traffic
• DNS resolves
• workloads become ready
• databases accept connections
• logs reach their destination
• alerts are active
• backups are usable
• customers can complete an order

Verification needs several layers.

Observability should arrive with the environment.

A reconstructed environment that cannot explain its failures is only partially restored.

Telemetry should identify:

• service
• environment
• region
• cluster
• deployment version
• image digest
• trace identity
• business-operation identity

The verification suite should check the observability path itself:

emit test log

record test metric

create test trace

confirm ingestion

A reconstruction process that exists only in documentation is still a hypothesis.

Teams should periodically create a replacement environment, run the verification suite and record every missing dependency, manual step or undocumented assumption they discover.

External prerequisites must be known

Not every dependency belongs to the environment being reconstructed.

A useful classification is:

ENVIRONMENT-OWNED

network, cluster, workloads,
roles, configuration


PLATFORM PREREQUISITES

organisation policy, identity provider,
registry, state backend, DNS, backup systems


EXTERNAL DEPENDENCIES

payment provider, email service,
third-party credentials

The reconstruction procedure should state which dependencies it creates, which it reconnects to and which must already exist.

Otherwise, “rebuild the environment” can quietly assume that its most important control systems survived elsewhere.

The same applies to service quotas, DNS delegations, registry retention, external contracts and break-glass access.

Hidden prerequisites are often the manual knowledge that makes an environment look reproducible until somebody else tries to create it.

A possible bfstore reconstruction path

A bfstore development environment might be reconstructed through this controlled flow:

1. Confirm platform prerequisites and external dependencies.

2. Select the target cloud account and region.

3. Obtain temporary Terraform credentials.

4. Initialise the approved backend and state boundary.

5. Use a controlled Terraform version,
   provider lock file and identified module revisions.

6. Apply organisation, network and platform declarations.

7. Create workload identities, policies and secret references.

8. Restore or initialise required data.

9. Deploy approved image digests using
   identified declarations and configuration.

10. Verify networking, platform and business flows.

11. Confirm logs, metrics and traces are ingested.

12. Test backup and recovery controls.

13. Store the resulting versions, approvals and evidence.

The output is not merely:

a cluster exists

It is:

the intended environment exists

using controlled inputs

under known authority

running approved artefacts

with recoverable data

and evidence that its capabilities work

That is the standard I now want.

A reproducibility contract

Cloud-generated resource IDs, timestamps, temporary credentials and physical placement do not need to remain identical unless the environment’s contract depends on them.

The environment should reproduce the properties that matter:

security boundary

availability model

network relationships

identity model

application version

data integrity

operational capability

The mental model I am keeping

My earlier model was:

REPRODUCIBLE ENVIRONMENT

Terraform can create
the resources again.

The stronger model is:

                         DECLARED ARCHITECTURE
                                  │
                                  ▼
                         CONTROLLED INPUTS
                                  │
                 ┌────────────────┼────────────────┐
                 │                │                │
                 ▼                ▼                ▼
             TOOLCHAIN          STATE          IDENTITY
                 │                │                │
                 └────────────────┼────────────────┘
                                  ▼
                         INFRASTRUCTURE APPLY
                                  │
                                  ▼
                       DATA AND CONFIGURATION
                                  │
                                  ▼
                       IMMUTABLE APPLICATIONS
                                  │
                                  ▼
                            VERIFICATION
                                  │
                 ┌────────────────┼────────────────┐
                 │                │                │
                 ▼                ▼                ▼
             TELEMETRY         RECOVERY         EVIDENCE
                                  │
                                  ▼
                         TRUSTED ENVIRONMENT

A reproducible cloud environment is not a photocopy of an earlier resource list.

It is a recoverable expression of an architecture.

Its networks can be rebuilt because their address plan is known.

Its identities can be restored because trust is declared.

Its workloads can be redeployed because artefacts and declarations are identifiable.

Its data can be recovered because backups have been tested.

Its behaviour can be examined because telemetry arrives with it.

Its history can be reconstructed because plans, approvals and versions remain available.

The environment is reproducible when another authorised engineer can rebuild it without relying on invisible knowledge, inherited credentials or a sequence of console clicks remembered by one heroic operator.

That is what I now expect:

Not merely infrastructure that automation can recreate, but a complete operating environment whose design, authority, software, data and evidence can be recovered together.

References and further reading

Terraform state Explains how Terraform maps configuration addresses to remote infrastructure.

Terraform dependency lock file Documents how Terraform records selected provider versions for repeatable installation.

AWS IAM roles Introduces role trust, permissions and temporary credentials.

Kubernetes declarative management Describes managing Kubernetes objects from declared configuration.

OpenTelemetry documentation Introduces vendor-neutral instrumentation, telemetry and context propagation.

Google SRE Workbook: Disaster recovery planning Discusses recovery planning, testing and operational readiness.