bfstore-network

The name appears straightforward.

Then almost every resource involved in communication begins making a case for residence there: load balancers, VPC endpoints, Route 53 zones, security groups and even EKS clusters.

By that reasoning, the network account becomes the reluctant landlord of half the AWS estate.

That is centralisation by vocabulary rather than responsibility.

The network account should not own every resource involved in communication.

It should own the shared connectivity capabilities through which independently operated accounts obtain addresses, exchange routes, reach common destinations and cross external network boundaries.

AWS guidance recommends a dedicated networking account for shared services such as Transit Gateway and Amazon VPC IP Address Manager. It also presents centralised egress and decentralised ingress as a useful multi-account pattern.

For bfstore, the opening boundary is:

NETWORK ACCOUNT

Owns network capabilities
shared by several accounts.


WORKLOAD ACCOUNT

Owns network resources
whose lifecycle follows one workload.

The network account should own the roads shared by several properties. It should not own every front door, driveway and room merely because people travel through them.

The central ownership boundary

The network account owns the shared connectivity control plane: address allocation, transit relationships, common egress, inspection, DNS distribution, external connectivity and resource sharing.

It may also operate shared data-plane services such as Transit Gateway, NAT gateways, Network Firewall, Resolver endpoints and VPN attachments.

Workload-specific data planes remain with the workloads:

• workload VPCs;
• application subnets;
• local route tables;
• internet-facing and internal load balancers;
• application security groups;
• Kubernetes network interfaces;
• database network interfaces;
• workload-specific VPC endpoints.

                         bfstore-network
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
        ▼                       ▼                       ▼
 ADDRESS AUTHORITY        TRANSIT AUTHORITY       SHARED DNS
        │                       │                       │
        ▼                       ▼                       ▼
       IPAM              TRANSIT GATEWAY        PROFILES / RULES
        │                       │                       │
        └───────────────────────┼───────────────────────┘
                                ▼
                    SHARED EGRESS AND INSPECTION
                                │
               ┌────────────────┼────────────────┐
               │                │                │
               ▼                ▼                ▼
          bfstore-dev     bfstore-staging    bfstore-prod
               │                │                │
               ▼                ▼                ▼
        WORKLOAD VPC       WORKLOAD VPC       WORKLOAD VPC

The network account sets the shared rules of movement.

The workload accounts retain ownership of the workloads being moved.

Address authority belongs centrally

The first capability I would place in bfstore-network is Amazon VPC IP Address Manager.

bfstore’s address plan must cover every environment, shared platform, future AWS Region and external cloud or network.

If each account chooses its own CIDRs independently, overlapping address space eventually becomes a connectivity constraint.

Transit Gateway does not make overlapping networks conveniently transitive.

Direct private connectivity between overlapping ranges requires redesign, translation or deliberate isolation.

AWS supports delegating IPAM administration to an AWS Organizations member account, allowing the management account to avoid everyday IPAM administration.

For bfstore:

AWS ORGANIZATIONS MANAGEMENT ACCOUNT
              │
              │ delegates IPAM administration
              ▼
       bfstore-network
              │
              ▼
             IPAM

IPAM can be bfstore’s central AWS allocation authority.

It can allocate and monitor AWS CIDRs, while externally owned ranges are recorded as reservations in the broader address plan.

BFSTORE ADDRESS PLAN
        │
        ▼
AMAZON VPC IPAM
│
├── allocates and monitors AWS CIDRs
├── reserves planned Azure ranges
├── reserves planned GCP ranges
└── records external network space


AZURE AND GCP

enforce their allocations
through their own provider controls

Amazon VPC IPAM does not automatically discover or enforce Azure and GCP allocations in the same way it integrates with AWS Organizations resources.

The AWS pools might begin with:

bfstore
│
├── eu-west-2
│   ├── infrastructure
│   │   ├── network
│   │   ├── shared-services
│   │   ├── observability
│   │   └── backup
│   │
│   ├── workloads
│   │   ├── development
│   │   ├── staging
│   │   └── production
│   │
│   ├── sandbox
│   └── policy-test
│
└── future-regions

The network account owns IPAM, pools, allocation rules, sharing, compliance, overlap detection and external reservations.

The workload account owns the VPC created from the allocation, its subnet design and the resources consuming those addresses.

NETWORK ACCOUNT

You may allocate a /20
from the production pool.


WORKLOAD ACCOUNT

This /20 becomes the production VPC.

Its subnets follow
the workload's availability needs.

Address governance belongs centrally.

Every subnet decision does not.

Transit authority belongs centrally

AWS Transit Gateway is a Regional routing hub for VPC, VPN, Direct Connect and other supported attachments.

bfstore should create it in bfstore-network and share it with participating accounts through AWS RAM.

The network account owns the Transit Gateway, its route tables, attachment acceptance, associations, propagation, static and blackhole routes, peering, VPN attachments and network-function integration.

When bfstore-network shares the Transit Gateway, a workload account creates the VPC attachment for the VPC it owns and selects one attachment subnet in each required Availability Zone.

The network account accepts the attachment where required and controls its Transit Gateway route-table association, propagation and transit routes.

bfstore-network

Transit Gateway
Transit Gateway route tables
route segmentation
attachment governance


bfstore-prod

production VPC
VPC attachment request
attachment subnets
local route tables
security groups

The Transit Gateway route tables express communication policy.

PRODUCTION TRANSIT ROUTE TABLE

may reach:
    shared services
    observability
    backup
    central egress

must not reach:
    development
    sandbox
    policy-test


NON-PRODUCTION TRANSIT ROUTE TABLE

may reach:
    shared services
    non-production observability
    central egress

must not reach:
    production

I would disable automatic default route-table association and propagation, then associate and propagate attachments explicitly.

ATTACHMENT ASSOCIATION

Which Transit Gateway route table
governs traffic arriving here?


ROUTE PROPAGATION

Which attachment destinations
may appear in this route table?

Those decisions affect relationships between accounts, so they belong centrally.

The workload team still owns the local VPC routes directing selected destinations towards Transit Gateway.

Workload-local networking stays with the workload

Production, development and observability VPCs should live in their respective workload or platform accounts.

Those accounts own VPC lifecycle, subnets, local routes, internet gateways, security groups, network ACLs, endpoint associations, Flow Logs, local DNS associations and connectivity tests.

The network account supplies address allocation, shared transit, common routing policy, egress, DNS configuration and external connectivity.

WORKLOAD VPC

belongs with the workload
because its topology is part
of the workload architecture.


TRANSIT NETWORK

belongs centrally
because it determines how
several VPCs interact.

AWS also supports VPC sharing, which can suit organisations requiring tight central control over topology.

It is not my preferred starting point for bfstore because separate VPC ownership makes teardown, infrastructure state, responsibility and isolation clearer.

The network account should establish the network contract.

It should not deploy every application subnet for every team.

Shared egress should be deliberate

Private workloads still need controlled outbound access for activities such as:

• operating-system updates;
• package repositories;
• payment-provider APIs;
• shipping-provider APIs;
• public identity endpoints;
• selected SaaS integrations;
• public AWS service endpoints where no private endpoint is used.

A central egress design sends internet-bound traffic from spoke VPCs through Transit Gateway towards a network-account egress path.

That path may contain:

• Transit Gateway attachment subnets;
• inspection;
• NAT gateways;
• an internet gateway;
• egress route tables;
• network telemetry;
• zonal failure-domain routing.

PRIVATE WORKLOAD
       │
       ▼
WORKLOAD VPC ROUTE TABLE
       │
       ▼
TRANSIT GATEWAY
       │
       ▼
INSPECTION
       │
       ▼
EGRESS VPC
       │
       ▼
NAT GATEWAY
       │
       ▼
INTERNET GATEWAY

The network account should own the shared route, NAT, internet gateway, inspection integration, availability design and network-level telemetry.

The workload account still owns:

• whether a subnet uses that route;
• application security groups;
• application-level destination controls;
• credentials;
• timeout and retry behaviour.

Central egress answers:

Which shared path carries
internet-bound traffic?

It does not answer:

Is the order service authorised
to call this payment operation?

Central egress can provide:

• shared inspection;
• fewer NAT gateways;
• stable public source addresses;
• common network policy;
• central flow visibility.

It also introduces:

• Transit Gateway processing charges;
• concentrated traffic;
• central failure modes;
• route complexity;
• possible cross-Availability-Zone cost;
• dependence on network-account changes.

The correct decision tree is:

OUTBOUND REQUIREMENT
        │
        ▼
CAN IT USE A PRIVATE SERVICE ENDPOINT?
        │
     ┌──┴──┐
     │     │
    YES    NO
     │     │
     ▼     ▼
 ENDPOINT  DOES IT NEED CONTROLLED
           INTERNET EGRESS?
                │
             ┌──┴──┐
             │     │
            YES    NO
             │     │
             ▼     ▼
       CENTRAL OR LOCAL   NO ROUTE
         EGRESS PATH

Gateway endpoints for Amazon S3 and DynamoDB are VPC-local.

They are associated with route tables in the consuming VPC and cannot be used from another VPC through Transit Gateway, VPC peering, VPN or Direct Connect.

They therefore belong in each consuming workload VPC.

Interface endpoints can be local or shared, depending on cost, DNS, availability and failure-boundary requirements.

A local NAT gateway may still be appropriate where workload independence, latency or cost analysis favours it.

The network account should provide a shared egress capability.

It should not force every byte through it when a simpler private or local path is better.

Inspection belongs in the shared transit architecture

Central inspection is part of the shared routing architecture.

Its account ownership should follow the organisation’s operational and security-policy model.

bfstore has a separate account called:

bfstore-security-tooling

That creates two reasonable ownership arrangements.

NETWORK ACCOUNT

owns routing integration
and shared transit behaviour


SECURITY TOOLING

owns policy objectives,
standards and exceptions

The firewall resource itself may live in either account, provided the Transit Gateway and firewall attachment workflow supports the chosen boundary and operational responsibilities are explicit.

AWS now provides two principal Network Firewall integration models.

Inspection VPC

The traditional design deploys Network Firewall endpoints inside a dedicated VPC.

SPOKE VPC
    │
    ▼
TRANSIT GATEWAY
    │
    ▼
INSPECTION VPC
    │
    ▼
NETWORK FIREWALL ENDPOINTS
    │
    ▼
TRANSIT GATEWAY
    │
    ▼
EGRESS OR DESTINATION

This is useful when bfstore needs:

• detailed control over VPC subnet routing;
• additional network appliances;
• Gateway Load Balancer integration;
• an established inspection-VPC model.

The Transit Gateway VPC attachment must use appliance mode so forward and return flows remain on a consistent Availability Zone path.

Transit Gateway network-function attachment

Transit Gateway can connect directly to AWS Network Firewall through a network-function attachment.

SPOKE VPC
    │
    ▼
TRANSIT GATEWAY
    │
    ▼
NETWORK FUNCTION ATTACHMENT
    │
    ▼
AWS NETWORK FIREWALL
    │
    ▼
TRANSIT GATEWAY DESTINATION

This removes the need to create and operate an inspection VPC.

It reduces subnet and VPC route-table components and keeps traffic steering in Transit Gateway route tables.

Appliance mode is handled by the attachment model.

A firewall attachment created from another account can require acceptance by the Transit Gateway owner, allowing the security account to operate the firewall while the network account retains transit authority.

The bfstore decision should compare:

• account ownership;
• route complexity;
• appliance requirements;
• change workflow;
• failure-domain design;
• operational skills;
• support for future third-party appliances.

The inventory should therefore describe an inspection capability, not assume that an inspection VPC must exist.

For policy rollout:

• begin Route 53 DNS Firewall rules with the ALERT action;
• for Network Firewall, begin with non-blocking or alert-producing rules appropriate to the selected rule type and policy;
• enable logging;
• characterise legitimate traffic;
• introduce drop or reject actions deliberately.

Network Firewall does not have one universal firewall-wide alert mode.

Shared name resolution and private service access

DNS is partly central and partly distributed.

The network account should own common resolution behaviour.

Workload and platform accounts should own names whose lifecycle follows their services.

Common DNS configuration

When required, the network account should own Resolver endpoints, shared rules, DNS Firewall groups, Route 53 Profiles, common query logging, hybrid integration and genuinely shared zones.

A Route 53 Profile can distribute:

• private hosted zones;
• Resolver forwarding and system rules;
• DNS Firewall rule groups;
• Resolver query logging;
• selected VPC DNS settings;
• interface VPC endpoint DNS associations.

Profiles can be shared through AWS RAM and associated with VPCs in other accounts.

bfstore-network

Route 53 Profile
│
├── common Resolver rules
├── DNS Firewall rule groups
├── query logging
├── shared private zones
└── selected endpoint DNS
        │
        ▼
shared with workload accounts

Only one Route 53 Profile can be associated with a VPC.

That makes Profile composition and ownership important.

Locally configured DNS resources can also take precedence when they conflict with Profile-provided configuration.

The network team therefore cannot treat Profiles as several independent policy layers that every central team attaches separately.

bfstore needs one coherent Profile design per VPC population.

Namespace ownership remains distributed

Workload and platform accounts may own service-specific zones, environment aliases, platform discovery, application records and local associations.

For bfstore:

NETWORK ACCOUNT

network.aws.internal.bfstore.example
shared forwarding rules
future hybrid DNS


SHARED-SERVICES ACCOUNT

platform.aws.internal.bfstore.example
Backstage
TerraKube
module registry


OBSERVABILITY ACCOUNT

observability.aws.internal.bfstore.example
telemetry ingestion
query endpoints


WORKLOAD ACCOUNTS

service and environment names
owned entirely by those workloads

Central DNS should provide a coherent namespace.

It should not turn a record for orders into a cross-team planning summit.

Resolver endpoints exist only for resolver boundaries

Resolver endpoints belong centrally when another DNS environment exists.

An inbound endpoint lets another resolver ask Route 53 VPC Resolver for AWS-private answers.

An outbound endpoint lets Route 53 Resolver forward selected namespaces to another authority.

They are useful for:

• on-premises DNS;
• future Azure DNS;
• future GCP DNS;
• private partner or identity services;
• bidirectional hybrid resolution.

Today, bfstore has no on-premises network and no active private connection to Azure or GCP.

network_account_dns:
  inbound_resolver_endpoints: []
  outbound_resolver_endpoints: []
  hybrid_forwarding_rules: []

The network account owns the future capability.

That does not mean it should deploy idle endpoints before another resolver boundary exists.

A central account should contain shared infrastructure with a job.

Not a museum of things bfstore may one day need.

Interface endpoints require an ownership test

Interface VPC endpoints can be deployed in each consuming VPC or centralised in a shared VPC reachable through Transit Gateway.

A consumer-owned endpoint is usually better for one workload, natural private DNS, low latency and independent failure behaviour.

A shared endpoint may be worthwhile when many VPCs consume the service, endpoint-hour cost is material, Transit Gateway already exists, and the DNS, policy and shared failure boundary are acceptable.

WHO CONSUMES THE ENDPOINT?
        │
   ┌────┴────┐
   │         │
ONE VPC    MANY VPCs
   │         │
   ▼         ▼
LOCAL      EVALUATE CENTRAL
ENDPOINT   SHARED ENDPOINT

Historically, central interface-endpoint designs often required custom private hosted zones and Resolver configuration because the endpoint’s managed private DNS applied within the endpoint VPC.

Route 53 Profiles can now distribute selected interface endpoint DNS associations to Profile-associated VPCs.

That can reduce custom private-zone work, but it does not remove routing, security-group, endpoint-policy, zonal availability, failure-boundary, capacity or cost requirements.

A PrivateLink endpoint service normally belongs with the provider service, not the network account.

bfstore-observability

owns:
    observability endpoint service
    provider load balancer
    telemetry application


bfstore-prod

owns:
    consumer interface endpoint
    endpoint security group


bfstore-network

owns:
    shared routing and naming standards

The network account enables connectivity.

It does not become the owner of every privately consumed service.

Public ingress stays with the workload

Customer traffic enters through workload-specific application infrastructure.

For production, that may include:

• Amazon CloudFront;
• AWS WAF;
• an internet-facing ALB;
• Envoy Gateway;
• EKS workloads;
• application certificates;
• host and path routing.

INTERNET
    │
    ▼
PRODUCTION INGRESS
inside bfstore-prod
    │
    ▼
PRODUCTION APPLICATION

The production account owns its load balancers, target groups, certificates, ingress routes, WAF association, DNS intent and customer-facing availability.

Security tooling may own organisation-wide WAF and Shield policy. The network account retains shared address planning, private external connectivity and any common inspection architecture.

It should not own the production ALB merely because an ALB is a network service.

That resource’s lifecycle is coupled to the production application.

Sharing and external boundaries

AWS RAM is one of the network account’s quiet workhorses.

bfstore may use it to share:

• Transit Gateway;
• IPAM pools;
• Route 53 Profiles;
• Resolver rules;
• DNS Firewall rule groups;
• selected prefix lists;
• other supported shared network resources.

Sharing and unsharing semantics are resource-specific. A Transit Gateway attachment, for example, can remain functional after the gateway is unshared even though the participant loses visibility of the gateway.

Every share therefore needs an owner, approved consumers, infrastructure-code evidence, a removal runbook, a tested impact model and rollback.

resource_share:
  name: bfstore-transit-gateway

  resource:
    type: ec2:TransitGateway
    owner: bfstore-network

  principals:
    - bfstore-dev
    - bfstore-staging
    - bfstore-prod
    - bfstore-shared-services
    - bfstore-observability

  change_repository: bfstore-network-infrastructure
  removal_runbook: transit-gateway-share-removal

Hybrid connectivity also belongs in the shared architecture when a real external network exists.

The network account is the natural owner of:

• Site-to-Site VPN;
• customer gateways;
• Transit Gateway VPN attachments;
• hybrid routing policy;
• advertised and accepted prefixes;
• connection monitoring;
• redundant paths.

A Site-to-Site VPN attachment must be created in the account that owns the Transit Gateway.

Direct Connect can involve a more distributed ownership model.

A Direct Connect gateway can be associated with a Transit Gateway in another account through an association proposal.

The Transit Gateway owner proposes the association, while the Direct Connect gateway owner accepts it and can control or override the allowed prefixes.

That is a useful reminder:

CENTRAL NETWORK AUTHORITY

does not always mean

ONE ACCOUNT OWNS
EVERY PROVIDER COMPONENT

bfstore currently has no on-premises data centre and no active Azure or GCP private network.

The circuit should not be created before the relationship exists.

Telemetry crosses the account boundary

The network account creates VPC and Transit Gateway Flow Logs, firewall logs, NAT and transit metrics, Resolver query logs, hybrid-connectivity metrics and CloudTrail change evidence.

The network account should own the configuration that emits these signals.

It should not necessarily store every signal locally.

bfstore-log-archive

Owns:

• durable security evidence;
• long-term network-flow archives;
• organisation CloudTrail;
• recovery copies;
• tightly controlled retention.

Where bfstore requires immutability, the archive design must provide it explicitly through controls such as:

• S3 versioning;
• S3 Object Lock;
• a defined governance or compliance retention mode;
• restricted retention administration;
• KMS key controls;
• CloudTrail log-file integrity validation;
• replication where required.

CloudTrail integrity validation can detect modification or deletion.

It does not prevent either action.

S3 Object Lock compliance mode prevents protected object versions from being overwritten or deleted during their retention period, including by the account root user.

bfstore-observability

Owns:

• searchable network telemetry;
• dashboards;
• route and firewall alerts;
• flow analysis;
• DNS investigations;
• operational correlations.

NETWORK ACCOUNT
      │
      ├── emits network telemetry
      │
      ├────────────► LOG ARCHIVE
      │                durable evidence
      │
      └────────────► OBSERVABILITY
                       operational analysis

A routing failure may need correlation with:

• a workload deployment;
• a DNS change;
• a firewall-rule change;
• an IAM session;
• an application timeout;
• an external-provider incident.

The network account supplies the source.

The observability and archive accounts provide the destinations appropriate to their responsibilities.

What does not belong in the network account

The following would not normally belong in bfstore-network.

The network account may implement shared enforcement resources.

It should not become the vault for application credentials, the home of workload databases or a general shared-services cluster.

A network account full of platform applications becomes another shared-services account with unusually dangerous routing permissions.

The proposed bfstore inventory

The physical VPC layout may evolve.

bfstore-network
│
├── IPAM
├── Transit Gateway
├── AWS RAM shares
│
├── network-services VPC
│   ├── future Resolver endpoints
│   └── selected shared endpoints
│
├── inspection capability
│   ├── inspection VPC
│   └── or TGW network-function attachment
│
└── egress VPC
    ├── NAT gateways
    ├── internet gateway
    └── shared egress routes

Combining functions initially may reduce cost.

Separating them later may improve clarity and blast-radius control.

Infrastructure code should preserve the roles even when the first implementation uses fewer physical VPCs.

Ownership follows the decision

One resource can involve several owners.

Central networking should centralise shared authority.

It should not centralise every task.

Otherwise the network team becomes the queue through which all application delivery must crawl.

The network account is a service

The network account operates production capabilities.

Its service indicators should cover:

• Transit: attachment availability, route correctness, reachability, loss, latency and unintended blackholes.
• Egress and inspection: connection success, NAT errors, zonal availability, firewall health and rejected flows.
• DNS: endpoint health, latency, SERVFAIL, target timeouts, query-log freshness and unusual NXDOMAIN growth.
• IPAM: pool utilisation, non-compliant allocations, overlaps, allocation failures and available capacity.
• Recovery: time to reconstruct transit, firewall and DNS configuration from infrastructure code, including emergency-access validation.

A central network is shared infrastructure.

Shared infrastructure eventually becomes somebody’s outage unless it is operated as a service.

A staged implementation

Phase 1: Address governance

• delegate IPAM administration to bfstore-network;
• define AWS pools and external range reservations;
• share approved pools;
• allocate workload VPC ranges.

Phase 2: Thin network baseline

• establish network-account roles;
• enable CloudTrail and Config integration;
• create telemetry destinations;
• create the infrastructure repository;
• define emergency access.

Phase 3: Transit routing

• create Transit Gateway;
• disable automatic default association and propagation;
• create segmented route tables;
• share the gateway through AWS RAM;
• attach development and shared services first;
• test allowed and forbidden reachability;
• add staging and production deliberately.

Phase 4: Shared DNS governance

• define internal namespace conventions;
• determine central and distributed zone ownership;
• enable Resolver query logging;
• design one Route 53 Profile per VPC population where common configuration justifies it.

Phase 5: Central egress

• create the egress path;
• deploy zonal NAT;
• route selected non-production traffic first;
• measure cost and failure behaviour;
• add production after testing.

Phase 6: Inspection

• compare an inspection VPC with a Transit Gateway network-function attachment;
• establish policy ownership;
• begin DNS Firewall with ALERT;
• begin Network Firewall with suitable non-blocking or alert-producing rules;
• enable logging;
• move to blocking only after legitimate traffic is understood.

Phase 7: External connectivity

Add Resolver endpoints, VPN, Direct Connect or multi-cloud transit only when an actual external environment needs the path.

The network account should grow in response to connectivity relationships.

Not in anticipation of every icon in the AWS console.

The mental model I am keeping

My earlier model was:

NETWORK ACCOUNT

contains AWS networking resources

The stronger model is:

                       AWS ORGANISATION
                              │
                              ▼
                        bfstore-network
                              │
       ┌──────────────────────┼──────────────────────┐
       │                      │                      │
       ▼                      ▼                      ▼
  ADDRESS AUTHORITY      ROUTING AUTHORITY       COMMON DNS
       │                      │                      │
       ▼                      ▼                      ▼
      IPAM              TRANSIT GATEWAY       PROFILES / RULES
       │                      │                      │
       └──────────────────────┼──────────────────────┘
                              ▼
                  SHARED EGRESS AND INSPECTION
                              │
       ┌──────────────────────┼──────────────────────┐
       │                      │                      │
       ▼                      ▼                      ▼
      DEV                  STAGING                 PROD
       │                      │                      │
       ▼                      ▼                      ▼
  WORKLOAD VPC           WORKLOAD VPC          WORKLOAD VPC
       │                      │                      │
       ├── subnets            ├── subnets            ├── subnets
       ├── load balancers     ├── load balancers     ├── load balancers
       ├── security groups    ├── security groups    ├── security groups
       └── applications       └── applications       └── applications

IPAM answers which address space an account may consume.

Transit Gateway answers which attached networks may exchange routes.

Central egress and inspection answer which shared path traffic uses and which flows cross enforcement.

Route 53 Profiles answer which common DNS configuration a VPC population inherits.

Resolver endpoints answer how DNS crosses into another resolver environment.

AWS RAM answers how other accounts consume centrally owned capabilities.

The workload VPC answers:

How is this application environment
segmented and operated locally?

The workload ingress answers:

How does traffic reach
this particular application?

The security account answers:

Which organisation-wide network
behaviour should be prohibited?

The observability account answers:

How do we know the shared
network is still behaving?

And the network account answers:

Which shared connectivity decisions
must remain consistent across bfstore?

That is the account’s real job.

The network account should own the map, the shared junctions and the bridges between territories. The workload accounts should still own the buildings, their entrances and the locks on their doors.

For bfstore, bfstore-network will own IPAM, Transit Gateway, shared route policy, AWS RAM network shares and the eventual shared egress, inspection, hybrid connectivity and Resolver infrastructure.

It will not own the production EKS cluster.

It will not own the order database.

It will not own the customer-facing ALB.

It will not become a general shared-services cluster.

It will not host Backstage merely because Backstage has a hostname.

The account should remain narrow, resilient and slightly dull.

When every other account depends on the roads, the road authority should not also run a furniture shop from the central reservation.

References and further reading

• Foundational infrastructure and the network account

• Centralised egress

• Decentralised ingress

• Work with shared Transit Gateways

• Amazon VPC attachments in Transit Gateway

• Amazon VPC IPAM and AWS Organizations

• Manual IPAM allocations for external reservations

• AWS Transit Gateway network-function attachments

• Transit Gateway attachment configuration for AWS Network Firewall

• Accepting cross-account firewall attachments

• Route 53 Profiles

• Associating interface VPC endpoints with Route 53 Profiles Covers distributing selected interface endpoint DNS associations.

• Centralised interface VPC endpoints Describes central endpoint patterns and their DNS implications.

• Gateway endpoints for Amazon S3 Documents that gateway endpoints are VPC-local and cannot be consumed through Transit Gateway.

• DNS Firewall rule actions Documents the ALERT, ALLOW and BLOCK actions.

• Network Firewall rule actions Covers alerting and enforcement behaviour for different rule types.

• Direct Connect gateways and Transit Gateway associations Covers cross-account association proposals and allowed prefixes.

• S3 Object Lock Covers governance and compliance retention modes for WORM protection.

I need:

• an AWS organisation;
• multiple accounts;
• central logging;
• security oversight;
• workforce access;
• account provisioning;
• organisation-wide controls;
• compliance visibility;
• protection against configuration drift.

AWS Control Tower says, in effect:

Give me the organisation.

I will assemble a governed
multi-account landing zone.

That is not an empty promise.

AWS Control Tower orchestrates services including AWS Organizations, AWS Service Catalog and AWS IAM Identity Center to establish and govern a multi-account environment. It creates and manages landing-zone resources, applies controls and provides a consolidated view of governance state.

For many organisations, this is an excellent starting point.

It converts a long list of foundational tasks into a supported AWS service.

It can reduce the amount of bespoke landing-zone engineering required before application teams receive usable accounts.

It can also make good multi-account patterns more accessible to a company that needs governance quickly.

bfstore still chose not to use it.

That choice was not based on the belief that Control Tower is bad, inflexible or only suitable for beginners.

The current service is much more flexible than its earlier versions. Landing Zone 4.0 allows organisations to define their own structure, makes several integrations optional and even supports a controls-focused environment without deploying a complete traditional landing zone.

My decision was narrower:

Control Tower was capable of producing much of the environment I needed. It would also have taken ownership of several mechanisms I specifically wanted bfstore to expose, test and explain.

The decision was about project purpose, operating model and learning value.

Not product quality.

A landing zone is more than an organisation

AWS Organizations provides the hierarchy.

It gives me:

• a management account;
• member accounts;
• organisational units;
• consolidated billing;
• organisation policies;
• delegated administration;
• trusted service integrations.

That is not yet a complete landing zone.

A production-ready multi-account environment also needs decisions about:

• account purposes;
• workforce access;
• central logging;
• security administration;
• approved Regions;
• account bootstrapping;
• policy attachment;
• detective controls;
• network ownership;
• backups;
• recovery;
• exception handling;
• account vending;
• evidence.

A simplified landing zone looks more like this:

AWS ORGANISATION
       │
       ├── account structure
       ├── identity and access
       ├── preventive controls
       ├── detective controls
       ├── security visibility
       ├── central logging
       ├── account provisioning
       ├── baseline configuration
       └── governance operations

AWS Control Tower coordinates many of these concerns through AWS-native services.

It does not replace AWS Organizations.

It extends it with an opinionated governance and account-management layer.

That distinction matters.

Control Tower is not a separate cloud-governance universe.

Its controls and resources are implemented through familiar AWS mechanisms such as:

• service control policies;
• resource control policies;
• AWS Config rules;
• CloudFormation hooks;
• CloudFormation StackSets;
• CloudTrail;
• IAM roles;
• IAM Identity Center;
• Service Catalog.

The value is orchestration.

AWS owns more of the assembly, lifecycle and conformity checking.

What Control Tower sets up

A full Control Tower landing zone can establish shared governance capabilities and managed resources around an AWS organisation.

Depending on the selected landing-zone version and configuration, Control Tower can provide or integrate:

• security and logging accounts;
• organisation-level CloudTrail;
• AWS Config recording and aggregation;
• IAM Identity Center access;
• baseline IAM roles;
• managed controls;
• account enrolment;
• Account Factory;
• dashboards for governance and drift.

Landing Zone 4.0 makes integrations such as AWS Config, CloudTrail, IAM Identity Center, security roles and AWS Backup more selectively configurable than in earlier versions. It also removes the earlier requirement that every deployment adopt the same mandatory organisational structure.

This is important because an old description of Control Tower might say:

Control Tower forces every organisation
into one fixed OU structure.

That is no longer a fair description of the current service.

A more accurate model is:

CONTROL TOWER
      │
      ├── managed landing-zone baselines
      ├── optional service integrations
      ├── managed control catalogue
      ├── OU and account governance
      ├── account lifecycle workflows
      └── defined drift workflows

The platform remains prescriptive.

It has become more modular about which prescriptions must travel together.

Those choices are connected. In Landing Zone 4.0, disabling the AWS Config integration also removes detective controls, the full Account Factory and Service Catalog path, Account Factory for Terraform, Account Factory Customization and OU baselines. AWS Organizations account creation, account movement and auto-enrolment provide a lighter alternative.

See the Landing Zone 4.0 feature comparison.

Controls are the governance vocabulary

Control Tower calls its governance rules controls.

Older documentation and discussions may call them guardrails. The terms refer to the same general Control Tower concept.

Controls are enabled against organisational units, but inheritance depends on their behaviour.

Preventive controls flow through nested OUs and affect their accounts, including accounts and OUs that are not registered with Control Tower. Detective and proactive controls do not automatically flow into nested OUs. They must be enabled for the relevant registered OUs and operate only in supported governed Regions.

See Nested OUs in AWS Control Tower.

Preventive controls

Preventive controls stop prohibited actions.

They are implemented through AWS Organizations service control policies, resource control policies or declarative policies.

A request that violates the applicable policy is denied before the action succeeds. Preventive controls work across AWS Regions because the underlying organisation policies govern the account rather than depending on a Regional resource assessment.

REQUEST
   │
   ▼
ORGANISATION POLICY
   │
   ├── permitted → continue
   └── denied    → stop

Detective controls

Detective controls evaluate resources for non-compliance.

They are commonly implemented using AWS Config rules.

They do not necessarily prevent the resource from being created. They identify that the resulting state violates the control.

RESOURCE EXISTS
      │
      ▼
CONFIGURATION EVALUATED
      │
      ├── compliant
      └── violation reported

Proactive controls

Proactive controls assess supported resources before CloudFormation deploys them.

They use CloudFormation hooks managed through Control Tower. A non-compliant resource can be rejected during provisioning rather than discovered afterwards.

CLOUDFORMATION CHANGE
        │
        ▼
PROACTIVE CONTROL
        │
        ├── compliant     → provision
        └── non-compliant → reject

These behaviours answer different governance questions:

That model is one of Control Tower’s strongest features.

It moves governance away from one giant policy document and towards controls with explicit enforcement behaviour.

See Control behaviour and guidance.

Managed controls reduce repeated policy engineering

Without Control Tower, a platform team may need to:

1. identify the policy objective;
2. select an enforcement mechanism;
3. write the SCP, RCP, declarative policy, Config rule or hook;
4. test and deploy it;
5. map it to compliance requirements;
6. monitor it;
7. update it as AWS behaviour changes.

The Control Catalog provides a consolidated catalogue of AWS-managed controls organised around objectives, services and compliance frameworks.

AWS currently describes the catalogue as containing more than 750 managed controls across areas such as security, operations, resilience, cost and compliance. The catalogue can be inspected through the console and APIs, and organisations can use the controls-focused experience without deploying a full traditional landing zone.

This can save substantial effort where an organisation needs:

• consistent control definitions;
• documented policy intent;
• compliance-framework mapping;
• broad service coverage;
• supported control updates;
• governance reporting.

A managed control does not remove the need to understand the policy.

It removes some of the repeated implementation work.

See AWS Control Tower features and the Control Catalog reference.

Mandatory controls protect the integrations you enable

Mandatory controls protect the governance resources on which Control Tower depends.

Landing Zone 4.0 changed how they are introduced. Mandatory controls are no longer enabled by default across every OU. AWS deploys the controls required by the integrations and managed resources selected for the environment.

Once an applicable mandatory control is deployed, it protects the associated Control Tower resources and generally cannot be disabled independently.

This is sensible.

A governance service cannot provide reliable oversight if ordinary account administrators can quietly dismantle its:

• roles;
• logging resources;
• policies;
• configuration recorders;
• cross-account access.

Control Tower therefore owns certain resources and expects them to be changed through supported Control Tower workflows. AWS warns that changing or deleting those resources outside supported methods can leave the environment drifted or in an unknown state.

CUSTOMER-MANAGED RESOURCES

change through ordinary
platform workflows


CONTROL-TOWER-MANAGED RESOURCES

change through supported
Control Tower workflows

For a company that wants AWS to own more of the landing-zone machinery, this is beneficial.

For bfstore, it would have obscured some of the machinery I wanted to operate directly.

See Control behaviour and guidance and considerations for controls and OUs.

Baselines make governance repeatable

A Control Tower baseline is a group of resources and configurations applied to a target.

The AWSControlTowerBaseline is the fuller OU-level baseline. It establishes resources and controls needed for compliance monitoring, auditing, security monitoring and optional access management. Accounts in the target OU receive the baseline through enrolment.

REGISTERED OU
      │
      ▼
CONTROL TOWER BASELINE
      │
      ▼
MEMBER ACCOUNTS RECEIVE
REQUIRED GOVERNANCE RESOURCES

That solves a real problem.

Without a baseline mechanism, every new account risks arriving without required roles, logging, Config recording, access configuration, governance policy or central evidence paths.

Landing Zone 4.0 makes the baseline model less monolithic.

ConfigBaseline can establish the narrower Config resources needed for detective controls without the complete AWSControlTowerBaseline. Central functions are represented through separate baselines, including:

• CentralConfigBaseline;
• CentralSecurityRolesBaseline;
• LogArchiveBaseline;
• IdentityCenterBaseline.

The baseline resources are versioned. Landing-zone or OU baseline changes may require updates, re-registration or account-update workflows.

That creates work.

It also makes the work visible and gives it a supported lifecycle.

See Types of baselines and the Landing Zone 4.0 migration guide.

Specific governance drift becomes a first-class concern

A landing zone begins compliant.

Humans and automation then interact with it.

Someone may edit or detach a managed policy, move an account, disable trusted access, remove a required role or create inheritance differences between an OU and its accounts.

Control Tower detects defined classes of governance drift, including:

• OU and account governance drift;
• control-policy drift;
• trusted-access and required-role drift;
• enabled-control inheritance drift;
• enabled-baseline inheritance drift.

The resulting model is:

EXPECTED GOVERNANCE STATE
          │
          ▼
ACTUAL ORGANISATION STATE
          │
          ▼
SUPPORTED DRIFT DETECTION
          │
          ├── no drift
          └── remediation workflow

Drift detection does not mean every deviation heals itself automatically. Administrators may need to reset a control or baseline, re-register an OU, update an account or repair the landing zone.

It is also not a comprehensive state reconciler.

Control Tower does not generally inspect drift in associated services such as CloudTrail, CloudWatch, IAM Identity Center, CloudFormation or AWS Config. It does not detect arbitrary resource changes inside a baseline.

A complete operating model still needs:

• CloudFormation or IaC drift detection;
• AWS Config resource assessment;
• periodic reconciliation;
• alerts for critical service changes.

Control Tower makes important governance deviations visible. It does not replace every other form of configuration assurance.

See Types of governance drift.

Account Factory solves repeated account vending

A growing organisation should not provision accounts through improvised emails and hand-entered console steps.

A new account normally needs:

• a unique email address;
• OU placement;
• workforce access;
• standard tags;
• security roles;
• logging;
• budgets;
• platform integration;
• ownership metadata.

Control Tower Account Factory provides a supported account-provisioning workflow. Accounts are provisioned into governed OUs and receive the corresponding baselines and controls.

Account Factory Customization can apply a CloudFormation or Terraform-backed Service Catalog blueprint during provisioning or enrolment.

ACCOUNT REQUEST
      │
      ▼
VALIDATED PARAMETERS
      │
      ▼
ACCOUNT FACTORY
      │
      ▼
GOVERNED AWS ACCOUNT
      │
      ▼
CUSTOM BLUEPRINT
      │
      ▼
READY FOR TEAM USE

AWS Organizations can create an account.

Account Factory’s value is delivering one that already belongs to the organisation’s operating model.

The full Account Factory, Service Catalog, AFT and AFC paths depend on the Landing Zone 4.0 AWS Config integration. Without it, AWS points to a lighter workflow based on Organizations and auto-enrolment.

See Account Factory and the Config integration comparison.

AFT adds a Terraform and GitOps workflow

Account Factory for Terraform, usually shortened to AFT, adds a Terraform-oriented account-provisioning and customization workflow around Control Tower.

AFT supports account requests through Git, stores account metadata and audit history, and runs global or account-specific customizations through Terraform, Python, Bash and AWS CLI stages.

GIT ACCOUNT REQUEST
        │
        ▼
AFT PROVISIONING PIPELINE
        │
        ▼
CONTROL TOWER ACCOUNT FACTORY
        │
        ▼
NEW ACCOUNT
        │
        ▼
GLOBAL AND ACCOUNT
CUSTOMISATIONS

AFT requires an existing Control Tower landing zone and deploys into a separate AFT management account. It is intended for account provisioning and customization, not as the general deployment system for application infrastructure.

AFT itself has no separate service fee, but the infrastructure it deploys incurs AWS charges.

For a company vending many accounts, it may be a sensible platform component.

For bfstore’s current account count and learning objectives, it would have introduced another provisioning control plane before the repeated journey had justified it.

See Account Factory for Terraform.

What Control Tower solves well

Control Tower is particularly strong when an organisation needs several capabilities together:

• a supported starting point for a governed multi-account environment;
• consistent account onboarding through Account Factory or AFT;
• a broad catalogue of AWS-managed controls;
• account, OU, baseline, control and drift status;
• staged adoption into an existing AWS Organization;
• optional auto-enrolment when accounts move into governed OUs;
• control mappings that support wider compliance programmes;
• an account and control lifecycle that can scale across teams.

None of these are small benefits.

Control Tower solves real platform work.

What Control Tower does not solve

A Control Tower landing zone is governed infrastructure.

It is not a complete cloud platform.

It does not decide bfstore’s:

• business domains;
• Kubernetes architecture;
• database ownership;
• Kafka topology;
• application identity model;
• network segmentation;
• SLOs;
• backup strategy;
• observability backend;
• deployment model;
• incident response;
• cost allocation;
• multi-cloud architecture.

It also does not remove the need to understand AWS permissions.

A preventive control implemented with an SCP still does not grant an IAM permission.

A detective control still requires someone to interpret and remediate the finding.

A compliant AWS Config rule does not prove the application is secure.

A vended account does not prove the workload belongs in that account.

CONTROL TOWER

governs the landing-zone machinery


PLATFORM ARCHITECTURE

decides how teams build and operate


APPLICATION ARCHITECTURE

decides how bfstore behaves

Control Tower shortens the first journey.

It does not complete the other two.

Why I chose not to use it for bfstore

The decision came from six project-specific considerations.

1. The landing zone is part of the curriculum

bfstore is not only an ecommerce system.

It is also a deliberately inspectable cloud-engineering project.

I wanted to understand and document:

• why each OU exists;
• why each account exists;
• how SCP inheritance works;
• how IAM Identity Center assignments work;
• how CloudTrail is centralised;
• how AWS Config should be organised;
• how delegated administration works;
• which responsibilities belong in the management account;
• how policies fail;
• how they are tested;
• how emergency access interacts with governance.

Control Tower would still require knowledge of those subjects.

It would also make several of their initial implementations arrive as managed baseline resources.

For an enterprise trying to reduce setup work, that is an advantage.

For a learning project trying to expose the setup work, it is a layer of smoked glass.

I did not merely want the answer. I wanted the workings in the margin.

2. The organisation structure was already deliberately designed

bfstore’s structure is not a generic Security, Sandbox and Workloads arrangement.

The organisation separates concerns across OUs and accounts including:

• management;
• security tooling;
• log archive;
• identity;
• network;
• shared services;
• observability;
• backup;
• development;
• staging;
• production;
• sandbox;
• policy testing.

The design is based on:

• failure boundaries;
• policy boundaries;
• operational ownership;
• data flows;
• recovery responsibilities;
• independent platform projects.

Control Tower 4.0 now supports customer-defined organisational structures, so the mere existence of this structure is no longer a strong argument against adopting the service.

The relevant point is sequence.

The structure, accounts, identity assignments and initial policies were already being built deliberately.

Introducing a complete landing zone would have required deciding which existing resources should become Control Tower-managed, which OUs should be registered and how current logging and identity choices should be reconciled with the selected baseline.

That migration would have added product work without addressing an immediate bfstore problem.

3. I wanted a policy-testing laboratory

bfstore has a dedicated bfstore-policy-test account.

Its purpose is to test:

• SCP allow and deny behaviour;
• RCP behaviour;
• policy inheritance;
• exceptions;
• policy syntax;
• service interactions;
• expected and unexpected denial paths.

The desired workflow is:

WRITE POLICY
     │
     ▼
STATIC VALIDATION
     │
     ▼
POLICY-TEST OU OR ACCOUNT
     │
     ▼
POSITIVE TESTS
     │
     ▼
NEGATIVE TESTS
     │
     ▼
EVIDENCE
     │
     ▼
PROMOTE TO REAL OUs

Control Tower provides managed controls whose implementations are already tested and supported by AWS.

That is useful.

It is not the same learning exercise as constructing a custom policy, predicting its evaluation and proving its effect through an explicit harness.

bfstore needs both concepts:

• awareness of AWS-managed controls;
• practical competence building and testing bespoke controls.

The dedicated policy-test account places the second objective in the foreground.

4. Account vending is not yet the bottleneck

Account Factory becomes especially valuable when account creation is repeated frequently.

bfstore currently has a known, finite set of accounts with deliberately assigned purposes.

It is not yet serving:

• dozens of product teams;
• frequent project onboarding;
• temporary account fleets;
• departmental account requests;
• customer-specific environments.

A full account-vending platform would therefore optimise work that is currently infrequent.

The account lifecycle still needs documentation and automation.

It does not yet need a large self-service factory.

This may change later.

A platform should be introduced when its repeated journey exists, not because its architecture diagram is handsome.

5. bfstore already has an infrastructure control-plane direction

bfstore selected:

• OpenTofu-compatible infrastructure;
• TerraKube for infrastructure delivery and governance;
• GitHub Actions;
• reusable cloud modules;
• a private module-registry path;
• later Crossplane capabilities.

AFT supports Terraform and GitOps-style account requests, so it is not philosophically opposed to this stack.

It would still introduce:

• an AFT management account;
• AFT pipelines;
• Service Catalog relationships;
• AFT-specific account metadata;
• customization repositories;
• another lifecycle to upgrade and observe.

That might be justified when AFT owns a large account-vending workload.

It would be excessive while bfstore is still establishing its broader infrastructure-delivery model.

I prefer one visible control plane for each responsibility.

ACCOUNT GOVERNANCE

currently:
explicit organisation and IaC workflow


INFRASTRUCTURE DELIVERY

TerraKube and OpenTofu direction


WORKLOAD DELIVERY

GitHub Actions and Kubernetes tooling

Adding AFT now would create another governance engine before bfstore had enough account churn to feed it.

6. The project needs cloud-neutral reasoning

bfstore is being implemented in this order:

AWS
 │
 ▼
AZURE
 │
 ▼
GCP
 │
 ▼
MULTI-CLOUD CAPSTONE

The objective is not to make all three providers identical.

Provider-native services should be used where they offer real value.

The objective is to understand the underlying pattern clearly enough to recognise its equivalents and differences elsewhere.

Building the AWS landing zone from primitives makes the pattern visible:

ORGANISATION HIERARCHY

ACCOUNT OR SUBSCRIPTION BOUNDARIES

CENTRAL IDENTITY

PREVENTIVE POLICY

DETECTIVE POLICY

CENTRAL LOGGING

ACCOUNT VENDING

DRIFT MANAGEMENT

Control Tower could still be documented and compared.

It simply would not be the machinery through which I first learned the pattern.

Manual does not mean unmanaged

The alternative to Control Tower is not:

Click around the management account
until the organisation looks convincing.

bfstore’s alternative is an explicitly engineered landing zone.

It should include:

• infrastructure as code;
• version-controlled policies;
• policy tests;
• documented account ownership;
• centralised CloudTrail;
• AWS Config design;
• IAM Identity Center;
• delegated administrators;
• account bootstrap procedures;
• drift detection;
• runbooks;
• evidence.

The difference is ownership.

CONTROL TOWER MODEL

AWS manages landing-zone
baselines and control lifecycle


BFSTORE MODEL

bfstore defines, tests and operates
those mechanisms directly

A custom landing zone must reproduce the operational discipline Control Tower would otherwise help provide.

Declining the service does not decline the problem.

It accepts more of it.

What bfstore built instead

The bfstore organisation begins with explicit account boundaries.

BFSTORE MANAGEMENT
        │
        ├── Security
        │   ├── security-tooling
        │   └── emergency-access
        │
        ├── Logging
        │   └── log-archive
        │
        ├── Identity
        │   └── identity
        │
        ├── Network
        │   └── network
        │
        ├── Shared Services
        │   └── shared-services
        │
        ├── Observability
        │   └── observability
        │
        ├── Backup
        │   └── backup
        │
        ├── Workloads
        │   ├── dev
        │   ├── staging
        │   └── prod
        │
        ├── Sandbox
        │   └── sandbox
        │
        └── Policy Test
            └── policy-test

The account names are less important than their responsibilities.

AWS Organizations

Owns:

• hierarchy;
• account membership;
• SCPs;
• RCPs;
• delegated-service access;
• consolidated billing.

IAM Identity Center

Owns:

• workforce sign-in;
• permission sets;
• group assignments;
• temporary account access.

Log archive

Owns:

• organisation CloudTrail;
• selected AWS Config evidence;
• immutable or tightly controlled central logs;
• long-term security evidence.

Security tooling

Owns:

• delegated security services;
• organisation-wide findings;
• security operations;
• investigation access.

Policy test

Owns:

• policy validation;
• expected-denial tests;
• expected-success tests;
• policy evidence before promotion.

Infrastructure repositories

Own:

• policy source;
• OU and account definitions;
• baseline modules;
• deployment workflows;
• change history.

This is not automatically better than Control Tower.

It is more educational and more directly aligned with bfstore’s current purpose.

The costs I accepted

Choosing not to use Control Tower creates work.

bfstore must:

• maintain the account baseline, including roles, logging, access, tags and policies;
• detect drift through AWS Config, CloudTrail, EventBridge, IaC plans and reconciliation;
• place, bootstrap and verify every new account;
• maintain custom SCPs, RCPs, Config rules and other controls as AWS changes;
• build governance reporting rather than receive a consolidated Control Tower view;
• test module, service and policy changes without a Control Tower landing-zone upgrade workflow;
• prove that every account received the intended baseline.

These are not footnotes.

They are the invoice for owning the machinery.

Control Tower is free, its ingredients are not

AWS does not charge an additional fee for Control Tower itself.

Customers pay for the services and resources it enables, including CloudTrail, Config, CloudWatch, Service Catalog and associated storage or processing.

The same is true of a custom landing zone.

Avoiding Control Tower does not avoid the cost of organisation trails, Config evaluations, log storage, security services, automation or engineering time.

CONTROL TOWER COST

underlying AWS services
+
managed design constraints


CUSTOM LANDING-ZONE COST

underlying AWS services
+
engineering and operational ownership

The cheaper option depends on scale, requirements and the value assigned to engineering time.

For bfstore, some of that engineering time is part of the project’s educational output. In a commercial platform team, it would be judged more sternly.

Where I would choose Control Tower

I would strongly consider Control Tower when:

• the organisation is beginning its multi-account journey and needs a supported baseline quickly;
• accounts are created regularly enough for Account Factory or AFT to become a platform product;
• a small cloud team benefits from AWS owning more of the baseline lifecycle;
• managed control definitions and compliance mappings remove meaningful repeated work;
• governance drift is already difficult to see and repair;
• AWS is the organisation’s primary cloud;
• several teams need consistent account and control workflows;
• the company wants AWS support around the landing-zone product.

Control Tower is not an admission that the platform team lacks skill.

It may be evidence that the team is applying its skill to the business-specific layers where differentiation matters.

Landing Zone 4.0 makes future adoption easier

The choice is not permanently binary.

Landing Zone 4.0 supports a controls-focused environment for organisations that already have an AWS Organizations structure and primarily want managed controls. It no longer requires every customer to adopt the complete earlier landing-zone shape.

That gives bfstore several future paths:

• keep managing the custom landing zone;
• use the Control Catalog for research and mapping;
• adopt a controls-focused environment on selected OUs;
• register suitable OUs under fuller governance;
• introduce Account Factory or AFT when account demand justifies it.

Architecture decisions should be revisited when their assumptions change.

See the Landing Zone 4.0 migration guide.

My decision record

The bfstore decision can be expressed compactly.

decision:
  service: AWS Control Tower
  outcome: not-adopted-for-initial-landing-zone

context:
  - organisation and accounts designed explicitly
  - project requires deep AWS governance learning
  - account creation frequency is currently low
  - custom policy-test workflow is a primary objective
  - OpenTofu and TerraKube form the infrastructure direction
  - existing identity and account setup is already in progress
  - AWS will later be compared with Azure and GCP

benefits_declined:
  - managed landing-zone baseline
  - Account Factory
  - managed drift workflows
  - consolidated governance dashboard
  - managed control lifecycle

responsibilities_accepted:
  - account bootstrap
  - policy implementation
  - policy testing
  - drift detection
  - governance reporting
  - baseline upgrades
  - account lifecycle automation

revisit_when:
  - account vending becomes frequent
  - team count grows
  - compliance-control mappings become operationally important
  - custom baseline maintenance becomes disproportionate
  - controls-focused adoption offers clear value

That is more precise than:

We prefer doing things ourselves.

Preferences are not architecture.

Trade-offs are.

The mental model I am keeping

My earlier model was:

AWS CONTROL TOWER

automatic AWS organisation


NO CONTROL TOWER

manual AWS organisation

The stronger model is:

                 GOVERNED MULTI-ACCOUNT FOUNDATION
                              │
              ┌───────────────┴───────────────┐
              │                               │
              ▼                               ▼
       AWS CONTROL TOWER              CUSTOM LANDING ZONE
              │                               │
              ▼                               ▼
      AWS-MANAGED BASELINES          CUSTOMER-MANAGED BASELINES
      MANAGED CONTROLS               CUSTOM CONTROLS
      ACCOUNT FACTORY                CUSTOM ACCOUNT WORKFLOW
      GOVERNANCE DRIFT STATUS        CUSTOM RECONCILIATION

AWS Organizations defines the hierarchy.

Control Tower assembles and operates a managed governance layer around it.

A custom landing zone accepts direct ownership of the same governance responsibilities.

I did not reject Control Tower because it hides poor engineering. I declined it because it successfully hides engineering that this project exists to learn.

For bfstore today, the custom route remains intentional.

Control Tower stays in the architectural catalogue, understood rather than dismissed. The service solves a real problem. I have chosen, for now, to keep that problem where I can see its gears.

References and further reading

• What is AWS Control Tower?
• Landing Zone 4.0 migration guide
• Feature comparison with and without AWS Config integration
• About AWS Control Tower controls
• Control behaviour and guidance
• Nested OUs and control inheritance
• The AWS Control Tower Control Catalog
• Types of baselines
• Types of governance drift
• Account Factory
• Account Factory for Terraform
• AWS Control Tower pricing

The platform also uses AWS-managed services and control planes:

• Application Load Balancers
• Network Load Balancers
• Amazon RDS for MySQL
• Amazon VPC
• AWS Transit Gateway
• AWS CloudTrail
• AWS Config
• AWS IAM
• Amazon EKS
• Route 53 Resolver

The platform is self-hosted.

Its backends may include:

• a Prometheus-compatible metrics system
• Thanos for durable metrics
• ClickHouse or OpenSearch for logs and events
• an OpenTelemetry-compatible trace store
• Grafana for visualisation

The first architecture sketch is tempting:

AWS
 │
 ▼
OpenTelemetry Collector
 │
 ▼
self-hosted observability

It is also too vague to build.

AWS telemetry does not originate through one universal interface.

A bfstore Go service can emit OTLP directly.

An EKS control plane cannot.

An RDS instance exposes metrics through CloudWatch and selected engine logs through CloudWatch Logs.

VPC and Transit Gateway Flow Logs can be delivered to CloudWatch Logs, Amazon S3 or Amazon Data Firehose.

CloudTrail trails primarily deliver durable event files to S3, with optional CloudWatch Logs and EventBridge integrations.

The phrase:

export AWS telemetry

therefore hides several different jobs.

A stronger architecture begins by separating three source classes.

WORKLOAD-NATIVE TELEMETRY

bfstore application logs
application metrics
distributed traces
runtime metrics


PLATFORM TELEMETRY

Kubernetes node metrics
Pod and container metrics
cluster object state
container logs


AWS-MANAGED TELEMETRY

CloudWatch metrics
managed-service logs
flow logs
CloudTrail
AWS Config
AWS service events

Self-hosted observability does not mean pretending CloudWatch does not exist. It means using the most appropriate AWS collection path while retaining control of the platform where telemetry is stored, correlated and analysed.

The source determines the collection path.

The platform determines the final destination.

Collection, transport and storage are different decisions

A telemetry pipeline has several distinct stages.

SOURCE
   │
   ▼
COLLECTION
   │
   ▼
TRANSPORT
   │
   ▼
BUFFER
   │
   ▼
PROCESSING
   │
   ▼
BACKEND

Source

Where did the signal originate?

Examples:

• Go application
• Kubernetes node
• EKS control plane
• RDS MySQL
• VPC network interface
• AWS API call

Collection

Which system can obtain the signal?

Examples:

• OpenTelemetry SDK
• OpenTelemetry Collector
• CloudWatch
• VPC Flow Logs
• CloudTrail
• AWS Config

Transport

How does the signal leave the collection system?

Examples:

• OTLP
• Prometheus remote write
• CloudWatch Logs subscription
• CloudWatch metric stream
• Kinesis Data Streams
• Amazon Data Firehose
• S3 object delivery

Buffer

Where can telemetry wait safely when the destination is unavailable?

Examples:

• Collector sending queue
• Kinesis retention
• Firehose buffering
• S3 landing bucket
• local persistent storage

Processing

What must happen before storage?

Examples:

• decompression
• schema parsing
• field removal
• resource enrichment
• deduplication
• sampling
• format conversion

Backend

Which system answers future questions?

Examples:

• Thanos
• ClickHouse
• OpenSearch
• trace store
• long-term S3 archive

Treating these stages separately prevents one product from owning every concern.

An OpenTelemetry Collector is excellent at receiving, enriching, batching and exporting telemetry, but it is not automatically a durable broker.

S3 provides durable retention, not low-latency metric queries.

CloudWatch is the native source for many AWS-managed signals.

It does not have to be bfstore’s final operational interface.

Application telemetry should take the shortest path

bfstore controls the code of its Go services.

That makes application telemetry the easiest category to design cleanly.

The services can emit:

• traces through OTLP
• metrics through OTLP or Prometheus
• structured logs through stdout or OTLP

The preferred path is:

GO SERVICE
    │
    ▼
LOCAL OR NODE COLLECTOR
    │
    ▼
CLUSTER GATEWAY COLLECTOR
    │
    ▼
PRIVATE OBSERVABILITY INGEST
    │
    ▼
SELF-HOSTED BACKENDS

OpenTelemetry documents an agent-to-gateway pattern in which agent Collectors run close to workloads and gateway Collectors provide centralised processing and export.

For bfstore, the node-level collector can handle:

• container log collection
• node metadata
• kubelet metrics
• local OTLP reception
• initial batching
• node and Pod enrichment

The gateway can handle:

• central filtering
• sensitive-field removal
• tenant or environment attribution
• retry and queue policy
• backend routing
• trace sampling

Tail sampling needs one important qualification.

A tail-sampling processor holds spans while it decides whether to retain the completed trace. When several gateway replicas receive different spans from the same trace, each replica can make an incomplete or conflicting decision.

TRACE SPANS
     │
     ▼
TRACE-ID-AWARE ROUTING
     │
     ▼
ONE TAIL-SAMPLING COLLECTOR
     │
     ▼
SAMPLING DECISION

A horizontally scaled tail-sampling tier should therefore sit behind trace-ID-aware routing, commonly an OpenTelemetry load-balancing exporter. Ordinary round-robin load balancing is not enough.

A simplified gateway configuration might look like:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

  batch:
    timeout: 5s
    send_batch_size: 4096

  resource:
    attributes:
      - key: cloud.provider
        value: aws
        action: upsert

exporters:
  otlp/metrics:
    endpoint: metrics-ingest.observability.internal:4317

  otlp/traces:
    endpoint: traces-ingest.observability.internal:4317

  otlp/logs:
    endpoint: logs-ingest.observability.internal:4317

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/metrics]

    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/traces]

    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/logs]

This example shows pipeline separation only. A production configuration also needs:

• TLS or mTLS
• client authentication
• bounded retry policies
• exporter sending queues
• persistent queue storage where required
• Collector health checks
• queue and rejection monitoring

The Collector’s default in-memory queue protects only short interruptions. Data can still be lost when the queue fills, retries expire or the Collector restarts. Persistent file_storage can preserve queued data across a Collector restart, but it still does not turn the Collector into a dedicated durable broker.

AWS Distro for OpenTelemetry is AWS’s supported distribution of the upstream project and can send data to OTLP-compatible and other observability backends. It does not require CloudWatch to be the destination.

The important principle is:

Do not send bfstore application telemetry through CloudWatch merely to export it again when the application can send OTLP directly to the self-hosted platform.

That extra path would add:

• another ingestion charge
• another schema conversion
• another delivery dependency
• additional delay
• duplicate retention

CloudWatch remains valuable for AWS-managed signals.

Application-native telemetry can take the more direct road.

Kubernetes telemetry belongs close to the node

Application instrumentation does not reveal everything happening around the process.

The platform also needs:

• node CPU and memory
• filesystem capacity
• Pod and container resource use
• container restarts
• kubelet behaviour
• Kubernetes object state
• container stdout and stderr
• scheduling and rollout state

A node-level Collector or equivalent agent can collect much of this.

OpenTelemetry’s Kubernetes guidance identifies the kubeletstats receiver for node, Pod and container metrics, the file-log receiver for container logs, and host-metrics collection for operating-system resources. These node-specific receivers are normally deployed in a DaemonSet so each instance observes its own node.

EKS NODE
│
├── bfstore containers
├── container log files
├── kubelet
├── node filesystem
│
└── telemetry DaemonSet
    ├── file-log receiver
    ├── kubelet metrics
    ├── host metrics
    └── OTLP receiver

Cluster-wide state requires another view.

Metrics such as:

• desired versus available replicas
• failed Jobs
• pending Pods
• node conditions
• resource requests and limits
• deployment generations

are commonly exposed through kube-state-metrics and scraped by a Prometheus-compatible collector.

The useful split is:

NODE-LOCAL OBSERVATION

DaemonSet collector


CLUSTER OBJECT STATE

kube-state-metrics


APPLICATION SEMANTICS

OpenTelemetry SDK

These sources overlap in places.

The platform should choose one authoritative source for each metric family rather than collecting the same series through several agents and later wondering which copy is the original goblin.

AWS-managed signals should follow their native exits

AWS services expose telemetry through different collection and delivery surfaces.

A compact source map makes the architecture clearer than pretending every service begins in CloudWatch Logs.

The reusable patterns are:

AWS SERVICE
    → CLOUDWATCH LOGS
    → SUBSCRIPTION
    → KINESIS OR FIREHOSE


AWS SERVICE
    → FIREHOSE
    → S3 OR HTTPS ADAPTER


AWS SERVICE
    → S3
    → SQS
    → PRIVATE IMPORTER

EKS control-plane logs

bfstore does not operate the EKS control-plane hosts.

The cluster can export five control-plane log types:

• API server
• audit
• authenticator
• controller manager
• scheduler

Amazon EKS sends enabled control-plane logs to CloudWatch Logs. The log types are disabled by default and can be enabled independently.

For bfstore, I would enable at least:

• audit
• authenticator
• api

The controller-manager and scheduler logs should also be enabled where their operational value justifies the volume.

eks_control_plane_logging:
  enabled:
    - api
    - audit
    - authenticator
    - controllerManager
    - scheduler

  cloudwatch_retention_days: 14
  export_to_observability: true
  archive_policy: security-long-term

CloudWatch is the collection surface here.

It does not have to become the permanent analytical home.

RDS logs

RDS can publish selected database logs to CloudWatch Logs.

For MySQL, this can include:

• error log
• general log
• slow-query log
• audit log when the required option is configured

RDS Enhanced Monitoring also writes operating-system metrics into CloudWatch Logs rather than ordinary CloudWatch metric series. Those records need to enter the log pipeline and be parsed into metrics if bfstore wants them in the metrics backend.

This is a useful reminder:

SIGNAL MEANING

does not always equal

AWS TRANSPORT TYPE

An operating-system metric can arrive inside a log stream.

Load-balancer logs

Publication note: When this article was originally published on 18 July 2026, the established ALB access-log path was S3. AWS added vended ALB delivery to CloudWatch Logs, Amazon Data Firehose and S3 on 23 July 2026.

The current ALB integration covers access, connection and health-check logs.

NLB also provides vended delivery to CloudWatch Logs, Firehose and S3, while its legacy S3 path remains available. The choice depends on latency, cost and retention.

LOWER-LATENCY SEARCH

ALB OR NLB
    → CLOUDWATCH LOGS
    → SUBSCRIPTION


MANAGED DELIVERY

ALB OR NLB
    → FIREHOSE
    → S3 OR ADAPTER


DURABLE BATCH

ALB OR NLB
    → S3
    → SQS IMPORTER

Flow logs and Resolver query logs

VPC Flow Logs, Transit Gateway Flow Logs and Route 53 Resolver query logs can all deliver to CloudWatch Logs, Firehose or S3.

For high-volume network records, I would avoid CloudWatch Logs unless CloudWatch querying or subscription processing is specifically useful.

FLOW OR DNS QUERY LOGS
          │
          ▼
      FIREHOSE OR S3
          │
          ▼
    SPECIALISED PARSER
          │
          ▼
CLICKHOUSE / SECURITY ANALYTICS

Flow logs are collected outside the packet path and do not affect traffic throughput or latency.

They are also aggregated and delivered on a best-effort basis.

They are excellent network evidence.

They are not packet capture.

Exporting AWS-managed metrics

Many AWS services publish operational metrics to CloudWatch.

Examples include:

• ALB request and target metrics
• NLB flow metrics
• RDS CPU, storage and connection metrics
• NAT gateway errors and bytes
• Transit Gateway metrics
• EBS volume metrics
• S3 request metrics where enabled
• EKS and EC2 infrastructure metrics

There are two primary export models.

Stream broadly

CloudWatch metric streams continually send selected metrics to Amazon Data Firehose.

Metric streams support JSON and OpenTelemetry formats, including OpenTelemetry 1.0.0. The Firehose delivery stream exists in the same account and Region as the metric stream, although its final destination can be elsewhere.

CLOUDWATCH METRICS
        │
        ▼
METRIC STREAM
        │
        ▼
AMAZON DATA FIREHOSE
        │
        ▼
S3 OR HTTPS ADAPTER
        │
        ▼
NORMALISATION
        │
        ▼
SELF-HOSTED METRICS

Streaming is the stronger choice when bfstore needs:

• broad namespace coverage
• low delivery latency
• automatic inclusion of matching new metrics
• no repeated metric discovery

Metric-stream payloads still need an ingestion component that understands the selected output format and maps AWS dimensions into the target model.

The words OpenTelemetry format do not mean that Firehose sends ordinary OTLP requests to any OTLP endpoint.

The consumer must implement the Firehose delivery contract and the metric-stream record format it actually receives.

Poll selectively

A service can periodically call CloudWatch APIs such as ListMetrics and GetMetricData.

SELF-HOSTED COLLECTOR
        │
        ▼
CLOUDWATCH API
        │
        ▼
GET METRIC DATA
        │
        ▼
PROMETHEUS-COMPATIBLE SERIES

Polling may be suitable when:

• the metric set is small
• a mature CloudWatch exporter already exists
• delivery can be delayed by the scrape interval
• bfstore wants precise metric selection
• a streaming decoder would be premature

Polling becomes harder as the estate grows.

It introduces:

• API request volume
• metric and dimension discovery
• delayed retrieval
• repeated historical windows
• duplicate-point handling
• cross-account credential management
• missing-series detection

The decision is not:

streaming is modern

polling is old

It is:

STREAM

for broad continuous export


POLL

for a curated metric set
or an early implementation

Multi-account metric options

bfstore’s AWS estate spans several accounts.

Three CloudWatch mechanisms should not be treated as interchangeable.

Source-account metric streams

Each account owns its own metric stream and Firehose delivery path.

Advantages:

• explicit source ownership
• independent failure domains
• straightforward account boundaries

Costs:

• repeated configuration
• more delivery streams
• more export monitoring

Cross-account observability with a monitoring-account stream

CloudWatch cross-account observability links source accounts to a monitoring account within a Region.

A metric stream created in the monitoring account can include metrics from linked source accounts. This is the clearest central streaming option when bfstore needs AWS-vended service metrics from several accounts without creating a separate stream in every source account.

SOURCE ACCOUNTS
      │
      ▼
CROSS-ACCOUNT OBSERVABILITY
      │
      ▼
MONITORING ACCOUNT
METRIC STREAM
      │
      ▼
FIREHOSE

This still creates a central policy and failure boundary.

The stream also remains Regional.

Metrics Centralization

CloudWatch Metrics Centralization copies metric data from selected AWS Organization accounts and Regions into a destination account while preserving source account and Region metadata.

The current documentation states that all metrics from the selected source accounts are centralized and explicitly identifies custom metrics, Embedded Metric Format metrics and OpenTelemetry metrics. Selective metric filtering is not currently supported.

Centralized metrics can be used by CloudWatch features including metric streams.

SELECTED ORGANIZATION ACCOUNTS
             │
             ▼
METRICS CENTRALIZATION
             │
             ▼
DESTINATION ACCOUNT COPY
             │
             ▼
OPTIONAL METRIC STREAM

This is a replication model, not merely permission to view source-account metrics. Use it when a CloudWatch-owned central copy is valuable and centralizing every supported metric from the selected accounts is acceptable. It is not a universal replacement for source-account streams or cross-account observability.

The architecture should compare:

• whether metrics are viewed or copied
• whether filtering is required
• whether AWS-vended metrics are in scope
• Regional coverage
• stream ownership
• replication cost
• shared failure boundaries

Firehose is a useful buffer, with one private-network catch

Amazon Data Firehose can buffer, transform and deliver records to several destinations.

For custom HTTP delivery, it can:

• send HTTPS requests
• retry failed delivery for a configured duration
• back up failed or all data to S3
• record delivery errors in CloudWatch Logs
• retrieve destination credentials from Secrets Manager

When retry duration expires for an HTTP endpoint, Firehose writes failed data to the configured S3 backup location.

That makes it attractive as a managed edge buffer.

However, Firehose does not support delivery to custom HTTP endpoints inside a VPC. Custom HTTP delivery uses HTTPS on port 443 and requires a Firehose-compatible request and response contract.

This rules out the deceptively neat design:

FIREHOSE
    │
    ▼
PRIVATE OTLP ENDPOINT
inside observability VPC

Firehose cannot simply call that private endpoint.

bfstore has three practical options.

S3 landing zone

FIREHOSE
    │
    ▼
S3 LANDING BUCKET
    │
    ▼
S3 EVENT
    │
    ▼
SQS
    │
    ▼
INGESTION WORKERS
    │
    ▼
SELF-HOSTED BACKEND

This provides durable replay and a private backend, at the cost of additional latency and importer logic.

Public hardened adapter

FIREHOSE
    │
    ▼
PUBLIC HTTPS ADAPTER
    │
    ▼
PRIVATE OBSERVABILITY PIPELINE

The adapter is not the backend itself.

It implements:

• Firehose request contract
• authentication
• request deduplication
• decompression
• payload validation
• rate limiting
• private forwarding

Avoid Firehose HTTP delivery

For CloudWatch Logs, use Kinesis Data Streams and run private consumers.

For a small metric set, poll CloudWatch instead of creating a metric stream.

Metric streams require Firehose, so a private near-real-time metric destination still needs an S3 landing path, a controlled public adapter or a different export model.

The constraint belongs in the design early.

It should not be discovered after bfstore has built a beautiful internal OTLP endpoint that Firehose can admire only from afar.

Exporting AWS logs, events and durable evidence

CloudWatch Logs subscriptions

CloudWatch Logs subscriptions can forward newly ingested events to:

• Kinesis Data Streams
• Amazon Data Firehose
• Lambda
• Amazon OpenSearch Service

The subscription payload is Base64-encoded and GZIP-compressed.

Delivery is at least once, so duplicate events can occur. Retryable failures are retried for a limited period, after which undelivered events can be dropped.

A private self-hosted path can use Kinesis:

CLOUDWATCH LOG GROUPS
        │
        ▼
ACCOUNT-LEVEL SUBSCRIPTION
        │
        ▼
CROSS-ACCOUNT KINESIS STREAM
        │
        ▼
PRIVATE CONSUMER
        │
        ▼
DECOMPRESS AND NORMALISE
        │
        ▼
CLICKHOUSE OR OPENSEARCH

The consumer must decode and decompress the payload, parse the CloudWatch Logs envelope, preserve source metadata, deduplicate where useful, submit events to the backend and checkpoint only after durable acceptance.

Account-level subscriptions reduce the chance that a newly created log group remains unexported.

They also increase the risk of:

• sending unwanted high-volume logs
• exceeding stream capacity
• creating recursive logging loops

The export workflow’s own log groups must be excluded, otherwise the pipeline can ingest its own operational logs and manufacture a surprisingly energetic invoice.

Central searchable logs and the archive are different systems

bfstore already has separate accounts for:

• log archive
• observability

Those accounts should not collapse into one vague “logging account”.

AWS SOURCE
    │
    ├──────────► LOG ARCHIVE
    │             durable evidence
    │             restricted mutation
    │
    └──────────► OBSERVABILITY
                  indexed and queryable
                  operational retention

bfstore-observability exists for:

• dashboards
• alerts
• operational search
• correlations
• service troubleshooting

bfstore-log-archive exists for:

• durable security evidence
• policy-driven retention
• restricted deletion
• central CloudTrail files
• AWS Config history
• selected network and service records
• forensic recovery

An S3 bucket is not immutable merely because it lives in a log-archive account.

Where immutability is required, the design should explicitly include:

• S3 Versioning
• S3 Object Lock
• compliance-mode retention where policy requires it
• restricted bucket and KMS administration
• lifecycle controls
• CloudTrail log-file integrity validation

Object Lock prevents protected object versions from being overwritten or deleted for the retention period. CloudTrail integrity validation helps detect whether delivered files were changed or deleted. It does not itself prevent modification.

Searchability is not durability.

Durability is not immutability.

Each property needs its own control.

CloudTrail and AWS Config

The primary CloudTrail path should remain:

AWS API ACTIVITY
       │
       ▼
ORGANISATION CLOUDTRAIL
       │
       ▼
bfstore-log-archive S3

Selected operational events can also travel through CloudWatch Logs or EventBridge.

The indexed copy improves search and correlation.

It should not replace the authoritative S3 trail and digest files.

AWS Config records resource configuration state and change history.

Its delivery channel can send:

• configuration histories and snapshots to S3
• change notifications to SNS

Config records answer questions such as:

Which security-group rule changed
before the connection failures began?

When was this bucket policy altered?

Which resource configuration existed
during the incident?

They are structured evidence, not ordinary application logs.

The ingestion platform must provide guarantees

The observability platform will sometimes fail.

Reasons include:

• upgrade
• cluster failure
• network partition
• backend overload
• storage outage
• certificate problem
• schema rejection

The source workload should not fail merely because the telemetry destination is unavailable.

APPLICATION
    │
    ▼
TELEMETRY PIPELINE FAILURE
    │
    ▼
APPLICATION CONTINUES
within defined limits

Bounded buffering and replay

A mature design often uses two levels.

SHORT INTERRUPTION

Collector queue


LONGER INTERRUPTION

Kinesis or S3

Collector queues are useful for brief failures.

Kinesis provides a replayable near-real-time stream with consumer lag and checkpoint responsibilities.

Firehose provides managed buffering, retry and S3 backup.

S3 provides the durable landing zone and the clearest replay source.

No buffer should grow without a limit.

An observability outage should not become a workload outage through an infinitely enthusiastic telemetry queue.

At-least-once delivery and deduplication

CloudWatch Logs subscriptions, Firehose, S3 notifications, Kinesis consumers and OpenTelemetry exporters can all retry.

A record may therefore arrive more than once.

A useful CloudWatch log identity can combine:

source account
source Region
log group
log stream
CloudWatch event ID

A Firehose adapter can retain the stable request ID used across retries.

Trace spans carry trace and span identifiers.

Metric duplicates need series, timestamp and value-aware handling.

The platform should state explicitly:

• where deduplication happens
• which signals are deduplicated
• how long the deduplication window lasts
• what happens when the same timestamp carries conflicting values

At-least-once delivery is a reliability feature.

It moves exactly-once fantasies into the consumer’s data model, where they can be dealt with honestly.

Source identity and schema

Every AWS record should preserve enough context to answer:

Which account?

Which Region?

Which environment?

Which service?

Which cluster?

Which resource?

Which original source?

A common envelope could include:

resource:
  cloud.provider: aws
  cloud.account.id: "123456789012"
  cloud.region: eu-west-2
  deployment.environment.name: production
  service.name: order-service
  service.namespace: bfstore
  k8s.cluster.name: bfstore-prod

source:
  type: aws.eks.audit
  aws.log.group: /aws/eks/bfstore-prod/cluster
  pipeline: cloudwatch-kinesis

The common envelope enables cross-source queries.

The original payload preserves source-specific evidence.

Normalisation should not discard fields merely because the first shared schema does not understand them.

Cardinality controls

Useful AWS and application dimensions include:

• load balancer
• target group
• database instance
• Availability Zone
• service
• route
• status
• cluster
• namespace

Dangerous metric labels include:

• customer ID
• order ID
• trace ID
• request ID
• full URL
• exception message
• unbounded SQL text

METRIC LABEL

bounded grouping field


LOG FIELD

searchable event context


TRACE ATTRIBUTE

request-scoped diagnostic context

An order ID is useful in a trace or log.

It is usually catastrophic as a Prometheus label.

Self-hosting does not make cardinality free.

It changes the person receiving the storage invoice and the 03:00 alert.

Security and cost

Private OTLP paths still need authentication.

A private route proves network reachability.

It does not prove the caller is an authorised producer.

The ingestion boundary should use:

• mTLS or short-lived workload identity
• scoped IAM roles
• narrow S3 prefixes and KMS permissions
• request validation
• sensitive-field redaction
• source-specific retention
• access auditing for security telemetry

The architecture should also count every handoff.

EKS AUDIT EVENT
      │
      ▼
CLOUDWATCH INGESTION
      │
      ▼
CLOUDWATCH RETENTION
      │
      ▼
SUBSCRIPTION DELIVERY
      │
      ▼
KINESIS
      │
      ▼
SELF-HOSTED STORAGE

Each copy should have a purpose.

A duplicate without a purpose is just telemetry practising mitosis on the invoice.

My proposed bfstore architecture

I would use several ingestion lanes rather than one universal pipe.

BFSTORE APPLICATIONS
      │
      ▼
OTEL AGENTS AND GATEWAYS
      │
      ▼
PRIVATE OTLP INGEST
      │
      ▼
SELF-HOSTED BACKENDS


AWS CLOUDWATCH METRICS
      │
      ▼
METRIC STREAM OR POLLER
      │
      ▼
NORMALISATION
      │
      ▼
METRICS BACKEND


AWS CLOUDWATCH LOGS
      │
      ▼
CENTRAL SUBSCRIPTION
      │
      ▼
KINESIS
      │
      ▼
PRIVATE LOG CONSUMERS
      │
      ▼
LOG BACKEND


AWS DURABLE SOURCES
      │
      ├── CLOUDTRAIL
      ├── CONFIG
      ├── LOAD-BALANCER LOGS
      ├── FLOW LOGS
      └── RESOLVER QUERY LOGS
              │
              ▼
          CENTRAL S3
              │
              ▼
         SQS IMPORTERS
              │
              ▼
    SEARCH AND ARCHIVE SYSTEMS

Lane 1: Application and Kubernetes telemetry

• Go services instrumented with OpenTelemetry
• Collector DaemonSet on each EKS node
• gateway Collectors
• trace-ID-aware routing before tail sampling
• private OTLP ingestion
• direct export to self-hosted metrics, log and trace backends
• no CloudWatch round trip

Lane 2: AWS service metrics

Initial phase:

• curated CloudWatch polling
• limited namespaces
• explicit metric inventory

Later phase:

• cross-account observability into bfstore-observability
• central monitoring-account metric stream for broad AWS metrics
• Firehose to an S3 landing zone
• private importer converts records into the self-hosted metric model

Metrics Centralization would be evaluated separately where a CloudWatch-owned copy of all supported metrics from selected accounts and Regions is genuinely required.

Lane 3: Managed-service logs

• EKS control-plane and RDS logs to CloudWatch Logs
• selected service logs to CloudWatch Logs
• organization-aware subscription into central Kinesis
• consumers in the observability cluster
• source metadata retained
• duplicates tolerated and removed where useful

Lane 4: Network and DNS telemetry

• VPC Flow Logs to Firehose or S3
• Transit Gateway Flow Logs to Firehose or S3
• Route 53 Resolver query logs to Firehose or S3
• source-specific schemas and parsers
• indexed operational copy
• durable archive copy

Lane 5: Security and configuration evidence

• organization CloudTrail to bfstore-log-archive
• log-file integrity validation
• S3 Object Lock where immutability is required
• selected CloudTrail events copied into the operational backend
• AWS Config histories and snapshots to archive S3
• Config change notifications ingested as structured events

This design uses AWS services where they are the only or strongest source.

It keeps the analytical destination self-hosted.

The pipeline needs its own SLOs

An observability platform cannot be trusted merely because its Grafana home page loads.

The ingestion system needs service-level indicators.

Application OTLP

• accepted spans per second
• rejected telemetry
• exporter queue length
• export failure rate
• end-to-end delay

CloudWatch Logs export

• subscription delivery throttling
• Kinesis iterator age
• consumer lag
• decode failures
• duplicate rate
• last event received per source

Metric streams

• Firehose delivery freshness
• S3 backup count
• decoder failures
• sample delay
• missing expected namespaces

S3 imports

• oldest unprocessed object
• SQS queue age
• failed object count
• replay backlog
• parser-version mismatch

Source coverage

• expected log groups exporting
• expected EKS clusters reporting
• expected accounts represented
• expected Regions represented
• expected CloudTrail files arriving

A meta-signal is:

time since last telemetry
from every expected source

A green ingestion service means little when one production account quietly stopped sending logs three days ago.

The platform should detect absence.

Silence is sometimes excellent news.

In telemetry pipelines, it is often a severed wire wearing a calm expression.

A staged implementation path

Phase 1: Application and Kubernetes telemetry

• instrument two bfstore services
• deploy Collector DaemonSet and gateway
• configure private, authenticated OTLP
• establish common resource attributes
• validate loss, retry and restart behaviour
• add node, kubelet and cluster-state metrics

Phase 2: EKS and selected managed logs

• enable EKS audit, API and authenticator logs
• publish selected RDS logs
• create CloudWatch Logs subscriptions
• run private Kinesis consumers
• preserve source metadata
• exclude export-pipeline log groups

Phase 3: Selected AWS metrics

Begin with:

• ALB
• NLB
• RDS
• NAT Gateway
• Transit Gateway
• EBS

Use polling initially when it provides the quickest testable route.

Measure API cost, delay and maintenance burden.

Phase 4: Durable AWS sources

• organization CloudTrail
• AWS Config
• VPC Flow Logs
• Transit Gateway Flow Logs
• Route 53 Resolver query logs
• load-balancer logs

Use central S3, SQS and replayable importers where batch ingestion is appropriate.

Phase 5: Broader metric streaming

Introduce cross-account observability and a monitoring-account metric stream when:

• the curated poller becomes difficult to maintain
• broader namespace coverage is required
• delivery latency matters
• the Firehose landing and decoding path has been tested

Evaluate Metrics Centralization only as a separate replication decision.

Phase 6: Test failure

Simulate:

• backend outage
• Collector restart
• tail-sampling routing failure
• Kinesis consumer failure
• Firehose delivery failure
• malformed record
• duplicate event
• S3 replay
• one source account stopping export

The pipeline is ready when telemetry failure is:

• detected
• bounded
• recoverable
• unable to break the applications it observes

The mental model I am keeping

My earlier model was:

AWS TELEMETRY
      │
      ▼
CLOUDWATCH
      │
      ▼
GRAFANA

The stronger model is:

                         TELEMETRY SOURCE
                                │
              ┌─────────────────┼─────────────────┐
              │                 │                 │
              ▼                 ▼                 ▼
         APPLICATION        KUBERNETES       AWS-MANAGED
              │                 │                 │
              ▼                 ▼                 ▼
             OTLP         NODE COLLECTOR      NATIVE AWS
                                                COLLECTION
              │                 │                 │
              └─────────────────┼─────────────────┘
                                ▼
                         TRANSPORT CHOICE
                                │
        ┌───────────────┬───────┼────────┬───────────────┐
        │               │       │        │               │
        ▼               ▼       ▼        ▼               ▼
       OTLP          KINESIS  FIREHOSE    S3         API POLLING
        │               │       │        │               │
        └───────────────┴───────┼────────┴───────────────┘
                                ▼
                         DURABLE BUFFER
                                │
                                ▼
                           PROCESSING
                                │
                 ┌──────────────┼──────────────┐
                 │              │              │
                 ▼              ▼              ▼
             ENRICHMENT     DEDUPLICATION    REDACTION
                 │              │              │
                 └──────────────┼──────────────┘
                                ▼
                       SELF-HOSTED BACKENDS
                                │
                 ┌──────────────┼──────────────┐
                 │              │              │
                 ▼              ▼              ▼
              METRICS          LOGS          TRACES
                                │
                                ▼
                             EVIDENCE

OpenTelemetry answers:

How should bfstore-owned
applications describe telemetry?

The node Collector answers:

How does the platform observe
what surrounds the process?

CloudWatch answers:

Where does AWS expose telemetry
for managed resources?

A metric stream answers:

How can broad CloudWatch metrics
leave continuously?

A subscription answers:

How can newly ingested CloudWatch
log events enter another stream?

Kinesis answers:

Where can near-real-time logs wait
for a private consumer?

Firehose answers:

Where can AWS manage buffering,
batching, retry and delivery?

S3 answers:

Which telemetry can be replayed
after the analytical system fails?

Object Lock answers:

Which protected archive objects
must resist deletion or overwrite?

The self-hosted platform answers:

Where are AWS, Kubernetes and
application signals queried together?

And the pipeline SLO answers:

How do we know the observer
has not quietly gone blind?

Getting AWS telemetry into a self-hosted platform does not require rejecting AWS-native observability mechanisms.

It requires refusing to confuse collection with destination.

CloudWatch can be the source without being the home. OpenTelemetry can be the common language without being the durable queue. S3 can be the archive without being the dashboard.

bfstore application traces can travel directly through OTLP.

Kubernetes telemetry can be collected beside the workloads.

EKS control-plane and RDS logs can leave CloudWatch through subscriptions.

AWS metrics can be streamed or polled.

Flow logs and Resolver query logs can land in Firehose or S3.

CloudTrail and Config can retain authoritative evidence in the log-archive account.

Each path can end in the same self-hosted operational platform without pretending that every signal began in the same place.

The result is not one pipe.

It is a small, deliberate railway system.

Every source has the right carriage.

Every important journey has a buffer.

Every record retains its origin.

And when one line closes, the telemetry waits somewhere safer than application memory and wishful thinking.

References and further reading

• AWS Distro for OpenTelemetry
• OpenTelemetry Collector deployment patterns
• Scaling the OpenTelemetry Collector
• OpenTelemetry Collector resiliency
• OpenTelemetry Collector Kubernetes components
• Amazon EKS control-plane logging
• CloudWatch metric streams
• CloudWatch cross-account observability
• CloudWatch Metrics Centralization
• Amazon Data Firehose HTTP delivery
• CloudWatch Logs subscriptions
• CloudWatch Logs cross-account subscriptions
• CloudWatch Logs centralisation
• Application Load Balancer vended logs announcement, 23 July 2026
• Network Load Balancer access logs
• RDS log publication
• VPC Flow Logs
• Transit Gateway Flow Logs
• Route 53 Resolver query logging
• CloudTrail trails
• CloudTrail log-file integrity validation
• AWS Config delivery channels
• S3 Object Lock

otel.prod.aws.internal.bfstore.example
        │
        ▼
primary internal load balancer
        │
        ▼
OpenTelemetry ingestion gateway
        │
        ▼
self-hosted observability platform

The service is private. Its name lives in a Route 53 private hosted zone associated with the production VPC, and the network path is available through bfstore’s private routing architecture.

Then the primary observability environment suffers a serious failure.

bfstore could change DNS manually, restore the primary in place, accept telemetry loss, or direct clients to a separately prepared recovery service. Route 53 failover routing addresses that last option.

Two records share the same name and type:

otel.prod.aws.internal.bfstore.example

primary
    → primary observability endpoint

secondary
    → recovery observability endpoint

When Route 53 considers the primary healthy, it returns the primary record. When the primary is unhealthy and the secondary is healthy, it returns the secondary. Failover routing is Route 53’s active-passive policy and is supported in private hosted zones.

That is useful, but incomplete.

DNS failover can change the answer returned to a future query. It cannot create or warm the recovery environment, replicate data, establish routes, copy credentials, repair dependencies, move existing connections, flush cached answers or prove that the secondary can carry production load.

DNS failover redirects discovery. Disaster recovery makes the discovered destination worth reaching.

Route 53 earns its keep when both sides of that sentence have been built.

What Route 53 actually decides

Failover routing is active-passive

Route 53 supports several routing policies that can consider health. Failover routing specifically models an active-passive relationship.

NORMAL OPERATION

clients
   │
   ▼
primary endpoint


FAILOVER OPERATION

clients
   │
   ▼
secondary endpoint

Only the primary should normally be returned. The secondary waits until the primary is considered unhealthy.

Active-active designs use other health-aware policies, such as weighted or latency routing, where several healthy records can be eligible at once. Those environments receive ordinary traffic and are exercised continuously.

An active-passive secondary may receive little or no ordinary traffic. Its readiness must therefore be demonstrated by probes, drills and operational controls.

ACTIVE-PASSIVE

one destination is ordinary

one destination is a promise
about future readiness

Route 53 implements the selection. It does not prove that the promise is true.

Two records form one decision

A basic failover group contains one primary record and one secondary record with the same name and type, distinct identifiers, and health behaviour that lets Route 53 choose between them.

records:
  - name: otel.prod.aws.internal.bfstore.example
    type: A
    routing_policy: failover
    failover_role: primary
    identifier: primary-observability
    target: primary-observability-endpoint

  - name: otel.prod.aws.internal.bfstore.example
    type: A
    routing_policy: failover
    failover_role: secondary
    identifier: recovery-observability
    target: recovery-observability-endpoint

The pair can use alias or non-alias records, and one may be an alias while the other is not, provided the names and record types match.

IS PRIMARY HEALTHY?
        │
    ┌───┴───┐
   YES      NO
    │        │
    ▼        ▼
PRIMARY   IS SECONDARY HEALTHY?
              │
          ┌───┴───┐
         YES      NO
          │        │
          ▼        ▼
      SECONDARY  SPECIAL FAILURE BEHAVIOUR

That final branch is easy to overlook.

If both destinations are unhealthy, Route 53 returns the primary

If both failover records have health checks and both are considered unhealthy, Route 53 still returns the primary record.

PRIMARY UNHEALTHY

SECONDARY UNHEALTHY
        │
        ▼
ROUTE 53 RETURNS PRIMARY

Route 53 needs to return a DNS response and treats all records as healthy as a last resort. The failover policy then selects the primary.

The operational consequence is important:

Failover routing does not guarantee that every returned endpoint is healthy.

When all options are unhealthy, clients still need deadlines, bounded retries, circuit breaking and useful error handling. DNS is one selector in the reliability system, not its final safety mechanism.

A secondary with no health evaluation is considered healthy

If the primary becomes unhealthy and the secondary has neither an associated health check nor alias target-health evaluation, Route 53 considers the secondary healthy and returns it.

PRIMARY

health checked
unhealthy


SECONDARY

no health check
no target-health evaluation
therefore considered healthy
        │
        ▼
SECONDARY ANSWER RETURNED

An alias record is different when EvaluateTargetHealth is enabled: Route 53 inherits health from the alias target.

An unchecked secondary can be appropriate for a deliberately simple destination, such as a static maintenance page. It deserves more caution for a private database, payment service or telemetry pipeline that may be under-sized, stale, missing credentials or blocked by a route.

An unchecked secondary means:

If the primary fails,
send traffic here without asking
whether this destination can receive it.

That should be an explicit architectural decision, not an accidental omission from the Terraform.

How to create a meaningful health signal

Health must represent the capability being failed over

The key question is not:

What metric do we already have?

It is:

What must be true for this endpoint
to fulfil the service contract?

For observability ingestion, useful health may require the load balancer to accept connections, at least one gateway target to be ready, authentication to work, the queue to be writable and downstream storage to remain within safe capacity.

A weak signal such as:

EC2 instance is running

can stay green while the process has crashed, the disk is full, the certificate has expired or every request fails.

A stronger signal asks whether a production-shaped telemetry request can be accepted and durably handed to the next stage.

Health checks should be meaningful enough to represent the capability, cheap enough to run continually, independent enough not to amplify an outage and stable enough not to flap. The best check is the one that answers whether traffic should continue arriving.

Alias health can follow a load balancer

When a failover alias points to an Application Load Balancer or Network Load Balancer, EvaluateTargetHealth can make Route 53 inherit the health of that destination.

FAILOVER ALIAS RECORD
        │
        ▼
EVALUATE TARGET HEALTH
        │
        ▼
INTERNAL LOAD BALANCER
        │
        ▼
TARGET-GROUP HEALTH

For an ALB or NLB to be considered healthy, every target group that contains targets must contain at least one healthy target. A target group with no registered targets is considered unhealthy.

This works well when load-balancer target health accurately represents the DNS name’s service contract.

It can be too broad when one load balancer serves several capabilities. A target group unrelated to the specific DNS name can influence the alias health decision. The health boundary and the DNS name should describe the same service.

Highly available AWS endpoints may need an application health check

EvaluateTargetHealth is not equally useful for every alias target.

AWS notes that highly available services such as VPC interface endpoints, API Gateway, S3 and VPC Lattice may provide little operational benefit from alias target evaluation. Failover involving these services may need a Route 53 health check that represents the application’s ability to use them.

For a bfstore PrivateLink endpoint, the interface endpoint may remain infrastructurally healthy while the provider application is unavailable, authentication fails, storage cannot be read or every request returns an error.

INFRASTRUCTURE HEALTH

interface endpoint exists
network interfaces available


APPLICATION HEALTH

approved module can be retrieved
and authenticated successfully

Failover should normally follow the second when that is the service promise being protected.

Standard endpoint health checks cannot target private addresses

Route 53 endpoint health checkers operate outside the VPC and cannot target local, private, non-routable or multicast IP address ranges.

For private hosted-zone failover, a common alternative is a Route 53 health check based on a CloudWatch alarm data stream.

PRIVATE SERVICE
      │
      ▼
IN-VPC PROBE OR SERVICE METRIC
      │
      ▼
CLOUDWATCH ALARM
      │
      ▼
ROUTE 53 HEALTH CHECK
      │
      ▼
FAILOVER RECORD

A purpose-built probe runs where the private service is reachable, publishes a metric and lets CloudWatch evaluate it. The probe can include private routing, transport, authentication and dependency behaviour without exposing the service merely so external health checkers can reach it.

The CloudWatch alarm needs an explicit insufficient-data policy

A Route 53 health check based on a CloudWatch alarm monitors the alarm’s data stream rather than simply reading its current label.

The health check is healthy when the stream indicates OK, unhealthy when it indicates ALARM, and follows the configured insufficient-data behaviour when the stream lacks enough evidence:

• healthy;
• unhealthy;
• last known status.

Route 53 does not support cross-account CloudWatch alarms for this relationship. The alarm must also use a standard-resolution metric. Metric math and “M out of N” alarms are not supported.

If several signals must be combined, publish a purpose-built readiness metric or use a calculated health check.

What should missing data mean?

HEALTHY

Assume the service is healthy
unless failure is proven.


UNHEALTHY

Assume missing evidence means
the service cannot be trusted.


LAST KNOWN

Continue using the previous state.

Healthy can keep traffic on an unobservable failure. Unhealthy can trigger failover because the monitoring path failed. Last known can preserve a stale decision. The choice should reflect the independence and reliability of the monitoring path.

Which account owns the alarm?

If the private hosted zone and health check live in bfstore-network while the metric lives in bfstore-observability or bfstore-prod, the design must account for the same-account requirement. Options include publishing a purpose-built metric into the health-check account, moving the health check and alarm with the workload, or revisiting which account owns the DNS decision.

Central ownership is useful only when the required signals can reach it cleanly.

Calculated health checks can express quorum

A calculated health check can monitor several child health checks and compare the number of healthy children with a threshold. The published Route 53 quota allows up to 255 child checks.

PRIMARY OBSERVABILITY HEALTH

child 1:
    primary-specific probe from production VPC

child 2:
    primary-specific probe from staging VPC

child 3:
    ingestion backlog below threshold

child 4:
    durable storage writable


parent healthy when:
    at least 3 children are healthy

This can reduce sensitivity to one faulty probe, but it can also create a formula nobody understands during an incident. Calculated health checks cannot monitor other calculated health checks, so the design must remain shallow.

Combine only independent evidence that genuinely determines whether the service can receive traffic, and make the formula explainable in a runbook.

What DNS cannot do

TTL determines how quickly new queries learn the decision

Route 53 may mark the primary unhealthy quickly, while resolvers and clients continue using a cached primary answer.

T0  primary fails
T1  health state changes
T2  Route 53 returns secondary
T3  cache expires
T4  client queries again
T5  client connects to secondary

Experienced failover time is closer to:

health detection
+ health-state evaluation
+ remaining DNS cache lifetime
+ application retry or reconnect
+ secondary connection establishment

For non-alias health-checked records, AWS recommends a TTL of 60 seconds or less when clients need to react quickly. Shorter TTLs increase query frequency, so the setting must match the recovery objective.

Alias records do not set their own TTL. Route 53 uses the TTL of the AWS resource or record that the alias targets.

Existing connections do not move

Suppose a Go service has an established gRPC connection to:

10.96.12.45

DNS later begins returning:

10.160.12.45

The existing TCP or HTTP/2 connection remains attached to the original destination until it fails, the client closes it, an idle timeout expires or the application refreshes it.

DNS failover redirects future discovery. It does not migrate live sessions.

Client behaviour therefore belongs in the recovery design:

• connection deadlines;
• bounded retries;
• reconnect logic;
• appropriate DNS refresh;
• idempotent operations;
• awareness of long-lived pools.

Telemetry collectors may keep persistent export connections. MySQL pools may retain sessions. Kafka clients may rely on broker metadata after the initial bootstrap lookup. The failover test must use the real client, not only dig.

Route 53 cannot repair a missing route

A workload may receive the secondary answer:

otel.prod.aws.internal.bfstore.example
    → 10.160.12.45

The production VPC still needs a route to the recovery network, a return route, security-group and firewall permission, the correct hosted-zone association and authentication accepted by the secondary.

DNS ANSWER
    │
    ▼
ROUTE TABLE
    │
    ▼
TRANSIT OR OTHER PRIVATE PATH
    │
    ▼
SECURITY CONTROLS
    │
    ▼
SECONDARY SERVICE

A correct failover record can return an unreachable address. Private recovery paths should exist and be tested before the incident. Creating them during the incident is emergency network construction, not automated failover.

The secondary must be able to carry the traffic

A cold standby may contain no running replicas, stale data, missing secrets, outdated infrastructure or untested network policy. DNS can still direct production traffic to it.

Before enabling a secondary failover record, bfstore should know:

• Capacity: can it handle full production demand?
• Data: is required state replicated, restored or deliberately absent?
• Identity: will current workloads and clients be accepted?
• Secrets: are certificates, tokens and keys available?
• Dependencies: can it reach the services it needs?
• Operations: are dashboards, alerts and runbooks active?
• Ownership: who declares it ready?

DNS SECONDARY

must represent

RECOVERY CAPABILITY


not merely

ANOTHER ADDRESS

For stateful order processing, readiness may also require database promotion, write fencing, event-offset reconciliation and idempotency state. Route 53 should move traffic only after those concerns have been handled.

Multi-AZ load balancing usually does not need Route 53 failover

An internal ALB or NLB can already operate across Availability Zones and route traffic to healthy targets. An Application Load Balancer normally spans at least two enabled zones.

orders.prod.aws.internal.bfstore.example
        │
        ▼
internal ALB
        │
        ├── targets in AZ A
        ├── targets in AZ B
        └── targets in AZ C

Adding Route 53 failover between two load balancers that duplicate the same multi-AZ failure boundary often adds complexity without creating meaningful independence.

Route 53 is more useful between independent service cells:

PRIMARY CELL              SECONDARY CELL

load balancer             load balancer
compute                    compute
dependencies               dependencies

The boundary may be another Region, VPC, service cell, recovery stack or maintenance destination. Two records pointing into the same failure domain do not create redundancy.

Shared dependencies can defeat failover

Suppose primary and secondary order-service endpoints both depend on the same regional database, Kafka cluster, KMS key, identity provider, Transit Gateway route or egress path.

If the primary fails because the shared dependency is unavailable, the secondary fails for the same reason.

PRIMARY ENDPOINT ───┐
                    ▼
               SHARED KAFKA
                    ▲
SECONDARY ENDPOINT ─┘

DNS failover protects only against failures outside the shared dependency set.

A recovery review should ask:

Which failure makes
the primary unhealthy?

Can that same failure
disable the secondary?

The secondary does not need to duplicate every component. It needs enough independence to survive the failure scenarios the policy claims to handle.

How to operate the recovery system

Automated health and operator-controlled failover solve different problems

Automated failover is attractive: the service fails, health changes and Route 53 redirects future resolutions.

The difficult question is whether every detected failure should move traffic automatically.

Some failures are clear:

• the primary endpoint is unreachable;
• all application targets are unhealthy;
• the environment has been deliberately isolated.

Others are ambiguous:

• latency is elevated;
• one dependency is degraded;
• replication is behind;
• the recovery environment may be less ready;
• the monitoring system is unstable.

Amazon Application Recovery Controller can provide operator-controlled routing switches integrated with Route 53 health checks. ARC routing controls change the associated health-check state and can use safety rules to prevent dangerous combinations. They are traffic switches, not application-health monitors.

AUTOMATED FAILOVER

observed service health
        │
        ▼
DNS changes automatically


CONTROLLED FAILOVER

operator or recovery automation
        │
        ▼
routing-control state changes
        │
        ▼
DNS changes deliberately

A stateless private ingestion endpoint may tolerate automatic failover. A stateful order system with asynchronous cross-Region replication may require verification before writes move.

Mature automation is not merely fast. It moves traffic only when the destination is ready.

Failback can be more dangerous than failover

The primary recovers and its health check turns green. Automatic failback may still be premature.

During the outage, the secondary may have accepted writes, advanced event consumers, changed caches or reversed replication direction. The primary may have missed events.

PRIMARY HEALTHY AGAIN
        │
        ▼
IS ITS STATE CURRENT?
IS REPLICATION SAFE?
IS IT READY FOR AUTHORITY?

Failback should consider data reconciliation, replication direction, event offsets, connection draining, customer impact and operational approval.

For stateful services, automated detection with controlled failback is often safer. A green process or load-balancer target does not prove that the primary should reclaim authority.

Health checks need hysteresis

If a service alternates between healthy and unhealthy, Route 53 can repeatedly change the returned destination. Clients hold answers for different TTL windows, so traffic may split across primary and secondary.

A stable design should account for check interval, failure threshold, CloudWatch evaluation periods, missing-data behaviour, client TTL, cooldown and the evidence required before failback.

Route 53 endpoint health checks can run at 10-second or 30-second intervals and use a configured failure threshold. A compatible CloudWatch alarm can require consecutive breaching periods, but Route 53 supports only standard-resolution metrics and does not support metric math or “M out of N” alarms.

If bfstore needs asymmetric failure and recovery thresholds, implement them upstream and publish one binary readiness metric or controlled routing state.

# Conceptual upstream readiness policy,
# not a direct Route 53-compatible alarm definition.

readiness_policy:
  declare_unready:
    success_rate_below_percent: 95
    sustained_minutes: 3

  declare_ready_again:
    success_rate_at_or_above_percent: 99
    sustained_minutes: 10
    operator_approval: true

The numbers require testing. The principle is to prevent one noisy measurement from moving production traffic repeatedly.

Testing must include cached answers

A failover drill should test the complete path.

Baseline

Confirm that the primary is returned, clients connect, the secondary is independently healthy and both network paths work.

Trigger failure

Use a controlled health-check override, failed synthetic probe, ARC routing control or target withdrawal in a test environment.

Observe Route 53 and caches

Confirm that the primary health changes, the secondary remains healthy and fresh queries return the secondary.

Run:

dig otel.prod.aws.internal.bfstore.example
getent hosts otel.prod.aws.internal.bfstore.example

from several clients and resolvers. Record the answer, TTL and time until cached primary addresses disappear.

Observe real applications

Confirm that long-lived clients reconnect, retries remain bounded, authentication works, no duplicate operation is created and latency remains acceptable.

Observe the secondary

Check capacity, queue depth, throttling and downstream storage.

Test failback separately

Restore the primary without immediately returning traffic. Validate it, perform controlled failback and test clients again.

THE TEST IS COMPLETE WHEN

real clients reach a valid secondary

and return safely

A DNS response proves routing intent. A completed operational or business transaction proves recovery.

Metrics and evidence

The design should expose several layers.

Route 53 health

• health-check state and transition time;
• child-check state;
• CloudWatch alarm data stream;
• insufficient-data events.

DNS

• Resolver query logs;
• response data and code;
• source VPC;
• old and new answers;
• observed TTL.

Resolver query logging records queries that reach Route 53 VPC Resolver. Queries answered from a resolver cache are not logged individually.

Client and recovery

• connection failures, retries and reconnect time;
• successful requests by destination;
• secondary target health, capacity and queue depth;
• replication state and failover approvals.

A useful incident timeline distinguishes detection, DNS selection, cache expiry, client reconnect and secondary stabilisation. That reveals where the recovery time was actually spent.

Route 53’s DNS data plane is separate from configuration work

Route 53 separates the DNS-serving data plane from the control plane used to create and update resources. AWS recommends relying on preconfigured health checks and failover records during recovery rather than making control-plane changes at the moment of failure.

WEAK RECOVERY

incident
   │
   ▼
engineer creates records
and health checks


STRONGER RECOVERY

incident
   │
   ▼
preconfigured health changes
   │
   ▼
existing records select secondary

The secondary configuration should exist before the primary fails. Recovery should execute prepared decisions rather than discover missing permissions, invalid targets or untested scripts during the incident.

Applying the model at bfstore

A bfstore private failover design

A useful first bfstore case is the observability ingestion endpoint.

Telemetry is important, but its front-door ingestion path can be largely stateless. Moving it between service cells is more manageable than moving the authoritative order database.

Stable service name

otel.prod.aws.internal.bfstore.example

Destinations

primary observability ingestion
eu-west-2

recovery observability ingestion
separate VPC or Region

The production, staging and platform VPCs can reach both destinations through approved private paths.

Separate control-loop probes

Each branch must be tested directly. A branch health probe must not resolve the failover name whose answer it controls.

otel-primary.prod.aws.internal.bfstore.example
        │
        ▼
primary ingestion endpoint
        │
        ▼
primary capability metric


otel-recovery.prod.aws.internal.bfstore.example
        │
        ▼
recovery ingestion endpoint
        │
        ▼
recovery capability metric

Each synthetic OpenTelemetry probe:

1. resolves its endpoint-specific name;
2. establishes the required transport;
3. authenticates with a probe identity;
4. submits a marked signal;
5. verifies acceptance by the intended gateway;
6. publishes a branch-specific metric.

Using otel.prod.aws.internal.bfstore.example for branch health would create a circular control loop. After failover, that name may resolve to the recovery endpoint, so the probe would no longer prove the health of the primary.

End-to-end service probe

A separate probe resolves the stable production name and measures the client path and service SLO. It confirms which branch Route 53 selected and whether clients can use it, but it does not directly determine either branch’s health.

BRANCH-SPECIFIC PROBES
        │
        ▼
FAILOVER DECISION


STABLE-NAME PROBE
        │
        ▼
END-TO-END RECOVERY EVIDENCE

Failover records

dns_failover:
  name: otel.prod.aws.internal.bfstore.example
  type: A

  primary:
    identifier: primary-observability
    target: primary-ingest-load-balancer
    health: primary-capability-health

  secondary:
    identifier: recovery-observability
    target: recovery-ingest-load-balancer
    health: recovery-capability-health

The recovery gateway maintains enough baseline capacity to accept production telemetry or can scale within the recovery window. It may use an independent queue and storage, or an explicitly accepted period of reduced retention.

Failback is operator-approved after the primary-specific probe stays healthy, storage and queues are reconciled, capacity is confirmed and the stable-name probe succeeds after traffic returns.

This is a realistic use of Route 53 failover: both destinations accept the same protocol and identity, the secondary is independently useful, and DNS has a precise job.

A weaker bfstore use case: failing over the order database name

Consider:

mysql.orders.prod.aws.internal.bfstore.example

Changing its DNS answer from a primary RDS MySQL writer to a recovery writer does not prove that the secondary has been promoted, contains current data, matches the schema or has fenced the old primary from writes. It also does not reconnect existing pools or reconcile event consumers and idempotency state.

replicate or restore
        │
        ▼
promote secondary
        │
        ▼
fence old primary
        │
        ▼
validate state
        │
        ▼
change DNS
        │
        ▼
reconnect and verify writes

DNS may be one step in the procedure. It should not be the entire procedure. A record that moves before the database can safely change authority is unsafe.

Where failover routing earns its keep

Route 53 failover is a strong fit when:

• clients must keep one stable name;
• both destinations accept the same protocol and identity;
• consuming networks can already reach both;
• health represents the service capability;
• TTL and reconnect behaviour fit the recovery objective;
• the secondary has tested capacity, state and dependencies;
• failback requires deliberate evidence.

It is weaker when the secondary shares the primary’s failure domain, needs a long rebuild, cannot be reached privately, or must complete stateful promotion before accepting traffic. It also adds little when a load balancer already handles the relevant boundary.

Failover routing is a selector between prepared alternatives.

Pre-deployment review

Failure domain

• Which failure should move traffic?
• Can it also disable the secondary?
• Which critical dependencies are shared?

Health

• Does each branch-specific check represent the service contract?
• Is the secondary checked, target-evaluated or intentionally unchecked?
• What does insufficient data mean?
• What happens when both branches are unhealthy?

Reachability, state and capacity

• Can every consuming VPC reach both destinations?
• Will the same identities and credentials work?
• Is the secondary’s state current enough?
• Can it carry the failover load?
• What promotion, fencing or reconciliation must happen first?

Clients and operations

• How long may resolvers retain the old answer?
• When do long-lived clients resolve again?
• Are retries bounded and operations idempotent?
• Is failover automatic or controlled?
• What evidence permits failback?
• Has the complete path been tested with real clients?

The mental model

The incomplete model is:

PRIMARY FAILS
      │
      ▼
ROUTE 53 NOTICES
      │
      ▼
EVERYTHING MOVES
TO SECONDARY

The useful model is:

PRIMARY CAPABILITY
        │
        ▼
BRANCH-SPECIFIC HEALTH
        │
        ▼
ROUTE 53 DECISION
        │
        ├── primary healthy
        │       └── primary answer
        │
        └── primary unhealthy
                ├── secondary healthy
                │       └── secondary answer
                └── secondary unhealthy
                        └── primary may still return
        │
        ▼
CACHE EXPIRES
        │
        ▼
CLIENT RESOLVES AND RECONNECTS
        │
        ▼
DESTINATION ACCEPTS WORK

The records decide where traffic may go. Health decides whether each destination can fulfil the contract. TTL and clients determine when the change takes effect. Network reachability and recovery readiness determine whether it succeeds.

Route 53 is the switch in a recovery system, not the recovery system itself.

It earns its keep between two already viable destinations.

References and further reading

• Route 53 failover routing
  Defines the failover policy and confirms that it can be used in private hosted zones.

• Active-active and active-passive failover
  Explains how failover routing implements active-passive service selection.

• How Route 53 chooses records when health checking is configured
  Documents unchecked records, alias target-health evaluation and the result when both failover records are unhealthy.

• Values specific for failover alias records
  Covers primary and secondary alias records and EvaluateTargetHealth.

• AliasTarget API reference
  Documents how Route 53 evaluates ALB and NLB target-group health and why highly available alias targets may need application-level health checks.

• Configuring failover in a private hosted zone
  Describes CloudWatch-based health checks for resources reachable only through private addresses.

• Values for Route 53 health checks
  Covers private-address restrictions, endpoint-specific names, calculated checks, CloudWatch alarm limitations and insufficient-data behaviour.

• Route 53 quotas
  Lists the published limit of 255 child checks for a calculated health check.

• Route 53 DNS best practices
  Discusses TTL choices and preconfigured recovery mechanisms.

• Choosing between alias and non-alias records
  Explains how alias records inherit the TTL of their target.

• How Elastic Load Balancing works
  Describes multi-Availability-Zone load balancing and zonal target handling.

• Application Recovery Controller routing controls
  Describes operator-controlled traffic movement through routing-control health checks.

• Route 53 Resolver query logging
  Documents DNS query evidence and the effect of resolver caching on logged requests.

Production workloads can resolve:

otel.prod.aws.internal.bfstore.example

modules.platform.aws.internal.bfstore.example

orders.prod.aws.internal.bfstore.example

The VPCs use AmazonProvidedDNS. Their DNS attributes are enabled, and the appropriate private hosted zones are associated with them.

Private DNS already works.

BFSTORE WORKLOAD
       │
       ▼
AmazonProvidedDNS
       │
       ▼
ASSOCIATED PRIVATE HOSTED ZONE
       │
       ▼
PRIVATE ANSWER

No Resolver endpoint is required.

Then bfstore begins its Azure implementation.

An Azure-hosted platform component needs to resolve:

modules.platform.aws.internal.bfstore.example

AWS workloads also need to resolve:

registry.platform.azure.internal.bfstore.example

The DNS questions now cross between two resolver environments.

AZURE DNS
    │
    │ asks for an AWS-private name
    ▼
AWS DNS


AWS DNS
    │
    │ asks for an Azure-private name
    ▼
AZURE DNS

This is where Route 53 Resolver endpoints become useful.

Route 53 VPC Resolver is available by default in every VPC. It answers public records, VPC-specific names and associated private hosted zones. Resolver endpoints extend that service so DNS queries can cross between VPC Resolver and resolvers on another connected network.

Resolver endpoints are DNS bridges. They become necessary when the question or answer lives on the other side of a resolver boundary.

An inbound endpoint carries questions into AWS.

An outbound endpoint carries selected questions away from AWS.

A forwarding or delegation rule decides which names should take the outbound bridge.

That is the architecture in one paragraph.

The rest is deciding where the bridges belong, who may use them and how to stop them quietly forwarding questions in circles.

Start with the authority, not the endpoint

Before creating an endpoint, I need to know which DNS system owns each namespace.

Suppose bfstore eventually has:

AWS Route 53

aws.internal.bfstore.example


Azure private DNS

azure.internal.bfstore.example


GCP Cloud DNS

gcp.internal.bfstore.example


External identity platform

id.internal.bfstore.example

The DNS architecture should answer three questions:

Which resolver receives
the client query first?


Which DNS system is authoritative
for the requested namespace?


How does the first resolver
reach that authority?

A Resolver endpoint is useful only when the answer to the final question crosses an AWS network boundary.

If an AWS workload queries a private hosted zone associated with its VPC, Route 53 already owns and can answer the name.

AWS CLIENT
    │
    ▼
AmazonProvidedDNS
    │
    ▼
AWS PRIVATE HOSTED ZONE

No endpoint is needed.

If an external resolver needs the AWS answer:

EXTERNAL CLIENT
    │
    ▼
EXTERNAL RESOLVER
    │
    ▼
AWS INBOUND ENDPOINT
    │
    ▼
AWS PRIVATE HOSTED ZONE

An inbound endpoint is needed.

If an AWS client needs an externally owned answer:

AWS CLIENT
    │
    ▼
AmazonProvidedDNS
    │
    ▼
OUTBOUND RULE
    │
    ▼
AWS OUTBOUND ENDPOINT
    │
    ▼
EXTERNAL DNS AUTHORITY

An outbound endpoint is needed.

The endpoint follows the authority relationship.

It should not be deployed first and assigned a purpose afterwards.

Inbound: another network asks AWS

An inbound Resolver endpoint creates private IP addresses inside an AWS VPC. AWS creates a managed network interface for each endpoint address.

DNS resolvers on another connected network send queries to those addresses. VPC Resolver then answers using the DNS context available to the endpoint VPC, including private hosted zones associated with that VPC.

OTHER NETWORK
      │
      ▼
EXTERNAL DNS RESOLVER
      │
      │ conditional forward
      ▼
INBOUND ENDPOINT IP ADDRESSES
      │
      ▼
ROUTE 53 VPC RESOLVER
      │
      ▼
AWS-PRIVATE DNS ANSWER

A practical enterprise resolver configuration might be:

aws.internal.bfstore.example
    → 10.112.10.25
    → 10.112.11.25

Those two addresses belong to an inbound endpoint in two Availability Zones.

When the external resolver receives:

orders.prod.aws.internal.bfstore.example

it forwards the query to one of the endpoint addresses. VPC Resolver finds the matching private hosted zone and returns the record.

The external client can use the returned address only if a separate network route and security policy allow it.

INBOUND ENDPOINT

external DNS
    → AWS DNS


VPN, DIRECT CONNECT
OR OTHER NETWORK PATH

external client
    → returned AWS address

The endpoint provides DNS reachability.

It does not provide application reachability.

The endpoint is a narrow service boundary

An inbound endpoint accepts DNS queries. It is not a router into the endpoint VPC and does not grant access to:

• application subnets;
• Kubernetes Pods;
• databases;
• arbitrary VPC addresses;
• AWS control-plane APIs.

EXTERNAL RESOLVER
        │
        ▼
DNS QUERY ONLY
        │
        ▼
INBOUND ENDPOINT
        │
        ▼
ROUTE 53 RESOLVER

The endpoint security group determines which source resolvers and protocols may reach it.

The associated private hosted zones determine which private answers VPC Resolver can provide.

The returned destination is governed separately.

This is a service-shaped boundary:

The external network may ask
approved DNS questions.

It does not join the AWS VPC
because those questions receive answers.

The inbound endpoint sees its VPC’s DNS view

An inbound endpoint does not automatically see every private hosted zone in every AWS account.

Suppose the endpoint lives in:

bfstore-dns-hub VPC

but the production zone is associated only with:

bfstore-prod VPC

An external query entering through the DNS hub does not inherit the production VPC’s zone association. The DNS hub also needs an appropriate association or shared DNS configuration.

PRIVATE HOSTED ZONE
prod.aws.internal.bfstore.example
          │
          ├── bfstore-prod VPC
          └── bfstore-dns-hub VPC

That association is also an audience decision. External resolver populations reaching the hub may be able to resolve records from the associated zone, subject to the forwarding and access design.

Zone associations should therefore express intended visibility, not merely make troubleshooting easier.

Outbound: AWS asks another authority

An outbound Resolver endpoint handles the opposite direction.

An AWS workload sends its normal query to AmazonProvidedDNS. VPC Resolver compares the queried name with its rules. When a matching forwarding rule exists, Resolver sends the query through the outbound endpoint to the configured target DNS servers.

AWS WORKLOAD
      │
      ▼
AmazonProvidedDNS
      │
      ▼
MATCHING RESOLVER RULE
      │
      ▼
OUTBOUND ENDPOINT
      │
      ▼
EXTERNAL DNS SERVER
      │
      ▼
EXTERNAL PRIVATE ZONE

Suppose bfstore’s Azure DNS environment owns:

azure.internal.bfstore.example

The AWS network account creates a forwarding rule:

rule:
  name: azure-private-dns
  type: FORWARD
  domain: azure.internal.bfstore.example

  targets:
    - ip: 10.160.10.53
      port: 53

    - ip: 10.160.11.53
      port: 53

The rule is associated with the VPCs whose workloads need that namespace.

A production Pod asks AmazonProvidedDNS for:

registry.platform.azure.internal.bfstore.example

Resolver finds the matching rule and sends the query through the outbound endpoint. The Azure-side resolver answers, and the result returns through VPC Resolver to the Pod.

The Pod continues using the resolver it always used. It does not need to know the addresses of the Azure DNS servers.

APPLICATION

asks for a name


ROUTE 53 RESOLVER

chooses the authority


OUTBOUND ENDPOINT

carries the question

Outbound rules decide which authority wins

An outbound endpoint does not forward every DNS query automatically.

Resolver rules determine which namespaces use it.

A rule can forward:

id.internal.bfstore.example

without forwarding:

aws.internal.bfstore.example

or ordinary public DNS.

QUERY

login.id.internal.bfstore.example
        │
        ▼
MATCHING RULE
        │
        ▼
OUTBOUND ENDPOINT


QUERY

orders.prod.aws.internal.bfstore.example
        │
        ▼
NO EXTERNAL RULE
        │
        ▼
ROUTE 53 PRIVATE HOSTED ZONE

When several rules match, Route 53 uses the rule containing the most-specific domain name. A rule for:

platform.azure.internal.bfstore.example

therefore takes precedence over:

azure.internal.bfstore.example

That resembles longest-prefix routing, except DNS labels rather than address bits determine the most-specific match.

The rule set should be reviewed as a routing table for names.

Forwarding rules can shadow private hosted zones

Suppose a VPC has:

• a private hosted zone for corp.example;
• a forwarding rule for corp.example.

The forwarding rule takes precedence. Resolver sends the query to the external DNS servers instead of answering it from the private hosted zone.

PRIVATE HOSTED ZONE

corp.example


OUTBOUND FORWARDING RULE

corp.example
    → external DNS


WINNER

outbound forwarding rule

This can be intentional. The external DNS environment may be the authoritative source while the private hosted zone is being retired.

It can also produce a baffling incident:

The record is clearly present
in Route 53.

The client still receives
the external answer.

The record is present. It is simply not on the winning resolution path.

Before creating a forwarding rule, I now ask:

• Does a private hosted zone use the same name?
• Does an AWS-defined system rule already own part of the namespace?
• Will a more-specific subdomain need to remain inside Route 53?
• Which VPCs should inherit the external authority?

A DNS forwarding rule is an authority decision. It should receive the same care as a network route.

System rules create deliberate exceptions

A forwarding rule normally applies to its domain and subdomains.

Suppose bfstore forwards:

example.com
    → external DNS

but wants VPC Resolver to use its normal resolution behaviour for:

aws.example.com

A more-specific SYSTEM rule can create that exception.

FORWARD RULE

example.com
    → external DNS


SYSTEM RULE

aws.example.com
    → Route 53 Resolver

This can preserve:

• a Route 53 private hosted zone;
• AWS-provided private naming;
• ordinary recursive behaviour for one branch.

The final answer may therefore depend on:

• private hosted zones;
• custom forwarding rules;
• delegation rules;
• custom system rules;
• AWS-defined system rules;
• the most-specific domain match.

DNS troubleshooting should inspect the complete rule set, not only the record the engineer expected to win.

Forwarding and delegation are different relationships

Conditional forwarding says:

For names beneath this domain,
send the query to these resolver IPs.

DNS delegation says:

These name servers are authoritative
for this child namespace.

Route 53 Resolver supports both models, but they are not interchangeable.

A FORWARD rule redirects matching queries to configured target resolver addresses.

A DELEGATE rule follows the DNS delegation chain. Resolver uses the outbound endpoint only when the delegation configuration matches the NS records returned during resolution. For private-hosted-zone-to-outbound delegation, the private hosted zone must contain the relevant NS records.

FORWARD

domain match
    → configured target resolvers


DELEGATE

DNS response contains matching NS records
    → reach those delegated name servers
      through the outbound endpoint

Inbound delegation is also distinct from ordinary inbound forwarding. It uses a delegation-category inbound endpoint, and the parent DNS environment needs glue NS records pointing at the endpoint addresses. Delegation inbound endpoints currently support Do53 only.

Forwarding is often simpler when integrating existing resolvers whose authority cannot be changed easily.

Delegation can create a cleaner authority model when parent and child namespaces are deliberately divided between teams or environments.

The architectural question is:

Are we telling the resolver
where to forward a question?


Or are we declaring
who owns the child zone?

Both can involve Resolver endpoints.

They express different governance and require different DNS records.

A shared Regional DNS hub

A central DNS design does not normally need one inbound endpoint per private hosted zone or one outbound endpoint per rule.

Within a Region, one inbound endpoint and one outbound endpoint are typically sufficient for their respective directions. Multiple rules can use the same outbound endpoint, and additional capacity can usually be added by adding endpoint addresses rather than creating another endpoint.

ONE INBOUND ENDPOINT
        │
        ├── queries for aws.internal...
        ├── queries for platform.internal...
        └── queries for other approved AWS-private names


ONE OUTBOUND ENDPOINT
        │
        ├── rule for azure.internal...
        ├── rule for gcp.internal...
        └── rule for id.internal...

Separate endpoints may still be justified by different:

• security boundaries;
• source networks;
• protocols;
• owning teams;
• routing paths;
• availability requirements;
• Regions.

The default should not be one endpoint per domain.

The endpoint is shared DNS transport. The rules provide namespace-specific routing.

The endpoint VPC becomes an audience and routing boundary

For bfstore, I would evaluate central Resolver endpoints in the bfstore-network account.

A dedicated DNS VPC or carefully designed network-services VPC could host:

BFSTORE-NETWORK ACCOUNT

DNS SERVICES VPC
│
├── inbound endpoint
│   ├── subnet-a
│   └── subnet-b
│
├── outbound endpoint
│   ├── subnet-a
│   └── subnet-b
│
└── external connectivity
    ├── Transit Gateway
    ├── VPN
    └── future cloud connections

The endpoint VPC is not selected merely because it is central. It needs:

• private hosted-zone associations for inbound answers;
• routes towards external resolvers;
• return routes from those resolvers;
• suitable subnets in multiple Availability Zones;
• security groups permitting the selected protocols;
• operational ownership.

A subnet labelled dns does not create connectivity through the force of nominative optimism.

The complete query and response route must exist.

Shared rules centralise outbound behaviour

A central networking account can create outbound Resolver rules and share them with other AWS accounts using AWS Resource Access Manager.

Consuming accounts associate the shared rules with their VPCs. Queries from those VPCs then use the centrally owned outbound endpoint named by the rule.

BFSTORE-NETWORK

outbound endpoint

shared rules:
    azure.internal...
    gcp.internal...
    id.internal...
          │
          ▼
AWS RAM
          │
          ├── bfstore-dev
          ├── bfstore-staging
          ├── bfstore-prod
          └── bfstore-shared-services

This creates a useful platform boundary.

The network team controls:

• external target resolvers;
• endpoint security;
• DNS transport;
• common namespace rules.

Workload accounts control:

• which shared rules are associated with their VPCs;
• which applications use the resulting names;
• application access to the resolved services.

A shared rule remains centrally owned. Deleting or unsharing it changes how associated VPCs process matching queries, so its lifecycle is a cross-account production concern.

Resolver endpoints and rules are Regional

A rule created in one Region is not automatically available in another.

EU-WEST-2

inbound endpoint
outbound endpoint
regional rules


EU-CENTRAL-1

inbound endpoint
outbound endpoint
regional rules

A multi-Region design should decide whether:

• each Region forwards directly to external resolvers;
• one Region acts as a DNS hub;
• a hub failure affects other Regions;
• authorities and private zones exist in both Regions;
• rules and target addresses are kept consistent.

Forwarding every Region through one distant DNS hub may reduce infrastructure while adding latency and a central dependency.

A Regional service should not be made global through wishful arrows.

Operating Resolver endpoints

Availability starts with endpoint addresses

Each endpoint requires at least two IP addresses. AWS recommends placing them in at least two Availability Zones. Each address creates a Resolver-managed network interface.

DNS SERVICES VPC

Availability Zone A
    inbound endpoint ENI
    outbound endpoint ENI


Availability Zone B
    inbound endpoint ENI
    outbound endpoint ENI

External resolvers using an inbound endpoint should be configured with every endpoint address.

The target DNS environment behind an outbound endpoint should also provide multiple healthy resolvers where availability requires it.

AWS OUTBOUND ENDPOINT
        │
        ├── external DNS server A
        └── external DNS server B

Two AWS interfaces forwarding towards one fragile external DNS server still produce one very carefully approached failure domain.

Match security groups to the endpoint protocol

Do53 uses DNS over UDP or TCP. A Do53 endpoint policy should normally account for both transports on port 53 because DNS can fall back to TCP for large or truncated responses and other protocol behaviours.

DO53 INBOUND ENDPOINT

allow UDP/53 and TCP/53
from approved external resolver ranges


DO53 OUTBOUND ENDPOINT

allow UDP/53 and TCP/53
to approved target resolver addresses

DoH uses HTTPS over TCP port 443. DoH-FIPS is supported for inbound endpoints only. Delegation inbound endpoints currently support Do53 only.

DOH INBOUND ENDPOINT

allow TCP/443
from approved external resolver ranges

The exact policy should follow the protocols enabled on the endpoint and supported by the connected resolver.

Encryption does not replace DNS access control. It protects the query in transit. Security groups and resolver configuration still decide who may submit it.

Overly broad rules increase the number of clients capable of using the bridge. Restrictive, connection-tracked security-group paths can also reduce effective endpoint throughput, so security and scaling should be reviewed together.

Address family and DNS transport are separate from the answer

Resolver endpoints can use IPv4, IPv6 or dual-stack addressing. Inbound endpoints can also support DNS64 for IPv6-only clients that need synthesized AAAA answers for IPv4-only destinations, when paired with NAT64.

The design should ask:

• Are external resolvers reachable over IPv4, IPv6 or both?
• Which address family does the endpoint expose?
• Does the network path carry that family?
• Do security groups permit it?
• Does the returned application path support the address family advertised by DNS?

A dual-stack DNS endpoint does not make an application dual-stack.

It only allows the DNS transport to use both address families.

Prevent forwarding loops

Consider:

VPC Resolver
      │
      ▼
outbound rule
      │
      ▼
external DNS server
      │
      ▼
inbound endpoint
      │
      ▼
VPC Resolver

If the external resolver sends the same namespace back into AWS, and the AWS rule sends it out again, the query begins a circular tour.

AWS asks external

external asks AWS

AWS asks external

external asks AWS

The solution begins with clear authority:

For each namespace,
one resolver path must terminate
at an authority capable of answering it.

The design record should state:

• authoritative DNS system;
• forwarding origin;
• forwarding targets;
• permitted return behaviour;
• more-specific exceptions;
• expected terminal response.

A diagram containing two arrows is not automatically resilient.

Sometimes it is a very tidy loop.

Scale per endpoint address, not per diagram

As of July 2026, AWS documents up to 10,000 UDP DNS queries per second per endpoint IP address under qualifying conditions. Actual capacity depends on query type, response size, latency, target health, protocol and connection tracking. AWS notes that tracked paths can reduce effective capacity substantially.

ENDPOINT

IP address A
    → DNS query capacity

IP address B
    → DNS query capacity

IP address C
    → additional capacity

DNS traffic can be bursty:

• workloads restarting after an outage;
• short-lived Kubernetes Jobs;
• low TTLs on popular names;
• an external resolver losing its cache;
• repeated negative lookups during a migration;
• search-suffix mistakes;
• random-name traffic.

Scaling should consider peak query rate per endpoint address, not only average daily volume.

Observe the bridge and the authority behind it

Route 53 Resolver publishes CloudWatch metrics for inbound and outbound endpoints, including endpoint-interface health and query volume. Optional detailed metrics include latency and response outcomes such as SERVFAIL, NXDOMAIN, refusals and timeouts.

Outbound endpoints can also publish target name-server metrics, including request count, latency and timeouts for each target.

A useful bfstore dashboard would show:

INBOUND

queries by endpoint IP
healthy endpoint interfaces
response latency
SERVFAIL
NXDOMAIN
timeouts


OUTBOUND

queries by endpoint IP
target resolver request count
target resolver latency
target resolver timeouts
healthy endpoint interfaces

Resolver query logs can add:

• queried name;
• source VPC;
• source address;
• response code;
• returned answer.

Application telemetry shows whether failed resolution affected a customer operation.

CLOUDWATCH ENDPOINT METRICS

Is the bridge operating?


QUERY LOGS

Which questions crossed it?


APPLICATION TELEMETRY

What did DNS failure do
to the customer journey?

A healthy endpoint with a misspelled forwarding rule is still a beautifully monitored wrong turn.

Cost and quotas favour shared infrastructure

Resolver endpoint pricing is based on provisioned endpoint addresses and processed DNS queries. Because each endpoint requires at least two addresses, even an idle endpoint has a baseline cost.

Related architecture may also cost money:

• VPN;
• Transit Gateway;
• Direct Connect;
• cross-Region traffic;
• external resolver infrastructure;
• optional detailed CloudWatch metrics;
• log storage.

As of July 2026, AWS documents default quotas including:

• four Resolver endpoints per account per Region;
• six IP addresses per endpoint;
• 1,000 rules per account per Region;
• 2,000 associations between rules and VPCs per account per Region;
• six target addresses per rule.

Several quotas can be increased.

The intended shape remains:

few shared endpoints

many namespace rules

many VPC associations

The endpoint is transport infrastructure.

The rule is the unit of DNS-routing policy.

The bfstore architecture now

bfstore currently has:

• AWS VPCs;
• Route 53 private hosted zones;
• AmazonProvidedDNS;
• no on-premises data centre;
• no active Azure or GCP private network;
• no external DNS authority requiring conditional forwarding.

The current design remains:

route53_resolver:
  inbound_endpoints: []
  outbound_endpoints: []
  forwarding_rules: []
  delegation_rules: []

Private hosted zones are associated directly with the VPCs that consume them.

That is enough.

BFSTORE AWS TODAY

workload
   → AmazonProvidedDNS
   → associated private hosted zone

Resolver endpoints should be introduced when an external DNS relationship becomes real.

Future AWS-to-Azure or AWS-to-GCP integration

AWS workloads need cloud-private names:

AWS Resolver
    → outbound rule
    → outbound endpoint
    → cloud-side resolver

Cloud workloads need AWS-private names:

cloud-side resolver
    → inbound endpoint
    → AWS Resolver

The design requires both directions only when the DNS relationship is genuinely bidirectional.

External identity platform

An externally hosted identity platform may need to resolve a private AWS service, or AWS may need to resolve a private identity namespace.

The product integration should determine whether private DNS is necessary. A public authenticated endpoint does not require a private Resolver bridge merely because bfstore enjoys network diagrams.

Hybrid enterprise network

An on-premises DNS environment may forward AWS-private names inbound while AWS forwards corporate domains outbound.

That is the classic bidirectional Resolver endpoint use case.

A practical implementation sequence

1. Document the authority

namespace: azure.internal.bfstore.example
authority: azure-private-dns
consumers:
  - bfstore-dev
  - bfstore-staging
  - bfstore-prod

2. Establish network connectivity

Prove that the endpoint VPC can reach each external resolver and receive the reply.

3. Create endpoint subnets

Use at least two Availability Zones and reserve addresses for growth.

4. Create protocol-specific security groups

Permit only the required Do53 or DoH traffic between intended resolver ranges.

5. Create the endpoint

Choose inbound or outbound according to where the query originates.

6. Create narrow rules

Begin with the smallest domain requiring external authority. Use delegation only when the DNS authority model and required NS records support it.

7. Share and associate rules

Associate only the VPCs that genuinely consume the namespace.

8. Configure the external resolver

For ordinary inbound use, configure conditional forwarding to every endpoint address. For inbound delegation, create the required parent-side NS and glue records.

9. Enable evidence

Add query logging, endpoint metrics and target name-server metrics where useful.

10. Test success, failure and isolation

Confirm:

• the intended name resolves;
• a child name uses the correct authority;
• an unrelated name remains on its ordinary path;
• an unapproved network cannot query the endpoint;
• loss of one endpoint interface does not remove resolution;
• loss of one target resolver is tolerated;
• no forwarding loop exists.

The test is not complete when:

dig returned an address once.

It is complete when the authority, failure behaviour and isolation match the design.

The mental model I am keeping

                       DNS QUERY
                           │
                           ▼
                  WHERE DID IT ORIGINATE?
                           │
             ┌─────────────┴─────────────┐
             │                           │
             ▼                           ▼
          INSIDE AWS                 OUTSIDE AWS
             │                           │
             ▼                           ▼
      AmazonProvidedDNS          EXTERNAL RESOLVER
             │                           │
             ▼                           ▼
       WHO OWNS THE NAME?         WHO OWNS THE NAME?
             │                           │
    ┌────────┴────────┐          ┌───────┴────────┐
    │                 │          │                │
    ▼                 ▼          ▼                ▼
ROUTE 53         EXTERNAL DNS  AWS DNS       EXTERNAL DNS
    │                 │          │                │
    ▼                 ▼          ▼                ▼
ANSWER         OUTBOUND RULE  INBOUND         ANSWER
DIRECTLY            │         ENDPOINT        DIRECTLY
                    ▼             │
             OUTBOUND ENDPOINT    ▼
                    │        ROUTE 53 RESOLVER
                    ▼
              EXTERNAL RESOLVER

AmazonProvidedDNS answers:

Can AWS already resolve
this name locally?

The private hosted-zone association answers:

Does this VPC receive
the AWS-private namespace?

The inbound endpoint answers:

How can another DNS environment
ask Route 53 Resolver a question?

The outbound endpoint answers:

How can Route 53 Resolver
ask another authority a question?

The Resolver rule answers:

Which namespace should cross
the outbound bridge?

The system rule answers:

Which more-specific branch
should remain with Route 53?

Delegation answers:

Which DNS authority owns
this child namespace?

The endpoint security group answers:

Which resolvers may send
or receive these questions?

CloudWatch answers:

Is the bridge healthy,
and is the authority responding?

Query logging answers:

Which names actually crossed it?

And the lack of an endpoint answers:

Is every current DNS question
already being answered inside AWS?

For bfstore today, most of them are.

For bfstore’s future multi-cloud environment, that answer will change.

Create an inbound endpoint when another resolver needs AWS answers. Create an outbound endpoint when AWS needs another resolver’s answers. Create both when the DNS relationship is genuinely bidirectional.

Do not create either merely because the record is private.

Do not create one endpoint per hosted zone.

Do not forward a namespace before deciding who owns it.

Do not join two forwarding paths without checking whether they form a circle.

Resolver endpoints are not the foundation of VPC DNS.

AmazonProvidedDNS already holds that job.

The endpoints are the carefully guarded doors added when DNS must finally leave the room.

References and further reading

• What is Route 53 VPC Resolver?
• Considerations when creating inbound and outbound endpoints
• Forwarding inbound DNS queries to your VPCs
• Values for inbound endpoints
• Forwarding outbound DNS queries to your network
• Values for outbound endpoints
• Using rules to control outbound DNS
• How VPC Resolver matches domain names to rules
• Private hosted-zone considerations
• Resolver delegation rules tutorial
• High availability for Resolver endpoints
• Resolver endpoint scaling
• Monitoring Resolver endpoints with CloudWatch
• Route 53 Resolver quotas
• Amazon Route 53 pricing

The observability service currently has this private address:

10.96.12.45

I could place that address directly in the application configuration:

telemetry:
  endpoint: 10.96.12.45:4317

That works until the service moves, a load balancer replaces it, recovery activates another destination or somebody must remember what 10.96.12.45 represents.

An address identifies a current network location.

The application needs a stable service identity:

otel.prod.aws.internal.bfstore.example

The name can point to:

• an internal load balancer;
• a PrivateLink interface endpoint;
• one or more intentional private addresses;
• another Route 53 record;
• a failover destination.

The application keeps the service name.

DNS absorbs the change in location.

BFSTORE APPLICATION
        │
        │ query:
        │ otel.prod.aws.internal.bfstore.example
        ▼
ROUTE 53 VPC RESOLVER
        │
        ▼
PRIVATE DNS VIEW
        │
        ▼
PRIVATE SERVICE DESTINATION

A Route 53 private hosted zone contains records that Route 53 VPC Resolver can use to answer queries for a domain and its subdomains. The private answer is available through the resolver contexts to which the zone is distributed, traditionally by direct VPC association and now also through Route 53 Profiles. AWS describes the basic private hosted-zone model here.

A private hosted zone gives selected resolver views a private interpretation of a DNS namespace.

This article covers VPC DNS through direct associations, Profiles and Resolver rules and endpoints. Route 53 Global Resolver is outside scope. AWS documents it separately.

Private DNS is one part of private service access

Suppose bfstore creates:

prod.aws.internal.bfstore.example

with these records:

orders.prod.aws.internal.bfstore.example
payments.prod.aws.internal.bfstore.example
otel.prod.aws.internal.bfstore.example

The zone tells Route 53 how to answer those names.

It does not establish a route to the returned address.

If:

otel.prod.aws.internal.bfstore.example

resolves to:

10.96.12.45

the client still needs a network path to 10.96.12.45.

That path may use:

• local VPC routing;
• Transit Gateway;
• VPC peering;
• AWS PrivateLink;
• another private network connection.

Security groups, network ACLs, firewalls and application authentication must also permit the connection.

DNS is not private access control. A strong design provides both a stable name and an intentional access path.

How the private namespace reaches a VPC

A private hosted zone must be associated with a VPC when it is created. More VPCs can be associated afterwards. The associated VPCs require enableDnsSupport and enableDnsHostnames to be enabled for private hosted-zone resolution. AWS documents those VPC requirements and the matching behaviour.

There are now two principal ways to distribute the namespace.

Direct hosted-zone association

The zone is associated directly with each consuming VPC.

PRIVATE HOSTED ZONE
prod.aws.internal.bfstore.example
            │
            ├── bfstore-prod VPC
            ├── bfstore-observability VPC
            └── bfstore-backup VPC

This model is explicit and works well when the VPC inventory is modest or when each association needs individual approval.

Route 53 Profile

A Route 53 Profile can collect DNS configuration and apply it to many VPCs, including VPCs in other accounts through AWS Resource Access Manager.

Profiles can distribute:

• private hosted-zone associations;
• Resolver rules;
• DNS Firewall rule groups;
• Resolver query-logging configurations.

A VPC can have one Profile association. When local VPC DNS configuration conflicts with Profile-delivered configuration, the local configuration takes precedence; the most-specific matching domain still determines the winner where names differ in specificity. AWS documents Profile resources and precedence here.

ROUTE 53 PROFILE
production-dns
        │
        ├── private hosted zones
        ├── Resolver rules
        ├── DNS Firewall policies
        └── query logging
                │
                ├── prod VPC
                ├── observability VPC
                └── backup VPC

A useful choice is:

The association mechanism defines resolution scope: which resolver views receive the namespace. It does not authorise workloads to use the services behind the records.

Transit Gateway does not distribute DNS

Suppose the production and observability VPCs are attached to the same Transit Gateway.

The routes work:

BFSTORE PROD VPC
       │
       ▼
TRANSIT GATEWAY
       │
       ▼
OBSERVABILITY VPC

That does not automatically make a private hosted zone available to production.

Transit Gateway provides Layer 3 connectivity.

A direct zone association, Route 53 Profile or deliberate forwarding design determines whether the source resolver view receives the private answer.

TRANSIT GATEWAY

Can the returned address
be reached?


DNS CONFIGURATION

Can this resolver view
obtain the private answer?

Routing joins address spaces. DNS configuration joins naming views.

They must often be designed together, but they are not the same control plane.

Workloads query Route 53 VPC Resolver

Workloads do not normally query the private zone’s displayed name servers directly.

They send queries to Route 53 VPC Resolver, the recursive DNS service available in each VPC. VPC Resolver answers local VPC names, associated private hosted zones and public recursive queries, and applies matching Resolver rules. AWS describes the resolver path here.

WORKLOAD
   │
   ▼
ROUTE 53 VPC RESOLVER
   │
   ├── local VPC names
   ├── private hosted zones
   ├── Resolver rules
   └── public recursive resolution

The ordinary DHCP option set normally directs resources towards AmazonProvidedDNS. Custom first-hop DNS servers need conditional forwarding, commonly through Resolver inbound endpoints, for names AWS should answer.

How VPC Resolver chooses the winning answer

Most private DNS mistakes become easier to understand when resolution is treated as a selection process.

The resolver considers:

1. local and Profile-delivered DNS configuration;
2. matching Resolver rules;
3. matching private hosted zones;
4. namespace specificity;
5. record presence;
6. public recursion only when no private namespace or rule claims the query.

Split-view DNS

Route 53 can use public and private hosted zones with the same name.

PUBLIC ZONE
bfstore.example

PRIVATE ZONE
bfstore.example

An internet client receives the public view.

A client using VPC Resolver in a resolver view containing the private zone receives the private view.

INTERNET CLIENT
      │
      ▼
PUBLIC ZONE
      │
      ▼
PUBLIC ENDPOINT


VPC CLIENT
      │
      ▼
PRIVATE ZONE
      │
      ▼
PRIVATE ENDPOINT

The zones do not share or inherit records.

A matching private zone prevents public fallback

Suppose the public zone contains:

docs.bfstore.example

The production resolver view also contains a private zone:

bfstore.example

but that private zone does not contain the docs record.

VPC Resolver finds a matching private namespace, searches it and returns NXDOMAIN.

It does not retry against the public zone. AWS documents this behaviour explicitly.

QUERY
docs.bfstore.example
        │
        ▼
MATCHING PRIVATE ZONE
bfstore.example
        │
        ▼
RECORD ABSENT
        │
        ▼
NXDOMAIN

A broad same-name private zone claims authority over the complete branch for that resolver view.

If internal clients still need public records, bfstore must:

• reproduce the required records privately;
• use a narrower private namespace;
• redesign forwarding;
• avoid creating the overlapping private zone.

The most-specific namespace wins

A resolver view may contain:

internal.bfstore.example

prod.internal.bfstore.example

payments.prod.internal.bfstore.example

For:

authorise.payments.prod.internal.bfstore.example

the most-specific matching zone is:

payments.prod.internal.bfstore.example

VPC Resolver searches that zone.

If the record is absent, it does not continue upwards and search the less-specific zones for the same record. AWS documents most-specific matching for overlapping private zones.

This allows independent authority for narrower namespaces, but can also create accidental shadowing. Review a new, more-specific zone like a routing change.

The word “delegation” should be reserved for designs that actually use DNS delegation semantics. A separately associated child private hosted zone can become the winning authority without the parent zone containing an NS delegation.

A Resolver rule can override the hosted zone

Suppose a VPC has:

Private hosted zone:
corp.example

Forwarding rule:
corp.example
    → enterprise DNS

The forwarding rule takes precedence over the private hosted zone.

Resolver also chooses the most-specific matching rule when several rules apply. AWS documents rule selection and private-zone interactions here.

QUERY
service.corp.example
        │
        ▼
MATCHING FORWARD RULE
corp.example
        │
        ▼
ENTERPRISE DNS

A correct private record can remain unused because a rule intercepted the namespace. Troubleshooting must inspect both zones and rules.

Choose zone boundaries around ownership and audience

One possible design is a broad zone:

internal.bfstore.example

It could contain production, non-production and platform records.

That minimises the number of zones.

It also centralises every change beneath one authority.

A more separated design might use:

prod.aws.internal.bfstore.example

nonprod.aws.internal.bfstore.example

platform.aws.internal.bfstore.example

Separate a zone when the namespace needs a different owner, audience, IAM policy, change lifecycle or incident boundary. Production records should not normally share the everyday development DNS role.

The displayed hierarchy does not imply that the zones are automatically delegated to one another. Each zone becomes authoritative in resolver views where it is directly or indirectly associated, subject to the resolver’s selection rules.

Names should express durable service identity rather than incidental infrastructure.

Prefer:

orders.prod.aws.internal.bfstore.example

over:

orders-eks-cluster-3-subnet-a.prod...

The service may move between clusters without requiring every client to learn a new name.

Records should point to durable service boundaries

Private hosted zones support ordinary Route 53 record types and alias records.

Common bfstore patterns include:

Internal load balancer

orders.prod.aws.internal.bfstore.example
    → internal ALB or NLB alias

PrivateLink endpoint

otel.prod.aws.internal.bfstore.example
    → interface endpoint alias

Database endpoint

mysql.orders.prod.aws.internal.bfstore.example
    → RDS endpoint through CNAME where appropriate

Intentional private address

legacy.prod.aws.internal.bfstore.example
    → 10.80.40.25

Alias records can target supported AWS resources and can be used at a zone apex. A CNAME cannot be used at the apex and cannot coexist with other data at the same owner name. AWS compares aliases and CNAME records here.

Use direct IP records only for intentionally stable, owned addresses, not as an inventory of ephemeral Kubernetes Pods.

KUBERNETES-INTERNAL DISCOVERY

use Kubernetes Service and cluster DNS


VPC OR CROSS-VPC BOUNDARY

consider Route 53 private DNS

Private hosted zones are strongest at durable service boundaries. Publishing a record does not grant permission to use the service.

DNS

discover the destination


NETWORK POLICY

permit the connection


SERVICE IDENTITY

authorise the operation

A useful internal name is a stable label, not an invitation.

TTL and alias records need different treatment

DNS resolvers cache ordinary record answers according to the record’s time to live.

RECORD CHANGED
      │
      ▼
ROUTE 53 HAS NEW ANSWER
      │
      ▼
CACHE MAY HOLD OLD ANSWER
      │
      ▼
TTL EXPIRES
      │
      ▼
CLIENT QUERIES AGAIN

Shorter TTLs make changes visible sooner; longer TTLs reduce query frequency. Alias records do not always expose a configurable TTL.

• When an alias points to an AWS resource, Route 53 uses the target resource’s default TTL.
• When an alias points to another record in the same hosted zone, Route 53 uses the target record’s TTL.
• Ordinary non-alias records such as A, AAAA and CNAME records have an explicitly configured TTL.

AWS documents these alias TTL rules here.

A team cannot assume it can lower an ALB alias TTL before migration; the target controls it.

The change process should distinguish:

ROUTE 53 CHANGE STATUS

Has Route 53 applied the new record?


CACHE LIFETIME

How long may resolvers and clients
retain the previous answer?

INSYNC confirms the Route 53 change has propagated through Route 53.

It does not flush answers already cached by clients or recursive resolvers.

Health and failover depend on the target

Private hosted zones support failover and other routing policies.

The health source should match the target and the capability being protected.

When EvaluateTargetHealth is enabled for a supported alias target, the alias inherits the health of the referenced AWS resource or record. For load balancers, Route 53 can use load-balancer and target-group health. AWS documents alias target health here.

This differs from a standalone Route 53 health check. Health checkers cannot directly monitor private, non-routable addresses, so private services can use a CloudWatch alarm-backed health check. AWS documents both points here. health-check-addresses

PRIVATE SERVICE
      │
      ▼
APPLICATION OR SYNTHETIC METRIC
      │
      ▼
CLOUDWATCH ALARM
      │
      ▼
ROUTE 53 HEALTH CHECK
      │
      ▼
FAILOVER DECISION

The signal should represent the capability the DNS decision is meant to protect.

An instance status check may prove that an instance is running.

It may not prove that:

• the service can process an order;
• its database is available;
• the recovery endpoint is ready;
• required credentials remain valid.

DNS failover affects future lookups.

It does not move established connections, and cached answers may remain in use until their TTL expires.

Private routing policies use AWS resolver context

Private hosted zones support several routing policies, including failover, weighted, latency and geolocation.

For private geolocation records, Route 53 uses the AWS Region of the originating VPC, not necessarily the human user’s location. Resolver endpoints can therefore shape the location Route 53 observes. AWS documents the private-zone behaviour here.

Before using private latency or geolocation routing, state explicitly:

Which resolver location does Route 53 see?

Does that location represent
the client population we intend to steer?

For many internal services, explicit regional names or application-level routing may be easier to reason about.

Multi-account ownership

bfstore may place the private hosted zone in bfstore-network while workload VPCs live in separate accounts.

For a direct cross-account association:

1. the zone-owning account authorises one VPC association;
2. the VPC-owning account completes the association;
3. the unused authorisation should be removed.

The cross-account workflow requires the API, CLI, SDK or infrastructure code rather than the Route 53 console alone. AWS documents the handshake here.

ZONE ACCOUNT
     │
     ▼
AUTHORISE VPC
     │
     ▼
VPC ACCOUNT
     │
     ▼
ASSOCIATE ZONE

For repeated organisation-wide configuration, a shared Route 53 Profile may remove much of that per-zone, per-VPC association work.

Profiles are shared through AWS RAM, and updates to the Profile propagate to its associated VPCs. AWS documents Profile sharing here.

The choice should be intentional:

DIRECT ASSOCIATION

explicit exceptional relationship


ROUTE 53 PROFILE

repeatable resolver view
for a class of VPCs

DNS write access is data-path authority

Changing:

payments.prod.aws.internal.bfstore.example

can redirect payment traffic towards:

• the wrong environment;
• an obsolete endpoint;
• a compromised destination;
• a recovery service that is not ready.

DNS write permission is production data-path authority.

The change model should distinguish:

Route 53 IAM conditions can restrict changes by normalised record name, record type and change action, and can restrict zone associations to particular VPCs. AWS documents these fine-grained conditions here.

That supports a model such as:

APPLICATION TEAM

declares a service-name requirement


PLATFORM PIPELINE

validates the namespace and destination


ROUTE 53 ROLE

updates only approved names and types

The key question is who may redirect the name and where that decision is reviewed.

Logging and inventory must cover the resolver view

VPC Resolver query logging can record queries from associated VPCs and deliver them to CloudWatch Logs, S3 or Firehose.

Useful fields include:

• VPC and source information;
• queried name and record type;
• response code;
• returned answer;
• DNS Firewall action.

The logs can identify clients using retired names, explain NXDOMAIN, show failover lookups and reveal unexpected domains.

VPC Resolver caches answers.

Query logging records unique queries that reach the resolver, not repeated queries answered from the resolver cache. AWS documents that limitation here.

ONE QUERY-LOG ENTRY

does not equal

ONE APPLICATION REQUEST

The inventory must also cover every way DNS configuration reaches the VPC:

resolver_view:
  vpc_id: vpc-example
  region: eu-west-2

private_zones:
  direct:
    - zone_id: Z123
      name: prod.aws.internal.bfstore.example

  profiles:
    - profile_id: rp-example
      private_zones:
        - platform.aws.internal.bfstore.example

resolver_rules:
  - domain: corp.example
    type: FORWARD

query_logging:
  destination: central-dns-logs

dns_firewall:
  policy: production-egress

endpoints:
  inbound: []
  outbound:
    - rslvr-out-example

This distinction matters because ListHostedZonesByVPC and GetHostedZone do not report private hosted-zone associations delivered through Route 53 Profiles. Profile resource-association APIs must also be queried. AWS documents the inventory limitation here.

The architecture must know which rule, association or Profile actually supplies the resolver view.

Quotas should inform, not dictate, the design

Route 53 applies quotas to hosted zones, records, associations, Profiles and other resources.

AWS currently documents:

• 10,000 records per hosted zone by default;
• 300 direct VPC associations per private hosted zone;
• Route 53 Profiles as the recommended model when more associations are required.

Because quotas can change or be increased, runbooks should link to the current page. Current quotas are documented here.

The design tension remains:

TOO BROAD

one zone
many unrelated owners
large blast radius


TOO FRAGMENTED

one zone per record
large inventory
complex authority tree


USEFUL

one zone per meaningful
owner and audience boundary

The objective is not the fewest zones.

It is the clearest authority model.

A bfstore private DNS design

I would place private DNS governance in the bfstore-network account and use a combination of Route 53 Profiles and exceptional direct associations.

aws.internal.bfstore.example
│
├── prod.aws.internal.bfstore.example
├── nonprod.aws.internal.bfstore.example
└── platform.aws.internal.bfstore.example

These are separate authority zones, not automatically delegated children.

Production Profile

A bfstore-production-dns Profile could contain:

• prod.aws.internal.bfstore.example;
• approved platform-zone access;
• central query logging;
• standard DNS Firewall policy;
• common enterprise forwarding rules.

It would be shared with production-class VPCs through AWS RAM.

Non-production Profile

A separate Profile could provide:

• nonprod.aws.internal.bfstore.example;
• development forwarding rules;
• non-production logging and firewall policy.

Development and staging can split when their ownership diverges.

Exceptional direct associations

A narrow zone or rule can be associated directly when one VPC needs an exceptional resolver view.

Because local configuration overrides conflicting Profile configuration, the exception must be visible in inventory.

A conceptual record could look like:

private_hosted_zone:
  name: prod.aws.internal.bfstore.example
  owner_account: bfstore-network
  owner_team: network-platform

distribution:
  profile:
    name: bfstore-production-dns
    shared_with:
      - bfstore-prod
      - bfstore-observability
      - bfstore-backup

records:
  - name: otel.prod.aws.internal.bfstore.example
    type: A_ALIAS
    target: observability-ingest-nlb
    owner: observability-platform
    health:
      evaluate_target_health: true
    ttl:
      managed_by: alias-target

  - name: orders.prod.aws.internal.bfstore.example
    type: A_ALIAS
    target: internal-orders-alb
    owner: order-service
    health:
      evaluate_target_health: true
    ttl:
      managed_by: alias-target

controls:
  query_logging: central-dns-logs
  public_dnssec_validation: evaluated-separately
  record_changes: infrastructure-pipeline
  resolver_view_inventory: required

public_dnssec_validation is deliberately scoped.

VPC Resolver DNSSEC validation applies to recursively resolved signed public domains. It is not a mechanism that signs or authenticates records inside this private hosted zone. AWS documents VPC DNSSEC validation here.

Test from the consuming VPC

A hosted zone showing the correct record in the console does not prove that a client receives it.

Test from the resolver view that consumes the namespace.

dig otel.prod.aws.internal.bfstore.example A

dig missing.prod.aws.internal.bfstore.example A

getent hosts otel.prod.aws.internal.bfstore.example

Record the source VPC and Region, resolver configuration, VPC DNS attributes, direct and Profile associations, matching rules, winning namespace, effective TTL, route and application result.

NAME RESOLVES
      │
      ▼
ADDRESS RETURNED
      │
      ▼
ROUTE EXISTS
      │
      ▼
NETWORK POLICY PERMITS
      │
      ▼
SERVICE AUTHORISES

A successful answer proves only the first two steps; a failed connection does not automatically mean DNS is broken.

Common failure patterns

Public name works, VPC receives NXDOMAIN

Likely cause:

• a matching private zone claims the namespace;
• the record is absent privately;
• Resolver does not fall back to public DNS.

Name resolves in production but not development

Likely cause:

• the production resolver view includes the zone;
• the development resolver view does not.

This may be intentional.

Correct record exists but is ignored

Likely causes:

• a forwarding rule takes precedence;
• local VPC configuration overrides a Profile;
• a more-specific zone owns the branch;
• the client uses a custom resolver without the required forwarding path.

Name resolves to the wrong environment

Likely causes:

• overly broad association or Profile;
• copied record points to staging;
• local override conflicts with the shared Profile;
• stale client or resolver cache.

Name resolves, connection times out

Likely causes:

• missing route;
• incorrect Transit Gateway propagation;
• security-group or firewall denial;
• return-path failure;
• service not listening.

DNS may have succeeded.

Failover does not switch as expected

Likely causes:

• the health source does not represent the application;
• EvaluateTargetHealth is unsupported or ineffective for the target;
• a CloudWatch-alarm health check is misconfigured;
• the old answer remains cached;
• existing connections do not repeat DNS resolution.

Inventory says the VPC has no zone, but it resolves

Likely cause:

• the zone arrived through a Route 53 Profile;
• the inventory queried only direct VPC associations.

Diagnose namespace selection, cache state, routing and service acceptance separately.

The decision model

The recurring questions are:

Which namespace are we claiming?

Which resolver views receive it?

Does a local setting, Profile or rule win?

Which zone is the most-specific authority?

What durable service boundary
does the record represent?

Who may redirect the name?

How is health evaluated?

What does the client cache?

Can the returned address be reached
and used by the caller?

The model is:

                       PRIVATE NAMESPACE
                               │
                               ▼
                      PRIVATE HOSTED ZONE
                               │
                 ┌─────────────┴─────────────┐
                 │                           │
                 ▼                           ▼
              RECORDS               RESOLUTION SCOPE
                                         │
                           ┌─────────────┴─────────────┐
                           │                           │
                           ▼                           ▼
                   DIRECT ASSOCIATION             PROFILE
                           │                           │
                           └─────────────┬─────────────┘
                                         ▼
                               VPC RESOLVER VIEW
                                         │
                         ┌───────────────┼───────────────┐
                         │               │               │
                         ▼               ▼               ▼
                    LOCAL OVERRIDE   RESOLVER RULE   PRIVATE ZONE
                         │               │               │
                         └───────────────┼───────────────┘
                                         ▼
                              MOST-SPECIFIC WINNER
                                         │
                                         ▼
                                   DNS RESPONSE
                                         │
                                         ▼
                                  CACHE AND TTL
                                         │
                                         ▼
                                  NETWORK ROUTE
                                         │
                                         ▼
                               NETWORK AUTHORISATION
                                         │
                                         ▼
                              APPLICATION AUTHORISATION

A private hosted zone creates private meaning, not private infrastructure.

The zone defines authority. Direct associations and Profiles define the resolver views. Rules can redirect the question, and the most-specific namespace decides which authority wins. Alias targets may control caching, while health determines which future answers remain eligible.

The network carries the resulting packet. Network and application controls decide whether the request should succeed.

DNS lets the architecture move without requiring every application to memorise its current address.

References and further reading

Reviewed against official AWS documentation on 31 July 2026.

• Working with private hosted zones
• Private hosted-zone considerations
• What is Route 53 VPC Resolver?
• What are Route 53 Profiles?
• Working with shared Route 53 Profiles
• Route 53 Global Resolver
• Resolver rule behaviour
• Choosing between alias and non-alias records
• Alias target health
• Failover in private hosted zones
• Health-check address restrictions
• Geolocation routing in private hosted zones
• Cross-account private hosted-zone associations
• Route 53 IAM conditions
• VPC Resolver query logging
• ListHostedZonesByVPC and Profile inventory
• Route 53 quotas
• VPC Resolver DNSSEC validation

The order service confirms the basket, reserves inventory, creates an order and asks the payment service to begin payment.

Events move through Kafka.

Records are written to MySQL.

A payment provider later sends a webhook.

From the customer’s perspective, this is one action:

Place order

From a security perspective, it is a procession of identities, messages, stores and trust decisions.

CUSTOMER
   │
   ▼
PUBLIC EDGE
   │
   ▼
ENVOY GATEWAY
   │
   ▼
ORDER SERVICE
   │
   ├── basket
   ├── inventory
   ├── payment
   ├── MySQL
   └── Kafka
            │
            ▼
     PAYMENT PROVIDER
            │
            ▼
         WEBHOOK

Each arrow makes an assumption.

The edge assumes the customer’s session is authentic.

The order service assumes the basket belongs to that customer.

Kafka consumers assume an event came from an approved producer and describes a permitted business transition.

The payment service assumes the provider webhook is authentic, fresh and relevant to the referenced order.

The delivery system assumes the deployed image is the artefact that passed review.

Threat modelling makes those assumptions visible before an attacker, outage or hurried administrator tests them on bfstore’s behalf.

AWS frames threat modelling around four questions:

1. What are we working on?
2. What can go wrong?
3. What are we going to do about it?
4. Did we do a good job?

The exercise should therefore produce more than a diagram. It should produce prioritised decisions, owners, tests and evidence. AWS Well-Architected threat-modelling guidance

A threat model is an architecture diagram whose assumptions have been asked difficult questions.

STRIDE supplies six families of those questions.

It does not answer them automatically.

Start with the system, not the acronym

A useful threat model moves through these stages:

STRIDE belongs mainly inside “What can go wrong?”

Beginning with:

What spoofing threats exist
in cloud applications?

usually produces generic advice.

The system must provide the detail.

STRIDE is a set of prompts

Microsoft’s threat-modelling guidance uses data-flow diagrams, trust boundaries and STRIDE to identify design risks while they remain cheaper to change. Microsoft Threat Modeling Tool Microsoft data-flow diagrams

The categories overlap.

A forged webhook may involve spoofing and tampering.

An unauthorised service publishing OrderPaid may involve spoofing, tampering or elevation of privilege.

Deleting audit evidence may involve tampering and repudiation.

The category keeps the discussion moving. The actual system provides the threats.

Scope the checkout journey

This first model covers:

Customer places and pays for an order.

Included:

• customer session use and public ingress
• basket, catalog, inventory, order and payment services
• gRPC communication, Kafka and MySQL
• payment-provider calls and webhooks
• workload identity
• build and deployment paths
• relevant audit and telemetry paths

Not analysed in detail:

• the customer’s device
• the payment provider’s internal infrastructure
• AWS physical facilities
• unrelated administration journeys

The provider’s internal systems remain outside bfstore’s control.

The integration boundary remains inside the model.

The identity provider and session-management components are abstracted from the first diagram. They need a related model covering credential issuance, account recovery, token validation, revocation, session storage and signing-key lifecycle.

Identify the assets

The checkout journey protects more than secrets.

The product price is intentionally public.

Changing it from £84 to 84 pence without permission is still a security event.

Draw flows and explain each trust boundary

A trust boundary is not simply another hop. It marks a meaningful change in control, identity, ownership or privilege. Microsoft data-flow diagrams

                         INTERNET
                            │
                   TRUST BOUNDARY A
                            │
                            ▼
                    ALB / PUBLIC EDGE
                            │
                   TRUST BOUNDARY B
                            │
                            ▼
                       ENVOY GATEWAY
                            │
                   TRUST BOUNDARY C
                            │
          ┌─────────────────┼─────────────────┐
          │                 │                 │
          ▼                 ▼                 ▼
       BASKET             ORDER            CATALOG
                           │
                    ┌──────┼──────┐
                    ▼      ▼      ▼
                INVENTORY MYSQL  KAFKA
                           │
                           ▼
                        PAYMENT
                           │
                  TRUST BOUNDARY D
                           │
                           ▼
                  PAYMENT PROVIDER
                           │
                           ▼
                    SIGNED WEBHOOK

The services should not trust one another merely because they run in the same cluster.

NETWORK POLICY

May traffic travel along this path?


WORKLOAD IDENTITY

Which workload is calling?


SERVICE AUTHORISATION

May that identity perform
this operation on this resource?

Kubernetes NetworkPolicy controls supported Layer 3 and Layer 4 paths and requires an enforcing network implementation. It is not workload identity or operation-level authorisation. Kubernetes NetworkPolicy

A second diagram covers delivery:

ENGINEER
   │
   ▼
GITHUB
   │
   ▼
GITHUB ACTIONS
   │
   ├── build and test
   ├── produce image
   ├── attest artefact
   └── request deployment
            │
            ▼
       AWS IAM ROLE
            │
            ▼
   TERRAKUBE / KUBERNETES
            │
            ▼
       BFSTORE RUNTIME

A model that covers only customer traffic may defend an application that the pipeline can replace with arbitrary code.

Spoofing: who are you really?

Customer session spoofing

An attacker steals or reuses a customer session and places an order as that customer.

Conditions may include:

• tokens in logs
• weak account recovery
• long-lived access
• missing issuer or audience validation
• no revocation after credential changes

Controls may include short-lived access, secure session storage, issuer and audience checks, reauthentication for sensitive operations, revocation and anomaly detection.

Workload spoofing

A compromised Pod calls the order service while claiming to be the payment service.

A private source IP is insufficient identity.

Controls include distinct Kubernetes service accounts, short-lived workload credentials, explicit service authorisation, network segmentation and producer-specific Kafka ACLs.

EKS can provide scoped temporary AWS credentials through IAM roles for service accounts or EKS Pod Identity. The resulting role still needs narrow permissions, and the node credential path still needs protection. EKS IRSA EKS Pod Identity

Payment webhook spoofing and replay

Plausible JSON is not proof that the payment provider sent an event.

The payment service should verify:

• the signature over the exact received payload, using the provider’s supported library or canonicalisation rules
• signed timestamp or equivalent freshness evidence
• unique event identifier and replay window
• event schema and type
• relationship between provider payment reference and bfstore order
• whether the current state permits the transition

A valid signature proves less than many implementations assume. It does not prove that the event is fresh, previously unprocessed, relevant to the named order or permitted to change its state.

PENDING_PAYMENT
       │
       ▼
PAYMENT_AUTHORISED
       │
       ▼
READY_FOR_FULFILMENT

Only permitted, idempotent transitions should succeed.

Tampering: what changed?

Request tampering

A customer changes:

{
  "product_id": "product-42",
  "quantity": 1,
  "unit_price_pence": 1
}

The order service must retrieve or validate authoritative product, discount and pricing data.

The customer may request:

Buy one product-42.

They may not declare:

Product-42 costs one penny.

Event tampering

A malicious or compromised service publishes OrderPaid for an order whose payment failed.

Controls include broker-authenticated producer identities, topic authorisation, schema validation, immutable event IDs, authoritative state checks, idempotent consumers and provider reconciliation.

An envelope field such as:

{
  "producer": "payment-service"
}

is only a claim unless tied to broker-authenticated identity or another trusted provenance mechanism.

TLS protects transit.

It does not prove that an authorised producer generated a truthful business event.

Database tampering

Prepared statements protect SQL structure from injection.

They do not stop an over-privileged runtime identity from performing operations it was granted.

Controls include service-owned schemas, separate migrator and runtime accounts, least-privilege grants, database constraints and reconciliation among order, payment and inventory state.

Supply-chain tampering

The delivery path must answer two questions:

WORKFLOW IDENTITY

Which repository, environment
and workflow may request deployment?


ARTEFACT IDENTITY

Which exact digest was reviewed,
built, attested and approved?

GitHub Actions can use OIDC to obtain short-lived AWS credentials. The AWS trust policy should constrain audience and subject claims, including the expected repository and protected environment or branch. GitHub recommends environment protection rules where environments are used. GitHub OIDC for AWS

OIDC trust does not prove that the correct artefact is deployed.

Use immutable image digests and verify build provenance or attestations before deployment. GitHub artefact attestations can bind a build to its repository, workflow, commit and digest, but only verification gives the attestation operational value. GitHub artefact attestations

The dangerous request may not enter through the ALB. It may arrive as a valid deployment issued by an unexpectedly powerful workflow.

Repudiation: prove enough to investigate

Repudiation appears when an actor denies an action and the system lacks enough trustworthy evidence to reconstruct it.

A useful application audit event records:

event:
  type: order.refund_requested
  event_id: audit-91c4
  occurred_at: 2026-06-24T11:14:32Z

actor:
  type: customer
  subject_id: customer-421
  session_reference: session-72a1

execution:
  workload_identity: order-service
  policy_decision: refund-owner-check

resource:
  type: order
  id: order-8042

outcome:
  decision: accepted
  amount_pence: 4200

correlation:
  trace_id: 6d2c4e1f

An audit event may contain an opaque session reference.

It must not contain a bearer token, refresh token or another reusable credential.

CloudTrail can provide AWS control-plane and configured data-event evidence. Its usefulness depends on enabled event sources, destinations, retention and protection against alteration. Management events are generally enabled for trails, while data events require explicit configuration. CloudTrail overview CloudTrail management events CloudTrail data events

bfstore must still create application-level evidence for business actions.

Logs should be centralised with narrow write paths, restricted deletion, access auditing, retention and integrity controls.

They provide evidence whose strength depends on the complete logging path.

Information disclosure: where did the data travel?

Sensitive data can escape through error responses, telemetry, Kafka, support exports, environment variables, broad queries or compromised workload identities.

Telemetry

A payment failure should not log:

customer=mariam@example.com
address=42 Example Road
token=pm_secret_value

Use structured logging, field allow-lists, token redaction, no raw request bodies by default, purpose-based retention and tests for sensitive-field leakage.

Customer IDs, emails and order IDs should not become metric labels merely because labels are convenient.

Events

An OrderCreated event should contain what its consumers need, not a travelling copy of the customer profile.

message OrderCreated {
  string event_id = 1;
  string order_id = 2;
  string customer_id = 3;
  int64 total_pence = 4;
}

A shipping workflow may need a delivery address through a narrow path.

A recommendation consumer probably does not.

Event schemas are disclosure boundaries.

Secrets and encryption

Kubernetes recommends encryption at rest, least-privilege access and limiting Secret visibility to the containers that need it. Base64 is not protection. Kubernetes Secrets

AWS KMS encryption context can bind non-secret context to ciphertext and support policy conditions and auditability. It appears in plaintext, so it must not contain customer or secret values. AWS KMS encryption context

Encryption reduces the value of unauthorised storage access.

It does not excuse over-collection or broad decryption permission.

Denial of service: can legitimate work finish?

STRIDE draws attention to loss of availability.

The cause may be hostile, accidental or external. The threat record should state which cause it analyses because security and resilience controls may differ.

Abuse and cost amplification

Threats include request floods, expensive searches, oversized bodies, baskets with thousands of items and repeated checkout calls.

Controls include rate and connection limits, request-size limits, bounded input collections, caching and inexpensive rejection before expensive work.

VALID TYPE

does not automatically mean

SAFE AMOUNT OF WORK

Retry storms

A slow provider can be multiplied by retries from callers, consumers and redelivery.

Use deadlines, bounded retries, backoff and jitter, concurrency limits, circuit breakers, idempotency, dead-letter handling and load shedding.

Platform exhaustion

Kafka may suffer oversized messages, hot partitions, lag or poison-message loops.

A compromised workload may consume shared CPU, memory or Pod capacity.

Use topic limits, ACLs, lag monitoring, resource requests and limits, LimitRange, ResourceQuota and controlled failure testing.

External dependency failure

The storefront may be healthy while the payment provider is unavailable.

That may result from ordinary failure, overload, network partition or attack.

The model should name the assumed cause and consider bounded queueing, explicit delayed states, reconciliation, alerting and provider failover where realistic.

Availability is end to end.

A green Pod is not proof that a customer can place an order.

Elevation of privilege: what can this identity become?

Customer to administrator

An ordinary customer manipulates an endpoint or claim and invokes an administration operation.

The control must be server-side authorisation tied to trusted identity.

Hiding the button is interface design, not access control.

Service to broader authority

The notification service may send messages.

It does not need to refund payments, modify inventory, decrypt customer profiles or publish OrderPaid.

Workload roles, database accounts and Kafka ACLs should match narrow service functions.

Pod to node or cluster authority

A compromised Pod may exploit an over-privileged service account, mounted credentials, host access, Linux capabilities or node credentials.

Kubernetes defines Privileged, Baseline and Restricted Pod Security Standards. Pod Security Admission can enforce, audit or warn against a version-pinned profile at namespace level. Pod Security Standards Pod Security Admission

bfstore application namespaces should target a version-pinned restricted posture unless a reviewed exception is required.

Additional bfstore hardening, such as read-only filesystems, narrow token use and workload-specific egress, should be distinguished from the controls defined by the standard itself.

Kubernetes RBAC must also account for indirect escalation through access to Secrets, workload creation, role bindings and certificate-signing operations. Kubernetes RBAC good practices

A deployment controller may need to update Deployments and read rollout status.

It probably does not need to read every Secret or create ClusterRoleBinding objects.

CI/CD to production authority

The GitHub OIDC trust relationship should restrict which workflow context can assume the production role.

The role itself should then permit the smallest useful deployment operation.

Deployment automation does not need to become an unrestricted production administrator.

Write precise threat statements

A useful threat statement follows this structure:

An actor

can perform an action

through a particular path

because a condition exists

causing a business or technical impact.

For example:

An attacker can replay a previously valid
payment webhook through the public endpoint
because bfstore validates the signature but
not the event identifier or timestamp, causing
duplicate state transitions and incorrect fulfilment.

This is more reviewable than:

Threat: webhook spoofing
Mitigation: encryption

Score inherent and residual risk

Priority labels need defined meaning.

CRITICAL

Mitigate, redesign or formally
accept before release.


HIGH

Assign an owner, delivery date
and tracked verification.


MEDIUM

Track and review against
planned architecture changes.


LOW

Document and reconsider when
assumptions or exposure change.

A threat record should separate untreated risk from what remains after current controls.

threat:
  id: BF-THREAT-014
  title: Replay of payment-success webhook
  stride:
    - spoofing
    - tampering

scenario:
  actor: external-attacker
  entry_point: payment-webhook
  asset: order-payment-state
  precondition: valid signed webhook captured

risk:
  inherent:
    likelihood: high
    impact: high
    priority: critical

current_controls:
  - provider signature verification

planned_controls:
  - verify signed timestamp
  - reject events outside replay window
  - enforce unique provider event ID
  - match provider payment reference
  - enforce idempotent state transition

residual_risk:
  likelihood: low
  impact: high
  priority: medium

verification:
  - replay the same signed event
  - replay an expired event
  - submit a valid event for the wrong order
  - attempt an invalid state transition

A planned control should not lower residual risk until it exists and has evidence.

A first threat register

Each record still needs an owner, assumptions, current controls, residual risk, verification evidence and review trigger.

A Critical row should cause one of four outcomes:

MITIGATE

REDESIGN

TRANSFER

FORMALLY ACCEPT

“Discussed during threat modelling” is not a fifth treatment.

Prevent, detect, respond and verify

One threat usually needs several controls.

For replayed payment events:

AWS recommends adapting controls to prevent, detect and respond rather than relying on one layer. AWS Well-Architected threat-modelling guidance

Every mitigation should produce a test or evidence requirement.

Examples include:

• use expired and wrong-audience customer tokens
• call a service using another service’s identity
• replay a correctly signed webhook
• alter a client-supplied price
• publish from an unauthorised Kafka principal
• deploy a mutable tag or unattested digest
• reconstruct a refund from application and infrastructure evidence
• scan telemetry for seeded personal data
• attempt a privileged Pod deployment
• introduce dependency latency and observe retries
• verify an unapproved branch cannot assume the production role

A model without verification can become a thoughtful list of controls that everybody assumes somebody else implemented.

Keep the model attached to delivery

A pull request introducing a new flow should answer:

What new asset exists?

Which actor can reach it?

Which boundary changed?

Which data crosses it?

What can go wrong?

Which controls and tests accompany it?

Reopen the model for changes such as:

• new public endpoints or Kafka topics
• new providers or customer-data fields
• new account connections or workload identities
• new controllers or CI/CD permissions
• changes to authentication, payments, refunds, exports or backups
• incidents that expose a false assumption

A stored model might contain:

docs/security/
├── checkout-threat-model.md
├── checkout-data-flow.txt
├── threat-register.yaml
├── assumptions.md
└── verification/
    ├── webhook-replay.md
    ├── workload-identity.md
    └── audit-evidence.md

The goal is to keep the reasoning attached to the system as it changes.

The mental model

                         BFSTORE JOURNEY
                                │
                                ▼
                      SCOPE, ASSETS, ACTORS
                                │
                                ▼
                   FLOWS, STORES AND BOUNDARIES
                                │
                                ▼
                              STRIDE
                                │
                                ▼
                         THREAT STATEMENTS
                                │
                                ▼
                 INHERENT RISK AND CURRENT CONTROLS
                                │
                                ▼
                 PREVENT, DETECT, RESPOND, VERIFY
                                │
                                ▼
                          RESIDUAL RISK
                                │
                                ▼
                     OWNER, EVIDENCE, REVIEW
                                │
                                └──────────► MODEL UPDATE

The recurring questions are:

What journey or change are we modelling?

Which assets and actors participate?

Where does trust, control,
identity or ownership change?

What can go wrong at each boundary?

Which controls prevent,
detect and contain it?

What evidence proves
those controls work?

A trust boundary is where bfstore stops accepting one promise on appearance and requires evidence before making the next decision.

That evidence may be a validated token, workload identity, authorised network path, fresh signed webhook, broker-authenticated principal, database constraint, immutable image digest, verified build provenance, narrow IAM role, protected audit record or tested rate limit.

Threat modelling does not make the system invulnerable.

It makes the security reasoning visible enough to challenge, implement, test and improve.

The attacker should not be the first person to notice where bfstore trusted the wrong thing.

References and further reading

Reviewed against official documentation on 31 July 2026.

• AWS Well-Architected: Identify threats and prioritise mitigations using a threat model
• AWS Well-Architected Security Pillar
• Microsoft Threat Modeling Tool overview
• Microsoft data-flow diagram guidance
• Kubernetes NetworkPolicy
• Kubernetes Pod Security Standards
• Kubernetes Pod Security Admission
• Kubernetes RBAC good practices
• Kubernetes Secrets
• Amazon EKS IAM roles for service accounts
• Amazon EKS Pod Identity
• AWS CloudTrail overview
• AWS CloudTrail management events
• AWS CloudTrail data events
• AWS KMS encryption context
• GitHub Actions: Configuring OIDC in AWS
• GitHub Actions artefact attestations

I place it in Kubernetes with no resource configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: basket-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: basket-service
  template:
    metadata:
      labels:
        app: basket-service
    spec:
      containers:
        - name: basket-service
          image: registry.example/basket-service:v1

The manifest is valid.

It is also silent about several important questions:

• How much CPU and memory should scheduling plan for?
• How far may the process burst?
• What happens during throttling or a memory spike?
• How many replicas can the namespace afford?
• What should the autoscaler interpret as high utilisation?
• Do the Go runtime’s CPU and memory controls match the container?

Kubernetes will run the container.

It cannot infer the service’s operating contract from the language used to build the binary.

Resource configuration is the point where application behaviour becomes a scheduling, reliability and cost agreement with the platform.

Throughout this article, bfstore is a hypothetical ecommerce platform. The examples use ordinary container-level resources and assume Go 1.25 or later. They are design examples, not universal production values.

The control map

Kubernetes resource controls answer different questions.

These controls cooperate.

They are not interchangeable.

Requests, limits, LimitRanges and ResourceQuotas

Resource requests

A request is a capacity claim used for scheduling:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"

For ordinary container-level resources, the scheduler accounts for container requests plus the documented init-container and Pod-overhead rules. A Pod is placed only when those requests fit within a node’s allocatable capacity. Kubernetes resource management

Resource limits

A limit is a runtime boundary enforced through operating-system resource controls:

resources:
  limits:
    cpu: "1"
    memory: "512Mi"

CPU and memory limits have different failure semantics.

LimitRange

A LimitRange applies admission policy to individual Pods or containers in one namespace. It can define defaults, minimums, maximums and request-to-limit ratios. Changes do not rewrite existing Pods. Kubernetes LimitRanges

ResourceQuota

A ResourceQuota limits aggregate namespace claims such as total CPU and memory requests or limits, Pod counts and persistent storage. It is admission control, not a namespace CPU throttle, shared memory cgroup or capacity reservation. Namespace quotas can exceed cluster capacity, leaving accepted workloads to contend for placement. Kubernetes ResourceQuotas

CONTAINER
request and limit
       │
       ▼
LIMIT RANGE
is this shape allowed?
       │
       ▼
RESOURCE QUOTA
is namespace budget available?
       │
       ▼
SCHEDULER
can an eligible node place it?

These controls are complementary. A container limit does not define namespace scale, a quota does not rightsize one container, and a default does not prove suitability.

Requests are capacity claims, not private hardware

Suppose a node has:

ALLOCATABLE CPU

4 CPU

It already hosts workloads requesting:

3.6 CPU

A new Pod requests:

500m CPU

The scheduler sees:

3.6 + 0.5 = 4.1 CPU

The Pod does not fit.

This remains true when existing services are idle. The scheduler is protecting the capacity claims represented by their requests.

Requests have different runtime implications for CPU and memory:

If requests are consistently too high:

• nodes appear full while resources remain idle;
• node autoscaling may add unnecessary capacity;
• deployments remain unplaced;
• namespace quotas are consumed artificially;
• cost rises.

If requests are consistently too low:

• too many Pods can be packed onto a node;
• CPU contention and memory pressure become more likely;
• eviction exposure increases for workloads using far more than requested;
• percentage-based autoscaling signals become misleading.

Requests should describe meaningful ordinary demand, not the smallest values that permit admission.

CPU and memory fail differently

CPU is compressible

A process can receive less CPU time and keep running more slowly.

When a container tries to consume more CPU than its limit permits, the kernel throttles its cgroup. The process is not normally killed merely for demanding additional CPU. Kubernetes resource management

CPU DEMAND ABOVE LIMIT
         │
         ▼
KERNEL THROTTLING
         │
         ▼
WORK CONTINUES MORE SLOWLY
         │
         ▼
LATENCY MAY INCREASE

For a Go HTTP or gRPC service, throttling can delay:

• request handling;
• garbage collection;
• goroutine scheduling;
• TLS and serialisation work;
• health checks;
• graceful shutdown;
• background consumers.

The service may remain alive while failing its latency objective.

Memory is incompressible

Memory cannot always be deferred.

Memory-limit enforcement is reactive. A container can briefly cross the configured value, but when the cgroup cannot satisfy an allocation, the kernel may OOM-kill a process in that cgroup. Kubernetes then observes the container termination. Whether it restarts depends on the Pod’s restart policy and its managing workload. Kubernetes resource management

MEMORY DEMAND
REACHES THE HARD BOUNDARY
         │
         ▼
CGROUP OOM CONDITION
         │
         ▼
PROCESS TERMINATION
         │
         ▼
POSSIBLE CONTAINER RESTART

CPU limits usually produce degraded execution. Memory limits may produce lost execution.

Both can damage availability, but they leave different evidence.

The units deserve respect

Kubernetes CPU is measured in CPU units:

1 CPU = 1000m CPU

Therefore:

250m  = 0.25 CPU
500m  = 0.5 CPU
1500m = 1.5 CPU

Memory is measured in bytes. I normally use binary units:

256Mi
512Mi
1Gi

The case matters:

memory: "400Mi"

means 400 mebibytes.

memory: "400m"

means 0.4 bytes. Kubernetes calls out this exact class of typo in its resource documentation. Kubernetes resource management

Quoting quantities is not required, but it keeps their intended units visually explicit.

A Go service has two memory boundaries

The Kubernetes memory limit is an external cgroup boundary.

The Go runtime also supports a soft memory limit through:

GOMEMLIMIT

or:

debug.SetMemoryLimit

GOMEMLIMIT includes the Go heap and other memory managed by the Go runtime, including goroutine stacks. It excludes external sources such as the binary’s mappings, memory managed by other languages and memory held by the operating system on the program’s behalf. It is not an RSS or cgroup-memory ceiling. Go runtime environment variables

KUBERNETES MEMORY LIMIT

hard cgroup boundary
a process may be killed
        │
        ▼
HEADROOM

cgo and native allocations
external memory mappings
shared libraries
kernel accounting differences
temporary overshoot and variance
        │
        ▼
GOMEMLIMIT

soft Go runtime target
GC responds more aggressively

Suppose the application container has:

limits:
  memory: "512Mi"

An initial hypothesis might be:

env:
  - name: GOMEMLIMIT
    value: "420MiB" # Illustrative; validate with measurements.

The correct margin depends on the application’s measured memory shape, including:

• Go runtime memory;
• cgo and native libraries;
• memory-mapped files;
• telemetry libraries;
• temporary allocation spikes;
• the difference between runtime and kernel accounting.

A sidecar normally has its own container memory cgroup and does not consume the application container’s 512Mi limit. It still contributes to the Pod’s scheduling total, namespace quota, node pressure and any Pod-level autoscaling metric.

Setting GOMEMLIMIT equal to the container limit leaves little protection against memory outside the runtime’s accounting.

Setting it far too low can drive near-continuous garbage collection. The Go garbage-collector guide describes this as a soft limit: the runtime attempts to respect it while preserving enough CPU for the program to make progress. Go garbage-collector guide

GOMEMLIMIT does not replace the Kubernetes limit.

The Kubernetes limit protects the node and neighbouring workloads.

GOMEMLIMIT gives the Go runtime an earlier signal.

Go 1.25 can use CPU limits for GOMAXPROCS, but not requests

Before Go 1.25, the runtime usually selected GOMAXPROCS from the logical CPUs visible to the process.

A small container on a large node could therefore attempt much more parallel Go execution than its cgroup CPU limit could sustain.

Go 1.25 changed the Linux default. The runtime considers a cgroup CPU bandwidth limit when selecting GOMAXPROCS, periodically checks for changes and rounds fractional CPU limits up to a valid integer. Kubernetes CPU limits normally provide that cgroup input. Kubernetes CPU requests do not. Manually setting GOMAXPROCS disables the automatic default and updates. Go 1.25 release notes Container-aware GOMAXPROCS

KUBERNETES CPU REQUEST

scheduler capacity claim
CPU weight under contention
HPA utilisation denominator


KUBERNETES CPU LIMIT

cgroup throughput ceiling
Go 1.25 runtime input


GOMAXPROCS

parallel Go execution limit

A CPU limit and GOMAXPROCS are not identical.

A CPU limit controls throughput over time.

GOMAXPROCS controls how many threads may execute Go code simultaneously.

The trade-off is therefore explicit:

For bfstore, I would not set GOMAXPROCS manually in every Deployment by habit.

I would choose the CPU operating model at platform level, measure throttling and tail latency, and override the runtime only when evidence supports it.

A baseline bfstore Deployment

A baseline should be an explicit, measurable hypothesis.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-service
  namespace: bfstore-prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: catalog-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: catalog-service
    spec:
      containers:
        - name: catalog-service
          image: registry.example/bfstore/catalog-service:v1.0.0

          env:
            - name: GOMEMLIMIT
              value: "420MiB" # Initial hypothesis, not a universal ratio.

          ports:
            - name: grpc
              containerPort: 8080

          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              memory: "512Mi"

          readinessProbe:
            grpc:
              port: 8080
            periodSeconds: 5

          livenessProbe:
            grpc:
              port: 8080
            periodSeconds: 10

This example deliberately omits a CPU limit.

That permits the service to use idle node CPU beyond its request and avoids hard cgroup throttling. It also means Go cannot derive GOMAXPROCS from the 250m request. On a large node, the runtime may select much greater parallelism unless CPU affinity, an explicit runtime value or another effective cgroup boundary narrows it.

A stricter environment might use:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1"
    memory: "512Mi"

That gives the runtime a cgroup CPU limit to consider and constrains prolonged consumption.

It can also create throttling.

Neither design is universally correct.

Sidecars are part of the resource shape

A Pod may contain:

• a service-mesh proxy;
• a telemetry collector;
• a certificate agent;
• a configuration reloader;
• a security agent.

Each container needs its own resource contract:

spec:
  containers:
    - name: order-service
      resources:
        requests:
          cpu: "300m"
          memory: "320Mi"
        limits:
          memory: "640Mi"

    - name: telemetry-sidecar
      resources:
        requests:
          cpu: "50m"
          memory: "64Mi"
        limits:
          cpu: "200m"
          memory: "128Mi"

For ordinary container-level resources, the Pod total includes both containers:

CPU REQUEST

300m + 50m = 350m


MEMORY REQUEST

320Mi + 64Mi = 384Mi


MEMORY LIMIT

640Mi + 128Mi = 768Mi

The sidecar also affects autoscaling when the HPA uses aggregate Pod resource utilisation.

That leads to a design decision:

Should scaling follow
the complete Pod cost?

or

Should scaling follow
the application container?

When a stable application container should drive scaling and sidecar usage is incidental, autoscaling/v2 supports a ContainerResource metric:

metrics:
  - type: ContainerResource
    containerResource:
      name: cpu
      container: catalog-service
      target:
        type: Utilization
        averageUtilization: 65

Container names then become part of the autoscaling contract. A rename must be coordinated with the HPA. Kubernetes HPA

Requests and limits create a QoS class

Kubernetes assigns each Pod a Quality of Service class.

Guaranteed

Every relevant container has:

• a CPU request;
• an equal CPU limit;
• a memory request;
• an equal memory limit.

Burstable

At least one request or limit exists, but the Pod does not meet all Guaranteed conditions.

BestEffort

No relevant container has a CPU or memory request or limit.

QoS influences node-pressure behaviour, but the common shorthand of “BestEffort, then Burstable, then Guaranteed” is not the complete eviction algorithm.

For node-pressure eviction, Kubernetes considers:

1. whether the Pod’s use of the starved resource exceeds its request;
2. Pod priority;
3. usage relative to the request.

A BestEffort Pod is especially exposed because any positive usage exceeds its zero request. A Guaranteed Pod has stronger protection, but it is not immune to eviction or to a container-level OOM. Kubernetes QoS classes Kubernetes node-pressure eviction Pod priority and eviction ranking

The baseline example is Burstable:

requests:
  cpu: "250m"
  memory: "256Mi"
limits:
  memory: "512Mi"

The gap says:

Plan for this much.

Allow this much memory.

Treat the difference
as deliberate burst space.

Using equal requests and limits to obtain Guaranteed QoS may suit workloads that need predictable placement and stronger pressure protection.

It also accounts the full limit during scheduling.

QoS should follow the workload’s operating requirements, not act as a badge.

The CPU request becomes the HPA denominator

For a percentage-based CPU target, the HorizontalPodAutoscaler compares observed CPU use with the relevant CPU request.

Suppose one Pod consumes:

200m CPU

With a request of:

250m

utilisation is:

200 / 250 = 80%

With a request of:

500m

the same process usage becomes:

200 / 500 = 40%

The application did not change.

The autoscaler’s interpretation did.

If the request is fictional, the target is fictional too.

A Pod-level CPU utilisation metric also depends on the relevant containers having requests. Missing requests can prevent the HPA from calculating utilisation normally. Kubernetes HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: catalog-service
  namespace: bfstore-prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: catalog-service

  minReplicas: 3
  maxReplicas: 12

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65

  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

CPU may not be the best demand signal.

Alternatives include:

• request rate per Pod;
• active requests;
• queue depth;
• event-processing lag;
• concurrency;
• latency;
• several metrics together.

autoscaling/v2 can evaluate several metrics and use the highest proposed replica count. Kubernetes HPA

ResourceQuota creates a namespace ceiling

A production namespace may contain:

• application services;
• workers;
• migration Jobs;
• gateways;
• telemetry components;
• old and new replicas during rollouts.

A ResourceQuota can express the namespace’s aggregate ceiling:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: bfstore-production-budget
  namespace: bfstore-prod
spec:
  hard:
    requests.cpu: "20"
    requests.memory: "40Gi"

    limits.cpu: "40"
    limits.memory: "80Gi"

    pods: "150"
    services: "40"
    persistentvolumeclaims: "30"

Kubernetes rejects a new or resized object when accepting it would exceed the applicable quota. A Deployment object itself may still be accepted while the controller fails to create all its Pods; the controller status and events expose the admission failure. Kubernetes ResourceQuotas

RESOURCE QUOTA

admission-time ceiling


NOT

namespace CPU throttling
shared memory enforcement
reserved cluster capacity
automatic rightsizing

The budget needs headroom for:

• rollout surge;
• autoscaling;
• Jobs;
• incident tooling;
• operational agents;
• controlled failover.

A quota calculated only from steady-state replicas can block the rollout intended to improve the service.

Quota is capacity governance, not capacity reservation.

LimitRange provides rails, not rightsizing

A LimitRange can reject clearly unsupported shapes and supply defaults:

apiVersion: v1
kind: LimitRange
metadata:
  name: bfstore-service-boundaries
  namespace: bfstore-prod
spec:
  limits:
    - type: Container

      min:
        cpu: "25m"
        memory: "32Mi"

      max:
        cpu: "4"
        memory: "4Gi"

      defaultRequest:
        cpu: "100m"
        memory: "128Mi"

      default:
        cpu: "1"
        memory: "512Mi"

This prevents an ordinary container from declaring a shape the namespace or node classes cannot reasonably support.

Defaults still require care.

If ten services silently inherit the same values, the platform has achieved consistency without proving accuracy.

For bfstore:

GOLDEN PATH

explicit reviewed values


LIMIT RANGE

supported individual boundaries


RESOURCE QUOTA

namespace-wide declaration ceiling

The golden path should render an explicit resource block.

The LimitRange should remain a final guardrail for third-party charts, Jobs and incomplete workloads.

Choosing the first numbers

No production service begins with perfect resource data. The first values are hypotheses.

Measure startup, steady-state and post-GC memory; credible memory peaks; background and loaded CPU; goroutine count; cgo or mapping behaviour; and telemetry overhead.

Then test realistic conditions:

• expected and peak traffic;
• large valid payloads and batches;
• slow downstream dependencies, retries and timeouts;
• database contention or Kafka bursts;
• startup and graceful shutdown.

For CPU, observe upper-percentile usage, throttling, GC CPU and latency at saturation. For memory, observe Go runtime memory, working set, RSS, post-GC floor, peaks and OOM events.

Choose requests from the capacity one replica needs during ordinary operation under its SLO, not from its lowest idle measurement.

For memory limits, keep measured headroom above credible peaks while retaining a useful node-protection boundary. For CPU, choose explicitly between a hard ceiling and request-based access to idle capacity.

MEASURE
   │
   ▼
SET AN INITIAL CONTRACT
   │
   ▼
DEPLOY AND OBSERVE
   │
   ▼
ADJUST

kubectl top provides a useful recent snapshot when Metrics Server is available, but durable evidence should come from the observability platform:

• CPU use and throttling;
• working set, RSS, OOM kills and restarts;
• Go runtime memory, GC CPU and goroutines;
• latency, request rate, queue depth or consumer lag;
• desired and admitted replica counts.

A service using little CPU because it is deadlocked is not efficient.

Reading common failures

The workload controller cannot create a Pod

Check ResourceQuota, LimitRange, other admission policies and the Deployment’s ReplicaFailure events. A Deployment can exist while its requested Pods do not.

A Pod exists but remains Pending

Check whether any eligible node can satisfy the requests, then inspect affinity, taints, tolerations, topology constraints and node-class availability. This is a scheduling failure after admission, not the same as a quota rejection.

A container is OOMKilled

Inspect which process was killed, Go runtime memory, cgo or mapping growth, transient allocations and the hard limit. Raising the limit may be correct, but it can also turn a leak into a slower incident.

Latency rises while CPU is throttled

Check the CPU limit, GC bursts, GOMAXPROCS, replica count and downstream retries. The service may be alive while users are already waiting.

HPA scales unexpectedly

Check the credibility of CPU requests, sidecar usage, missing metrics, quota headroom and whether CPU actually represents demand. Consider a ContainerResource metric when the application container should drive scaling.

A node experiences memory pressure

Check whether the Pod exceeds its request, its priority, relative overuse and the node’s reservations and eviction thresholds. QoS is useful context, not the complete eviction algorithm.

Workload profiles are starting ranges

A golden path should reduce blank-page work without pretending to know every service.

Example starting ranges might be:

The platform should attach governance to the profile:

workload:
  profile: standard-go-service
  profile_version: "2"
  review_after: "30d"

autoscaling:
  metric: cpu
  target_utilisation: 65

resource_overrides:
  requests:
    memory: "384Mi"
  limits:
    memory: "768Mi"

evidence:
  load_test: required
  owner: catalog-team

The platform can then:

• render explicit Kubernetes values;
• validate them against LimitRange;
• calculate quota and rollout headroom;
• add standard dashboards;
• require evidence for unusually large overrides;
• surface throttling and OOM behaviour after deployment.

A profile is a starting point.

It is not production truth.

The decision checklist

Before approving a resource contract, I now ask:

1. What resource shape does the workload exhibit under realistic load?
2. Which CPU and memory values must scheduling plan for?
3. Which hard limits protect the platform without breaking the SLO?
4. How do GOMEMLIMIT and GOMAXPROCS relate to those boundaries?
5. Can rollout surge and maximum autoscaling fit beneath quota?
6. Which production evidence will trigger rightsizing?

The mental model

                         GO SERVICE
                              │
                              ▼
                    OBSERVED RESOURCE SHAPE
                              │
                 ┌────────────┴────────────┐
                 ▼                         ▼
            CPU REQUEST              MEMORY REQUEST
       scheduling, weight, HPA    scheduling, eviction
                 │                         │
                 └────────────┬────────────┘
                              ▼
                 ┌────────────┴────────────┐
                 ▼                         ▼
             CPU LIMIT                MEMORY LIMIT
             throttling                 OOM wall
                 │                         │
                 ▼                         ▼
            GOMAXPROCS                 GOMEMLIMIT
                 │                         │
                 └────────────┬────────────┘
                              ▼
                 ┌────────────┴────────────┐
                 ▼                         ▼
             LIMITRANGE              RESOURCEQUOTA
        individual guardrail        namespace ceiling
                              │
                              ▼
                      CLUSTER CAPACITY

Requests make placement and autoscaling more honest. Limits define selected runtime boundaries. GOMEMLIMIT gives the garbage collector an earlier signal, while Go 1.25 can align default parallelism with a CPU limit. LimitRange constrains individual shapes, and ResourceQuota constrains the namespace total.

Observability decides whether any of those assumptions were true.

The numbers will change. That is expected.

The failure would be choosing them once, copying them into every service and calling the resulting uniformity a platform.

References and further reading

• Kubernetes resource management for Pods and containers
• Kubernetes ResourceQuotas
• Kubernetes LimitRanges
• Kubernetes Pod Quality of Service classes
• Kubernetes node-pressure eviction
• Kubernetes Pod priority and eviction ranking
• Kubernetes Horizontal Pod Autoscaling
• Go 1.25 release notes
• Container-aware GOMAXPROCS
• Go runtime environment variables
• Go garbage-collector guide

The customer profile is deleted from MySQL.

The job reports:

customer_deleted = true

The application has removed one row.

The customer may still exist in:

• order records
• shipping events
• payment-provider references
• Kafka topics
• search indexes
• application logs and traces
• support exports
• database snapshots
• cross-account backups
• disaster-recovery copies
• a CSV downloaded during an incident

Deleting the live row is not the same as eliminating every recoverable representation of the data.

Encryption adds another possible control. If the remaining copies are ciphertext and every key path required to decrypt them is permanently removed, the ciphertext may remain physically present while the protected plaintext becomes infeasible to recover.

That is the idea behind cryptographic erase.

NIST SP 800-88 Revision 2 describes cryptographic erase as a purge technique based on sanitising the keys that encrypt data, or keys that prevent access to those encryption keys. It also places strict conditions on the claim: sensitive data must not previously have been stored in plaintext on the affected storage, all relevant key copies must be sanitised, and backed-up, escrowed or unwrapped keys must be handled separately.¹

That is much stricter than:

We deleted the KMS alias.

or:

We turned encryption on.

Cryptographic erasure is not a button. It is an evidence-backed claim that no usable route from the remaining ciphertext to its plaintext still exists.

AWS KMS can control an important part of that route. The architecture must still know where the data, derived copies and relevant keys live.

Throughout this article, bfstore is a hypothetical AWS-hosted ecommerce platform used to make the design choices concrete. The intended audience is application, platform and security engineers designing access, recovery and deletion boundaries. This is not legal advice or a complete KMS operations manual.

AWS service behaviour in this article was checked against official documentation on 31 July 2026. Service state transitions and permission requirements should be confirmed again when a production runbook is used.

The decision in one table

The recurring question is not simply:

Is this encrypted?

It is:

Which data must be accessible,
recoverable and destroyed together?

Define the erasure claim before choosing the mechanism

“No usable route remains” needs a scope.

For bfstore, the technical claim should cover:

• the identified ciphertext;
• every retained plaintext or derived representation;
• KMS keys and application data keys controlled by bfstore;
• wrapped-key copies in databases, events, exports and backups;
• grants, replicas and external key stores that can restore access;
• plaintext keys still held in application or service memory.

It does not prove that plaintext was never copied or exfiltrated before the erasure operation. It does not protect against a future failure of the cryptographic algorithm. It does not automatically establish compliance with every legal or contractual deletion requirement.

The evidence should therefore separate three claims:

TECHNICAL CLAIM

The identified ciphertext cannot be
decrypted through any retained key path.


DATA-LIFECYCLE CLAIM

Live, replicated, exported and backup
representations were handled under
the approved retention policy.


COMPLIANCE CLAIM

The organisation reviewed whether
that process satisfies the applicable
legal and contractual obligations.

Keeping those claims separate makes each one more credible.

Encryption at rest is only one stage of the lifecycle

AWS services provide broad encryption-at-rest support. For example, Amazon S3 encrypts new objects by default, RDS encryption covers database storage, logs, automated backups and snapshots, and DynamoDB encrypts table data, indexes, streams and backups.²³⁴

That protection matters. Someone who obtains encrypted storage should not automatically obtain readable data.

But encrypted does not answer:

• Who may request decryption?
• Which workloads share the key?
• Which backup has another key?
• What happens during recovery?
• Which records become unreadable if the key is deleted?
• Does the same plaintext exist elsewhere?
• Is the data being retained longer than necessary?

The UK Information Commissioner’s Office says organisations should establish retention periods, erase or anonymise data that is no longer needed, and address backup data rather than treating deletion from the live system as the whole process.⁵

A useful sequence is:

DATA CLASSIFICATION
        │
        ▼
RETENTION RULE
        │
        ▼
ENCRYPTION BOUNDARY
        │
        ▼
RECOVERY DESIGN
        │
        ▼
ERASURE DESIGN

Encryption should be designed with the final stage already visible.

KMS usually protects the keys that encrypt the data

Large application payloads are generally protected through envelope encryption.

PLAINTEXT DATA
      │
      ▼
DATA KEY
      │
      ▼
CIPHERTEXT DATA


DATA KEY
      │
      ▼
KMS KEY
      │
      ▼
ENCRYPTED DATA KEY

AWS KMS can generate a plaintext data key for immediate use and an encrypted copy that can be stored with the encrypted data.⁶ The application or AWS service then stores:

ciphertext

+

encrypted data key

When the data is read:

1. the encrypted data key is sent to KMS;
2. KMS authorises the operation;
3. KMS returns the plaintext data key;
4. the data key decrypts the application data;
5. the plaintext key should be removed from memory as soon as possible.

KMS does not retain an inventory of every data key, row, object, export or backup protected through that process. It also does not store the application ciphertext.⁷

KMS CAN RECORD

the KMS key
the cryptographic request
the caller
the policy and encryption context


KMS DOES NOT INVENTORY

every database row
every S3 object version
every backup
every exported encrypted data key
every plaintext copy

Deleting a KMS key can make the data keys wrapped by it unusable. It cannot prove that another data-key copy or plaintext representation does not exist.

Key topology becomes deletion topology

Suppose every production resource uses:

alias/bfstore-prod

It protects:

• customer and order databases;
• application logs;
• Kafka storage;
• snapshots;
• backup vaults;
• sensitive exports.

The policy inventory is small. The destruction boundary is enormous.

DELETE ONE KEY
      │
      ├── customer records unreadable
      ├── orders unreadable
      ├── logs unreadable
      ├── backups unreadable
      └── operational recovery lost

The key cannot selectively erase one customer. It also cannot retire one service without affecting unrelated workloads.

At the other extreme, one KMS key per row would create a large key inventory, difficult policies, recovery complexity and avoidable cost.

The useful boundary usually lies between those extremes.

A KMS key should protect data that shares an access policy, recovery policy and destruction lifecycle.

For bfstore, a first layout might be:

alias/bfstore-prod-customer-data
alias/bfstore-prod-order-data
alias/bfstore-prod-payment-tokens
alias/bfstore-prod-application-logs
alias/bfstore-prod-sensitive-exports
alias/bfstore-prod-backups

This is not a rule that every resource needs a unique key. It is a warning against placing unrelated destruction and recovery requirements under one cryptographic roof.

Ecommerce data does not have one lifecycle

The data model should let these lifecycles diverge.

An order may need to survive account closure, but it may not need to keep the customer’s email and full shipping address for the same period. Public catalogue data may still be encrypted, but key deletion is unlikely to be its main end-of-life mechanism.

Record-level crypto-shredding needs a dedicated envelope

Deleting a database-level KMS key affects the encrypted resource, not one customer.

CUSTOMER DATABASE
        │
        ▼
alias/bfstore-prod-customer-data

For narrower control, bfstore can selectively apply application-level envelope encryption to high-risk fields or records.

customer_id: customer-8042

encrypted_profile:
  ciphertext: AQICAH...
  encrypted_data_key: AQIDAH...
  algorithm: AES-256-GCM
  schema_version: 2

The sensitive profile is encrypted with a data-encryption key. The wrapped copy of that data key is stored with the ciphertext.

At customer deletion, bfstore may remove every recoverable copy of:

• the wrapped data key;
• plaintext projections and indexes;
• duplicate envelopes in events or exports;
• cached plaintext keys;
• the ciphertext itself, where the retention policy requires deletion.

For shared KMS keys, this narrower pattern is often called record-level crypto-shredding. NIST’s guidance supports sanitising wrapped versions of data-encryption keys as a way to prevent decryption, but the assurance conditions still apply: every wrapped or unwrapped copy must be addressed.¹

Deleting one wrapped-key row from the live database is not enough when the same envelope survives in a snapshot, Kafka event, support export or replicated store.

Also keep two actions distinct:

DATA DELETION

remove the ciphertext or plaintext record


CRYPTOGRAPHIC ERASURE

eliminate every usable decryption-key path

A robust customer-erasure workflow may use both. They are not the same operation.

Application envelope encryption should be selective. Good candidates include saved addresses, high-risk profile attributes, sensitive support attachments and confidential exports. Encrypting every ordinary column independently without an access or deletion reason creates complexity rather than useful control.

Payment data: minimise before encrypting

Encryption is not permission to collect every payment field.

PCI DSS requires retention to be limited to legal, regulatory and business needs. Card verification codes or values cannot be retained after authorisation, even if encrypted. Encryption of stored cardholder data also does not, by itself, remove the environment from PCI DSS scope.⁸⁹¹⁰

A preferred application record looks like:

payment:
  provider: example-processor
  provider_customer_id: cus_7f42
  provider_payment_method_id: pm_91a8
  payment_intent_id: pi_4431
  card_brand: visa
  card_last_four: "4242"
  status: authorised

Not:

payment:
  pan: "4242424242424242"
  expiry: "12/30"
  card_verification_value: "123"

The safer sequence is:

DO NOT COLLECT UNNECESSARY DATA
            │
            ▼
USE PROVIDER TOKENISATION
            │
            ▼
ENCRYPT RETAINED REFERENCES

Provider tokens and transaction identifiers still need protection. They may permit refunds, account actions or valuable operations in the wrong application context.

Encryption context binds purpose, not identity

AWS KMS supports an encryption context for symmetric cryptographic operations. It is non-secret additional authenticated data that is cryptographically bound to the ciphertext. The same exact, case-sensitive context must be supplied for decryption. Context values may also be used in policy conditions and appear in plaintext in CloudTrail.¹¹

Suitable:

{
  "service": "customer",
  "record-type": "shipping-address",
  "environment": "production"
}

Unsuitable:

{
  "customer-email": "mariam@example.com",
  "shipping-address": "42 Example Road"
}

A policy fragment might require the expected service and environment:

{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111122223333:role/bfstore-customer-runtime"
  },
  "Action": [
    "kms:Decrypt",
    "kms:GenerateDataKey"
  ],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "kms:EncryptionContext:service": "customer",
      "kms:EncryptionContext:environment": "production"
    }
  }
}

This is an illustrative key-policy fragment, not a complete production policy. A full design may also need account and service constraints, grant controls, organisation policies and protection against confused-deputy paths.

Encryption context improves integrity, audit evidence and purpose restriction. It does not create a new destruction boundary. One KMS key may still wrap many data keys carrying different contexts.

Key policy is part of the data boundary

Every KMS key has one key policy, and AWS describes key policies as the primary mechanism for controlling access. IAM policies, grants and VPC endpoint policies may also participate.¹²

Separate key administration from ordinary data use:

KEY ADMINISTRATOR

may:
    update key policy
    manage rotation
    disable key
    schedule deletion

does not automatically need:
    decrypt customer data


APPLICATION ROLE

may:
    generate data keys
    decrypt approved envelopes

must not:
    change key policy
    disable the key
    schedule deletion

A backup service may need another constrained path. The exact permissions depend on the service integration.

The ability to decide whether data may ever be decrypted again is more powerful than permission to decrypt it today. kms:ScheduleKeyDeletion should be rare, reviewed and monitored.

What does not establish erasure

Deleting an alias

Aliases are mutable names for keys. Removing an alias does not delete the underlying KMS key or its material.

Inventory and evidence should use the stable key ARN and key ID where exact identity matters.

Rotation

KMS rotation changes the key material used for new encryption operations. KMS retains the older material required to decrypt existing ciphertext.¹³

BEFORE ROTATION

ciphertext A → material version 1


AFTER ROTATION

ciphertext A → still decryptable with version 1
ciphertext B → encrypted with version 2

Rotation does not re-encrypt existing application data, rotate stored data keys or erase old ciphertext.

Temporary disablement

Disabling a customer managed key makes KMS cryptographic operations fail, but it is reversible. It is useful for incident containment and dependency testing.

The impact on workloads may be delayed. Services or applications that already hold plaintext data keys can continue until they need KMS again.¹⁴

Disablement is therefore a rehearsal, not destruction.

Scheduling deletion without waiting

AWS KMS requires a deletion waiting period of 7 to 30 days, with 30 days as the default. During PendingDeletion, the key cannot perform cryptographic operations, but deletion can still be cancelled. Once the waiting period completes, the KMS key, key material and metadata are irreversibly deleted.⁷

A rehearsal must run long enough to exercise actual key reuse, service caches and normal recovery paths.

Recovery deliberately complicates erasure

A backup exists so that data survives loss of the primary copy. Cryptographic erasure exists so that data cannot survive loss of its required key. Those goals can conflict.

If production and backup use the same key:

PRIMARY ── key A
BACKUP  ── key A

Deleting key A may remove both the workload and its recovery path.

If the backup copy uses an independent key:

PRIMARY             ── production key
CROSS-ACCOUNT COPY  ── backup-account key

Deleting the production key does not erase the backup. That improves recovery isolation, but final erasure must also address the recovery point, destination key, retention schedule and any locked vault policy.

AWS Backup generally encrypts cross-account and cross-Region copies using the target vault’s KMS key, with resource-specific caveats.¹⁵

This is not a flaw. It is the expected result of separating recovery credentials from production credentials.

PRODUCTION KEY COMPROMISED

preserve independent recovery


DATASET RETENTION EXPIRES

remove live data and every retained copy


CUSTOMER ERASURE REQUEST

handle that customer's fields under
the approved retention basis

One KMS-key deletion cannot safely represent all three events.

Service behaviour is not uniform

The architectural principle is stable: a restore requires usable data and a usable key. The exact failure states are service-specific.

A backup encrypted under a deleted key is not an escape hatch. Recovery needs a key path too.

Multi-Region and imported keys widen the boundary

Related KMS multi-Region keys share key material and can decrypt one another’s ciphertext across Regions. Deleting one relative does not establish erasure while another remains usable. AWS KMS keeps a primary key in PendingReplicaDeletion until all replicas have been deleted, after which the ordinary deletion waiting period begins.¹⁸

Inventory:

• the primary;
• every replica;
• Region-specific policies and grants;
• disaster-recovery resources;
• replicated and backup data.

Imported key material adds an external copy by design. AWS recommends retaining the material outside AWS so it can be reimported if the KMS copy expires or is deleted.¹⁹

KMS COPY

deleted


EXTERNAL HSM COPY

still exists

That is temporary withdrawal, not permanent erasure. The erasure process must include external HSM backups, escrow and runbooks capable of reimporting the material.

Deleting imported material from a KMS key can be reversed by reimporting the same material. Deleting the KMS key after its waiting period is different and irreversible.²⁰⁷

Plaintext keys and values can outlive a KMS request

Envelope encryption puts plaintext data keys outside KMS while data is being encrypted or decrypted. They may exist in:

• application memory;
• service caches;
• cryptographic-library state;
• hardware registers;
• crash dumps;
• diagnostic captures.

NIST specifically requires assurance that unwrapped key copies in volatile memory or encryption engines are eliminated as part of a cryptographic-erase operation.¹

Application code should:

• use reviewed encryption libraries;
• minimise plaintext-key lifetime;
• never log key material;
• avoid support workflows that collect unrestricted memory dumps;
• clear mutable buffers where the language and library make that meaningful;
• restart or drain workloads when a high-assurance erasure requires cached keys to disappear.

Go does not offer a universal guarantee that overwriting one variable removes every compiler-created or runtime copy. That is another reason not to build custom cryptographic protocols in business code.

KMS does not know every dependency

Before deleting a KMS key, AWS recommends determining its past usage. KMS does not maintain a list of all ciphertexts protected by the key, and local use of a previously generated data key is not visible as a new KMS operation.⁷²¹

Evidence must combine:

• CloudTrail;
• infrastructure-as-code state;
• AWS Config or resource inventory;
• RDS, S3, DynamoDB and AWS Backup metadata;
• application envelope registries;
• grants and cross-account access;
• disaster-recovery inventories;
• retention and deletion records.

A key record could contain:

key:
  alias: alias/bfstore-prod-customer-data
  arn: arn:aws:kms:eu-west-2:111122223333:key/example
  region: eu-west-2
  type: symmetric
  regionality: single-region
  material_origin: AWS_KMS

ownership:
  data_owner: customer-service
  key_administrator: security-platform
  backup_owner: bfstore-backup

data:
  classification: customer-personal-data

resources:
  primary:
    - rds-customer-prod
    - customer-export-bucket
  backups:
    - customer-rds-snapshot-policy
    - bfstore-backup-account-copy

deletion:
  waiting_period_days: 30
  disable_rehearsal_required: true
  restore_test_required: true
  approvers:
    - data-owner
    - security
    - backup-owner

No key should reach deletion with its purpose described as:

probably old

Key deletion needs a two-person-shaped process

A safe production process might be:

1. The data owner confirms that the retention basis has ended.
2. The resource owner inventories every dependent resource and application envelope.
3. The backup owner confirms every recovery point and destination key.
4. Security reviews policy, grant and usage evidence.
5. The key is disabled.
6. Workloads are observed through a meaningful rehearsal period.
7. Restore tests confirm which recovery paths remain.
8. A separate privileged role schedules deletion.
9. Attempted use during the waiting period produces alarms.
10. Unexpected dependencies trigger cancellation.
11. Final deletion evidence is retained.

RETENTION APPROVAL
        │
        ▼
DEPENDENCY INVENTORY
        │
        ▼
DISABLE AND OBSERVE
        │
        ▼
RESTORE TEST
        │
        ▼
SCHEDULE DELETION
        │
        ▼
ALARMED WAITING PERIOD
        │
        ├── unexpected use → cancel
        └── no dependency  → continue
        │
        ▼
FINAL DELETION
        │
        ▼
ERASURE EVIDENCE

AWS records CloudTrail events when deletion is scheduled and when the key is deleted. Policy can also constrain the requested waiting period using kms:ScheduleKeyDeletionPendingWindowInDays.⁷²²

The waiting period is the last recovery window before an irreversible action.

Customer erasure starts with a data map, not a KMS command

CUSTOMER ID
    │
    ├── customer service
    ├── order service
    ├── payment references
    ├── shipping service
    ├── notification service
    ├── Kafka and search
    ├── logs and traces
    ├── analytics
    └── backups

Each representation then receives one of four decisions.

Delete now

Examples:

• live customer profile;
• saved addresses;
• marketing preferences;
• inactive sessions;
• unnecessary search projections.

Transform

An order may initially contain:

order:
  order_id: order-8042
  customer_id: customer-421
  customer_email: mariam@example.com
  shipping_name: Mariam Example
  shipping_address: 42 Example Road
  items:
    - product_id: product-11
      quantity: 2
  total_pence: 8900

A later retained form might become:

order:
  order_id: order-8042
  customer_reference: deleted-customer
  items:
    - product_id: product-11
      quantity: 2
  total_pence: 8900
  fulfilment_region: GB
  personal_data_removed_at: 2026-06-22

Whether this is anonymous or merely pseudonymous depends on the remaining data and bfstore’s ability to reconnect it with other records. The architecture should not label data anonymous without assessing re-identification.

Retain under an approved basis

Examples may include tax records, chargeback evidence, fraud investigation or legal claims. This is a legal and business decision, not one KMS can make.

Put beyond ordinary use until backup expiry

The live system no longer exposes the data. Backup procedures prevent it from being used for another purpose and allow it to expire under an established schedule. The ICO describes this “beyond use” approach for backup data that cannot be overwritten immediately, while stressing that the organisation must be clear about the process.²³

Where a dedicated field or record envelope exists, bfstore can also destroy every wrapped data-key copy and remove the corresponding plaintext projections. That does not normally require deleting the shared service KMS key.

Restores must replay erasure decisions

Suppose a customer is erased on 22 June.

A backup from 15 June still contains the profile.

In August, an incident restores that backup.

15 JUNE BACKUP
      │
      ▼
22 JUNE CUSTOMER ERASED
      │
      ▼
10 AUGUST RESTORE
      │
      ▼
CUSTOMER DATA REAPPEARS

The restore process needs an erasure ledger, or an equivalent durable reconciliation mechanism, containing deletion and retention changes made after the recovery point.

Before customer traffic resumes:

1. restore the backup;
2. replay ordinary database events;
3. replay the erasure ledger;
4. rebuild search, cache and analytics projections;
5. verify that removed data has not returned;
6. record the reconciliation evidence.

RESTORE BACKUP
      │
      ▼
REPLAY APPLICATION CHANGES
      │
      ▼
REPLAY ERASURE LEDGER
      │
      ▼
REBUILD PROJECTIONS
      │
      ▼
VERIFY
      │
      ▼
RETURN TO SERVICE

A restore that resurrects data the business committed to removing is technically successful and operationally wrong.

Backup tests should therefore verify privacy and retention behaviour, not only whether MySQL starts.

Proposed bfstore approach

For bfstore, I would use the following model:

• Use AWS-service encryption for production storage, logs, event infrastructure and backups.
• Use customer managed KMS keys where bfstore needs control over policy, audit, disablement or deletion.
• Separate keys by environment, service or data owner, classification and recovery boundary.
• Keep payment-account data outside bfstore through hosted collection and provider tokenisation.
• Apply application envelope encryption selectively to data with a clear access or erasure requirement.
• Keep encryption context non-sensitive and purpose-oriented.
• Give production backups independent keys where recovery isolation requires it, while documenting their separate erasure lifecycle.
• Treat key deletion as a production change requiring inventory, rehearsal, restore testing, multiple approvals and retained evidence.
• Make erasure-ledger replay a mandatory restore gate.

The final mental model is:

DATA MINIMISATION
        │
        ▼
RETENTION DECISION
        │
        ▼
ACCESS AND ENCRYPTION BOUNDARY
        │
        ▼
RECOVERY COPIES AND KEYS
        │
        ▼
END-OF-LIFE ACTION
        │
        ├── delete
        ├── transform
        ├── retain under an approved basis
        └── cryptographically erase
        │
        ▼
VERIFIABLE EVIDENCE

A KMS key is not a padlock attached to one database row.

It is an authority, recovery and blast-radius boundary with cryptography inside it.

The key hierarchy should reflect not only who may read data, but which data must live, recover and die together.

The ciphertext may remain.

The route home should not.

References