Cloud Sandbox
All notes

AmazonProvidedDNS: the resolver already hiding inside the VPC

Every Amazon VPC already contains access to Route 53 VPC Resolver through AmazonProvidedDNS. Understanding its addresses, DHCP configuration, VPC attributes, private hosted zones, forwarding boundaries and query evidence makes AWS DNS architecture much easier to reason about.

about 14 minutes min read

I create a VPC.

I create three private subnets.

I launch a workload.

The workload resolves:

sts.eu-west-2.amazonaws.com

catalog.internal.bfstore.example

ip-10-80-20-34.eu-west-2.compute.internal

I did not create a DNS server.

There is no EC2 instance called bfstore-dns-01, no resolver appliance in a subnet and no DNS interface I manage.

The resolver was already there.

AWS calls the built-in service Route 53 VPC Resolver. It is also described as:

  • the Amazon DNS server;
  • AmazonProvidedDNS;
  • Route 53 Resolver in older documentation and architecture material.

The name changed when AWS introduced Route 53 Global Resolver. The products should not be confused.

ROUTE 53 VPC RESOLVER

built into every VPC
reachable through link-local
and VPC-plus-two addresses


ROUTE 53 GLOBAL RESOLVER

internet-reachable anycast resolver
for authorised remote, branch
and on-premises clients

This article covers the VPC service.

Resources reach it through:

169.254.169.253

fd00:ec2::253

primary VPC IPv4 CIDR plus two

A VPC using:

10.80.0.0/16

can therefore reach it at:

10.80.0.2

BFSTORE WORKLOAD
       │
       │ DNS query
       ▼
Route 53 VPC Resolver
       │
       ├── AWS-provided private names
       ├── Route 53 private hosted zones
       └── public recursive DNS

The resolver does not need to be added to the VPC architecture. It needs to be understood, configured and observed because it was part of the architecture from the beginning.

That small shift cleared up several of my earlier DNS questions.

How a VPC resource reaches the Resolver

The .2 address is not a little DNS appliance

Suppose bfstore’s production VPC uses:

10.80.0.0/16

Its familiar Resolver address is:

10.80.0.2

It is tempting to imagine one resolver living at that address inside the first subnet.

That mental model invites questions about its Availability Zone, route, interface and redundancy.

The address is a VPC-provided access point to the managed Resolver service.

AWS describes two highly available components behind it:

EC2 OR WORKLOAD
      │
      ▼
NITRO RESOLVER
      │
      ├── local cache
      │
      ▼
ZONAL RESOLVER FLEET
      │
      ▼
AUTHORITATIVE OR RECURSIVE ANSWER

Repeated queries may be answered from a local cache. Uncached queries pass to a highly available resolver fleet, normally in the same Availability Zone.

AWS transports these link-local queries privately outside the customer-visible VPC data path.

The VPC+2 address is derived from the VPC’s primary IPv4 CIDR.

Adding a secondary CIDR does not create another Resolver address for that block.

DHCP tells the client which resolver to use

AmazonProvidedDNS is a special DHCP option value.

It normally tells compatible VPC resources to use Route 53 VPC Resolver. It does not create the managed resolver.

The default DHCP option set includes:

domain-name-servers = AmazonProvidedDNS

Each VPC can have only one DHCP option set associated with it at a time.

A custom option set can instead specify customer-managed DNS server addresses. AWS allows custom servers or AmazonProvidedDNS, but warns that mixing them can produce unexpected behaviour.

For bfstore’s ordinary VPCs:

dhcp_options:
  domain_name_servers:
    - AmazonProvidedDNS

If no DHCP option set is associated, Nitro-based EC2 instances still receive:

169.254.169.253

as their DNS server. Older Xen-based instances do not receive the same fallback.

In an IPv6-only subnet, the IPv4 link-local Resolver address remains reachable when AmazonProvidedDNS is present in the DHCP option set.

A custom option set matters when the VPC should query Active Directory DNS, an enterprise filtering resolver or another internal DNS platform first.

Where that custom resolver runs matters.

CUSTOM DNS INSIDE THE VPC

may conditionally forward
AWS-private namespaces
to the VPC-plus-two address


DNS OUTSIDE THE VPC

must use an inbound endpoint
or another supported resolver service

An on-premises resolver, or a resolver in another VPC, should not forward private queries directly to a VPC’s CIDR-plus-two address.

Two VPC attributes control different behaviour

The VPC has two DNS attributes whose names are similar enough to invite configuration fog.

enableDnsSupport

This controls whether DNS resolution through the Amazon-provided resolver is supported for the VPC.

enableDnsHostnames

This controls AWS hostname behaviour, including public DNS hostnames for instances with public IPv4 addresses when the required conditions are met.

For private hosted zones and private DNS names associated with interface VPC endpoints, both attributes should be enabled.

For bfstore:

vpc_dns:
  enable_dns_support: true
  enable_dns_hostnames: true

enableDnsSupport defaults to true.

enableDnsHostnames defaults to false for a non-default VPC.

The attributes enable Resolver behaviour. They do not create bfstore’s private records.

How VPC Resolver chooses where to answer

A Route 53 private hosted zone stores records for one or more associated VPCs.

Suppose bfstore creates:

internal.bfstore.example

with records such as:

otel.internal.bfstore.example

modules.internal.bfstore.example

orders.internal.bfstore.example

A workload in an associated VPC sends its query to Route 53 VPC Resolver.

The client does not query the private zone’s displayed name servers over the public internet. Resolver uses the VPC association and private AWS connectivity to obtain the answer.

The simplified decision path is:

QUERY
  │
  ▼
MATCHING RESOLVER RULE?
  │
  ├── yes → follow the most-specific rule
  │
  ▼
MOST-SPECIFIC PRIVATE HOSTED ZONE?
  │
  ├── yes → search that zone
  │           │
  │           ├── record present → answer
  │           └── record absent  → NXDOMAIN
  │
  ▼
PUBLIC RECURSION

A forwarding rule for the same domain takes precedence over a private hosted zone.

Where overlapping private hosted zones exist, Resolver uses the most-specific matching zone.

For example, a query for api.accounting.example.com uses accounting.example.com rather than the broader example.com zone.

If that selected zone does not contain the requested record and type, Resolver returns:

NXDOMAIN

It does not retry against the broader private zone or the public namespace.

This is how split-view DNS can appear to break one public hostname while other public names still resolve. The private namespace has taken authority for that branch.

Private subdomains can also be delegated deliberately through NS records.

A private answer proves only that DNS returned an address. Routes, security controls, endpoint policies and the destination service still determine whether it can be used.

Resolver endpoints extend the boundary

Route 53 VPC Resolver already resolves:

  • public DNS names;
  • AWS-provided VPC names;
  • associated private hosted zones.

Resolver endpoints solve a different problem.

Inbound endpoint

An inbound endpoint allows a resolver on another connected network to ask VPC Resolver for AWS-private answers.

EXTERNAL RESOLVER
       │
       ▼
INBOUND ENDPOINT
       │
       ▼
VPC RESOLVER

Use it when on-premises DNS or another cloud must query AWS-private namespaces.

External DNS systems should not forward directly to a VPC’s CIDR-plus-two address.

Outbound endpoint

An outbound endpoint allows VPC Resolver to send selected namespaces to DNS servers elsewhere.

VPC RESOLVER
       │
       ▼
OUTBOUND RULE
       │
       ▼
OUTBOUND ENDPOINT
       │
       ▼
EXTERNAL DNS AUTHORITY

For example:

corp.example
    → enterprise DNS

legacy.internal
    → custom resolver

bfstore currently has no on-premises network.

It does not need Resolver endpoints merely because it uses private hosted zones.

The endpoints matter when a real external DNS boundary appears. No ceremonial resolver furniture is required.

Route 53 Global Resolver is a separate option for authorised remote, branch and on-premises clients that need internet-reachable anycast resolution of public and associated private domains. It does not replace the VPC’s built-in Resolver or make private-network endpoints unnecessary for every hybrid design.

Operating and observing the Resolver

Security groups cannot block it

Traffic to and from the Amazon DNS server cannot be filtered with security groups or network ACLs.

This rule does not create complete DNS control:

Application security group:

allow UDP 53 to approved resolver
deny everything else

The workload can still reach Route 53 VPC Resolver. Domain policy for queries handled there belongs in Resolver DNS Firewall.

DNS Firewall can allow, block or alert on domains using managed and custom lists.

Its default failure mode is fail closed.

If DNS Firewall does not return a decision, VPC Resolver blocks the query and returns:

SERVFAIL

A VPC can instead be configured to fail open where availability is preferred over enforcement during a firewall impairment.

DNS Firewall sees only queries that pass through VPC Resolver.

A workload that sends a query directly to another reachable DNS server does not automatically pass through those rules.

It also filters names, not the resulting application traffic.

Blocking a lookup does not prevent:

  • direct-IP connections;
  • HTTPS to an already known address;
  • SSH;
  • another protocol reaching the destination.

A complete egress design may also require routing controls, security groups, Network Firewall, a proxy or application-level policy.

DNS is one path towards a destination, not the destination itself.

Query logs reveal Resolver activity

Resolver query logging can record:

  • queries originating from selected VPCs;
  • queries entering through inbound endpoints;
  • recursive queries using outbound endpoints;
  • DNS Firewall decisions.

Records can include:

  • VPC ID;
  • source address and resource identity where available;
  • queried name;
  • record type;
  • response code;
  • response data;
  • DNS Firewall action.

Logs can be sent to:

  • CloudWatch Logs;
  • Amazon S3;
  • Amazon Data Firehose.

For high-volume environments, S3 may be the more durable destination.

Resolver logs queries it processes rather than answers from its cache.

ONE LOGGED QUERY

does not necessarily mean

ONE APPLICATION LOOKUP

The log is not a count of every request made by applications, operating systems or CoreDNS.

The hidden resolver still has a quota

Route 53 VPC Resolver is managed and highly available.

It is not an infinite packet funnel.

AWS provides a non-adjustable allowance of:

1,024 packets per second
per network interface

for link-local services.

The aggregate includes:

  • VPC Resolver DNS queries;
  • Instance Metadata Service;
  • Amazon Time Service;
  • Windows licensing traffic.

Packets beyond the allowance are rejected.

The practical diagnostic is linklocal_allowance_exceeded, exposed through ethtool and publishable through the CloudWatch agent.

This matters for workloads that disable local caching, resolve names for every request, create many short-lived processes or repeatedly query nonexistent names.

Kubernetes commonly adds CoreDNS:

application
    │
    ▼
CoreDNS
    │
    ▼
Route 53 VPC Resolver
    │
    ▼
private or public authority

When DNS throttling appears in EKS, useful responses may include scaling CoreDNS or adopting NodeLocal DNSCache.

A DNS incident can therefore exist in the application resolver, CoreDNS, VPC Resolver, a private hosted zone, a forwarding rule or the authoritative DNS system.

The phrase:

DNS is broken

contains several separate systems, even when one is intentionally invisible.

The bfstore starting design

For bfstore’s initial VPCs, I would use:

dns:
  vpc_attributes:
    enable_dns_support: true
    enable_dns_hostnames: true

  dhcp:
    domain_name_servers:
      - AmazonProvidedDNS

  private_hosted_zones:
    - internal.bfstore.example

  resolver_endpoints:
    inbound: none
    outbound: none

  query_logging:
    enabled: true
    destination: central-log-archive

  dns_firewall:
    phase: later-controlled-egress
    failure_mode: fail-closed

The private hosted-zone structure should follow real ownership boundaries rather than creating one giant internal namespace full of unrelated records.

Possible zones might include:

platform.internal.bfstore.example

prod.internal.bfstore.example

nonprod.internal.bfstore.example

The final names should align with VPC associations, account ownership, environment isolation, disaster recovery and future Azure or GCP naming.

Resolver endpoints should remain absent until bfstore has an actual forwarding or delegation requirement.

Transit Gateway connectivity alone does not solve private DNS naming across VPCs and external environments.

Routing answers whether an address is reachable.

Resolver configuration answers how the client discovers it.

The mental model I am keeping

My earlier model was:

VPC DNS

something Route 53 does
after I configure it

The stronger model is:

                       VPC WORKLOAD
                             │
                             ▼
                    OPERATING-SYSTEM DNS
                             │
                             ▼
                  Route 53 VPC Resolver
                             │
            ┌────────────────┼────────────────┐
            │                │                │
            ▼                ▼                ▼
      AWS PRIVATE       PRIVATE HOSTED      PUBLIC
        NAMES               ZONES           RECURSION
                             │
                             ▼
                    OPTIONAL EXTENSIONS
                             │
              ┌──────────────┴──────────────┐
              │                             │
              ▼                             ▼
      INBOUND ENDPOINT              OUTBOUND ENDPOINT
              │                             │
              ▼                             ▼
    QUERIES INTO AWS             QUERIES TO EXTERNAL DNS

The DHCP option answers:

Which resolver should
the resource use?

enableDnsSupport answers:

May the VPC use
the Amazon-provided resolver?

enableDnsHostnames answers:

Which AWS hostname behaviour
is enabled in this VPC?

A Resolver rule answers:

Which namespace should follow
a custom forwarding path?

A private hosted-zone association answers:

Which private namespace
is authoritative here?

An inbound endpoint answers:

How may an external resolver
query AWS-private names?

An outbound endpoint answers:

Which names should AWS send
to another DNS authority?

Query logging answers:

Which DNS questions did
Resolver process?

DNS Firewall answers:

Which Resolver-handled queries
should be permitted?

And the link-local metric answers:

Is one network interface
overwhelming the hidden path?

AmazonProvidedDNS was easy to overlook because it did not ask me to deploy anything. That is precisely why it deserves a place in the architecture.

The resolver is hidden operationally, not architecturally.

It helps workloads discover public AWS APIs, answers private hosted zones, supports interface endpoint private DNS, sits beneath Kubernetes DNS and becomes one side of future hybrid forwarding.

And when it cannot answer, an application can appear network-isolated even though every route and security rule is correct.

The VPC already had a resolver.

My job is not to install it.

My job is to know which names it owns, which rules take precedence, which queries it forwards, which evidence it records and when bfstore genuinely needs to extend it beyond the VPC.

References and further reading