Advanced networking: CIDR, NAT, peering & private connectivity

The last lesson gave you the shape of a cloud network — VPC, public and private subnets, security groups, load balancer, DNS. What it deliberately skipped is the part that trips up real engineers the first time they design a network themselves: the addressing math, and the connectivity primitives that join networks together or reach managed services privately. You can't carve a VPC into subnets without reading CIDR, you can't let a private server reach the internet without NAT, and you can't connect two networks without understanding peering, VPN, and private endpoints. This lesson fills exactly those gaps. Everything here still happens inside the VPC model you already know — we're now sizing it and wiring it to the outside world.

CIDR notation: how an address range is written

Every machine on a network has an IP address — four numbers 0–255 separated by dots, like 10.0.4.17. A CIDR block (Classless Inter-Domain Routing) is how you write a range of those addresses: an address followed by a slash and a number, like 10.0.0.0/16.

The slash number is a prefix length: it says how many of the 32 bits of the address are fixed (the network part), leaving the rest free (the host part). The fixed bits are the same for every address in the range; the free bits vary across the addresses inside it.

/16 fixes the first 16 bits → the first two octets (10.0.) are locked, the last two are free. That's 10.0.0.0 through 10.0.255.255.
/24 fixes the first 24 bits → the first three octets (10.0.4.) are locked, only the last octet is free. That's 10.0.4.0 through 10.0.4.255.

The smaller the slash number, the bigger the block. A /16 is large; a /24 is small; a /32 is a single address (all 32 bits fixed). This is the one mental flip beginners miss: a bigger prefix length means a smaller network.

Subnetting math: counting usable IPs

The number of addresses in a block is 2^(32 − prefix):

Block	Free (host) bits	Total addresses	Usable addresses
`/16`	16	65,536	65,531
`/24`	8	256	251
`/28`	4	16	11

"Usable" is smaller than "total" because some addresses are reserved. The math you can do in your head: 2^(32 − prefix) total addresses. The reservations are where the cloud differs from a textbook:

:::note Why "usable" isn't the whole block On a plain network, two addresses are always reserved per subnet — the network address (all host bits 0) and the broadcast address (all host bits 1). The big three cloud providers reserve five addresses in every subnet: the first (network), the last (broadcast), plus three more at the bottom of the range that the provider uses for the implied router, DNS, and future use. So a /24 gives 256 − 5 = 251 usable, not 254. Always subtract 5 when sizing a cloud subnet. :::

A worked carve: 10.0.0.0/16 into /24 subnets

You're handed a VPC CIDR of 10.0.0.0/16 — 65,536 addresses — and asked to carve it into public and private subnets across two availability zones. Going from a /16 to /24 subnets frees up the third octet (10.0.<this>.0) as the "which subnet" selector: 256 possible /24 subnets, each with 251 usable IPs. Pick four:

Subnet	CIDR	AZ	Role	Usable IPs
Public A	`10.0.0.0/24`	AZ-a	load balancer, NAT gateway	251
Public B	`10.0.1.0/24`	AZ-b	load balancer, NAT gateway	251
Private A	`10.0.10.0/24`	AZ-a	app servers, database	251
Private B	`10.0.11.0/24`	AZ-b	app servers, database	251

Three things to read off this:

The subnets do not overlap — each owns a distinct third octet, so no address belongs to two subnets. Overlap is the cardinal sin of network design; it makes routing ambiguous and blocks peering later.
Each subnet lives in one AZ (a subnet never spans AZs), which is how the public/private split combines with multi-AZ resilience from Chapter 1.
We used 0,1 for public and 10,11 for private on purpose — leaving a gap means you can add 10.0.2.0/24 later without re-numbering. Leave room to grow; a too-tight VPC CIDR is painful to widen after the fact.

NAT gateways: letting private servers reach out (one way)

A private subnet has no route from the internet — that's the point. But your app servers in that private subnet still need to reach out: download OS patches, call a third-party API, pull a container image. They need outbound internet without becoming inbound-reachable. That asymmetry is exactly what a NAT gateway provides.

NAT (Network Address Translation) sits in a public subnet and forwards outbound traffic from private instances to the internet, rewriting the source address to its own public IP. Return traffic for those connections comes back through it. Crucially, it only allows connections the private instance initiated — nobody on the internet can open a connection inward through a NAT gateway. Outbound yes, inbound no.

:::warning The NAT cost trap (FinOps Chapter 9) A managed NAT gateway bills two ways: an hourly charge just to exist, plus a per-GB charge on every byte it processes — on top of the normal internet egress fee. So a chatty private service behind a NAT gateway pays twice for the same bytes, and the per-GB processing cost hides inside service-specific charges where cost estimates miss it. This is one of the five hidden-cost categories FinOps tells you to always check. The fixes: route heavy provider-bound traffic through a VPC/gateway endpoint (next section) instead of the NAT, and don't put a NAT gateway in every AZ if cross-AZ traffic to one is cheaper than the per-AZ hourly fee. :::

Brand names: NAT Gateway (AWS), Cloud NAT (GCP), NAT Gateway (Azure).

VPC peering: joining two private networks

Sometimes two VPCs need to talk privately — your production VPC and a shared-services VPC, or two teams' networks. VPC peering is a direct, private connection between two VPCs so resources in each can reach the other using private IPs, without traversing the public internet. Traffic stays on the provider's backbone.

The one rule that bites everyone: the two VPCs must not have overlapping CIDR blocks. If both VPCs use 10.0.0.0/16, an address like 10.0.10.5 exists in both — and the router can't decide which side a packet is bound for. Peering between overlapping CIDRs is simply not allowed. This is why organizations plan a non-overlapping address scheme up front (e.g. prod gets 10.0.0.0/16, staging gets 10.1.0.0/16, shared gets 10.2.0.0/16) before they ever need to connect them.

Peering is also non-transitive: if A peers with B and B peers with C, A still can't reach C. Each pair needs its own peering. When the mesh of pairs gets unwieldy, providers offer a hub: Transit Gateway (AWS), Network Connectivity Center (GCP), Virtual WAN / vNet peering with a hub (Azure).

Brand names for peering itself: VPC Peering (AWS), VPC Network Peering (GCP), VNet Peering (Azure).

VPN: bridging the cloud to elsewhere over the internet

A VPN (Virtual Private Network) builds an encrypted tunnel over the public internet, so two networks (or a user and a network) talk as if directly connected, while the bytes on the wire are encrypted. Two flavors matter:

Site-to-site VPN connects a whole network to your VPC — classically an on-premises data center or office to the cloud, so the two behave like one private network. Same non-overlapping-CIDR rule as peering applies. (For higher bandwidth and lower latency than an internet tunnel, the dedicated-line upgrade is Direct Connect / Cloud Interconnect / ExpressRoute — a private physical link instead of a VPN over the internet.)
Client VPN connects an individual user's device into the VPC — how a remote engineer reaches private resources without exposing them publicly. The user authenticates, gets an address inside (or routable to) the VPC, and can reach private subnets directly.

Brand names: Site-to-Site VPN / Client VPN (AWS), Cloud VPN (GCP), VPN Gateway / Point-to-Site (Azure).

Private endpoints / PrivateLink: reaching managed services without the public internet

Here's a subtle problem. When your private app server calls a managed service — an object store, a managed database, another team's API — the service's endpoint is usually a public DNS name. By default, reaching it means going out through the NAT gateway, across the public internet, and back — paying NAT processing, taking the latency, and exposing the traffic to the internet path. For a service that lives in the same cloud, that's absurd.

A private endpoint (the general concept; AWS brands it PrivateLink) solves it: it places a private IP inside your VPC that maps to the managed service, so traffic to that service stays entirely on the provider's private network. Your private instance reaches the service over a private IP — no NAT, no public internet, no egress charge for that path. There are two shapes:

Gateway endpoint — a route-table entry for high-volume first-party services (notably object storage); free and the standard fix for "my private app reads from the bucket a lot."
Interface endpoint / Private Endpoint — an actual private IP (a network interface) in your subnet that fronts the service; used for most other services and for exposing your own service privately to another VPC.

Brand names: VPC Endpoints / PrivateLink (AWS), Private Service Connect / Private Google Access (GCP), Private Endpoint / Private Link (Azure).

Putting it together: trace a packet two ways

Take the carved VPC above and follow real traffic.

Outbound via NAT — a private instance patches itself:

App server at 10.0.10.5 (private subnet A) runs apt update, opening a connection to a public package mirror.
Its subnet's route table sends internet-bound traffic to the NAT gateway in public subnet A (10.0.0.0/24).
The NAT gateway rewrites the source to its own public IP and forwards the request to the internet — billing the per-GB processing fee on the way.
The mirror replies; the NAT gateway maps the return traffic back to 10.0.10.5. The mirror never learned the instance's private address, and could not have initiated the connection inward.

Inbound via private endpoint — the same instance reads object storage:

App server at 10.0.10.5 requests an object from the managed object store.
DNS for the object store resolves to a private IP inside the VPC (the gateway/interface endpoint), not the public endpoint.
The request goes straight to that private IP over the provider's backbone — never touching the NAT gateway or the public internet, so no NAT processing charge and no internet exposure.
The object comes back over the same private path.

Same instance, two destinations, two completely different paths — and knowing which is which is the difference between a cheap, private design and a surprise bill.

Common pitfalls

Overlapping CIDRs that block peering later. Two VPCs sized 10.0.0.0/16 each can never be peered or VPN-joined, because addresses collide. Plan a non-overlapping scheme org-wide before you need connectivity — widening or re-numbering a live VPC is painful.
Public subnets by accident. A subnet is "public" only because its route table sends 0.0.0.0/0 to an internet gateway. Attach that route (or a public IP) to the wrong subnet and a database you thought was private is now internet-reachable. Verify the route table, not just the name.
NAT cost surprises. Forgetting the per-GB processing charge, or running a high-throughput private service (logs, backups, image pulls) through NAT instead of a private endpoint, turns a "small" networking line into a top-three bill item. Route first-party traffic through endpoints.
A /24 is not 256 usable. It's 251 in the cloud (five reserved). Size with the 2^(32 − prefix) − 5 rule or you'll run out of addresses in a subnet you thought had headroom.
Forgetting peering is non-transitive. Adding a third VPC doesn't get connectivity for free through an existing peer — you need a new pairing or a transit hub.

Why it matters

The basic VPC model tells you where things go; this lesson tells you how to size and connect it. CIDR notation writes an address range as a prefix length — bigger slash number, smaller network — and 2^(32 − prefix) (minus 5 reserved in the cloud) counts the usable IPs, which is how you carve a /16 VPC into non-overlapping /24 subnets across AZs. NAT gateways give private instances outbound-only internet, at an hourly plus per-GB cost that's a classic FinOps trap. VPC peering and VPN (site-to-site for networks, client for users) join private networks together — both demanding non-overlapping CIDRs — and private endpoints / PrivateLink reach managed services over the provider's backbone instead of the public internet, killing both the NAT cost and the exposure. The public/private split, least-privilege firewalls, and private connectivity here are the network foundation that Cloud Security (Chapter 8) builds zero-trust on top of.

Where this leads: private endpoints and egress control reappear as security controls in Network security & zero trust, and the NAT cost trap is a worked example in how cloud pricing works (Chapter 9). For the network shape this builds on, see Networking: VPCs, subnets, load balancers & DNS.

Quiz

Required checkpoint