Open a terminal and trace a route to a large content network and something odd shows up. The hops belong to three different companies, a couple of internet exchange points, and a final router in a data centre you have never heard of. Yet the round trip is under 15 milliseconds, which puts that machine within a few hundred kilometres of your chair. Nobody configured that path. The routing system worked it out on its own.

Two mechanisms do the heavy lifting: anycast, an addressing trick that lets many servers share one IP address, and BGP, the protocol that decides which of them your packets actually reach. Together they are the reason an edge network behaves like a place rather than a marketing tier. Here is how the machinery works.

What Anycast Actually Does

With ordinary unicast, one IP address lives on one machine. Anycast breaks that assumption. The same address is configured on servers in dozens or hundreds of locations, and each of those servers announces it into BGP. Your packets are not sent to all of them, and no central appliance load-balances between them. Every router along the way simply forwards toward its own best path for that prefix — and different routers have different best paths.

The result: two users in the same city often land on the same node, while a user 400 kilometres away lands somewhere else entirely, even though everyone typed the same hostname. Nothing in the packet header says "nearest server." The behaviour comes entirely out of route selection.

So anycast is not a protocol and not a load balancer. It is a deployment pattern, and you will find it in DNS resolvers, root and TLD name servers, DDoS scrubbing centres, CDN front doors, and UDP backends for voice and gaming.

BGP in One Page

Border Gateway Protocol is the glue between autonomous systems. Your ISP, the transit providers it buys from, the CDN, and the exchange points where they meet all exchange reachability information: "I can reach 203.0.113.0/24, and here is the path I would use." Each announcement carries attributes, and those attributes are the steering wheel.

How a router picks a path

BGP follows a defined decision order. Implementations differ slightly in the middle of the list, but the shape is consistent:

• Highest local preference, set inside your own network — this overrides almost everything below it
• Shortest AS path, meaning fewest autonomous systems, not fewest milliseconds
• Lowest origin code, where a route learned from an interior protocol beats an exterior one
• Lowest multi-exit discriminator from the same neighbouring AS
• Prefer routes learned from eBGP over iBGP
• Lowest interior cost to the next hop, which is where "hot potato" routing comes from
• Tie-breakers: oldest route, then lowest router ID

Engineers push these levers with communities, MED values, and AS path prepending. Prepending is the blunt instrument: repeat your own AS number a few times and neighbours will generally prefer a different path to reach you.

Anycast does not tell the internet where your servers are. It tells the internet that this address is reachable from many places, then trusts each network to pick the one that suits it.

Why the Closest Node Is Not Always the Fastest

BGP is a policy protocol, not a performance protocol. It counts AS hops and respects commercial agreements. Someone in Amsterdam can be routed to Frankfurt, or even to Ashburn, because their ISP's cheapest transit exit points that way. The path may be two AS hops long and still cross an ocean.

Peering topology decides a lot. If a network does not peer at the exchange point in your city, your traffic leaves town. If two carriers have a congested interconnect, the shorter path is the slower one. Hot-potato routing makes it worse: an ISP hands traffic to a peer at the earliest opportunity to save its own capacity, with no regard for the end-to-end result.

That is why mature edge platforms rarely rely on anycast alone. They measure latency from real users and steer with DNS, or they take the first packet by anycast and pin the rest of the session to a specific node.

The Stateful Problem with Anycast

A one-shot DNS query over UDP fits anycast perfectly. Send, get an answer, done. If the next query lands on a different node, nobody notices.

A TLS connection is the opposite. The handshake happens on one machine, keys are derived there, and the session state lives in its memory. If BGP reconverges halfway through — a link flaps, a node is withdrawn, a better path appears — later packets can arrive in a different city where the connection does not exist. The client sees a reset or a long stall.

Common mitigations:

• Flow affinity: ECMP hashing on the five-tuple keeps an established flow on one path, so only withdrawals and new flows move.
• First-packet anycast, then unicast: the initial request hits a nearby node, which redirects or proxies the session to a named server.
• Short-lived connections: aggressive timeouts mean a reset costs a retry rather than a broken transaction.
• QUIC and HTTP/3, which tolerate path changes better than TCP — though the server-side state problem does not disappear.
• Keep TCP anycast inside a single metro, where the odds of a mid-session path change are low, and use unicast between regions.

DNS Steering and Hybrid Approaches

The alternative is to make the name resolve differently depending on who asks. A latency-aware authoritative DNS server answers with the address of the node closest to the resolver. It is easier to reason about, it suits long-lived TCP sessions, and you can drain a node simply by removing it from answers.

It has weaknesses too. Recursive resolvers are often far from the users they serve, so a resolver in a capital city may cover a whole country. EDNS Client Subnet passes a truncated client prefix through, which helps, but not every resolver supports it and it raises privacy questions. TTLs mean steering changes take effect slowly, and every lookup adds a round trip before the real connection begins.

In practice the two are stacked. Anycast handles the lookups quickly; latency-based answers or an anycast front door handle the data path. A failed health check should trigger both a DNS change and a BGP withdrawal, so a bad node stops attracting traffic from both directions.

When Routing Goes Wrong

Withdrawals take time. A node that fails can keep receiving traffic for seconds or minutes while the removal propagates, so well-built platforms drain a node deliberately before maintenance rather than yanking it offline. A node that keeps announcing while its application is dead is worse: it becomes a blackhole that looks healthy to every router on the internet.

Prefix leaks and accidental hijacks are the other failure mode. Filtering your announcements with RPKI origin validation and keeping your IRR records current is basic hygiene, and it protects everyone downstream of you, not just your own traffic.

Practical Checks and Questions to Ask

You can learn a lot in ten minutes. From a couple of different networks, run a traceroute or mtr to the hostname and look at where the last few hops sit and what the RTT implies. Check the address a DNS query returns from different resolvers by querying them directly. If an operator publishes a looking glass, use it to see how their prefix is seen from other networks — that view is closer to the truth than anything you can measure from one desk.

When you evaluate an edge provider, ask:

1. How many points of presence do you run, and which exchange points do you peer at?
2. Do you use anycast only, latency-based DNS only, or both?
3. How fast do you withdraw a failed node at the BGP level, and do you drain before maintenance?
4. How do sessions survive a route change?
5. What does your RPKI and route filtering posture look like?

Then test from real user locations, not just your office. The map on the marketing page tells you where servers exist; the routing table tells you where your traffic actually goes, and those are rarely the same picture.

Photo: PawinG / Pixabay