At OSSO, we've been using a spine-leaf architecture in the datacenter, using BGP and Layer 3 to the host. This means that we can have any IP address of ours just pop up anywhere in our network, simply by adding a prefix on a leaf switch. We sacrifice half of our IP space for this. But we gain simplicity by avoiding all Layer 2 tricks.
TL;DR: Avoid IP addresses ending in
.255 for endpoints.
31-bit prefixes on Point-to-Point Links
Do we want IP
18.104.22.168 to show up somewhere? We'll just enable the
22.214.171.124/31 prefix on the relevant port on leafX. BGP handles that
both IPs are now reachable on the leaf switch and
126.96.36.199 can be
reached at the selected switch port.
This makes deploying and moving services a breeze. And we don't have to deal with Layer 2 hacks. But as you can see, we have to sacrifice one IP (the switch end) per prefix.
Normally, networks have to contain a network address (
x.x.x.0 for a
class C network) and a broadcast address (
x.x.x.255). Thus, the
smallest network would be 2 bits: e.g.
The network address is obsolete and never used. And the broadcast
address equates to “the other host”, so it is not needed for networks
with only two parties. Therefore RFC
3021 specifies that for such
small networks, we only need two addresses: a
Wasting 75% of our address space would've been awful, but wasting 50% is a good trade-off for what we're winning.
The problem with Windows
Microsoft unfortunately did not read RFC 3021, so for those hosts
we're stuck wasting two additional IP addresses. I've seen reports
that one could use a
/32 on the Windows side, and it will
effectively work as if it was a
/31, but this has not been verified by
The problem with some routers
Recently we ran into an issue where someone could connect to a service
from work, but not from home. Investigation revealed that the problem
arose because the endpoint was on an IP address that ends in
Following the setup above, we can hand out all 128 prefixes from
188.8.131.52/31 up to
184.108.40.206/31. By convention we assigned the even IP
220.127.116.11 — to the switch: this meant that the
endpoint/service got address
This had been running fine for years, until now.
The user with issues was using the KPN network (
an Experia box V8 (Arcadyan, Astoria Networks VGV7519), S/N
A30401xxxx, MAC 84:9C:A6:xx:xx:xx. Apparently that device treats
addresses that end in
.255 specially, and drops packets headed in that
We tested this with two different public IP addresses that ended in
.255. Neither IP would receive UDP or ICMP from this NAT-router.
Traffic to neighbouring IPs had no issues. Maybe this is some overeager
Smurf Attack protection?
(Here's a note to KPN: please fix your firmware!)
Avoid using 255 for endpoints
Moral of the story: we'll keep the convention of assigning the uneven
address to the endpoint, except for the
.255 address, which we'll
assign to the switch instead. This should avoid problems with crappy
routers in the future. We know the real network hardware beyond that