The IPv4 address is independent of any one server. Think of it kind of like L2 / L3, when a packet gets to its final network the L3 address is translated to a L2 address (ARP typically with ethernet / IP) with the L3 frame wrapped by a L2 frame. So think of this more like L2 is now a combination of classic L2 and the lower end of L3. It's sort of halfway between home router style NAT and switching; and yes, I'm sure it runs as close to full speed as switching does. This kind of set up is very common in cloud based services that support live migration (not that this service does; it's bare metal).