Can't ping own floating IP from instance

[Evan Callicoat]

After allocating and associating a floating IP with an instance, traffic to the floating IP from within the instance fails due to the iptables rules on the host. The desired behavior is commonly known as NAT reflection or hairpin NAT, and is part of a VEPA's bridging behavior.

This use-case is primarily interesting as a way to avoid using split-DNS; instead of maintaining both public and private DNS, instances can use hostnames which resolve to public floating IPs to connect to whatever services an application needs, regardless of which instance they may happen to exist on. This provides flexibility in being able to move floating IPs and instances without the configuration in other instances changing. Being unable to communicate with your own floating IP means you're restricted to only communicating with services on other instances by their public DNS records, which defeats the purpose of using floating IPs and not splitting DNS in the first place.

In order for hairpin NAT to work with standard Linux bridges at Layer 2, the sysfs flag "hairpin_mode" on a port in a bridge (located at /sys/class/net/<bridge>/brif/<port>/hairpin_mode or /sys/class/net/<port>/brport/hairpin_mode) needs to be set to 1, allowing a frame to exit the same bridge port it entered on. At Layer 3, we need to make use of a DNAT in PREROUTING and a SNAT in POSTROUTING, to go from [fixed source, float dest] -> DNAT -> [fixed source = fixed dest] -> SNAT -> [float source, fixed dest], with the routing step inbetween PRE/POST selecting the same bridge port.

Currently, nova-network's linux_net.py inserts the appropriate DNAT/SNAT rules when a floating IP is assigned to an instance, but in the POSTROUTING chains, a local traffic exception is inserted prior to the floating SNAT, as follows:

-A nova-network-POSTROUTING -s <fixed range> -d <fixed range> -j ACCEPT

Since ACCEPT is a terminating target, and after our DNAT the source and dest will match this rule, the floating SNAT rule is never matched and when the instance receives the traffic, it doesn't match what the instance originally sent, and is seen as new external traffic, except the source is the same as the instance, ultimately resulting in a TCP RST (or equivalent for other protocols) and lost traffic. In the case of a ping, for instance, the flow is basically like this:

[Instance: ping process (local to float) -> routing] -> [Host: DNAT -> routing -> ACCEPT (no SNAT)] -> [Instance: interface ICMP response -> routing] -> [Instance: interface ICMP response (local to local) -> error on unrelated response traffic] -> [Instance: interface ICMP process (nothing was expecting error response) -> drop packet]

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/4231
Committed: http://github.com/openstack/nova/commit/b61e1ea12cd41ea507b1f6496ec1413c93bd679b
Submitter: Jenkins
Branch: master

commit b61e1ea12cd41ea507b1f6496ec1413c93bd679b
Author: Evan Callicoat <email address hidden>
Date: Thu Feb 16 07:28:31 2012 +0000

    Enables hairpin_mode for virtual bridge ports, allowing NAT reflection

    * enables hairpin_mode on virtual bridge ports on instance spawn
    * adds conntrack DNAT state criteria to fixed/fixed SNAT exception so reflected traffic SNATs
    * updates get_interface ElementTree to work with Python 2.6/2.7
    * fixes bug 933640

    Change-Id: I63b3e91b41898fcffda8a288be503f9b740b4b4e