metadata IP 169.254.169.254 routing breaks RFC3927 and does not work on Windows starting from WS 2008

[Alessandro Pilotti]

The Quantum L3 Linux Agent handles metadata IP access with the following rule:

-A quantum-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697

obtained with: sudo ip netns exec qrouter-<router-id> iptables-save

169.254.x.x link local addresses are described in RFC3927 whose section 2.6.2 clearly states:

"The host MUST NOT send a packet with an IPv4 Link-Local destination address to any router for forwarding."

And on section 2.7:

"An IPv4 packet whose source and/or destination address is in the 169.254/16 prefix MUST NOT be sent to any router for forwarding, and any network device receiving such a packet MUST NOT forward it, regardless of the TTL in the IPv4 header."

Ref: http://tools.ietf.org/html/rfc3927#section-2.6.2

Linux does not enforce this rule, but Windows starting with 2008 and Vista does, which means that the metadata IP 169.254.169.254 is not accessible from a Windows guest (tested on Windows Server 2012 on Hyper-V).

The current workaround consists in adding explicitly a static route on the Windows guest with:

route add 169.254.169.254 mask 255.255.255.255 <router-ip>

[Mark McClain (markmcclain)]

The metadata service already supports passing the static metadata route via DHCP. Have you enabled this support?

[Salvatore Orlando (salvatore-orlando)]

We might want to implicitly add this route anyway with option 121, having the router ip as next hop for routed metadata access.
This might also be helpful for images configured with zeroconf, which have a preinjected metadata route.

Only problem is that this won't work with images which do not support option 121.

[Alessandro Pilotti (alexpilotti)]

Please add both 121 and 249 as Windows supports DHCP option 121 starting with 2008 while previous versions (including 2003) use option 249.

[Alessandro Pilotti (alexpilotti)]

As a workaround we are explicitly adding the route on the guest side in Cloudbase-init:

https://github.com/cloudbase/cloudbase-init/blob/master/cloudbaseinit/metadata/services/httpservice.py#L60

[Allister Maguire (allister-m)]

We have updated dhcp_agent.py and dhcp.py to allow another config option isolated_metadata_always=True.

If this is set and gateway_ip is set for the subnet the static metadata route will still be injected by DHCP.

This solves this issue for us plus allows us to use an external router for our Private Cloud (gateway_ip set in DHCP) and provides metadata redundancy with multiple DHCP Agents. We run DHCP Agent and Metadata Agent together on a number of our Compute nodes, then whichever DHCP Agent responses with a lease will also include the static metadata route to itself.

[Jeff Escamilla (jeff-escamilla)]

Allister,

That sounds exactly like what I am looking for. We have an external router, but we get DHCP from the Quantum DHCP agent and would love to be able to automatically inject that route in, but in it's current incarnation it's one or the other. Do you have that code around somewhere, or has a request been made to merge it into the code base?

[Allister Maguire (allister-m)]

I haven't done anything with it so far, but did respond to this bug in case anyone was interested.

The attachment contains two patches for the following two file:

/usr/share/pyshared/quantum/agent/dhcp_agent.py
/usr/share/pyshared/quantum/agent/linux/dhcp.py

It is a super simple patch, not a lot of code.

Regards
Allister

[Kevin Fox (kevin-fox-y)]

I just bumped into this issue with RHEL 5 / CentOS 5. It automatically adds a route to 169.254.0.0/16 out eth0. This forces the packets out eth0 instead of to the router. Quantum sees the packets and rejects them like:

Sep 16 17:32:21 vam kernel: martian source 169.254.169.254 from 10.0.1.2, on dev qbr7a45cbaa-0d
Sep 16 17:32:22 vam kernel: ll header: ff:ff:ff:ff:ff:ff:fa:16:3e:68:17:01:08:06

Perhaps the right solution is for quantum-ns-metadata-proxy to be actually bound to 169.254.169.254 instead of localhost with a firewall rewriting rule.

I tried adding a static route on the quantum network to force it out the router, but dhcp in RHEL 5 doesn't support that feture.

As a temporary work around, the follwoing worked for me:
echo >> /etc/sysconfig/network <<EOF
NOZEROCONF=yes
EOF

[Florian Haas (fghaas)]

I'm having trouble digging up the Gerrit change that fixes this. Could someone with better insight into the codebase please add the appropriate cross-reference and mark this bug fixed? As far as i can tell the 169.254.169.254 host route workaround has been in the Neutron DHCP agent for several releases now.

[Eugene Nikanorov (enikanorov)]

Fixed here: https://bugs.launchpad.net/neutron/+bug/1460793

[Florian Haas (fghaas)]

Er nope, that can't be right. I see this route being injected on an Icehouse cloud. The commit you're referring to landed in Liberty and was backported to Kilo. So what you're referring to is probably the last modification of the codebase in this regard, but the 169.254.169.254 host route has definitely been pushed for longer than that.

[Alessandro Pilotti (alexpilotti)]

@Florian: Cloudbase-Init creates the route if it's missing, in case you were using it on your Icehouse image

[Florian Haas (fghaas)]

I'm not using cloudbase-init. And I did verify (with tcpdump) that the host route is in the DHCP offer.

[Rossella Sblendido (rossella-o)]

Is the subnet isolated? That route for isolated subnets were added by https://review.openstack.org/#/c/21187/ back in Grizzly