Instance in isolated network could not get metadata

[Ilya Shakhat]

In MOS 7.0 metadata proxy is enabled for isolated networks. However instance fails to retrieve metadata, curl reports "couldn't connect to host" error.

Steps to reproduce:
1. Create network
2. Launch instance and plug into the network
3. Via Horizon login to instance console and run "curl http://169.254.169.254/"

It's expected that curl shows the root of metadata tree (list of versions) however it shows error.

[Eugene Nikanorov (enikanorov)]

I can't confirm the bug with this description.
I've tested the particular case with isolated networks, everything seems to be in place: metadata-proxy listening inside a namespace, 169.254 url is responding.

VMs are provided with 169.254.169.254/32 route to DHCP ip (each agent Provides IP of its port)

[Eugene Nikanorov (enikanorov)]

However I found a possible issue with metadata:

By default we're creating a router-plugged network with a subnet with /22 prefix, having 2 IP allocation pools.
I see that for some reason the subnet has gateway ip being the first one of the _second_ pool while dnsmasq is providing a route to 169.254 via a gateway ip being the first ip of the first interval.
It was like if gateway ip address was changed.

Restart of DHCP agent however seems to fix this.

Initial state might have resulted in inability to get metadata.

[Ilya Shakhat (shakhat)]

The issue is caused by Cirros DHCP client which does not handle option that sets static route. The workaround is to add route manually:
  sudo ip ro add 169.254.169.254 via <metadata-gateway>
where metadata-gateway is IP address of the port with DHCP attached (device type network:dhcp)

The issue is not reproducible on Ubuntu, i.e. it applies route successfully:
ci-info: +++++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++++
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 11.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 11.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 169.254.169.254 | 11.0.0.2 | 255.255.255.255 | eth0 | UGH |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+

According to traffic capture the DHCP server sends route via option 121:
   Classless-Static-Route Option 121, length 14: (169.254.169.254/32:11.0.0.2),(default:11.0.0.1)
This support of this option in Cirros was released in 0.3.2 (https://bugs.launchpad.net/cirros/+bug/1190372), but MOS contains version 0.3.1

[Ilya Shakhat (shakhat)]

The issue should be fixed by https://review.fuel-infra.org/#/c/9887/ which was done as https://bugs.launchpad.net/fuel/+bug/1475603. Thus re-assigning this bug to Fuel and mark it as Fix Committed