Instance in isolated network could not get metadata

Bug #1476612 reported by Ilya Shakhat on 2015-07-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
MOS Neutron
Mirantis OpenStack
Medium
MOS Neutron

Bug Description

In MOS 7.0 metadata proxy is enabled for isolated networks. However instance fails to retrieve metadata, curl reports "couldn't connect to host" error.

Steps to reproduce:
1. Create network
2. Launch instance and plug into the network
3. Via Horizon login to instance console and run "curl http://169.254.169.254/"

It's expected that curl shows the root of metadata tree (list of versions) however it shows error.

Ilya Shakhat (shakhat) on 2015-07-21
Changed in mos:
assignee: nobody → MOS Neutron (mos-neutron)
importance: Undecided → Medium
tags: added: neutron
Changed in mos:
milestone: none → 7.0
Changed in mos:
status: New → Confirmed
Changed in mos:
status: Confirmed → Incomplete
Eugene Nikanorov (enikanorov) wrote :

I can't confirm the bug with this description.
I've tested the particular case with isolated networks, everything seems to be in place: metadata-proxy listening inside a namespace, 169.254 url is responding.

VMs are provided with 169.254.169.254/32 route to DHCP ip (each agent Provides IP of its port)

Eugene Nikanorov (enikanorov) wrote :

However I found a possible issue with metadata:

By default we're creating a router-plugged network with a subnet with /22 prefix, having 2 IP allocation pools.
I see that for some reason the subnet has gateway ip being the first one of the _second_ pool while dnsmasq is providing a route to 169.254 via a gateway ip being the first ip of the first interval.
It was like if gateway ip address was changed.

Restart of DHCP agent however seems to fix this.

Initial state might have resulted in inability to get metadata.

Ilya Shakhat (shakhat) wrote :

The issue is caused by Cirros DHCP client which does not handle option that sets static route. The workaround is to add route manually:
  sudo ip ro add 169.254.169.254 via <metadata-gateway>
where metadata-gateway is IP address of the port with DHCP attached (device type network:dhcp)

The issue is not reproducible on Ubuntu, i.e. it applies route successfully:
ci-info: +++++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++++
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 11.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 11.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 169.254.169.254 | 11.0.0.2 | 255.255.255.255 | eth0 | UGH |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+

According to traffic capture the DHCP server sends route via option 121:
   Classless-Static-Route Option 121, length 14: (169.254.169.254/32:11.0.0.2),(default:11.0.0.1)
This support of this option in Cirros was released in 0.3.2 (https://bugs.launchpad.net/cirros/+bug/1190372), but MOS contains version 0.3.1

Ilya Shakhat (shakhat) wrote :

The issue should be fixed by https://review.fuel-infra.org/#/c/9887/ which was done as https://bugs.launchpad.net/fuel/+bug/1475603. Thus re-assigning this bug to Fuel and mark it as Fix Committed

Changed in fuel:
status: New → Fix Committed
importance: Undecided → Medium
milestone: none → 7.0
assignee: nobody → Fuel for Openstack (fuel)
Changed in mos:
status: Incomplete → Invalid
Changed in fuel:
assignee: Fuel for Openstack (fuel) → MOS Neutron (mos-neutron)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers