metadata route is absent and metadata server unavailable for instance with network w/o gateway

[Andrey Pavlov]

ubuntu 14.04
latest devstack (28.08.2016)
enabled services: nova, glance, cinder, keystone, horizon, neutron, neutron-vpnaas, ec2-api

steps to reproduce:
0) source demo credentials
1) create network
2) create subnet without gateway
3) boot instance from cirros image
4) run vnc console
5) run 'route -n':
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.0.0 10.10.2.1 255.255.0.0 UG 0 0 0 eth0
10.10.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

6) run 'curl http://169.254.169.254/latest/'
curl: (7) Failed to connect to 169.254.169.254: Network is unreachable

if user runs instance with predefined network that all works well:
$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.1.1 0.0.0.0 UG 0 0 0 eth0
10.10.0.0 10.10.1.1 255.255.0.0 UG 0 0 0 eth0
10.10.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.169.254 10.10.1.1 255.255.255.255 UGH 0 0 0 eth0

according to gating jobs it was broken between "Aug 25 2:05 PM" (last check pipeline - https://review.openstack.org/#/c/360230/)
and "Aug 26 1:05 AM" (patch 4 first check pipeline - https://review.openstack.org/#/c/357766/)

[Sean Dague (sdague)]

This was when we changed the default to use metadata server all the time. It sounds like previously the ec2api didn't work if it was using metadata server instead of config drive.

Is there some default additional metadata route that needs to be put into neutron during such a config?

[Andrey Pavlov (apavlov-e)]

Sean,
this doesn't work even if ec2api disabled.

neutron net-create nnn
neutron subnet-create
neutron subnet-create --name nnns --no-gateway 4cee85c7-48ed-427f-83a6-e0c43399aa07 10.10.0.0/24
nova boot --flavor 1 --image cirros-0.3.4-x86_64-uec --nic net-id=4cee85c7-48ed-427f-83a6-e0c43399aa07 ttt
nova console-log ttt | tail
nova get-vnc-console ttt novnc

then open vnc console and try to do
'curl http://169.254.169.254/latest'

[Matt Riedemann (mriedem)]

Related changes that sdague is referring to in the gate setup:

https://review.openstack.org/#/c/357446/

https://review.openstack.org/#/c/357443/

So it actually sounds like this is a latent bug, only recently exposed by the default change in the job setup, not actually a regression in nova.

[Brian Haley (brian-haley)]

Old bug. I'm going to mark this as Invalid for neutron since there is a flag to control the behavior, forcing the dhcp-agent to provide a route if the network is isolated.

In the dhcp-agent.ini file you should un-comment and set this value to True:

# The DHCP server can assist with providing metadata support on isolated
# networks. Setting this value to True will cause the DHCP server to append
# specific host routes to the DHCP request. The metadata service will only be
# activated when the subnet does not contain any router port. The guest
# instance must be configured to request host routes via DHCP (Option 121).
# This option doesn't have any effect when force_metadata is set to True.
# (boolean value)
enable_isolated_metadata = true

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]