Instances fail to get metadata: The 'service_metadata_proxy' option must be enabled.

[Syed Armani]

Instances are not getting metadata from the nova-api-metadata service.

Starting network...
udhcpc (v1.20.1) started
Sending discover...
Sending select for 10.5.5.3...
Lease of 10.5.5.3 obtained, lease time 86400
cirros-ds 'net' up at 4.00
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 4.02. request failed
failed 2/20: up 6.53. request failed
failed 3/20: up 8.60. request failed
failed 4/20: up 10.64. request failed
failed 5/20: up 12.75. request failed
failed 6/20: up 14.79. request failed
failed 7/20: up 16.83. request failed
failed 8/20: up 18.94. request failed
failed 9/20: up 20.97. request failed
failed 10/20: up 23.01. request failed
failed 11/20: up 25.12. request failed
failed 12/20: up 27.15. request failed
failed 13/20: up 29.20. request failed
failed 14/20: up 31.30. request failed
failed 15/20: up 33.34. request failed
failed 16/20: up 35.41. request failed
failed 17/20: up 37.49. request failed
failed 18/20: up 39.52. request failed
failed 19/20: up 41.59. request failed
failed 20/20: up 43.69. request failed

training@charlie:/var/log/nova$ tail -n100 nova-api-metadata.log

2015-06-25 22:36:35.323 25438 TRACE nova.api.metadata.handler
2015-06-25 22:36:37.396 25439 WARNING nova.api.metadata.handler [-] X-Instance-ID present in request headers. The 'service_metadata_proxy' option must be enabled to process this header.
2015-06-25 22:36:37.400 25439 ERROR nova.api.metadata.handler [req-f4352218-5b88-401f-ad60-68a5c937d1d0 - - - - -] Failed to get metadata for ip: 192.168.133.113
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler Traceback (most recent call last):
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/api/metadata/handler.py", line 159, in _handle_remote_ip_request
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler meta_data = self.get_metadata_by_remote_address(remote_address)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/api/metadata/handler.py", line 88, in get_metadata_by_remote_address
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(self.conductor_api, address)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py", line 511, in get_metadata_by_address
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1157, in get_fixed_ip_by_address
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler uuid_maps = self._get_instance_uuids_by_ip(context, address)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1056, in _get_instance_uuids_by_ip
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler data = get_client(context).list_ports(**search_opts)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 205, in get_client
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler auth_token = _ADMIN_AUTH.get_token(_SESSION)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 104, in get_token
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler return self.get_access(session).auth_token
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 144, in get_access
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler self.auth_ref = self.get_auth_ref(session)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 78, in get_auth_ref
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler authenticated=False, log=False)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 488, in post
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler return self.request(url, 'POST', **kwargs)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler return func(*args, **kwargs)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 374, in request
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler resp = send(**kwargs)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 417, in _send_request
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler raise exceptions.ConnectionRefused(msg)
2015-06-25 22:36:37.400 25439 TRACE nova.api.metadata.handler ConnectionRefused: Unable to establish connection to http://localhost:5000/v2.0/tokens

tail -n200 neutron-metadata-agent.log

2015-06-25 22:36:29.146 25427 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.
2015-06-25 22:36:31.228 25427 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.
2015-06-25 22:36:33.255 25427 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.
2015-06-25 22:36:35.327 25427 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.
2015-06-25 22:36:37.409 25427 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.
2015-06-25 22:36:39.452 25427 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.

syed@charlie:~$ sudo initctl list | grep metadata
nova-api-metadata start/running, process 25421
neutron-metadata-agent start/running, process 25374

root@charlie:~# cat /etc/nova/nova.conf | grep metadata
enabled_apis=metadata
neutron_metadata_proxy_shared_secret=5f15f9bc-5731-461a-8b77-0a8e04226c17
service_neutron_metadata_proxy=True

NOTE: Error message suggests that "service_metadata_proxy" must be set while nova.conf has "service_neutron_metadata_proxy" set which i think got deprecated in last release.

[Thiago Martins (martinx)]

Just installed Kilo on Trusty.

I am facing this problem too...

[Iman Darabi (iman-darabi)]

hi.
i had this problem in liberty, i solved this problem by adding "service_metadata_proxy" and "metadata_proxy_shared_secret" variables in both nova.conf of compute and controller like this:

[neutron]
...
service_metadata_proxy = True
metadata_proxy_shared_secret = METADATA_SECRET

[Iman Darabi (iman-darabi)]

in documentation "metadata_proxy_shared_secret" variable suggest set in /etc/neutron/metadata_agent.ini but it must be set in nova.conf file.

[Liam Young (gnuoy)]

I think this bug may have been reported against the wrong project. metadata_proxy_shared_secret has been set in juju deployed gateway since icehouse at least.