| 2013-11-27 15:50:29 |
Roman Podoliaka |
description |
In TripleO devtest story we are using Nova + Baremetal Driver + Neutron. The provisioned baremetal instance obtains its configuration from Metadata API. Currently all requests to Metadata API fail with error 500.
In nova-api log I can see the following traceback:
2013-11-27 11:44:01,423.423 5895 ERROR nova.api.metadata.handler [req-0d22f3c7-663e-452e-bfa9-747b728fc13b None None] Failed to get metadata for ip: 192.0.2.2
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler Traceback (most recent call last):
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 136, in _handle_remote_ip_request
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler meta_data = self.get_metadata_by_remote_address(remote_address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 78, in get_metadata_by_remote_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(self.conductor_api, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/base.py", line 466, in get_metadata_by_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 680, in get_fixed_ip_by_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler uuid_maps = self._get_instance_uuids_by_ip(context, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 582, in _get_instance_uuids_by_ip
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler data = neutronv2.get_client(context).list_ports(**search_opts)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 69, in get_client
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler raise exceptions.Unauthorized()
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler Unauthorized: Unauthorized: bad credentials
Analyzing this issue we found that Metadata API stopped working since change https://review.openstack.org/#/c/56174/4 was merged (it seems that change of line 57 in https://review.openstack.org/#/c/56174/4/nova/network/neutronv2/__init__.py is the reason).
The commit message looks pretty sane and that fix seems to be the right thing to do, because we don't want to do neutron requests on behalf of neutron service user we have in nova config, but rather on behalf of the admin user instead who made the original request to nova api. So it seems that context.is_admin should be extended to make it possible to distinguish between those two cases of admin users: the real admin users, and the cases when nova api needs to talk to neutron.
The problem is that all metadata queries are handled using default admin context (user and other vars are set to None while is_admin=True), so with https://review.openstack.org/#/c/56174/4 applied, get_client() always raises an exception. |
In TripleO devtest story we are using Nova + Baremetal Driver + Neutron. The provisioned baremetal instance obtains its configuration from Metadata API. Currently all requests to Metadata API fail with error 500.
In nova-api log I can see the following traceback:
2013-11-27 11:44:01,423.423 5895 ERROR nova.api.metadata.handler [req-0d22f3c7-663e-452e-bfa9-747b728fc13b None None] Failed to get metadata for ip: 192.0.2.2
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler Traceback (most recent call last):
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 136, in _handle_remote_ip_request
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler meta_data = self.get_metadata_by_remote_address(remote_address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 78, in get_metadata_by_remote_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(self.conductor_api, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/base.py", line 466, in get_metadata_by_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 680, in get_fixed_ip_by_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler uuid_maps = self._get_instance_uuids_by_ip(context, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 582, in _get_instance_uuids_by_ip
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler data = neutronv2.get_client(context).list_ports(**search_opts)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 69, in get_client
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler raise exceptions.Unauthorized()
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler Unauthorized: Unauthorized: bad credentials
Analyzing this issue we found that Metadata API stopped working since change https://review.openstack.org/#/c/56174/4 was merged (it seems that change of line 57 in https://review.openstack.org/#/c/56174/4/nova/network/neutronv2/__init__.py is the reason).
The commit message looks pretty sane and that fix seems to be the right thing to do, because we don't want to do neutron requests on behalf of neutron service user we have in nova config, but rather on behalf of the admin user instead who made the original request to nova api. So it seems that context.is_admin should be extended to make it possible to distinguish between those two cases of admin users: the real admin users, and the cases when nova api needs to talk to neutron.
The problem is that all metadata queries are handled using default admin context (user and other vars are set to None while is_admin=True), so with https://review.openstack.org/#/c/56174/4 applied, get_client() always raises an exception when Metadata API requests are handled. |
|
