Requests to Metadata API fail with 500 if Neutron network plugin is used

[Roman Podoliaka]

In TripleO devtest story we are using Nova + Baremetal Driver + Neutron. The provisioned baremetal instance obtains its configuration from Metadata API. Currently all requests to Metadata API fail with error 500.

In nova-api log I can see the following traceback:

2013-11-27 11:44:01,423.423 5895 ERROR nova.api.metadata.handler [req-0d22f3c7-663e-452e-bfa9-747b728fc13b None None] Failed to get metadata for ip: 192.0.2.2
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler Traceback (most recent call last):
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 136, in _handle_remote_ip_request
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler meta_data = self.get_metadata_by_remote_address(remote_address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 78, in get_metadata_by_remote_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(self.conductor_api, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/base.py", line 466, in get_metadata_by_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 680, in get_fixed_ip_by_address
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler uuid_maps = self._get_instance_uuids_by_ip(context, address)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 582, in _get_instance_uuids_by_ip
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler data = neutronv2.get_client(context).list_ports(**search_opts)
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 69, in get_client
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler raise exceptions.Unauthorized()
2013-11-27 11:44:01,423.423 5895 TRACE nova.api.metadata.handler Unauthorized: Unauthorized: bad credentials

Analyzing this issue we found that Metadata API stopped working since change https://review.openstack.org/#/c/56174/4 was merged (it seems that change of line 57 in https://review.openstack.org/#/c/56174/4/nova/network/neutronv2/__init__.py is the reason).

The commit message looks pretty sane and that fix seems to be the right thing to do, because we don't want to do neutron requests on behalf of neutron service user we have in nova config, but rather on behalf of the admin user instead who made the original request to nova api. So it seems that context.is_admin should be extended to make it possible to distinguish between those two cases of admin users: the real admin users, and the cases when nova api needs to talk to neutron.

The problem is that all metadata queries are handled using default admin context (user and other vars are set to None while is_admin=True), so with https://review.openstack.org/#/c/56174/4 applied, get_client() always raises an exception when Metadata API requests are handled.

[Derek Higgins (derekh)]

Also note that the first occurance of this was a
CI job that started at Nov 26 23:00:02 UTC

The Job the kicked off 2 hours previous was ok, its been failing ever sense

[Davanum Srinivas (DIMS) (dims-v)]

Short Story - Roman has a working patch.

Long Story (discussion on IRC)

[11:05] <dansmith> derekh: yeah, I know, not intending to blame you or anyone
[11:05] <rpodolyaka> dims: so it's called from code that has no idea, if admin=True must be set
[11:05] <dansmith> derekh: but it's clearly pretty fragile so I just think something more complicated must be going on and I think folks familiar with neutronapi should be taking a pretty deep look before we freak out this time
[11:06] <dims> rpodolyaka, i see that the "special" admin context is created here - get_metadata_by_address
[11:07] <dims> rpodolyaka, will you be able to try something (give me 10 mins)?
[11:07] <rpodolyaka> dims: sure!
[11:12] <rpodolyaka> dims: yeah, it's kind of special. What I want to say, is that PhilD's patch is all about making sure we pass admin=True to get_client() explicitly (and it's cool :) ), but e.g. the code of get_fixed_ip_by_address() have no idea about passing admin=True (and it seems, that it should not have, because it can be called to handle the request of a regular user too)
[11:12] * garyk has quit (Quit: Leaving.)
[11:12] <dansmith> yep
[11:12] <dims> rpodolyaka, i know. try this - http://paste.openstack.org/show/54078/
[11:13] <dims> rpodolyaka, i had mentioned this in the review for Phil's patch
[11:13] * rpodolyaka is applying the patch and restarting nova
[11:17] <rpodolyaka> dims: http://paste.openstack.org/show/54080/
[11:17] <rpodolyaka> dims: looking at the patch itself now :)
[11:18] <dansmith> change the context param,
[11:18] <dims> rpodolyaka, heh :) yes please
[11:18] <dansmith> so that it doesn't conflict with the context module
[11:18] <dansmith> s/context/ctxt/ in that method
[11:18] <dims> +1 to dansmith
[11:20] <rpodolyaka> dims: dansmith: that was an easy one :)
[11:20] <rpodolyaka> voila!
[11:21] <dansmith> dims: can you explain why this fixes it?
[11:21] <rpodolyaka> dims: looks likes it works, thanks!
[11:21] <dansmith> I don't have the context, but this seems sketchy to me
[11:21] <rpodolyaka> it makes the admin context really special now, I think :)
[11:21] <dims> rpodolyaka, nice
[11:21] <dims> rpodolyaka, right
[11:21] <rpodolyaka> not by is_admin=True
[11:22] <dims> yep
[11:22] <rpodolyaka> but making it a new type
[11:22] <dansmith> heh, I can read :)
[11:22] <dansmith> isinstance() and foo.is_admin==True is roughly equivalent
[11:23] <dims> we have admin contexts we create by hand and there are contexts from users who are really admins
[11:23] <dims> there only flag is is_admin for both cases
[11:23] <dims> here we are adding another way to distinguish between the two
[11:24] <dansmith> and that's a good thing why?
[11:24] <dims> if it is an admin context created by hand, we use the neutron credentials in nova.conf
[11:24] <dansmith> because this seems like a good way to be confused in a few months
[11:24] <dims> dansmith, if you peek at phil's original patch you will see the issue
[11:25] <rpodolyaka> not sure, if it's the best to do that, but must distinguish them somethow
[11:25] <rpodolyaka> *we
[11:26] <dansmith> is the problem that if the user is an admin we use his credentials instead of the neutron credentials w...

[Davanum Srinivas (DIMS) (dims-v)]

Original Stack trace from Roman

  /usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py(80)_spawn_n_impl()
-> func(*args, **kwargs)
  /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py(594)process_request()
-> proto.__init__(sock, address, self)
  /usr/lib/python2.7/SocketServer.py(649)__init__()
-> self.handle()
  /usr/lib/python2.7/BaseHTTPServer.py(340)handle()
-> self.handle_one_request()
  /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py(285)handle_one_request()
-> self.handle_one_response()
  /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py(389)handle_one_response()
-> result = self.application(self.environ, start_response)
  /usr/lib/python2.7/dist-packages/paste/urlmap.py(203)__call__()
-> return app(environ, start_response)
  /usr/lib/python2.7/dist-packages/webob/dec.py(130)__call__()
-> resp = self.call_func(req, *args, **self.kwargs)
  /usr/lib/python2.7/dist-packages/webob/dec.py(195)call_func()
-> return self.func(req, *args, **kwargs)
  /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/ec2/__init__.py(88)__call__()
-> return req.get_response(self.application)
  /usr/lib/python2.7/dist-packages/webob/request.py(1296)send()
-> application, catch_exc_info=False)
  /usr/lib/python2.7/dist-packages/webob/request.py(1260)call_application()
-> app_iter = application(self.environ, start_response)
  /usr/lib/python2.7/dist-packages/webob/dec.py(130)__call__()
-> resp = self.call_func(req, *args, **self.kwargs)
  /usr/lib/python2.7/dist-packages/webob/dec.py(195)call_func()
-> return self.func(req, *args, **kwargs)
  /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/ec2/__init__.py(100)__call__()
-> rv = req.get_response(self.application)
  /usr/lib/python2.7/dist-packages/webob/request.py(1296)send()
-> application, catch_exc_info=False)
  /usr/lib/python2.7/dist-packages/webob/request.py(1260)call_application()
-> app_iter = application(self.environ, start_response)
  /usr/lib/python2.7/dist-packages/webob/dec.py(130)__call__()
-> resp = self.call_func(req, *args, **self.kwargs)
  /usr/lib/python2.7/dist-packages/webob/dec.py(195)call_func()
-> return self.func(req, *args, **kwargs)
  /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py(115)__call__()
-> meta_data = self._handle_remote_ip_request(req)
  /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py(136)_handle_remote_ip_request()
-> meta_data = self.get_metadata_by_remote_address(remote_address)
  /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py(78)get_metadata_by_remote_address()
-> data = base.get_metadata_by_address(self.conductor_api, address)
  /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/api/metadata/base.py(466)get_metadata_by_address()
-> fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)
  /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py(680)get_fixed_ip_by_address()
-> uuid_maps = self._get_instance_uuids_by_ip(context, address)
  /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py(582)_get_instance_uuids_by_ip()
-> ...

[KaZeR (kazer)]

Applying the patch from https://review.openstack.org/#/c/58843 did solve this issue for me.

Thanks Dims!

[Mark McLoughlin (markmc)]

Just to confirm that reverting https://review.openstack.org/56174 fixes the tripleo seed for me too

Simply put, applying this change:

 - if admin or context.is_admin:
 + if admin: # or context.is_admin:

and restarting the nova-api service, is enough to make a simple EC2 metadata GET start working again:

    $> curl http://169.254.169.254/latest/meta-data/instance-id

Without that change, the GET fails with the nova-api traceback above

[Mark McLoughlin (markmc)]

Bah, got that reversed - this is the change that fixes it for me:

 + if admin:
 - if admin or context.is_admin:

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/59243

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/59243
Committed: http://github.com/openstack/nova/commit/652620d12f3afe6845e41d9762b52d23f44fd557
Submitter: Jenkins
Branch: master

commit 652620d12f3afe6845e41d9762b52d23f44fd557
Author: Phil Day <email address hidden>
Date: Fri Nov 29 23:24:51 2013 +0000

    Fix Neutron Authentication for Metadata Service

    A recent change to the way Nova creates a Neutron client
    51e5f52e4cb60e266ccde71f205c91eb8c97b48b changed the conditions
    under which it re-authenticates using the neutron admin credentials
    from "if admin" to "if admin or context.is_admin". However this
    meant that any user with admin role in Nova interacted with Neutron
    as a different tenant, preventing them from non-admin actions such
    as allocating floating IPS.

    This was then addressed by 1c1371c78b990447aeaa4377b512f8887e6ff3ce
    which only reauthenticated as admin on the explicit use of
    "admin=True" parameter to neutonv2.get_client().

    However the metadata service does not explicitly create the neutron
    client so it has to pass in a context created by
    context.get_admin_context() which has is_admin=True but no auth_token.

    This change therefore accepts this combination of context values
    as a valid reason to re-authenticate as admin, while still allowing
    users with the admin role to use their own token.

    As a tidy up two tests for getting an admin client are moved from
    TestNeutonClient to join the other admin client tests in
    TestNeutronClientForAdminScenarios

    Change-Id: Ic2a6f383a85f6bb606d8d65feaefa0d23329adb9
    Closes-Bug: 1255577

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54736
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bdc7519862299e43af55d7a7a221669de905d15c
Submitter: Jenkins
Branch: stable/havana

commit bdc7519862299e43af55d7a7a221669de905d15c
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 21 09:52:28 2013 -0500

    Pass thru credentials to allow re-authentication

    This is a backport of 4 fixes squashed into one because:

    1. They need to all be merged together as they build on each
       other as problems were found in each change after it was
       merged on master.
    2. The 3rd change won't pass Jenkins on it's own so it has
       to be squashed with the 4th and final change, so I'm just
       going to squash the entire topic branch together.

    The change bugs fixed and cherry pick commit hashes are left
    intact for each change.

    Closes-Bug: #1241275
    (cherry picked from commit 51e5f52e4cb60e266ccde71f205c91eb8c97b48b)

    Cache Neutron Client for Admin Scenarios

    Closes-Bug: #1250580
    (cherry picked from commit 85332012dede96fa6729026c2a90594ea0502ac5)

    Users with admin role in Nova should not re-auth with Neutron

    Closes-Bug: 1250763
    (cherry picked from commit 1c1371c78b990447aeaa4377b512f8887e6ff3ce)

    Fix Neutron Authentication for Metadata Service

    Closes-Bug: 1255577
    (cherry picked from commit 652620d12f3afe6845e41d9762b52d23f44fd557)

    ============

    Change-Id: I2858562b180f3e058a2da9d67bef02af80927177