network_get_all_by_uuids fails when requesting a network as admin

[Alvaro Lopez]

Hi there.

I'm using network.VlanManager.

I'm trying to get all the cloudpipes that are running on a system as an admin. If there's only one cloudpipe and it is associated to the admin tenant, it works perfectly. However, every time I get another cloudpipe, not associated with the admin's tenant I get the following error in nova-network:

    (nova.rpc.common): TRACE: Traceback (most recent call last):
    (nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 250, in _process_data
    (nova.rpc.common): TRACE: rval = node_func(context=ctxt, **node_args)
    (nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 239, in wrapped
    (nova.rpc.common): TRACE: return func(self, context, *args, **kwargs)
    (nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1475, in get_network
    (nova.rpc.common): TRACE: networks = self._get_networks_by_uuids(context, [network_uuid])
    (nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1748, in _get_networks_by_uuids
    (nova.rpc.common): TRACE:
    (nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 825, in network_get_all_by_uuids
    (nova.rpc.common): TRACE: return IMPL.network_get_all_by_uuids(context, network_uuids, project_id)
    (nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 101, in wrapper
    (nova.rpc.common): TRACE: return f(*args, **kwargs)
    (nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2063, in network_get_all_by_uuids
    (nova.rpc.common): TRACE: raise exception.NoNetworksFound()
    (nova.rpc.common): TRACE: NoNetworksFound: No networks defined.

And the following in nova-api:

    (nova.api.openstack): TRACE: Traceback (most recent call last):
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 41, in __call__
    (nova.api.openstack): TRACE: return req.get_response(self.application)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    (nova.api.openstack): TRACE: application, catch_exc_info=False)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    (nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 166, in __call__
    (nova.api.openstack): TRACE: return self.app(env, start_response)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    (nova.api.openstack): TRACE: return resp(environ, start_response)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    (nova.api.openstack): TRACE: return resp(environ, start_response)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    (nova.api.openstack): TRACE: return resp(environ, start_response)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    (nova.api.openstack): TRACE: response = self.app(environ, start_response)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    (nova.api.openstack): TRACE: return resp(environ, start_response)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    (nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    (nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 800, in __call__
    (nova.api.openstack): TRACE: content_type, body, accept)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 848, in _process_stack
    (nova.api.openstack): TRACE: action_result = self.dispatch(meth, request, action_args)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 924, in dispatch
    (nova.api.openstack): TRACE: return method(req=request, **action_args)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/cloudpipe.py", line 152, in index
    (nova.api.openstack): TRACE: for x in self._get_all_cloudpipes(context)]
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/cloudpipe.py", line 103, in _vpn_dict
    (nova.api.openstack): TRACE: network = self.network_api.get(elevated, vif['network']['id'])
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 45, in get
    (nova.api.openstack): TRACE: 'args': {'network_uuid': network_uuid}})
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 69, in call
    (nova.api.openstack): TRACE: return _get_impl().call(context, topic, msg, timeout)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 671, in call
    (nova.api.openstack): TRACE: return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 336, in call
    (nova.api.openstack): TRACE: rv = list(rv)
    (nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 304, in __iter__
    (nova.api.openstack): TRACE: raise result
    (nova.api.openstack): TRACE: RemoteError: Remote error: NoNetworksFound No networks defined.

This is due to the filter in network_get_all_by_uuids in nova/db/sqlalchemy/api.py that filters the networks by the project_id received as a parameter, which is filled in the appropriate calls with the context project (that's the reason of the above exceptions when I have a cloudpipe not in the admin tenant).

If I look at network_get_all or network_get code, there is no such filter. What is the purpose of this filter? Can it be removed or is it really needed?

[Vish Ishaya (vishvananda)]

ugh, nasty little bug. We were using project checking in the db as an added level of access control, but it does lead to problems when we are trying to do administrative commands across tenants.

Can you see if the following change fixes the issue?

diff --git a/nova/api/openstack/compute/contrib/cloudpipe.py b/nova/api/openstack/compute/contrib/cloudpipe.py
index b5d8747..115dc70 100644
--- a/nova/api/openstack/compute/contrib/cloudpipe.py
+++ b/nova/api/openstack/compute/contrib/cloudpipe.py
@@ -85,6 +85,7 @@ class CloudpipeController(object):

     def _vpn_dict(self, context, project_id, instance):
         elevated = context.elevated()
+ elevated.project_id = project_id
         rv = {'project_id': project_id}
         if not instance:
             rv['state'] = 'pending'

This fakes the context to be for the project. should only need to make the change on the api host.

[Vish Ishaya (vishvananda)]

it messed up the spaces but essentially it is to add a line that sets the elevated contexts project_id to the project_id that owns the vpn.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/6189

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6189
Committed: http://github.com/openstack/nova/commit/12d9a5a38b9968f488b26822dc082ecbc484bbe9
Submitter: Jenkins
Branch: master

commit 12d9a5a38b9968f488b26822dc082ecbc484bbe9
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Apr 3 13:47:40 2012 -0700

    Make sure cloudpipe extension can retrieve network

     * includes failing test
     * fixes bug 972583

    Change-Id: Idadac82c6a0fda8a1b912fb974e5754a1e82df39

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/6204

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6204
Committed: http://github.com/openstack/nova/commit/e36a1381e25b19200a2a0cf23ffc0212d09d0984
Submitter: Jenkins
Branch: milestone-proposed

commit e36a1381e25b19200a2a0cf23ffc0212d09d0984
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Apr 3 13:47:40 2012 -0700

    Make sure cloudpipe extension can retrieve network

     * includes failing test
     * fixes bug 972583

    Change-Id: Idadac82c6a0fda8a1b912fb974e5754a1e82df39

[Alvaro Lopez (aloga)]

Hi Vish.

Sorry for the delay in answering. Just for the record, that patch solved the problem.

Cheers,
Alvaro.