auth_url was not provided to the Neutron client

[TonyWang]

I got the following error in nova-api:
2015-02-05 14:50:59.176 13279 ERROR nova.api.openstack [req-d2a7c982-077a-4200-83e7-5bf9e4a7ef75 None] Caught error: auth_url was not provided to the Neutron client
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack Traceback (most recent call last):
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 124, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack return req.get_response(self.application)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack application, catch_exc_info=False)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack return resp(environ, start_response)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token.py", line 748, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack return self._call_app(env, start_response)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token.py", line 684, in _call_app
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack return self._app(env, _fake_start_response)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack return resp(environ, start_response)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack return resp(environ, start_response)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack response = self.app(environ, start_response)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack return resp(environ, start_response)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 910, in __call__
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack content_type, body, accept)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 1001, in _process_stack
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack request, action_args)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 880, in post_process_extensions
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack **action_args)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 608, in detail
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack self._extend_servers(req, list(resp_obj.obj['servers']))
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 557, in _extend_servers
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack servers))
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 346, in get_instances_security_groups_bindings
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack ports = self._get_ports_from_server_list(servers, neutron)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 305, in _get_ports_from_server_list
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack ports.extend(neutron.list_ports(**search_opts).get('ports'))
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 312, in list_ports
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack **_params)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1338, in list
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack for r in self._pagination(collection, path, **params):
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1351, in _pagination
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack res = self.get(path, params=params)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1324, in get
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack headers=headers, params=params)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1301, in retry_request
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack headers=headers, params=params)
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1244, in do_request
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack content_type=self.content_type())
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 183, in do_request
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack self.authenticate()
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 247, in authenticate
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack self._authenticate_keystone()
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 220, in _authenticate_keystone
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack raise exceptions.NoAuthURLProvided()
2015-02-05 14:50:59.176 13279 TRACE nova.api.openstack NoAuthURLProvided: auth_url was not provided to the Neutron client

"/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py"
def _get_client(token=None, admin=False):
    params = {
        'endpoint_url': CONF.neutron.url,
        'timeout': CONF.neutron.url_timeout,
        'insecure': CONF.neutron.api_insecure,
        'ca_cert': CONF.neutron.ca_certificates_file,
        'auth_strategy': CONF.neutron.auth_strategy,
        'token': token,
    }

    if admin:
        if CONF.neutron.admin_user_id:
            params['user_id'] = CONF.neutron.admin_user_id
        else:
            params['username'] = CONF.neutron.admin_username
        if CONF.neutron.admin_tenant_id:
            params['tenant_id'] = CONF.neutron.admin_tenant_id
        else:
            params['tenant_name'] = CONF.neutron.admin_tenant_name
        params['password'] = CONF.neutron.admin_password
        params['auth_url'] = CONF.neutron.admin_auth_url <<<<<If it isn't an admin, no "auth_url" in "params".
    return clientv20.Client(**params)

The caller:
"/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py"
class SecurityGroupAPI(security_group_base.SecurityGroupBase):

    id_is_uuid = True

    def create_security_group(self, context, name, description):
        neutron = neutronv2.get_client(context) <<<<< Doesn't give "admin=True"

The version of nova:
class VersionInfo(object):
    release = "1.el7.centos"
    version = "2014.2.1"

"master" has the same problem, but the code is moved to "nova/network/neutronv2/api.py"

[Steve Baker (steve-stevebaker)]

Moved to nova, not a heat bug

[Jordan Pittier (jordan-pittier)]

I have the exact same issue.

For me the caller is :

TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1917, in _deallocate_network
TRACE oslo.messaging.rpc.dispatcher context, instance, requested_networks=requested_networks)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 531, in deallocate_for_instance
TRACE oslo.messaging.rpc.dispatcher data = neutron.list_ports(**search_opts)

The neutron object is instantiated in the 'deallocate_for_instance' method (neutron = neutronv2.get_client(context).

get_client is defined in nova/network/neutronv2/__init__.py 'auth_url' is not passed to the neutron client. The resulting exception is

TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 220, in _authenticate_keystone
TRACE oslo.messaging.rpc.dispatcher raise exceptions.NoAuthURLProvided()
TRACE oslo.messaging.rpc.dispatcher NoAuthURLProvided: auth_url was not provided to the Neutron client

I have short lived token (1hour). I am on OpenStack Juno.

[Matthew Gilliard (matthew-gilliard-u)]

We see this problem a lot during scale testing with juno/stable. Sometimes as many as 10% of instances fail to come up. I suspect a race condition in either the nova neutronv2 code or the python neutronclient.

[Davanum Srinivas (DIMS) (dims-v)]

Hi,

So this bug should have been fixed already by the following review in trunk/kilo:
https://review.openstack.org/#/c/136931/

Please try that and let us know if you still see this.

-- dims