deletion of token for client causes failure

[Phil Nelson]

Used a script to get a token, use that token to delete a server, then delete the token (in an effort to cleanup). The server deletion got stuck, and the following error was logged:

unsupported operand type(s) for +: 'NoneType' and 'str'
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 231, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1401, in terminate_instance
    do_terminate_instance(instance, bdms)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 242, in inner
    retval = f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1393, in do_terminate_instance
    reservations=reservations)
  File "/usr/lib/python2.6/site-packages/nova/hooks.py", line 88, in inner
    rv = f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1356, in _delete_instance
    project_id=project_id)
  File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    self.gen.next()
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1329, in _delete_instance
    self._shutdown_instance(context, instance, bdms)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1254, in _shutdown_instance
    network_info = self._get_instance_nw_info(context, instance)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 691, in _get_instance_nw_info
    instance, conductor_api=self.conductor_api)
  File "/usr/lib/python2.6/site-packages/nova/network/quantumv2/api.py", line 363, in get_instance_nw_info
    result = self._get_instance_nw_info(context, instance, networks)
  File "/usr/lib/python2.6/site-packages/nova/network/quantumv2/api.py", line 371, in _get_instance_nw_info
    nw_info = self._build_network_info_model(context, instance, networks)
  File "/usr/lib/python2.6/site-packages/nova/network/quantumv2/ibmpowervm_api.py", line 27, in _build_network_info_model
    networks)
  File "/usr/lib/python2.6/site-packages/nova/network/quantumv2/api.py", line 794, in _build_network_info_model
    instance['project_id'])
  File "/usr/lib/python2.6/site-packages/nova/network/quantumv2/api.py", line 118, in _get_available_networks
    nets = quantum.list_networks(**search_opts).get('networks', [])
  File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 108, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 294, in list_networks
    **_params)
  File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 1002, in list
    for r in self._pagination(collection, path, **params):
  File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 1015, in _pagination
    res = self.get(path, params=params)
  File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 988, in get
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 973, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 907, in do_request
    resp, replybody = self.httpclient.do_request(action, method, body=body)
  File "/usr/lib/python2.6/site-packages/quantumclient/client.py", line 154, in do_request
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/quantumclient/client.py", line 183, in authenticate
    token_url = self.auth_url + "/tokens"

Deleting the token when the client is done with it should not cause a problem. If OpenStack needs a token internally, it should get one itself. And indeed we found that the code tried to do just that when the token created by the test script stopped working, but the reauthentication code is flawed... it does not have the auth_url, which led to the error shown in the stacktrace.

[Mark McClain (markmcclain)]

Adding the bug to Nova since it's the Quantum driver in Nova that is throwing the error.

[melanie witt (melwitt)]

When the quantum client instantiated by the nova quantum driver goes to list networks, auth fails as the token has been deleted. Quantum client has logic to try authenticating when Unauthorized is raised but quantum driver doesn't give params: username, password, tenant_name, auth_url required for the client to achieve this when it creates the client object.

[Satyanarayana Patibandla (satya-patibandla)]

I could not reproduce the issue in my local environment. Could you please provide me the detailed steps you followed to reproduce the issue.

[Matt Riedemann (mriedem)]

The tester created a script which gets a token, runs a rest call to delete a server and then deletes the token. They do this several times quickly and by the time quantum needs to do it's thing, the token is invalid so it fails and can't re-authenticate because the auth_url isn't passed down to the client. I believe this is the fix (I created this from stable/grizzly 2013.1.1 but it should be the same for master):

diff --git a/nova/network/quantumv2/__init__.py b/nova/network/quantumv2/__init__.py
index bda0392..f14d007 100644
--- a/nova/network/quantumv2/__init__.py
+++ b/nova/network/quantumv2/__init__.py
@@ -51,6 +51,7 @@
         'endpoint_url': CONF.quantum_url,
         'timeout': CONF.quantum_url_timeout,
         'insecure': CONF.quantum_api_insecure,
+ 'auth_url': CONF.quantum_admin_auth_url,
     }
     if token:
         params['token'] = token

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29402

[Matt Riedemann (mriedem)]

The nova side of this problem is fixed by this change:

https://review.openstack.org/#/c/23198/

I worked with Phil on a patch for that change to verify it resolves this bug.

That change is for this blueprint in havana: https://blueprints.launchpad.net/nova/+spec/fewer-networking-token-checks

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/23198
Committed: http://github.com/openstack/nova/commit/dd9c27f999221001bae9faa03571645824d2a681
Submitter: Jenkins
Branch: master

commit dd9c27f999221001bae9faa03571645824d2a681
Author: Stanislaw Pitucha <email address hidden>
Date: Thu Feb 28 19:50:21 2013 +0000

    Delegate authentication to quantumclient

    Quantumclient can deal with authentication, getting the token and
    refreshing it when needed. There should be no need for nova to do
    those explicitly. Additionally nova can save some time not having
    to recreate the Client object on each request.

    Fix a couple of places that relied on the exceptions module being
    imported inside quantumv2.

    Part of blueprint fewer-networking-token-checks
    Fixed bug 1177579

    Change-Id: I007cef1f0bd688036fa45d79792e6e350c145f05