Nova / Neutron Client failing upon re-authentication after token expiration

[Drew Thorstensen]

By default, the token length for clients is 24 hours. When that token expires (or is invalidated for any reason), nova should obtain a new token.

Currently, when the token expires, it leads to the following fault:
    File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 136, in _get_available_networks
      nets = neutron.list_networks(**search_opts).get('networks', [])
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 108, in with_params
      ret = self.function(instance, *args, **kwargs)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 325, in list_networks
      **_params)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1197, in list
      for r in self._pagination(collection, path, **params):
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1210, in _pagination
      res = self.get(path, params=params)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1183, in get
      headers=headers, params=params)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1168, in retry_request
      headers=headers, params=params)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1103, in do_request
      resp, replybody = self.httpclient.do_request(action, method, body=body)
    File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 188, in do_request
      self.authenticate()
    File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 224, in authenticate
      token_url = self.auth_url + "/tokens"
    TRACE nova.openstack.common.rpc.amqp TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

This error is occurring because nova/network/neutronv2/__init__.py obtains a token for communication with neutron. Nova is then authenticating the token (nova/network/neutronv2/__init__.py - _get_auth_token). Upon authentication, it passes in the token into the neutron client (via the _get_client method). It should be noted that the token is the main element passed into the neutron client (auth_url, username, password, etc... are not passed in as part of the request)

Since nova is passing the token directly into the neutron client, nova does not validate whether or not the token is authenticated.

After the 24 hour period of time, the token naturally expires. Therefore, when the neutron client goes to make a request, it catches an exceptions.Unauthorized block. Upon catching this exception, the neutron client attempts to re-authenticate and then make the request again.

The issue arises in the re-authentication of the token. The neutron client's authenticate method requires that the following parameters are sent in from its users:
 - username
 - password
 - tenant_id or tenant_name
 - auth_url
 - auth_strategy

Since the nova client is not passing these parameters in, the neutron client is failing with the exception above.

Not all methods from the nova client are exposed to this. Invocations to nova/network/neutronv2/__init__.py - get_client with an 'admin' value set to True will always get a new token. However, the clients that invoke the get_client method without specifying the admin flag, or by explicitly setting it to False will be affected by this. Note that the admin flag IS NOT determined based off the context's admin attribute.

Methods from nova/network/neutronv2/api.py that are currently affected appear to be:
 - _get_available_networks
 - allocate_for_instance
 - deallocate_for_instance
 - deallocate_port_for_instance
 - list_ports
 - show_port
 - add_fixed_ip_to_instance
 - remove_fixed_ip_from_instance
 - validate_networks
 - _get_instance_uuids_by_ip
 - associate_floating_ip
 - get_all
 - get
 - get_floating_ip
 - get_floating_ip_pools
 - get_floating_ip_by_address
 - get_floating_ips_by_project
 - get_instance_id_by_floating_address
 - allocate_floating_ip
 - release_floating_ip
 - disassociate_floating_ip
 - _get_subnets_from_port

[Matt Riedemann (mriedem)]

This is essentially a duplicate of bug 1177579.

[Matt Riedemann (mriedem)]

The patch is here: https://review.openstack.org/#/c/52954/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/53461

[Joe Gordon (jogo)]

nova should not be doing token management on behalf on neutronclient. Neutronclient should handle that on its own. We had a similar issue in novaclient that has been fixed (uses python-keyring).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/52954
Committed: http://github.com/openstack/nova/commit/51e5f52e4cb60e266ccde71f205c91eb8c97b48b
Submitter: Jenkins
Branch: master

commit 51e5f52e4cb60e266ccde71f205c91eb8c97b48b
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 21 09:52:28 2013 -0500

    Pass thru credentials to allow re-authentication

    Nova's network client obtains a token from keystone for authenitcation
    to the neutron services. When the timeout of keystone is set low, it
    creates a condition where some of the invocations to neutron start
    failing.

    This is because the neutron client detects that the token is expired,
    but during the re-authorization attempt fails due to not having enough
    information to properly reauthorize. Nova also assumes that within a
    given context, the token will always be valid. It does not perform the
    authorization check (and doing so would be double verification as the
    neutron client is already validating this).

    Since the neutron client has logic in place to re-authorize the token,
    this change set will pass thru the credentials to the neutron client
    for the re-authorization attempts. This change also updates the unit
    tests.

    Note: the change only will pass in the admin credential information
    if the context indicates that it is an admin.

    Change-Id: I2858562b180f3e058a2da9d67bef02af80927177
    Closes-Bug: #1241275

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/54736

[Matt Riedemann (mriedem)]

Abandoned the stable/havana cherry pick since this is required for this to be backported:

https://review.openstack.org/#/c/56174/

[Adrian Smith (adriansmith)]

Is there a workaround for this while the fix is being worked on? I tried deleting all tokens from the database and restarting everything but that doesn't seem to have worked.

[Matt Kennedy (mattkenn4545)]

I found a workaround. Logout... and to do that go to /horizon/auth/logout

This allowed me to log back in.

[kraig (kamador)]

Matt's workaround works for me on Horizon but it doesn't help my compute nodes throwing this error.

[piotrrr (piotrrr)]

Guys, is there any known workaround for this for Havana? Dashboard works fine for me but I get this error when I try to start an instance (using Neutron + monolithic Openvswitch plugin).

Does nova cache the old tokens somewhere somewhere on the disk? I ask, as restarting all the OpenStack services on each node and even manually deleting all the tokens from the Keystone's database does not seem the resolve the issue for me. Once I start seeing the issue the only solution seems to be reinitializing the whole OpenStack deployment on a clean drive. Reinitializing only the OpenStack DBs does not seem to do the trick, I actually have to reinstall the whole system from scratch. But then, after reinstalling, after some time I start seeing the issue again, and again...

[piotrrr (piotrrr)]

Solved.

It turns that in my case this error was caused by Neutron using a stale (cached locally) cacert.pem from the "/var/lib/neutron/keystone-signing/".

Flushing the content of that directory before reinstalling my OpenStack cluster (without re-installing the entire Operating System) solved my issue.

However I would argue that neutron should do a better job on reporting errors...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/53461
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=e49819caf95fc6985036231b1e5717f0ff7b6c61
Submitter: Jenkins
Branch: master

commit e49819caf95fc6985036231b1e5717f0ff7b6c61
Author: Drew Thorstensen <email address hidden>
Date: Wed Oct 23 16:41:45 2013 -0500

    New exception when auth_url is not specified

    Certain scenarios into the neutron client will not specify the
    auth_url. This is typically when a token is specified. However, when
    the token is expired the neutron client will attempt to refresh the
    token. Users of this may not have passed in all of the required
    information for this reauthentication to properly occur.

    This code fixes an error that occurs in this flow where the auth_url
    (which is None) is appended to another string. This results in a core
    Python error.

    The update will provide a more targetted error message specifying to
    the user that the auth_url needs to be specified. An associated unit
    test is also included to validate this behavior.

    Change-Id: I577ce0c009a9a281acdc238d290a22c5e561ff82
    Closes-Bug: #1241275

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54736
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bdc7519862299e43af55d7a7a221669de905d15c
Submitter: Jenkins
Branch: stable/havana

commit bdc7519862299e43af55d7a7a221669de905d15c
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 21 09:52:28 2013 -0500

    Pass thru credentials to allow re-authentication

    This is a backport of 4 fixes squashed into one because:

    1. They need to all be merged together as they build on each
       other as problems were found in each change after it was
       merged on master.
    2. The 3rd change won't pass Jenkins on it's own so it has
       to be squashed with the 4th and final change, so I'm just
       going to squash the entire topic branch together.

    The change bugs fixed and cherry pick commit hashes are left
    intact for each change.

    Closes-Bug: #1241275
    (cherry picked from commit 51e5f52e4cb60e266ccde71f205c91eb8c97b48b)

    Cache Neutron Client for Admin Scenarios

    Closes-Bug: #1250580
    (cherry picked from commit 85332012dede96fa6729026c2a90594ea0502ac5)

    Users with admin role in Nova should not re-auth with Neutron

    Closes-Bug: 1250763
    (cherry picked from commit 1c1371c78b990447aeaa4377b512f8887e6ff3ce)

    Fix Neutron Authentication for Metadata Service

    Closes-Bug: 1255577
    (cherry picked from commit 652620d12f3afe6845e41d9762b52d23f44fd557)

    ============

    Change-Id: I2858562b180f3e058a2da9d67bef02af80927177