AttributeError: 'NoneType' object has no attribute 'get_token'

[Matt Riedemann]

Someone reported this in nova IRC:

http://paste.openstack.org/show/626966/

They were doing a snapshot of an instance with an NFS-backed volume and using the nova service user configuration in the [service_user] section of nova.conf, however, it looks like their configuration was incomplete which resulted in this:

Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver [req-3278cb0b-5c22-4874-905c-7f19ef735188 req-9e15496a-ae04-49ad-adbd-dd65b1bc9aaf admin admin] Failed to send updated snapshot status to volume service.: AttributeError: 'NoneType' object has no attribute 'get_token'
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1952, in _volume_snapshot_update_status
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver status)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/opt/stack/nova/nova/volume/cinder.py", line 241, in wrapper
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver res = method(self, ctx, *args, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/opt/stack/nova/nova/volume/cinder.py", line 291, in wrapper
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver res = method(self, ctx, snapshot_id, *args, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/opt/stack/nova/nova/volume/cinder.py", line 540, in update_snapshot_status
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver 'progress': '90%'}
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/v3/volume_snapshots.py", line 166, in update_snapshot_status
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver base.getid(snapshot), update_dict)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/v3/volume_snapshots.py", line 161, in _action
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver resp, body = self.api.client.post(url, body=body)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 202, in post
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver return self._cs_request(url, 'POST', **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 190, in _cs_request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver return self.request(url, method, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 173, in request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver return self.session.request(url, method, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver auth_headers = self.get_auth_headers(auth)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver return auth.get_headers(self, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver token = self.service_auth.get_token(session, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver AttributeError: 'NoneType' object has no attribute 'get_token'
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver

Looking at the KSA code, this can happen if auth_type isn't configured:

https://github.com/openstack/keystoneauth/blob/3.2.0/keystoneauth1/loading/conf.py#L124

But nova doesn't handle this and it results in a NoneType error later.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/521947

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/526468

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/521947
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cff8b088370c9190e402ece1a9ff48eae27677f7
Submitter: Zuul
Branch: master

commit cff8b088370c9190e402ece1a9ff48eae27677f7
Author: Matt Riedemann <email address hidden>
Date: Tue Nov 21 12:01:33 2017 -0500

    Fix NoneType error when [service_user] is misconfigured

    If the [service_user]/send_service_user_token option is
    set to True but the actual auth options are incomplete,
    like missing to set the auth_type option, we eventually
    fail to re-auth with keystone due to a NoneType error
    in keystoneauth1.

    We can detect this issue because load_auth_from_conf_options
    will return None and we can just log a warning and continue
    as if the service user was never configured in the first place.

    Co-Authored-By: Eric Fried <email address hidden>
    Change-Id: I0a96c835d620307f1ab34736ba42c2deb1321a23
    Closes-Bug: #1733642

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526468
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6522d383320f5fda27635ee1ea86ea381c3858a1
Submitter: Zuul
Branch: stable/pike

commit 6522d383320f5fda27635ee1ea86ea381c3858a1
Author: Matt Riedemann <email address hidden>
Date: Tue Nov 21 12:01:33 2017 -0500

    Fix NoneType error when [service_user] is misconfigured

    If the [service_user]/send_service_user_token option is
    set to True but the actual auth options are incomplete,
    like missing to set the auth_type option, we eventually
    fail to re-auth with keystone due to a NoneType error
    in keystoneauth1.

    We can detect this issue because load_auth_from_conf_options
    will return None and we can just log a warning and continue
    as if the service user was never configured in the first place.

    Co-Authored-By: Eric Fried <email address hidden>
    Change-Id: I0a96c835d620307f1ab34736ba42c2deb1321a23
    Closes-Bug: #1733642
    (cherry picked from commit cff8b088370c9190e402ece1a9ff48eae27677f7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.0 release.