OpenStack Compute (nova)

resource tracking fails with Unauthorized from placement API (keystone v3)

Bug #1654107 reported by Matt Riedemann on 2017-01-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Matt Riedemann
	Newton	Fix Committed	Medium	Matt Riedemann

Bug Description

Seen here:

http://logs.openstack.org/65/416765/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-ubuntu-xenial-nv/faf1363/logs/screen-n-cpu.txt.gz#_2017-01-04_23_15_06_674

2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager [req-832f822c-02b3-453b-b80b-61ad7f06f401 - -] Error updating resources for node ubuntu-xenial-osic-cloud1-disk-6483179.
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager Traceback (most recent call last):
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/compute/manager.py", line 6537, in update_available_resource_for_node
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 540, in update_available_resource
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager self._update_available_resource(context, resources)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return f(*args, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 564, in _update_available_resource
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager self._init_compute_node(context, resources)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 451, in _init_compute_node
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager self.scheduler_client.update_resource_stats(self.compute_node)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 60, in update_resource_stats
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager self.reportclient.update_resource_stats(compute_node)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 462, in update_resource_stats
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager compute_node.hypervisor_hostname)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 282, in _ensure_resource_provider
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager rp = self._get_resource_provider(uuid)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 47, in wrapper
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return f(self, *a, **k)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 195, in _get_resource_provider
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager resp = self.get("/resource_providers/%s" % uuid)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 160, in get
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager endpoint_filter=self.ks_filter, raise_exc=False)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 710, in get
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return wrapped(*args, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager auth_headers = self.get_auth_headers(auth)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 770, in get_auth_headers
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return auth.get_headers(self, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager token = self.get_token(session)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return self.get_access(session).auth_token
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager self.auth_ref = self.get_auth_ref(session)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return self._plugin.get_auth_ref(session, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager authenticated=False, log=False, **rkwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 718, in post
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return self.request(url, 'POST', **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return wrapped(*args, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 607, in request
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager raise exceptions.from_response(resp, method, url)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-38ce6df3-1dca-4b66-a680-d2691bb4955f)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager

The scheduler report client's safe_connect method isn't handle Unauthorized exceptions, which are happening in the keystone v3 jobs now with the placement API, at least until this devstack patch is merged:

https://review.openstack.org/#/c/416599/

We handle Unauthorized in the new nova-status upgrade check command but not the safe_connect() decorator, which we need to do.

Tags:

Revision history for this message

Matt Riedemann (mriedem) wrote on 2017-01-05:

I've marked this high severity because what happens when we create the compute node record but fail to set free_disk_gb is the scheduler fails to update the HostState object and then we can't schedule anything.

Changed in nova:
assignee:	nobody → Matt Riedemann (mriedem)
importance:	Undecided → High
status:	New → Triaged
tags:	added: compute placement scheduler

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-05: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/416810

Changed in nova:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-05: Fix merged to nova (master)

Reviewed: https://review.openstack.org/416810
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=180e6340a595ec047c59365465f36fed7a669ec3
Submitter: Jenkins
Branch: master

commit 180e6340a595ec047c59365465f36fed7a669ec3
Author: Matt Riedemann <email address hidden>
Date: Wed Jan 4 21:11:10 2017 -0500

Handle Unauthorized exception in report client's safe_connect()

If nova is misconfigured for placement auth, as seen in:

I6351c1b2ca7ea4448e13eb87455bff4058df4fa7

    Keystone returns an Unauthorized 401 exception which the
    scheduler report client's safe_connect() method wasn't handling
    and disabling future attempts to talk to the placement service.

    This patch adds that check to the report client. Note also that
    we already check for this in the "nova-status upgrade check"
    command.

Change-Id: Id52de64528383de70fc5a8951dde764ba34e369d
Closes-Bug: #1654107

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-12: Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/419216

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-13: Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/419216
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9fb203f17e358760e9e40ef55ee79965343581f8
Submitter: Jenkins
Branch: stable/newton

commit 9fb203f17e358760e9e40ef55ee79965343581f8
Author: Matt Riedemann <email address hidden>
Date: Wed Jan 4 21:11:10 2017 -0500

Handle Unauthorized exception in report client's safe_connect()

If nova is misconfigured for placement auth, as seen in:

I6351c1b2ca7ea4448e13eb87455bff4058df4fa7

    Keystone returns an Unauthorized 401 exception which the
    scheduler report client's safe_connect() method wasn't handling
    and disabling future attempts to talk to the placement service.

    This patch adds that check to the report client. Note also that
    we already check for this in the "nova-status upgrade check"
    command.

    Change-Id: Id52de64528383de70fc5a8951dde764ba34e369d
    Closes-Bug: #1654107
    (cherry picked from commit 180e6340a595ec047c59365465f36fed7a669ec3)