DiscoveryFailure when trying to get resource providers from the scheduler report client

[Matt Riedemann]

I noticed this in a TripleO job:

http://logs.openstack.org/04/419604/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/5d95a8c/logs/nova/nova-compute.txt.gz#_2017-01-12_18_53_43_459

2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager [req-59098025-7c99-41b2-aaa9-0e5714770b3a - - - - -] Error updating resources for node centos-7-osic-cloud1-s3500-6631948.
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager Traceback (most recent call last):
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6537, in update_available_resource_for_node
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 540, in update_available_resource
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self._update_available_resource(context, resources)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return f(*args, **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 564, in _update_available_resource
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self._init_compute_node(context, resources)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 451, in _init_compute_node
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self.scheduler_client.update_resource_stats(self.compute_node)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 60, in update_resource_stats
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self.reportclient.update_resource_stats(compute_node)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 476, in update_resource_stats
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager compute_node.hypervisor_hostname)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 296, in _ensure_resource_provider
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager rp = self._get_resource_provider(uuid)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 53, in wrapper
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return f(self, *a, **k)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 209, in _get_resource_provider
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager resp = self.get("/resource_providers/%s" % uuid)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 174, in get
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager endpoint_filter=self.ks_filter, raise_exc=False)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 710, in get
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return wrapped(*args, **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 467, in request
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager auth_headers = self.get_auth_headers(auth)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 770, in get_auth_headers
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return auth.get_headers(self, **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager token = self.get_token(session)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return self.get_access(session).auth_token
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self.auth_ref = self.get_auth_ref(session)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 196, in get_auth_ref
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self._plugin = self._do_create_plugin(session)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager DiscoveryFailure: Could not determine a suitable URL for the plugin
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager

Looks like another type of keystoneauth1 exception that we're not handling in the safe_connect() decorator. At some point we'll probably just want to handle all KSA failures globally...

[Matt Riedemann (mriedem)]

In this case the keystone stuff was done after nova-compute was started:

http://logs.openstack.org/04/419604/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/5d95a8c/logs/puppet-20170112_185646.txt.gz#_2017-01-12_18_55_40

2017-01-12 18:55:40 +0000 /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_endpoint[RegionOne/placement::placement]/ensure (notice): created

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/422780

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422780
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4dbc0ddc740881a409085feeca2a1771a9ea95f3
Submitter: Jenkins
Branch: master

commit 4dbc0ddc740881a409085feeca2a1771a9ea95f3
Author: Roman Dobosz <email address hidden>
Date: Thu Jan 19 18:26:45 2017 +0100

    Prevent compute crash on discovery failure

    Keystoneauth1 may crash during discovery of URL for placement API. This is
    often happen on situation, when all services are started at once, compute will
    first be up while keystone still not. This patch fixes this by catching
    DiscoveyFailure as additional expected failures.

    Closes-Bug: #1656075

    Change-Id: I48aa6802286b408260b9d10f9d13860b44bd7d34

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0rc1 release candidate.