AttributeError: 'NoneType' object has no attribute 'generation' in stable/ocata CI

[Matt Riedemann]

We've been seeing this intermittently:

http://logs.openstack.org/33/559033/2/gate/legacy-tempest-dsvm-neutron-full/ad0acef/logs/screen-n-cpu.txt.gz?level=TRACE#_2018-04-19_21_26_05_260

2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager [req-f2f68994-4031-465c-b1cc-0e995617b860 - -] Error updating resources for node ubuntu-xenial-inap-mtl01-0003624986.
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager Traceback (most recent call last):
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/compute/manager.py", line 6582, in update_available_resource_for_node
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 551, in update_available_resource
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager self._update_available_resource(context, resources)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager return f(*args, **kwargs)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 575, in _update_available_resource
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager self._init_compute_node(context, resources)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 438, in _init_compute_node
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager self.scheduler_client.update_resource_stats(cn)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 60, in update_resource_stats
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager self.reportclient.update_resource_stats(compute_node)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 37, in __run_method
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 642, in update_resource_stats
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager self._update_inventory(compute_node.uuid, inv_data)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 55, in wrapper
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager return f(self, *a, **k)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 563, in _update_inventory
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager if self._update_inventory_attempt(rp_uuid, inv_data):
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 471, in _update_inventory_attempt
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager cur_rp_gen = self._resource_providers[rp_uuid].generation
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager AttributeError: 'NoneType' object has no attribute 'generation'
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager

Since this change merged in stable/ocata:

https://review.openstack.org/#/c/525309/

We have a revert up for it: https://review.openstack.org/#/c/562739/

This bug is for tracking the revert and the failure in elastic-recheck.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttributeError%3A%20'NoneType'%20object%20has%20no%20attribute%20'generation'%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20build_branch%3A%5C%22stable%2Focata%5C%22&from=7d

[Matt Riedemann (mriedem)]

I know what the problem probably is, copied from bug 1767139 which was reported against the same change but in pike 16.1.1:

(9:59:34 AM) mriedem: set_inventory_for_provider -> _ensure_resource_provider -> _create_resource_provider -> safe_connect returns None because it can't talk to placement yet
(9:59:41 AM) mriedem: https://review.openstack.org/#/c/524618/2/nova/scheduler/client/report.py@516
(9:59:44 AM) mriedem: so we put None in the cache

[Matt Riedemann (mriedem)]

https://review.openstack.org/#/c/566096/ is the fix on master.