TypeError: unsupported operand type(s) for *: 'int' and 'NoneType' in CPUWeigher

[Matt Riedemann]

This is similar to bug 1834691 in that the CPUWeigher can hit this error:

Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server [None req-df4ac006-fddf-4e6f-9c5e-1c352be60d9b admin admin] Exception during message handling
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server hosts = self._get_sorted_hosts(spec_obj, hosts, num)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server spec_obj)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/host_manager.py", line 601, in get_weighed_hosts
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server hosts, spec_obj)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server weights = weigher.weigh_objects(weighed_objs, weighing_properties)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server weight = self._weigh_object(obj.obj, weight_properties)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/weights/cpu.py", line 44, in _weigh_object
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server host_state.vcpus_total * host_state.cpu_allocation_ratio -
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server

If the HostState.cpu_allocation_ratio is not set here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L256

Because we return early here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L193

[Matt Riedemann (mriedem)]

This goes back to Rocky when the CPUWeigher was added:

https://review.opendev.org/#/c/379525/

[Matt Riedemann (mriedem)]

Patch: https://review.opendev.org/#/c/668252/

[Matt Riedemann (mriedem)]

I've done debugging to figure out the root cause of this and those notes are here:

https://review.opendev.org/#/c/668252/1/nova/scheduler/host_manager.py@626

tl;dr we fail to save free_disk_gb to the compute node, but later create the resource provider with inventory, so the scheduler gets the provider from placement but the node doesn't have free_disk_gb so the HostState isn't updated and gets run through the filters/weighers anyway and blows up in CPUWeigher.

[Matt Riedemann (mriedem)]

I've reported bug 1834712 for the issue in the ResourceTracker mentioned in comment 3.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/668252
Reason: I'm no longer working on this.