TypeError: unsupported operand type(s) for *: 'int' and 'NoneType' in CPUWeigher

Bug #1834694 reported by Matt Riedemann on 2019-06-28
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Matt Riedemann
Rocky
Medium
Unassigned
Stein
Medium
Unassigned

Bug Description

This is similar to bug 1834691 in that the CPUWeigher can hit this error:

Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server [None req-df4ac006-fddf-4e6f-9c5e-1c352be60d9b admin admin] Exception during message handling
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server hosts = self._get_sorted_hosts(spec_obj, hosts, num)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server spec_obj)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/host_manager.py", line 601, in get_weighed_hosts
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server hosts, spec_obj)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server weights = weigher.weigh_objects(weighed_objs, weighing_properties)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server weight = self._weigh_object(obj.obj, weight_properties)
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/weights/cpu.py", line 44, in _weigh_object
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server host_state.vcpus_total * host_state.cpu_allocation_ratio -
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'
Jun 28 20:26:07 devstack nova-scheduler[24099]: ERROR oslo_messaging.rpc.server

If the HostState.cpu_allocation_ratio is not set here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L256

Because we return early here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L193

Matt Riedemann (mriedem) wrote :

This goes back to Rocky when the CPUWeigher was added:

https://review.opendev.org/#/c/379525/

tags: added: scheduler
Changed in nova:
status: New → Triaged
importance: Undecided → High
Matt Riedemann (mriedem) on 2019-06-28
Changed in nova:
importance: High → Medium
Matt Riedemann (mriedem) wrote :

I've done debugging to figure out the root cause of this and those notes are here:

https://review.opendev.org/#/c/668252/1/nova/scheduler/host_manager.py@626

tl;dr we fail to save free_disk_gb to the compute node, but later create the resource provider with inventory, so the scheduler gets the provider from placement but the node doesn't have free_disk_gb so the HostState isn't updated and gets run through the filters/weighers anyway and blows up in CPUWeigher.

Changed in nova:
status: Triaged → In Progress
Matt Riedemann (mriedem) wrote :

I've reported bug 1834712 for the issue in the ResourceTracker mentioned in comment 3.

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers