AttributeError: 'HostState' object has no attribute 'failed_builds'

Bug #1834691 reported by Matt Riedemann on 2019-06-28
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Matt Riedemann
Pike
Medium
Matt Riedemann
Queens
Medium
Matt Riedemann
Rocky
Medium
Matt Riedemann
Stein
Medium
Matt Riedemann

Bug Description

I'm not sure how this happens, but I hit this in a devstack environment with 35 nova-compute services using the fake virt driver, one scheduler worker and then trying to schedule a server:

Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts = self._get_sorted_hosts(spec_obj, hosts, num)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server spec_obj)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts, spec_obj)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weights = weigher.weigh_objects(weighed_objs, weighing_properties)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weight = self._weigh_object(obj.obj, weight_properties)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return host_state.failed_builds
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds'
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server

The HostState.failed_builds attribute gets set outside of __init__ in _update_from_compute_node here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261

And I'd expect the _update_from_compute_node to get called from update() which is called from here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787

Ah bingo if the compute node hasn't had resource usage updated yet we don't set the failed_builds attribute:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191

Matt Riedemann (mriedem) wrote :

The HostState.failed_builds change was introduced in Rocky and backported through to Pike:

https://review.opendev.org/#/q/I71c56fe770f8c3f66db97fa542fdfdf2b9865fb8

Changed in nova:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Matt Riedemann (mriedem)
tags: added: scheduler

Fix proposed to branch: master
Review: https://review.opendev.org/668243

Changed in nova:
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.opendev.org/668252

Reviewed: https://review.opendev.org/668243
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d540903463aa9b0cf69cefac7cc60e5b70e40a1c
Submitter: Zuul
Branch: master

commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/668282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=725b37f515e4ad01e1f4491b6d8137ce1416f6d6
Submitter: Zuul
Branch: stable/stein

commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)

Reviewed: https://review.opendev.org/668520
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5acbea506a137492511a762e454d785810365bd8
Submitter: Zuul
Branch: stable/rocky

commit 5acbea506a137492511a762e454d785810365bd8
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)
    (cherry picked from commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6)

Reviewed: https://review.opendev.org/668911
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9bc95675325498a7d30b67089e19b5b953d77e75
Submitter: Zuul
Branch: stable/queens

commit 9bc95675325498a7d30b67089e19b5b953d77e75
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)
    (cherry picked from commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6)
    (cherry picked from commit 5acbea506a137492511a762e454d785810365bd8)

This issue was fixed in the openstack/nova 19.0.2 release.

This issue was fixed in the openstack/nova 18.2.2 release.

This issue was fixed in the openstack/nova 17.0.12 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers