AttributeError: 'HostState' object has no attribute 'failed_builds'

[Matt Riedemann]

I'm not sure how this happens, but I hit this in a devstack environment with 35 nova-compute services using the fake virt driver, one scheduler worker and then trying to schedule a server:

Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts = self._get_sorted_hosts(spec_obj, hosts, num)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server spec_obj)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts, spec_obj)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weights = weigher.weigh_objects(weighed_objs, weighing_properties)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weight = self._weigh_object(obj.obj, weight_properties)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return host_state.failed_builds
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds'
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server

The HostState.failed_builds attribute gets set outside of __init__ in _update_from_compute_node here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261

And I'd expect the _update_from_compute_node to get called from update() which is called from here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787

Ah bingo if the compute node hasn't had resource usage updated yet we don't set the failed_builds attribute:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191

[Matt Riedemann (mriedem)]

The HostState.failed_builds change was introduced in Rocky and backported through to Pike:

https://review.opendev.org/#/q/I71c56fe770f8c3f66db97fa542fdfdf2b9865fb8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/668243

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/668252

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668243
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d540903463aa9b0cf69cefac7cc60e5b70e40a1c
Submitter: Zuul
Branch: master

commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/668282

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=725b37f515e4ad01e1f4491b6d8137ce1416f6d6
Submitter: Zuul
Branch: stable/stein

commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/668520

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668520
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5acbea506a137492511a762e454d785810365bd8
Submitter: Zuul
Branch: stable/rocky

commit 5acbea506a137492511a762e454d785810365bd8
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)
    (cherry picked from commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/668911

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668911
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9bc95675325498a7d30b67089e19b5b953d77e75
Submitter: Zuul
Branch: stable/queens

commit 9bc95675325498a7d30b67089e19b5b953d77e75
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)
    (cherry picked from commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6)
    (cherry picked from commit 5acbea506a137492511a762e454d785810365bd8)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/669538

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.2.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.12 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669538
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=61104f3974c61bffb500c268fb4e1d6e65c6b3e6
Submitter: Zuul
Branch: stable/pike

commit 61104f3974c61bffb500c268fb4e1d6e65c6b3e6
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)
    (cherry picked from commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6)
    (cherry picked from commit 5acbea506a137492511a762e454d785810365bd8)
    (cherry picked from commit 9bc95675325498a7d30b67089e19b5b953d77e75)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova pike-eol release.