Late affinity check failre counted as failed build

[Balazs Gibizer]

The late anti-affinity checks runs in the compute manager to avoid parallel scheduling requests to invalidate the anti-affinity server group policy. When the check fails the instance is re-scheduled. However this failure counted as a real instance boot failure of the compute host[1][2][3] and can lead to de-prioritization of the compute host in the scheduler via BuildFailureWeigher[4]. As the late anti-affinity check is does not indicate any fault of the compute host itself it should not be counted towards the build failure counter.

[1] https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/manager.py#L2496
[2] https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/manager.py#L1808
[3] https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/manager.py#L2265
[4] https://docs.openstack.org/nova/latest/configuration/config.html#compute.consecutive_build_service_disable_threshold

[sean mooney (sean-k-mooney)]

this is a bug yes and a trivially fixable one.

we just need to add a new exception that inherits from the existing build failed exception and skip incrementing the fail
count if its for affinity.

setting this to low as the impact is low and this is more an optimisation then a bug.

[Yusuke Okada (yusokada)]

Hi Amit,

How is the bug fix going?
If you are busy to fix this issue, I can propose fix for this issue.
Could you let me know the current status?

[Amit Uniyal (auniyal)]

Hi yusokada,

sure, please reassign it to yourself.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/873216

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/885343

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/885344

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/885345

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/885347

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/885348

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/885349

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/885353

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/885355

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/873216
Committed: https://opendev.org/openstack/nova/commit/56d320a203a13f262a2e94e491af222032e453d3
Submitter: "Zuul (22348)"
Branch: master

commit 56d320a203a13f262a2e94e491af222032e453d3
Author: Yusuke Okada <email address hidden>
Date: Wed Feb 8 22:10:31 2023 -0500

    Fix failed count for anti-affinity check

    The late anti-affinity check runs in the compute manager to avoid
    parallel scheduling requests to invalidate the anti-affinity server
    group policy. When the check fails the instance is re-scheduled.
    However this failure counted as a real instance boot failure of the
    compute host and can lead to de-prioritization of the compute host
    in the scheduler via BuildFailureWeigher. As the late anti-affinity
    check does not indicate any fault of the compute host itself it
    should not be counted towards the build failure counter.
    This patch adds new build results to handle this case.

    Closes-Bug: #1996732
    Change-Id: I2ba035c09ace20e9835d9d12a5c5bee17d616718
    Signed-off-by: Yusuke Okada <email address hidden>

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/885355
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 28.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/885344
Committed: https://opendev.org/openstack/nova/commit/2f1d65774fbcf5c25c4ba53583b6a802a03f4c4d
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 2f1d65774fbcf5c25c4ba53583b6a802a03f4c4d
Author: Yusuke Okada <email address hidden>
Date: Wed Feb 8 22:10:31 2023 -0500

    Fix failed count for anti-affinity check

    The late anti-affinity check runs in the compute manager to avoid
    parallel scheduling requests to invalidate the anti-affinity server
    group policy. When the check fails the instance is re-scheduled.
    However this failure counted as a real instance boot failure of the
    compute host and can lead to de-prioritization of the compute host
    in the scheduler via BuildFailureWeigher. As the late anti-affinity
    check does not indicate any fault of the compute host itself it
    should not be counted towards the build failure counter.
    This patch adds new build results to handle this case.

    Closes-Bug: #1996732
    Change-Id: I2ba035c09ace20e9835d9d12a5c5bee17d616718
    Signed-off-by: Yusuke Okada <email address hidden>
    (cherry picked from commit 56d320a203a13f262a2e94e491af222032e453d3)
    (cherry picked from commit 1b56714e9119ab4152e6f33985a499b2d83a491b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/885345
Committed: https://opendev.org/openstack/nova/commit/cd0403dd3b1099bd13da503500a50249db8e49ea
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit cd0403dd3b1099bd13da503500a50249db8e49ea
Author: Yusuke Okada <email address hidden>
Date: Wed Feb 8 22:10:31 2023 -0500

    Fix failed count for anti-affinity check

    The late anti-affinity check runs in the compute manager to avoid
    parallel scheduling requests to invalidate the anti-affinity server
    group policy. When the check fails the instance is re-scheduled.
    However this failure counted as a real instance boot failure of the
    compute host and can lead to de-prioritization of the compute host
    in the scheduler via BuildFailureWeigher. As the late anti-affinity
    check does not indicate any fault of the compute host itself it
    should not be counted towards the build failure counter.
    This patch adds new build results to handle this case.

    Closes-Bug: #1996732
    Change-Id: I2ba035c09ace20e9835d9d12a5c5bee17d616718
    Signed-off-by: Yusuke Okada <email address hidden>
    (cherry picked from commit 56d320a203a13f262a2e94e491af222032e453d3)
    (cherry picked from commit 1b56714e9119ab4152e6f33985a499b2d83a491b)
    (cherry picked from commit 2f1d65774fbcf5c25c4ba53583b6a802a03f4c4d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 25.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 27.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.2.1 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/885353
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/885349
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/885348
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/885347
Reason: stable/xena branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/xena if you want to further work on this patch.