Service version check breaks FFU

[Dan Smith]

As reported on the mailing list:

http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026603.html

The service version check at startup can prevent FFUs from being possible without hacking the database. As implemented here:

https://review.opendev.org/c/openstack/nova/+/738482

We currently filter "forced down" computes from the check, but we should probably also eliminate those down long enough due to missed heartbeats (i.e. offline during the upgrade). However, a fast-moving FFU where everything is switched from an old container to a new one would easily still find computes that are considered "up" and effectively force a wait.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/826097

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/826097
Committed: https://opendev.org/openstack/nova/commit/7d2e4815892ddd523c21bf1785cc113981871998
Submitter: "Zuul (22348)"
Branch: master

commit 7d2e4815892ddd523c21bf1785cc113981871998
Author: Dan Smith <email address hidden>
Date: Fri Jan 21 12:51:35 2022 -0800

    Add service version check workaround for FFU

    We recently added a hard failure to nova service startup for the case
    where computes were more than one version old (as indicated by their
    service record). This helps to prevent starting up new control
    services when a very old compute is still running. However, during an
    FFU, control services that have skipped multiple versions will be
    started and find the older compute records (which could not be updated
    yet due to their reliance on the control services being up) and refuse
    to start. This creates a cross-dependency which is not resolvable
    without hacking the database.

    This patch adds a workaround flag to allow turning that hard fail into
    a warning to proceed past the issue. This less-than-ideal solution
    is simple and backportable, but perhaps a better solution can be
    implemented for the future.

    Related-Bug: #1958883

    Change-Id: Iddbc9b2a13f19cea9a996aeadfe891f4ef3b0264

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/831174

[melanie witt (melwitt)]

Marking this as Confirmed with Medium priority given that a workaround has been implemented and is now available.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/831174
Committed: https://opendev.org/openstack/nova/commit/3a5f8924ff42ce3f691f33abf2c2daee88e90fe5
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 3a5f8924ff42ce3f691f33abf2c2daee88e90fe5
Author: Dan Smith <email address hidden>
Date: Fri Jan 21 12:51:35 2022 -0800

    Add service version check workaround for FFU

    We recently added a hard failure to nova service startup for the case
    where computes were more than one version old (as indicated by their
    service record). This helps to prevent starting up new control
    services when a very old compute is still running. However, during an
    FFU, control services that have skipped multiple versions will be
    started and find the older compute records (which could not be updated
    yet due to their reliance on the control services being up) and refuse
    to start. This creates a cross-dependency which is not resolvable
    without hacking the database.

    This patch adds a workaround flag to allow turning that hard fail into
    a warning to proceed past the issue. This less-than-ideal solution
    is simple and backportable, but perhaps a better solution can be
    implemented for the future.

    Related-Bug: #1958883

    Change-Id: Iddbc9b2a13f19cea9a996aeadfe891f4ef3b0264
    (cherry picked from commit 7d2e4815892ddd523c21bf1785cc113981871998)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/844202

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/844202
Committed: https://opendev.org/openstack/nova/commit/e8b079a91ee723d0dc45e3d8b80f4efa2c1ce34d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit e8b079a91ee723d0dc45e3d8b80f4efa2c1ce34d
Author: Dan Smith <email address hidden>
Date: Fri Jan 21 12:51:35 2022 -0800

    Add service version check workaround for FFU

    We recently added a hard failure to nova service startup for the case
    where computes were more than one version old (as indicated by their
    service record). This helps to prevent starting up new control
    services when a very old compute is still running. However, during an
    FFU, control services that have skipped multiple versions will be
    started and find the older compute records (which could not be updated
    yet due to their reliance on the control services being up) and refuse
    to start. This creates a cross-dependency which is not resolvable
    without hacking the database.

    This patch adds a workaround flag to allow turning that hard fail into
    a warning to proceed past the issue. This less-than-ideal solution
    is simple and backportable, but perhaps a better solution can be
    implemented for the future.

    Related-Bug: #1958883

    Change-Id: Iddbc9b2a13f19cea9a996aeadfe891f4ef3b0264
    (cherry picked from commit 7d2e4815892ddd523c21bf1785cc113981871998)
    (cherry picked from commit 3a5f8924ff42ce3f691f33abf2c2daee88e90fe5)