HA: moving VIP away during minor update can leave some VIP stopped

[Damien Ciabrini]

in a HA control plane, during minor updates, controller nodes are stopped and updated one after the other. To minimise service outage, we pro-actively move the VIP away from the node we're updating.

There is a race between when we determine the VIP to move and when we execute the move action, that can result in the VIP to be moved twice, and that can leave a ban flag in the cluster.

# seen while updating controller-0. Note that the ban of the ovn VIP is performed for controller-2 instead of controller-0

[root@messaging-0 pacemaker]# grep cli-ban /var/log/pacemaker/pacemaker.log
Mar 25 00:32:44 messaging-0.redhat.local pacemaker-based [2019] (cib_perform_op) info: ++ /cib/configuration/constraints: <rsc_location id="cli-ban-ovn-dbs-bundle-on-controller-0" rsc="ovn-dbs-bundle" role="Started" node="controller
-0" score="-INFINITY"/>
Mar 25 00:32:47 messaging-0.redhat.local pacemaker-based [2019] (cib_perform_op) info: ++ /cib/configuration/constraints: <rsc_location id="cli-ban-ip-10.0.0.150-on-controller-0" rsc="ip-10.0.0.150" role="Started" node="controller-$
" score="-INFINITY"/>
Mar 25 00:33:25 messaging-0.redhat.local pacemaker-based [108231] (cib_perform_op) info: -- /cib/configuration/constraints/rsc_location[@id='cli-ban-ip-10.0.0.150-on-controller-0']
Mar 25 00:33:25 messaging-0.redhat.local pacemaker-based [108231] (cib_perform_op) info: + /cib/configuration/constraints/rsc_location[@id='cli-ban-ovn-dbs-bundle-on-controller-0']: @node=controller-0, @rsc=ovn-dbs-bundle
Mar 25 00:33:25 messaging-0.redhat.local pacemaker-based [108231] (cib_perform_op) info: ++ /cib/configuration/constraints: <rsc_location id="cli-ban-ip-172.17.4.150-on-controller-0" node="controller-0" role="Started" rsc="ip-172.17.$
.150" score="-INFINITY"/>
Mar 25 00:33:25 messaging-0.redhat.local pacemaker-based [108231] (cib_perform_op) info: ++ /cib/configuration/constraints: <rsc_location id="cli-ban-ip-172.17.1.40-on-controller-2" node="controller-2" role="Started" rsc="ip-172.17.1$
40" score="-INFINITY"/>

When that happens, the VIP can no longer be hosted on the node which is left with the ban flag. Under some circumstances, this can lead to the cluster hosting the ovn_dbs on the node with the left over ban, and prevent the VIP from moving with it, causing an outage of the ovn service.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/785205
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/de98fdb208dac1cf01d474c4363f6e2177e22c7e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit de98fdb208dac1cf01d474c4363f6e2177e22c7e
Author: Damien Ciabrini <email address hidden>
Date: Thu Mar 25 10:23:48 2021 +0100

    HA: fix race when moving VIP during minor update

    During a minor update of a controller node, we list the
    VIP hosted on the node and we force them to move away
    with a pcs command.

    The way we call the pcs command is racy, so change the
    action to make sure the VIP move is performed only if
    the VIP is still hosted on the node when we run the action.

    Change-Id: Id379f4fe1668d01fdd5f91b46e2f75d7cdb577ae
    Closes-Bug: #1921351
    (cherry picked from commit 3da1e7661036a73c9f222bcdcc489912502d23cb)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/788601

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/788601
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/eafdee6aedaab6d210cd29970b08932f48221495
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit eafdee6aedaab6d210cd29970b08932f48221495
Author: Damien Ciabrini <email address hidden>
Date: Thu Mar 25 10:23:48 2021 +0100

    HA: fix race when moving VIP during minor update

    During a minor update of a controller node, we list the
    VIP hosted on the node and we force them to move away
    with a pcs command.

    The way we call the pcs command is racy, so change the
    action to make sure the VIP move is performed only if
    the VIP is still hosted on the node when we run the action.

    Change-Id: Id379f4fe1668d01fdd5f91b46e2f75d7cdb577ae
    Closes-Bug: #1921351
    (cherry picked from commit 3da1e7661036a73c9f222bcdcc489912502d23cb)
    (cherry picked from commit de98fdb208dac1cf01d474c4363f6e2177e22c7e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/782931
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/2416eb3b145bb2d0b2e93ec2bbf1ebf0f95abadb
Submitter: "Zuul (22348)"
Branch: stable/train

commit 2416eb3b145bb2d0b2e93ec2bbf1ebf0f95abadb
Author: Damien Ciabrini <email address hidden>
Date: Thu Mar 25 10:23:48 2021 +0100

    HA: fix race when moving VIP during minor update

    During a minor update of a controller node, we list the
    VIP hosted on the node and we force them to move away
    with a pcs command.

    The way we call the pcs command is racy, so change the
    action to make sure the VIP move is performed only if
    the VIP is still hosted on the node when we run the action.

    Change-Id: Id379f4fe1668d01fdd5f91b46e2f75d7cdb577ae
    Closes-Bug: #1921351
    (cherry picked from commit 3da1e7661036a73c9f222bcdcc489912502d23cb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.4.4 release.