MariaDB deployment does not respect quorum and may break cluster

[Radosław Piliszek]

Current code does not wait for MariaDB to recalculate quorum when starting and stopping MariaDB containers. This may lead to WSREP issues (and failures) and in the worst case require a recovery.

This most notably affects upgrades of MariaDB clusters but also its reconfigurations (due to user action or our change). Hence high importance.

[Radosław Piliszek (yoctozepto)]

The fix is to separate bootstrap, deployment of new members and restart of old members, ensuring restart uses 3 phases (aka batches) not to break quorum.

[Radosław Piliszek (yoctozepto)]

And also wait for status Synced for already existing members.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701010
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=9f14ad651a9e6516d02c90d9eb0ec4b7a4702e7e
Submitter: Zuul
Branch: master

commit 9f14ad651a9e6516d02c90d9eb0ec4b7a4702e7e
Author: Radosław Piliszek <email address hidden>
Date: Fri Jan 3 11:20:00 2020 +0100

    Fix multiple issues with MariaDB handling

    These affected both deploy (and reconfigure) and upgrade
    resulting in WSREP issues, failed deploys or need to
    recover the cluster.

    This patch makes sure k-a does not abruptly terminate
    nodes to break cluster.
    This is achieved by cleaner separation between stages
    (bootstrap, restart current, deploy new) and 3 phases
    for restarts (to keep the quorum).

    Upgrade actions, which operate on a healthy cluster,
    went to its section.

    Service restart was refactored.

    We no longer rely on the master/slave distinction as
    all nodes are masters in Galera.

    Closes-bug: #1857908
    Closes-bug: #1859145
    Change-Id: I83600c69141714fc412df0976f49019a857655f5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/705414

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/706078

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705414
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=8acf5c132df02002e05a17c1754f5d79143a8d75
Submitter: Zuul
Branch: stable/train

commit 8acf5c132df02002e05a17c1754f5d79143a8d75
Author: Radosław Piliszek <email address hidden>
Date: Fri Jan 3 11:20:00 2020 +0100

    Fix multiple issues with MariaDB handling

    These affected both deploy (and reconfigure) and upgrade
    resulting in WSREP issues, failed deploys or need to
    recover the cluster.

    This patch makes sure k-a does not abruptly terminate
    nodes to break cluster.
    This is achieved by cleaner separation between stages
    (bootstrap, restart current, deploy new) and 3 phases
    for restarts (to keep the quorum).

    Upgrade actions, which operate on a healthy cluster,
    went to its section.

    Service restart was refactored.

    We no longer rely on the master/slave distinction as
    all nodes are masters in Galera.

    Backport includes also the:
    Followup on MariaDB handling fixes

    This fixes issues reported by Mark:
    - possible failure with 4-node cluster (however unlikely)
    - failure to stop all nodes from progressing when conditions are
      not valid (due to: "any_errors_fatal: False")

    Closes-bug: #1857908
    Closes-bug: #1859145
    Change-Id: I83600c69141714fc412df0976f49019a857655f5
    (cherry picked from commit 9f14ad651a9e6516d02c90d9eb0ec4b7a4702e7e)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/713501

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/713501
Reason: no time to pursue, rocky already em and code diverged much - could have different characteristics regarding stability

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/706078
Reason: not pursuing, stein is oldie ;-)