containerized galera bootstrap randomly fails

[Damien Ciabrini]

As stated in https://bugs.launchpad.net/tripleo/+bug/1714057, old version of mariadb (5.5.x) don not considier the command line flag --wsrep-on=OFF and have the WSREP replication started as soon as wsrep-provider is configured in galera.cnf.

The proposed fix is to unset explicitly the wsrep provider during bootstrap when we set the users' credentials. However the patch only partially fixed the issue because the kolla bootstrap script also starts the mysql server to set up the root password. In that case, failure happended randomly with the following logs from docker logs:

[...]
Set root password? [Y/n] y
New password:
Re-enter new password:
ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Password update failed!
Cleaning up...

This shows that WSREP replication is still enabled when kolla start mysql, while it shouldn't.

The wsrep provider should be disabled during all the bootstrap run.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/504049

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/504049
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b0f50db80b10e9cd6263c4d6b3ca8dd818b658ba
Submitter: Jenkins
Branch: master

commit b0f50db80b10e9cd6263c4d6b3ca8dd818b658ba
Author: Damien Ciabrini <email address hidden>
Date: Thu Sep 14 14:49:04 2017 +0200

    Disable all uses of wsrep-provider in mysql_bootstrap container

    During the bootstrap of the mariadb database, galera replication
    must be disabled while the users credentials are being set up. This
    is done by setting wsrep-provider=none when starting mysqld_safe.

    Icf67fd2fbf520e8a62405b4d49e8d5169ff3925b already disabled it
    when the clustercheck credentials are being set up, but Kolla also
    start a temporary server for setting up the root password.

    Disable the setting directly at the end of the mysql.cnf in the
    running container. That way, the default setting from galera.cnf will
    be overriden, all mysqld_safe calls will disable WSREP and the setting
    will stay ephemeral.

    Change-Id: If14e22992b46a35a05a16a9db5ecb360ea13df8f
    Closes-Bug: #1717250

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/505663

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505663
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ae4525574181865127ce2f2086b7729748b0f2fc
Submitter: Jenkins
Branch: stable/pike

commit ae4525574181865127ce2f2086b7729748b0f2fc
Author: Damien Ciabrini <email address hidden>
Date: Thu Sep 14 14:49:04 2017 +0200

    Disable all uses of wsrep-provider in mysql_bootstrap container

    During the bootstrap of the mariadb database, galera replication
    must be disabled while the users credentials are being set up. This
    is done by setting wsrep-provider=none when starting mysqld_safe.

    Icf67fd2fbf520e8a62405b4d49e8d5169ff3925b already disabled it
    when the clustercheck credentials are being set up, but Kolla also
    start a temporary server for setting up the root password.

    Disable the setting directly at the end of the mysql.cnf in the
    running container. That way, the default setting from galera.cnf will
    be overriden, all mysqld_safe calls will disable WSREP and the setting
    will stay ephemeral.

    Change-Id: If14e22992b46a35a05a16a9db5ecb360ea13df8f
    Closes-Bug: #1717250
    (cherry picked from commit b0f50db80b10e9cd6263c4d6b3ca8dd818b658ba)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.