HA scenario is broken: galera bootstrap fails

[Emilien Macchi]

This new error happens when deploying a controller with Pacemaker and MySQL Galera:

Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql/Exec[galera-set-root-password]/returns: change from notrun to 0 failed

After some investigation, it looks like it's related to the latest update of Galera package.

[Emilien Macchi (emilienm)]

See postci logs: http://logs.openstack.org/54/367754/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/691ed85/logs/postci.txt.gz#_2016-09-12_12_44_19_000

More logs for the controller: http://logs.openstack.org/54/367754/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/691ed85/logs/overcloud-controller-0/var/log/

[Emilien Macchi (emilienm)]

Here's a package difference between a very recent working job and a failing job:
https://www.diffchecker.com/t6HnUxle

[Damien Ciabrini (dciabrin)]

Looking at the /var/log/mysql.log, on this particular run, controller0 was selected as galera's bootstrap node, and succesfully started.

The other two nodes controller1 and controller2 acted as "joiners" and none of them could connect to the bootstrap node to join the galera cluster.

Investigating why this could be the case

[Ryan Hallisey (rthall14)]

I'm also seeing this.

[Emilien Macchi (emilienm)]

so tripleo CI will be fixed by removing EPEL, so we can download the version we know working from RDO repository: https://review.openstack.org/#/c/347499/

I'm closing the bug, but feel free to re-open a new one if you think something is wrong in our Galera deployment manifests.

[Damien Ciabrini (dciabrin)]

Update as to why the bootstrap of Galera cluster was failing:

We found out with Mike Bayer that version 25.3.12 of the galera WSREP provider behaves differently than version 25.3.5 when processing the provider option:

   wsrep_provider_options = gmcast.listen_addr=tcp://[172.16.2.15]:4567

This value is being set by t-h-t and has square brackets in the address to tell galera to listen to either an IPv4 or IPv6 address at startup.

In newer version of the galera provider, joining nodes won't connect to the cluster if the listen address is an IPv4 and contains brackets.

Removing brackets makes the bootstrap work again for IPv4 use cases.
Checks for IPv6 would need to be done to check whether brackets are still working as expected.

[Ben Nemec (bnemec)]

We're going to merge https://review.openstack.org/#/c/347499/ to unblock master ci, but we need to discuss a solution for the stable branches since there are deployments out there (including our ci cloud) that already have epel enabled and will break if they update to this newer package. If there's a reasonably backportable way to remove the brackets in the ipv4 case that would be excellent.

[Emilien Macchi (emilienm)]

See https://bugs.launchpad.net/tripleo/+bug/1622755 for following-up the actual galera issue.

[Ben Nemec (bnemec)]

I'm going to mark this as fix released per https://bugs.launchpad.net/tripleo/+bug/1622755 since that should address the problem.