Galera does not start due to using wrong xtrabackup version

[William Konitzer]

Detailed bug description:
Fuel MU5 upgrade appears to work, but afterwards the mysql cluster isn't running.

Clone Set: clone_p_mysqld [p_mysqld]

show stopped nodes. There is no way to get the nodes to start.

Problem was diagnosed as having the wrong version of xtrabackup installed. The latest version of Galera uses SST v3 and so requires xtrabackup with SST v3 support (xtrabackup=2.3.7)

Reproducibility:
 unknown

Workaround:
 1. Run "pcs status" on a controller node and see which node(s) is stopped under "Clone Set: clone_p_mysqld [p_mysqld]". Call this nodeA

  2. Update percona-xtrabackup on nodeA see:
  apt-cache policy percona-xtrabackup

  verify it will upgrade to percona-xtrabackup v2.3.7 and version is available. As long as version is available you can force an upgrade via "apt-get install percona-xtrabackup=<version>".

  3. Clean up mysql service on nodeA:

  export OCF_ROOT=/usr/lib/ocf
  export OCF_RESOURCE_INSTANCE=p_mysqld
  export OCF_RESKEY_socket=/var/run/mysqld/mysqld.sock

  psc resource ban p_mysql [nodeA]

  export OCF_RESOURCE_INSTANCE=p_mysqld
  export OCF_ROOT=/usr/lib/ocf
  export OCF_RESKEY_socket=/var/run/mysqld/mysqld.sock

  bash -x /usr/lib/ocf/resource.d/fuel/mysql-wss start

  4. Check if mysql replicated ok (see fuel /var/log/remote/nodeA.../oss-mysqlwss log file)

  5. Find constraint of mysql on nodeA in pcs:
  pcs constraint list --full
  pcs clear p_mysqld [nodeA]

  6. Check pcs status
  pcs status

  7. Upgrade percona-xtrabackup on other nodes by repeating steps above on other nodes

Impact:
 mysql cluster doesn't work

Additional information:
Current theory is that mysql-wsrep respository has a dependency on xtrabackup, but the minimum version of xtrabackup either doesn't exist or is too low.

[Oleksiy Molchanov (omolchanov)]

Feel free to use MU6, it contains xtrabcakup version 2.3.7

https://docs.mirantis.com/openstack/fuel/fuel-9.2/maintenance-updates/published-updates/maint-update-6/updated-packages-6.html

[Oleksiy Molchanov (omolchanov)]

percona-xtrabackup comes from here [0]. Manifest triggers galera module [1], that triggers package percona-xtrabackup with ensure => installed. Under MU6 we have only percona-xtrabackup package with version 2.3.7, so it should work as supposed.

[0] https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/manifests/database/database.pp#L274

[1] https://github.com/fuel-infra/puppet-galera/blob/master/manifests/init.pp#L296

[William Konitzer (wkonitzer)]

But what is the interaction if you already have percona-xtrabackup installed which is a lower version? I don't think it will force an upgrade to the latest 2.3.7 version or will it?

[Denis Meltsaykin (dmeltsaykin)]

William, it won't be forced to update since our wsrep packages do not have percona-xtrabackup as a dependency, but our update procedure updates xtrabackup alongside with wsrep so the problem is not triggered. What I see as a problem here is that at least the update procedure you use in not the official one and is not supported by us. On the other hand wsrep should have a dependency on xtrabackup in packages since de-facto wsrep demands xtrabackup >= 2.3.7 to work properly. Also changing the dependencies set of the wsrep packages might force us to use "dist-upgrade" since this is pretty common for apt/dpkg when there are new dependencies in a package. I'm trying to sort it all out to make a decision on package's specification change.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/38782

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/38782
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 61426e2d55c96963598e1d11f603f490f512a54c
Author: Denis V. Meltsaykin <email address hidden>
Date: Tue Jun 26 17:26:19 2018

Add xtrabackup as a dependency

mysql-wsrep doesn't contain a dependency for percona-xtrabackup,
but this package must be installed on fuel otherwise during the update
of mysql-wsrep it is possible to run recent mysql-server with old
xtrabackup which will lead to broken database.

Change-Id: I397823110e9886ea1c874e4af1c6a3269328983a
Closes-Bug: #1775266

[Dmitry (dtsapikov)]

Verified on 9.2+mu7