Percona XtraDB Cluster not working with v5.6

[Doug Shelley]

The current implementation of the pxc manager supports Percona XtraDB Cluster v5.5 and XtraBackup v2.1. Testing with v5.6 and XtraBackup v2.3 (a dependency of v5.6) yielded issues - i.e. cluster-create resulted in a cluster in error state. The donor bootstrap via xtrabackup would fail.

We need to:
- update the redstack elements for pxc to pull v5.6 from the Percona repo (much like the way that Percona Server 5.6 is installed)
- determine the issues related to bootstrap via XtraBackup v2.3

[Doug Shelley (0-doug)]

The guest with pxc5.6/pxb2.3 is failing during prepare. The failure manifests as an authentication failure during SST (Snapshot State Transfer). This process is currently setup to use xtrabackup. One workaround would be to switch to using rsync. This has been tested to work but has one drawback - the source DB has to be in read only mode during SST - this is likely not an issue during a create operation but could be an issue during cluster-grow.

Issues/possible solutions with SST using xtrabackup

1. The core issue with using pxb2.3 for SST is because Percona changed the process to pass in the mysql password using the env var "MYSQL_PWD". Historically, they were just passing the password on the backup command (innobackupex) command line. The problem is that the guest stores its mysql credentials (os_admin) in the my.cnf file, in a [client] section. This overrides the use of MYSQL_PWD and causes the authentication failure above. (Clearly, percona didn't test this scenario or expect anyone to do this). This issue can be corrected by moving the [client] section settings from the file under /etc/mysql/conf.d into ~/.my.cnf. I tested this and it will make pxc startup the cluster.

2. We need to switch to using sst method "xtrabackup-v2". This is recommended by Percona when using this combination (and if you don't do it, you just get a different failure). To set this, we just need to change templates/pxc/cluster.config.template:
               wsrep_sst_method=xtrabackup-v2

3. pxb2.3 seems to want to connect to an unknown socket when it gets "localhost" for the host. Fix is to put "host=127.0.0.1" in [client] section.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/266485

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/266485
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=293dafb56501702fc35c79d7c3c4efcf1fdaeab3
Submitter: Jenkins
Branch: master

commit 293dafb56501702fc35c79d7c3c4efcf1fdaeab3
Author: Doug Shelley <email address hidden>
Date: Tue Jan 12 16:54:19 2016 +0000

    Fix Percona XtraDB Cluster guest to work with v5.6

    The current implmentation of Percona clusters only supports
    PXC 5.5. To make the implementation work with v5.6, some changes
    need to be made to the base Mysql manager. The biggest change
    is that we can no longer store the os_admin client credentials
    in the server my.cnf file. This commit moves those credentials to
    a .my.cnf file stored in the trove homedir. This will be read
    by the guest when attempting sqlchemy connections to the local
    mysql database and will also be picked up automatically by other
    mysql clients (such as mysqladmin).

    Also, since PXC 5.6 requires PXB 2.3, this commit fixes the issue
    previously discovered with that combination. The previous fix was
    to pin PXB to v2.2. The fix is to specify host=127.0.0.1 in the
    client section of my.cnf.

    Change-Id: Ib73b4f8ba40ddf211d0ec88069c4807186fce139
    Parial-bug: 1532969