Comment 17 for bug 1624013

1) We've tried this with a variety of different hardware and it's always reproduced. Typically we're giving the controller nodes 80-100GB disk and 6GB RAM and 4 CPUs. Hardware is connected through an integrated switch (e.g. virtual connect) with VLANs defined by the switch.

2) No modifications of the snapshot archives were made - I've verified that the logs are indeed included. For example:
C:\Users\Bob\Downloads\fuel-snapshot-2016-09-15_16-56-42\fuel.domain.tld\var\log\remote\node-4.domain.tld\puppet-apply.log contains failures such as the following (which is attempted 3 times with a 10 minute timeout)
2016-09-15T14:33:06.393264+00:00 err: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]) Failed to call refresh: mysql -uclustercheck -pOObsCqCTtkLkRHK52n0H0N8O -Nbe "show status like 'wsrep_local_state_comment'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0]

As this is referring to node-4, I also checked some other node-4 logs at the offending time and mysqld.log contains failures such as:
2016-09-15T14:25:24.509238+00:00 err: 2016-09-15 14:25:24 17678 [ERROR] WSREP: Requesting state transfer failed: -77(File descriptor in bad state)
2016-09-15T14:25:24.509238+00:00 err: 2016-09-15 14:25:24 17678 [ERROR] WSREP: State transfer request failed unrecoverably: 77 (File descriptor in bad state). Most likely it is due to inability to communicate with the cluster primary component. Restart required.
2016-09-15T14:25:24.509238+00:00 err: 2016-09-15 14:25:24 17678 [Note] WSREP: /usr/sbin/mysqld: Terminated.

Mysqld log file actually contains a *lot* of errors here - but I don't understand what it's trying to do well enough.

3) We are using the mysql included in the ubuntu repositories in MOS9 - e.g.:
/var/www/nailgun/mitaka-9.0/ubuntu/x86_64/pool/main/m/mysql-wsrep-5.6/mysql-wsrep-common-5.6_5.6.23-1~u14.04+mos1_all.deb

MOS9 of course pulls mysql from this location to install on the nodes and therefore this location would need to be updated. We have tried a few methods to replace mysql files in this directory but get package verification errors when attempting to regenerate the contents. Advice for how to update this for MOS9 would be greatly appreciated.