Fuel update env from 9.0 to 9.1 results in MySQL unreachable

[Sam Stoelinga]

Steps to reproduce:
1. Install Fuel 9.0
2. Create virtual env on vbox with 3 controllers(4gb mem) and 1 compute node
3. Update Fuel master to 9.1 according to https://docs.mirantis.com/openstack/fuel/fuel-master/operations/update-from-9-to-9-1.html
$ vim /etc/yum.repos.d/mos-updates.repo
Change baseurl to
baseurl=http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-10-02-090322/x86_64/
$ yum clean all
$ yum install python-cudet
$ update-prepare prepare master
$ update-prepare prepare env 1
$ fuel2 update --env 1 install

Current result:
The update fails and MySQL is unreachable because haproxy failed. See log file below

Expected result:
Update can be applied without resulting in any downtime.

Log file:
http://paste.openstack.org/show/584921/

[Maksim Malchuk (mmalchuk)]

Paste file don't show the root cause of the problem.
IMO this is the env performance problem while MySQL Galera got split-brain.
Please attach the diagnostic snapshot.

[Sam Stoelinga (sammiestoel)]

Maksim, I ran this env for a few days and diagnostic snapshot is already at 2.4Gb... So no I can't share the diagnostic snapshot.
Anyone within Mirantis can access the env with:
ssh root@172.19.12.8 -p 2222

Please raise importance to critical as this is impacting the whole 9.1 release and will result in cloud downtime.

[Maksim Malchuk (mmalchuk)]

root@node-2:~# haproxy -v
HA-Proxy version 1.5.3 2014/07/25
Copyright 2000-2014 Willy Tarreau <[e-mail]>

root@node-2:~# apt-cache policy haproxy
haproxy:
  Installed: 1.5.3-1~u14.04+mos2
  Candidate: 1.5.3-1~u14.04+mos2
  Version table:
 *** 1.5.3-1~u14.04+mos2 0
       1050 http://10.20.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.24-2ubuntu0.4 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.4.24-2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

9.0 cluster not upgraded to 9.1 because of no proposed repo provided.