[Update] Update cluster from 9.1 to 9.2 failed on nova-db task

[Ilya Bumarskov]

Fuel 9.1

Detailed bug description:
Update cluster from 9.1 to 9.2 (snapshot id#515) failed on nova-db task.

Steps to reproduce:
   - Deploy 9.1 env with following nodes:
         * Controller
         * Compute, cinder
         * Compute, cinder
   - Add proposed repo for env (http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-11-14-194322)
   - Download mos-mu tool on master (git clone https://github.com/aepifanov/mos_mu.git)
   - Make a perform a preparation playbook (ansible-playbook playbooks/mos9_prepare.yml -e '{"env_id":<env_id>, "mos9_centos_repo":"http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-11-14-194322/x86_64"}, "mos9_GPG_KEY":"http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-11-14-194322/RPM-GPG-KEY-mos9.0"')
   - Update fuel node (ansible-playbook playbooks/update_fuel.yml)
   - Update env (fuel2 update --env <ENV_ID> install)

Observed behavior:
Deployment has failed. All nodes are finished. Failed tasks: Task[nova-db/2] Stopping the deployment process!
Task [nova-db/2] is failed because MySQL doesn't reachable (via vip__management ip):
(/Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Mysql_database[nova_api]/ensure) change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe create database if not exists `nova_api` character set `utf8` collate `utf8_general_ci`' returned 1: ERROR 2003 (HY000): Can't connect to MySQL server on '10.109.1.6' (113)

[Ilya Bumarskov (ibumarskov)]

vip_managment ip becomes available through several minutes (1-2 minutes) after stopped deployment.

[Oleksiy Molchanov (omolchanov)]

Ilya, please provide diagnostic snapshot.

[Ilya Bumarskov (ibumarskov)]

Snapshot - https://drive.google.com/open?id=0B8nyPqe6rrN1a3FDbzZORVFSazA

[Anton Chevychalov (achevychalov)]

I see a lot of br-mgmt flapping in logs:

cat fuel-snapshot-2016-11-15_14-11-11/node-2-10.109.0.3/var/log/messages | grep br-mgmt | grep 'Nov 15 12:' | wc -l
176

I can't reproduce it on my lab. Can you confirm problem on another lab instance?

[Anton Chevychalov (achevychalov)]

That bug related to pacemaker hang. See https://bugs.launchpad.net/fuel/+bug/1644152.