[swarm] mysql service does not start after termination

[Andrey Lavrentyev]

Detailed bug description:
Mysql service does not start after termination within 2 minutes. It looks like the box where it's running is a bit slow. All services are OK after restoring from the snapshot: all controllers run mysql services, no errors in logs. It can be considered as performance issue though.

Traceback: http://paste.openstack.org/show/549255/

Swarm failure: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/16/testReport/%28root%29/ha_neutron_mysql_termination/

Increasing the timeout might help a little bit for slow boxes, however it's not the best way to go.

Steps to reproduce:
1. Terminate mysql
2. Wait while it is being restarted
3. Verify it is restarted
4. Go to another controller
5. Run OSTF

Expected results:
mysql service gets up after termination withing 2 minutes

Actual result:
mysql service is down

Impact:
swarm failure

Description of the environment:
9.1 snapshot #90
[root@nailgun log]# shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 python-packetary-9.0.0-1.mos142.noarch
 fuelmenu-9.0.0-1.mos275.noarch
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-openstack-metadata-9.0.0-1.mos8748.noarch
 fuel-ostf-9.0.0-1.mos938.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8496.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8748.noarch
 fuel-utils-9.0.0-1.mos8496.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-library9.0-9.0.0-1.mos8496.noarch
 nailgun-mcagents-9.0.0-1.mos753.noarch
 rubygem-astute-9.0.0-1.mos753.noarch
 fuel-nailgun-9.0.0-1.mos8748.noarch
 fuel-notify-9.0.0-1.mos8496.noarch
 fuel-mirror-9.0.0-1.mos142.noarch
 fuel-migrate-9.0.0-1.mos8496.noarch

MOS_CENTOS_OS_MIRROR_ID: os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID: proposed-2016-08-03-150320
MOS_CENTOS_UPDATES_MIRROR_ID: updates-2016-06-23-135916
MOS_CENTOS_SECURITY_MIRROR_ID: security-2016-06-23-140002
MOS_CENTOS_HOLDBACK_MIRROR_ID: holdback-2016-06-23-140047
MOS_UBUNTU_MIRROR_ID: 9.0-2016-08-03-144320
UBUNTU_MIRROR_ID: ubuntu-2016-08-02-170656
CENTOS_MIRROR_ID: centos-7.2.1511-2016-05-31-083834

Logs: https://drive.google.com/open?id=0B5HPBFb7K7gXMVNUSmFBOWpiX2c

[Kyrylo Galanov (kgalanov)]

Hi,

We have a grace period of 10 minutes to reassemble cluster after it was broken. Could you please increase timeout in tests?

--
Kyrylo

[Nastya Urlapova (aurlapova)]

Green for last five runs:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/88/testReport/(root)/ha_neutron_mysql_termination/history/