[System tests] Helth check failed after slaves reboot: 'Lost connection to MySQL server'

[Artem Panchenko]

api: '1.0'
astute_sha: 644d279970df3daa5f5a2d2ccf8b4d22d53386ff
build_id: 2014-07-03_00-31-14
build_number: '88'
fuellib_sha: 385d713b569bc0633e695b44ff7eedf3417f0575
fuelmain_sha: 1072bc723d14427d5fdc24662ffe1af0641e0d9a
mirantis: 'yes'
nailgun_sha: 5c18e962d85b878e53ff6eb6eeeb14658814c5b8
ostf_sha: d0fe60e0eba61685008b86d101f459fc2d3bb654
production: docker
release: 5.0.1

The issue was reproduced during system tests on CI:

http://jenkins-product.srt.mirantis.net:8080/view/5.0_swarm/job/5.0_fuelmain.system_test.centos.thread_5/31/testReport/(root)/ceph_ha_restart/ceph_ha_restart/

In the 'ceph_multinode_restart' test after slaves restart we perform OSTF tests and expect that all of them pass:

https://github.com/stackforge/fuel-main/blob/master/fuelweb_test/tests/tests_strength/test_restart.py#L160

But sometimes OSTF checks are started before Mysql is up and it causes the following error:

http://paste.openstack.org/show/85398/

We need to wait until Mysql service is ready before performing helth checks.

Also, 'Check that required services are running' OSTF test will fail because one compute node is down, so 'should_fail=1' should be added.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104600

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104600
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=e52519ef9da87335ccef597994f64bd4bff07998
Submitter: Jenkins
Branch: master

commit e52519ef9da87335ccef597994f64bd4bff07998
Author: Artem Panchenko <email address hidden>
Date: Thu Jul 3 18:53:38 2014 +0300

    Wait until MySQL is ready before running OSTF

    Check that MySQL Galera is ready on primary
    controller before performing OSTF checks to avoid
    MySQL connection error from keystoneclient.
    Also, one OSTF test will fail because one compute
    node is down, so need to add it to should_fail.

    Change-Id: Id42abbf822f9bc51402f64ddedd1a6903260f977
    Closes-bug: #1337298

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/105253

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/105253
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=b0f5151d12751b9b55dcd69bd1445d0d480012d6
Submitter: Jenkins
Branch: stable/5.0

commit b0f5151d12751b9b55dcd69bd1445d0d480012d6
Author: Artem Panchenko <email address hidden>
Date: Thu Jul 3 18:53:38 2014 +0300

    Wait until MySQL is ready before running OSTF

    Check that MySQL Galera is ready on primary
    controller before performing OSTF checks to avoid
    MySQL connection error from keystoneclient.
    Also, one OSTF test will fail because one compute
    node is down, so need to add it to should_fail.

    Change-Id: Id42abbf822f9bc51402f64ddedd1a6903260f977
    Closes-bug: #1337298
    (cherry picked from commit e52519ef9da87335ccef597994f64bd4bff07998)