[system-tests] Test Mysql kill test wasn't wait enough on sync check

[Andrey Sledzinskiy]

{

    "build_id": "2015-01-23_22-54-45",
    "ostf_sha": "95054878eff472eb6a0d9d185ce65e3105dde632",
    "build_number": "76",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "e75ac13a46d6c40fe7c90af51c053c0e0a818808",
    "production": "docker",
    "fuelmain_sha": "",
    "astute_sha": "48829a27b8041c36b14e34cc2ae760c8f57703a9",
    "feature_groups":

[

    "mirantis"

],
"release": "6.1",
"release_versions":
{

    "2014.2-6.1":

{

    "VERSION":

{

    "build_id": "2015-01-23_22-54-45",
    "ostf_sha": "95054878eff472eb6a0d9d185ce65e3105dde632",
    "build_number": "76",
    "api": "1.0",
    "nailgun_sha": "e75ac13a46d6c40fe7c90af51c053c0e0a818808",
    "production": "docker",
    "fuelmain_sha": "",
    "astute_sha": "48829a27b8041c36b14e34cc2ae760c8f57703a9",
    "feature_groups":

                [
                    "mirantis"
                ],
                "release": "6.1",
                "fuellib_sha": "4c61bf87a96453a750df3e81606ba9fe21c2059f"
            }
        }
    },
    "fuellib_sha": "4c61bf87a96453a750df3e81606ba9fe21c2059f"

}

Steps:
1. Create next cluster - HA, CentOS, Neutron Vlan, 3 controllers, 2 compute nodes
2. Deploy cluster
3. After deployment ssh to first controller (node-1) and do:
pkill -9 -x "mysqld"
4. Wait until - crm resource status clone_p_mysql | grep -q "is running on: $HOSTNAME" return is Running
and galera state is synced
5. Ssh to next controller (node-4) and do:
pkill -9 -x "mysqld"

Actual result - after mysql killing on second controller mysql resiurce is stopped in pacemaker

Logs are attached

[Bogdan Dobrelya (bogdando)]

Do we have an automated test case for that running periodically?

[Andrey Sledzinskiy (asledzinskiy)]

yes, logs are provided from CI run
environment is still available

[Bogdan Dobrelya (bogdando)]

Deployment was finished at:
2015-01-26T03:16:21.602343 node-4 ./node-4.test.domain.local/puppet-apply.log:2015-01-26T03:16:21.602343+00:00 notice: Finished catalog run in 0.31 seconds

MySQL connection lost from node-4 at:
2015-01-26T03:46:23.674219 node-4 ./node-4.test.domain.local/nova-conductor.log:2015-01-26T03:46:23.674219+00:00 err: model server went away

Recovered at:
2015-01-26T03:46:24.036423 node-4 ./node-4.test.domain.local/cinder-scheduler.log:2015-01-26T03:46:24.036423+00:00 err: Recovered model server connection!

No issues here

[Tatyanka (tatyana-leontovich)]

Actually I revert env - wait more than 600 sec and galera synced with success, so we need to set timeout bigger, again

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/170182

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/170182
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=b3baa5b7074b1668c797b41948bd90448781b2e6
Submitter: Jenkins
Branch: master

commit b3baa5b7074b1668c797b41948bd90448781b2e6
Author: Dmitry Tyzhnenko <email address hidden>
Date: Thu Apr 2 18:42:01 2015 +0300

    Increase timeout in ha_destroy_controllers method

    Add timeout parameter into wait_mysql_galera_is_up with 120s by
    default

    Change-Id: I28bfc776288b587cb08f56350cc5e04f3d15ce39
    Closes-bug: #1414663

[Dmitry Tyzhnenko (dtyzhnenko)]

Varified on CI - http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.ubuntu.ha_neutron_destructive/105/consoleFull

Test failed on OSTF assertion. Galera and RabbitMQ was alive

ISO 6.1-345

{
    u'build_id': u'2015-04-24_21-59-41',
    u'build_number': u'345',
    u'auth_required': True,
    u'fuel-ostf_sha': u'b38602c841deaa03ddffc95c02f319360462cbe3',
    u'fuel-library_sha': u'6bdf783e2bffdce80ecffcca2915e6d32a8ccdd7',
    u'nailgun_sha': u'20bb824a6b4c3be156bf370a28d20995b8048ec9',
    u'openstack_version': u'2014.2.2-6.1',
    u'production': u'docker',
    u'api': u'1.0',
    u'python-fuelclient_sha': u'2b311b3b82a1e2df1dc3484a0f37e282273cd988',
    u'astute_sha': u'c1793f982fda7e3fc7b937ccaa613c649be6a144',
    u'fuelmain_sha': u'01288380950bc89d572cf5902141c9a393ada950',
    u'feature_groups': [u'mirantis'],
    u'release': u'6.1',
    u'release_versions': {u'2014.2.2-6.1': {u'VERSION': {
        u'build_id': u'2015-04-24_21-59-41',
        u'build_number': u'345',
        u'fuel-library_sha': u'6bdf783e2bffdce80ecffcca2915e6d32a8ccdd7',
        u'nailgun_sha': u'20bb824a6b4c3be156bf370a28d20995b8048ec9',
        u'fuel-ostf_sha': u'b38602c841deaa03ddffc95c02f319360462cbe3',
        u'production': u'docker',
        u'api': u'1.0',
        u'python-fuelclient_sha': u'2b311b3b82a1e2df1dc3484a0f37e282273cd988',
        u'astute_sha': u'c1793f982fda7e3fc7b937ccaa613c649be6a144',
        u'fuelmain_sha': u'01288380950bc89d572cf5902141c9a393ada950',
        u'feature_groups': [u'mirantis'],
        u'release': u'6.1',
        u'openstack_version': u'2014.2.2-6.1',
        }}},
    }