[sytem-tests] Improve RabbitMQ failover destructive test case

[Bogdan Dobrelya]

0) deploy any HA cluster of 3 controllers.
Assume we have a node-1 as a primary controller with the rabbit multistate clone resource master running, and
node-2, node-3 as running multistate resource slaves.

1) Move rabbit master resource to node-2
2) wait for ostf ha passed
3) kill the node-2, which should be a rabbit master now
4) wait for ostf ha passed
5) power on node-2
6) wait for it joined the rabbit cluster (wait for ostf ha passed)
7) repeat 1-6

Expected:
A. The rabbitmq cluster assembles from two remaining nodes, having 1 master and 1 slave after the step #4, no longer than in 5 minutes.
B. The rabbitmq cluster assembles from three nodes, having 1 master and 2 slaves after the steps #2, #6, no longer than in 5 minutes.
C. At least one node is always available for AMQP connections and it's queues and messages synced with other nodes, if any available as well.

Here is an ugly bash example http://pastebin.com/eANHxrHV

Updated the script source https://github.com/bogdando/rabbit_chirt_test

[Dennis Dmitriev (ddmitriev)]

Mark as duplicate because this case covered by https://review.openstack.org/#/c/185065/ that use the following scenario:

Add tests to cover scenario 3-1 rabbit failover:
 * SSH to controller and get rabbit master
 * Suspend not rabbit master node
 * Check that rabbit master stay as was
 * Run ostf
 * Resume slave
 * Check rabbit master is the same
 * Run ostf
 * Suspend rabbit master node
 * Check that new rabbit-master appears
 * Run ostf
 * Resume node
 * Check that new rabbit-master was not re-elected
 * Run ostf

[Bogdan Dobrelya (bogdando)]

This is not a duplicate. The bug https://bugs.launchpad.net/fuel/+bug/1435250 :
a) Describes no loop for the test
b) has nothing to the test case implementation but the failed result reported instead
c) anyway, I see no success runs for the existing 3-1 test case in the TestRail

[Bogdan Dobrelya (bogdando)]

Raised to high as it seems there is no adequate destructive test cases exist for the rabbit cluster in Fuel QA. At least I can't find a such one in the TestRail

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/239930

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/239930
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7a28d9040ee7f6dc072f98d15cd32d77825f020c
Submitter: Jenkins
Branch: master

commit 7a28d9040ee7f6dc072f98d15cd32d77825f020c
Author: Tatyana Leontovich <email address hidden>
Date: Wed Oct 28 14:36:23 2015 +0200

    Add stability rabbit test

    Add stability tests with next scenario:
    * Get rabbit master
    * move it to other node
    * delete contraint for rabbit resource
    * Run OSTF
    * get new rabbit master
    * destroy it
    * run OSTF
    * power on destroyed node
    * run OSTF
    * repeat failover cases COUNT_REPEAT times,

    By defaut COUNT_REPEAT is 2
    Closes-Bug: #1458830

    Change-Id: I6fb8d540e930c4d46ef73913e442100424e05047

[Tatyanka (tatyana-leontovich)]

test is executed on swarm - so move to fix released status

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280798
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=f770d426c81fca60ea9ded6e48256c3700d4a3ab
Submitter: Jenkins
Branch: master

commit f770d426c81fca60ea9ded6e48256c3700d4a3ab
Author: Volodymyr Shypyguzov <email address hidden>
Date: Tue Feb 16 18:24:11 2016 +0200

    Fix duplicate pacemaker constraint command invocation

    Edit test docstring according to the test script
    Add show_step to the test

    Related-Bug:#1458830
    Closes-Bug:#1544973

    Change-Id: I05322f648440447f7e23df0cfb9adffbfe7e7aec

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/282229

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/282229
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=e29ffb3087d296f3ed3ba9fbbd68a321f2139bf7
Submitter: Jenkins
Branch: stable/8.0

commit e29ffb3087d296f3ed3ba9fbbd68a321f2139bf7
Author: Volodymyr Shypyguzov <email address hidden>
Date: Tue Feb 16 18:24:11 2016 +0200

    Fix duplicate pacemaker constraint command invocation

    Edit test docstring according to the test script
    Add show_step to the test

    Related-Bug:#1458830
    Closes-Bug:#1544973

    Change-Id: I05322f648440447f7e23df0cfb9adffbfe7e7aec
    (cherry picked from commit f770d426c81fca60ea9ded6e48256c3700d4a3ab)