After changing pacemaker parameter rabbit master node does not leave cluster

[ElenaRossokhina]

Detailed bug description:
According to logs, rabbit master node does not reboot and always stays in cluster after command to change max_rabbitmqctl_timeouts. At the same time, rabbit slaves are restarted.
Parameter changed correctly.
Steps to reproduce:
 https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/44/testReport/(root)/change_pacemaker_parameter_not_break_rabbitmq/change_pacemaker_parameter_not_break_rabbitmq/
Expected results:
test passed
Actual result:
system test failed: All nodes are staying in the cluster
Reproducibility:
on swarm, since 9.1 snapshot #169

logs https://drive.google.com/open?id=0B2ag_Bf-ShtTZlp1NkxVVWZ1cFE

[Dmitry Belyaninov (dbelyaninov)]

Hot fail:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/52/testReport/(root)/change_pacemaker_parameter_not_break_rabbitmq/change_pacemaker_parameter_not_break_rabbitmq/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/368771

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/371443

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/368771
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=83a87f05929b22d04bfbc57957c2fba99b677731
Submitter: Jenkins
Branch: master

commit 83a87f05929b22d04bfbc57957c2fba99b677731
Author: Dmitry Mescheryakov <email address hidden>
Date: Mon Sep 12 14:14:23 2016 +0300

    Do not expect RabbitMQ to be down on Pacemaker parameter change

    Right now the test expects that all RabbitMQ nodes will be down
    after parameter change. That is not always the case and moreover we
    don't really care if it restarts or not. What we need instead is to
    verify that cluster either does not break at all, or is fixed after
    some time. So now we wait for a minute to give Pacemaker time to
    shut down some nodes if it decides to do so and after that we wait
    for RabbitMQ cluster to come up with timeout.

    Closes-Bug: #1618909
    Change-Id: I50f65f73c6937d23721c9e236f2a8059313e760f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/371443
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=8b636e5a7001c2c9b811dfeccacd9fdc55780afb
Submitter: Jenkins
Branch: stable/mitaka

commit 8b636e5a7001c2c9b811dfeccacd9fdc55780afb
Author: Dmitry Mescheryakov <email address hidden>
Date: Mon Sep 12 14:14:23 2016 +0300

    Do not expect RabbitMQ to be down on Pacemaker parameter change

    Right now the test expects that all RabbitMQ nodes will be down
    after parameter change. That is not always the case and moreover we
    don't really care if it restarts or not. What we need instead is to
    verify that cluster either does not break at all, or is fixed after
    some time. So now we wait for a minute to give Pacemaker time to
    shut down some nodes if it decides to do so and after that we wait
    for RabbitMQ cluster to come up with timeout.

    Closes-Bug: #1618909
    Change-Id: I50f65f73c6937d23721c9e236f2a8059313e760f

[Alexey Galkin (agalkin)]

Verification 9.1 on build #87.

Passed

change_pacemaker_parameter_not_break_rabbitmq.change_pacemaker_parameter_not_break_rabbitmq (from nosetests)

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/87/testReport/(root)/change_pacemaker_parameter_not_break_rabbitmq/change_pacemaker_parameter_not_break_rabbitmq/