[SWARM][8.0] RabbitMQ availability test failure after removing rabbit node

[Vladimir Jigulin]

Reproduced on CI:
https://patching-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/23/testReport/(root)/separate_rabbit_service_add_delete_node/separate_rabbit_service_add_delete_node/

Steps to reproduce:
1. Revert snapshot separate_rabbit_service
2. Add one rabbit node and re-deploy cluster
3. Run network verification
4. Run OSTF
5. Check hiera hosts are the same for different group of roles
6. Delete one rabbit node
7. Run network verification
8. Run OSTF

better use separate_rabbit_service_add_delete_node test from fuel-qa
test fail at line:
https://github.com/openstack/fuel-qa/blob/stable/8.0/fuelweb_test/tests/tests_separate_services/test_separate_rabbitmq.py#L266

Expected results: ostf tests passed successfully

Actual result: 2 ostf tests failed:
- RabbitMQ availability (failure) Number of RabbitMQ nodes is not equal to number of cluster nodes.
- RabbitMQ replication (failure) Failed to establish AMQP connection to 5673/tcp port on 10.109.1.11 from controller node!

Reproducibility: rarely

Snapshot: https://drive.google.com/file/d/0Bw7ZahkM7_sJZVBkd0VxRVRiVmM/view?usp=sharing

[Dmitry Belyaninov (dbelyaninov)]

The similar issue is present on 9.x SWARM test(s)

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.repetitive_restart/15/testReport/(root)/ceph_partitions_repetitive_cold_restart/ceph_partitions_repetitive_cold_restart/

Test scenario:
1. Revert snapshot 'prepare_load_ceph_ha'
2. Wait until MySQL Galera is UP on some controller
3. Check Ceph status
4. Run ostf
5. Fill ceph partitions on all nodes up to 30%
6. Check Ceph status
7. Disable UMM
8. Run RALLY
9. 100 times repetitive reboot:
10. Cold restart of all nodes
11. Wait for HA services ready
12. Wait until MySQL Galera is UP on some controller
13. Run ostf

Note: there is one more similar bug
https://bugs.launchpad.net/fuel/+bug/1495885