Redeployment leads to unaccessible rabbitmq

[Vladimir Khlyunev]

Swarm test failed:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.install_plugin_after_cluster_create/46/testReport/(root)/five_ctrl_enable_installed_after_create_add/five_ctrl_enable_installed_after_create_add/
ISO 9.0mos-465
Snapshot https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.install_plugin_after_cluster_create/46/artifact/logs/fail_error_five_ctrl_enable_installed_after_create_add-fuel-snapshot-2016-06-10_04-23-00.tar.gz

Steps to reproduce:
1. Enable plugin
2. Deploy 2 new controller nodes at cluster (Nodes Under Test)
3. Run network verification
4. Check plugin on ALL controller nodes
5. Run OSTF

Expected result: deploy passed, all nodes are fine

Actual result: new added node is marked as failed, in puppet.log:
 curl -k --noproxy localhost --retry 30 --retry-delay 6 -f -L -o /var/lib/rabbitmq/rabbitmqadmin http://nova:hDqUiRl24D4U2MvFnPSQZTcW@localhost:15672/cli/rabbitmqadmin returned 7 instead of one of [0]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/333334

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/333334
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=31869eae65b712da2f353648e85f0b885edca01f
Submitter: Jenkins
Branch: master

commit 31869eae65b712da2f353648e85f0b885edca01f
Author: Oleksiy Molchanov <email address hidden>
Date: Thu Jun 23 16:00:01 2016 +0300

    Add rabbitmq service check

    * Check whether rabbitmq is online before doing any rabbitmq queries
    * Add new provider for custom service checks

    Change-Id: I1fb27ab2d731dc3fa7daa3694c9da3a01f318695
    Closes-Bug: 1591213

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/345276

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/345276
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=db218d4b64aa9fb43030c2633f45c162b5c9c559
Submitter: Jenkins
Branch: stable/mitaka

commit db218d4b64aa9fb43030c2633f45c162b5c9c559
Author: Oleksiy Molchanov <email address hidden>
Date: Thu Jun 23 16:00:01 2016 +0300

    Add rabbitmq service check

    * Check whether rabbitmq is online before doing any rabbitmq queries
    * Add new provider for custom service checks

    Change-Id: I1fb27ab2d731dc3fa7daa3694c9da3a01f318695
    Closes-Bug: 1591213

[Tatyana Kuterina (tkuterina)]

Verified on 9.1 snapshot #119

FUEL_QA_COMMIT=251ffb1dc50d76ebfaec069283e2b41084eb9782
UBUNTU_MIRROR_ID=ubuntu-2016-08-03-174238
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-08-10-160322
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-08-10-164321
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.

[Dmitry Belyaninov (dbelyaninov)]

Verified on Newton.
Passed run:
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.plugins.install_plugin_after_cluster_create/211/