Rabbitmq::Install::Rabbitmqadmin fails during deploy after replacing primary controller

[ElenaRossokhina]

Detailed bug description:
Rabbitmq::Install::Rabbitmqadmin fails during deploy after replacing primary controller and adding one more controller.
Steps to reproduce:
1. Create cluster
2. Add 3 controller nodes and 1 compute
3. Deploy the cluster
4. Destroy primary controller
5. Wait controller offline
6. Remove offline controller from cluster (I used GUI, button "Remove" near node status label)
7. Add 1 new controller
8. Deploy changes
Expected results:
Deploy OK
Actual result:
All nodes have Error status;
puppet.log on new controller node contains:
(/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) Could not evaluate: Command is still failing after 180 seconds expired!
(/Stage[main]/Rabbitmq::Install::Rabbitmqadmin/Staging::File[rabbitmqadmin]/Exec[/var/lib/rabbitmq/rabbitmqadmin]/returns) change from notrun to 0 failed: curl -k --noproxy localhost --retry 30 --retry-delay 6 -f -L -o /var/lib/rabbitmq/rabbitmqadmin http://nova:ZVpDslAthkbNtSF8yhroxIrn@localhost:15672/cli/rabbitmqadmin returned 7 instead of one of [0]

fuel-9.0-mos-427-2016-06-01_12-18-59.iso:
cat /etc/fuel_build_id:
 427
cat /etc/fuel_build_number:
 427
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-misc-9.0.0-1.mos8415.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8415.noarch
 rubygem-astute-9.0.0-1.mos747.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8718.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-openstack-metadata-9.0.0-1.mos8718.noarch
 fuel-notify-9.0.0-1.mos8415.noarch
 nailgun-mcagents-9.0.0-1.mos747.noarch
 python-fuelclient-9.0.0-1.mos319.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8415.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-library9.0-9.0.0-1.mos8415.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2710.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8718.noarch
Diagnostic snapshot is available here https://drive.google.com/open?id=0B2ag_Bf-ShtTMjdUQTBNc0lPR3M

[ElenaRossokhina (esolomina)]

Also, issue was reproduced on another scenario:

1. Deploy cluster: 1 controller, Neutron Vlan, default storages
2. Run OSTF
3. Add 2 controllers
4. Re-deploy cluster (FAIL)

full snapshot https://drive.google.com/open?id=0B2ag_Bf-ShtTZ3dhTzVSX2VHUVk

[Oleksiy Molchanov (omolchanov)]

Not reproduced on latest ISO. Please provide env if it is possible.

[Oleksiy Molchanov (omolchanov)]

We have checked environment provided by TS. Env is extremely slow, it caused such failures. That's why I was not able to reproduce. Marking this as Invalid.