[BVT] [rabbitmq v5.0.0 module] failed do detect management plugin

[Bogdan Dobrelya]

First seen at:
 http://jenkins-product.srt.mirantis.net:8080/job/6.1.centos.bvt_1/52/
 http://jenkins-product.srt.mirantis.net:8080/job/6.1.ubuntu.bvt_2/54/

The issue is that puppet provider fails to execute curl to the management plugin:
  ./node-4.test.domain.local/lrmd.log:2015-01-13T01:16:30.676389+00:00 notice: notice: operation_finished: p_rabbitmq-server_start_0:26298:stderr [ Err
  or: {"need_to_load_plugins", ]
  ./node-4.test.domain.local/puppet-apply.log:2015-01-13T01:16:31.024470+00:00 err: curl -k --noproxy localhost -f -L -o /var/lib/rabbitmq/rabbitmqadmin http://nova:s3VJIAsW@localhost:15672/cli/rabbitmqadmin returned 7 instead of one of [0]

But the deployed environment after reverting of the failed deployment snaphot shows no errors. The same curl command executes OK at node-4 and rabbitmq-plugins list shows management plugin as active. There are also no issues with rabbit cluster.

[Bogdan Dobrelya (bogdando)]

Here is the log of events http://paste.openstack.org/show/156886/
It looks like everything is ok in the events flow:
- firewall rules were created at 00:46 (not shown in logs)
- the management plugin was enabled successfully,
- the service was started and was operational (list users ok)
But for some reason curl have failed for a while...

It looks like staging::file exec should be wrapped in retries.

[Bogdan Dobrelya (bogdando)]

related upstream bug https://tickets.puppetlabs.com/browse/MODULES-1650

[Bogdan Dobrelya (bogdando)]

related upstream bug https://tickets.puppetlabs.com/browse/MODULES-1651

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/146825

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/146825
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5551b1467b2f6aeff97699ee6de71a1d8318b436
Submitter: Jenkins
Branch: master

commit 5551b1467b2f6aeff97699ee6de71a1d8318b436
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Jan 13 12:25:45 2015 +0100

    Add retries for rabbitmqadmin file

    Staging::file used by Rabbitmq::Install::Rabbitmqadmin cannot handle
    non transient (connectivity) errors.

    The solution is to add retries for the file transfer when non
    transient connection errors exist, which is:

    * sync upstream rabbitmq module fix
      for https://tickets.puppetlabs.com/browse/MODULES-1650
    * sync upstream staging module fix
      for https://tickets.puppetlabs.com/browse/MODULES-1651

    Closes-bug: #1410119

    Change-Id: I7e38754d0d65480a71e3caaed723a41fb8784cbf
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148494
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=47e89058bf02c567b77f5366a79a192633354edb
Submitter: Jenkins
Branch: master

commit 47e89058bf02c567b77f5366a79a192633354edb
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Jan 20 11:25:11 2015 +0100

    Add retries for rabbitmqadmin curl command

    W/o this patch, Rabbitmq::Install::Rabbitmqadmin
    sometimes fails to install the rabbitmqadmin script if
    there are some connectivity (which is non-transient) errors.

    The solution is to add retries for exec command as curl allows
    retries only for transient errors and fails for connectivity
    errors.

    Closes-bug: #1412712
    Related-bug: #1410119

    Change-Id: I681599ea4a2756e7a442c194c68cf0a484f24f7f
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Bogdan Dobrelya (bogdando)]

This fix is bound for puppet-rabbitmq v5.0.0 module and cannot be backported for 6.0 as we don't have it there