unable to connect to rabbit on redeployment

[Andrey Grebennikov]

Fuel 5.1, Centos, Neutron+VLANs

Puppet fails on the first attempt during the deployment, but it was able to create rabbitmq resource under pacemaker. During the redeployment happens the next:

Fri Nov 07 00:02:07 +0000 2014 Service[p_rabbitmq-server](provider=pacemaker) (debug): STATUS IS: running
Fri Nov 07 00:02:13 +0000 2014 Puppet (debug): Executing '/usr/sbin/crm resource manage master_p_rabbitmq-server'
Fri Nov 07 00:02:14 +0000 2014 /Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]/enable (notice): enable changed 'true' to 'true'
Fri Nov 07 00:02:14 +0000 2014 /Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server] (debug): The container Class[Rabbitmq::Service] will propagat
e my refresh event
Fri Nov 07 00:02:14 +0000 2014 /Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server] (info): Evaluated in 13.25 seconds
Fri Nov 07 00:02:14 +0000 2014 Class[Rabbitmq::Service] (info): Starting to evaluate the resource
Fri Nov 07 00:02:14 +0000 2014 Class[Rabbitmq::Service] (debug): The container Stage[main] will propagate my refresh event
Fri Nov 07 00:02:14 +0000 2014 Class[Rabbitmq::Service] (info): Evaluated in 0.01 seconds
Fri Nov 07 00:02:14 +0000 2014 /Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest] (info): Starting to evaluate the resource
Fri Nov 07 00:02:14 +0000 2014 Puppet (debug): Executing '/usr/sbin/rabbitmqctl list_users'
Fri Nov 07 00:02:15 +0000 2014 /Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest] (err): Could not evaluate: Execution of '/usr/sbin/rabbitmqctl list_
users' returned 2: Listing users ...
Error: unable to connect to node 'rabbit@node-45': nodedown

It restarts the resource, and it needs some time for getting it back. Puppet doesn't wait and tries to make the request to rabbit immediately.

[Stanislav Makar (smakar)]

Could you please provide more info?
How many node in this env( logs show node-45) ?
What type of node did you deploy?

[Andrey Grebennikov (agrebennikov)]

I see this issue appears during the setup of the primary controller. In general I'm going to deploy with 4 controllers, but I'm not sure whether it really matters.

The workaround is:
Execute "crm resource stop p_rabbitmq_server" when the deployment fails, then next time it goes fine.

[Bogdan Dobrelya (bogdando)]

It is a known issue with puppet-rabbitmq module. Its provider should retry any rabbitmqctl list* requests, see https://tickets.puppetlabs.com/browse/MODULES-1452.

The solution for 6.1 was already addressed as a part of upstream module sync request https://review.openstack.org/#/c/127166/