Rabbitmq fails to start if mongodb+ceilometer is enabled

[Zoltan Langi]

Detailed bug description:
Fuel 11 and Ocata. If you turn on the "Install Ceilometer and Aodh" option in fuel, assign mongodb to your nodes, rabbitmq fails to start on 2 nodes out of a 3 control node cluster which makes your deployment to fail.

/var/log/puppet.log -> Puppet (err): Timeout waiting for custom service: 'rabbitmq' to become: 'online' after 600 seconds!

After you manually try to start rabbitmq on the failed hosts:
/etc/init.d/rabbitmq-server start
[....] Starting rabbitmq-server (via systemctl): rabbitmq-server.serviceJob for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
 failed!

journalctl -xe:
rabbitmq-server[6183]: mkdir: cannot create directory '/var/run/rabbitmq': Permission denied
rabbitmq-server[6183]: Failed to create directory: /var/run/rabbitmq

The /var/run/rabbitmq directory is missing from 2 out of 3 control nodes.

If you create the directory manually and set the permission, the rabbitmq starts, but if you try to redeploy your cluster and you got compute and storage nodes which don't have public ip assigned, they will fail to access to the repos.

If you don't deploy the ceilometer feature, everything works and the deployment goes through.

Expected results:
working deployment with ceilometer included.

Actual result:
failure in deployment due to rabbitmq fails to start.

Reproducibility:
turn on the feature, add mongodb to your control nodes and hit deploy

Workaround:
couldn't work it around, turned off ceilometer which is not the same result

Impact:
Unable to install ceilometer, no telemetry in the deployment at all

Description of the environment:
Fuel 11, Ubuntu 16.04 and Ocata

[Oleksiy Molchanov (omolchanov)]

Please attach diagnostic snapshot, marking as Incomplete.

[Zoltan Langi (zlangi)]

Alright, here is the diagnostic snapshot. Thanks for looking into it!

https://www.dropbox.com/s/8sa8m7mmb0za0yl/fuel-snapshot-2017-04-12_15-48-21.tar?dl=0

[Zoltan Langi (zlangi)]

Did you get anywhere with this bug? I'm happy to provide anything else you need to help this to be tracked down.