[library] rabbitmq container failed to load after fuel install

[Moshe Levi]

api version
{"ostf_sha": "9863db951a6e159f4fa6e6861c8331e1af069cf8", "auth_required": false, "api": "1.0", "nailgun_sha": "8325f4e7d4a23f6db96fe9edd61d6e0905b4af49", "production": "docker", "fuelmain_sha": "570033475c838fc17cb0befd1d9a586d65911810", "astute_sha": "88b41cec2afeb41780877f7ac13a3b18510e0052", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "eafceef139bc5559b8199d04ebb90a5ce0bced9b"}

Steps to reproduce
1builld custom iso with latest master of 15/7 and external repos "stable,http://fuel-repository.mirantis.com/repos/centos-fuel-5.1-stable/centos"
and http://fuel-repository.mirantis.com/repos/ubuntu-fuel-5.1-stable/ubuntu /"

2. install fuel iso

3. run dockerctl check

[root@fuel ~]# dockerctl check rabbitmq
checking container rabbitmq
checking with command "curl -f -L -i -u "naily:nKs7F066" http://10.20.0.2:15672/api/nodes 1>/dev/null 2>&1"
try number 1
return code is 22
try number 2
return code is 22
try number 3
return code is 22
try number 4
return code is 22
try number 5
return code is 22
try number 6

It also fail to generate Diagnostic snapshot
log are attached

[Matthew Mosesohn (raytrac3r)]

Moshe, this is a new issue from the init issue you had before. It seems that the exchanges and users didn't get created for both astute and mcollective. Maybe puppet ran with errors. Could you run these commands and send the output?
dockerctl destroy rabbitmq
dockerctl start rabbitmq
(wait 2 minutes)
dockerctl logs rabbitmq > rabbitmq.log
Upload the rabbitmq.log file (or pastebin)

[Moshe Levi (moshele)]

rabbitmq.log is attched

[Matthew Mosesohn (raytrac3r)]

Based on our conversation, this issue didn't reproduce. Let's mark this incomplete until we can find another reproduction.

[Moshe Levi (moshele)]

I will try to reproduce it with nightly build ISO

[Moshe Levi (moshele)]

I seen this issue with this fuel iso http://seed.fuel-infra.org/fuelweb-iso/fuel-community-master-44-2014-07-27_05-27-27.iso.torrent

logs attached

this issue is intermittent from 3 installation of the same iso it occurs once.

[Matthew Mosesohn (raytrac3r)]

Can we get the output of dockerctl logs rabbitmq while it's broken? If you destroy and recreate the container, it will overwrite this log data. I'm thinking there is some weird race condition here, but it's hard to figure out what is breaking without this log data.

[Moshe Levi (moshele)]

we reproduce it today again but running
dockerctl destroy rabbitmq
dockerctl start rabbitmq

fix the problem is there away to give you the logs without doing
dockerctl destroy rabbitmq
dockerctl start rabbitmq ?

[Matthew Mosesohn (raytrac3r)]

Moshe, of course. Just don't destroy the rabbitmq container. Run dockerctl logs `rabbitmq > rabbitmq.log`. Then you can destroy and start (as the workaround).

[Moshe Levi (moshele)]

reproduced with new ISO fuel-community-master-50-2014-08-02_05-27-27.iso

[Vladimir Kuklin (vkuklin)]

Moshe, could you also add /var/log/rabbitmq directory from the container?

[Moshe Levi (moshele)]

I will when we reproduce it again

[Moshe Levi (moshele)]

logs attached

[Mike Scherbakov (mihgen)]

Have it been reproduced? Bug title is pretty scary.

[Matthew Mosesohn (raytrac3r)]

This should be fixed now with https://review.openstack.org/#/c/116873/ . If you still see this bug in newer builds, please reopen this bug.

[Anastasia Palkina (apalkina)]

I checked all of containers and they were ready.

Verified on ISO #39

"build_id": "2014-10-23_00-01-06", "ostf_sha": "de177931b53fbe9655502b73d03910b8118e25f1", "build_number": "39", "auth_required": true, "api": "1.0", "nailgun_sha": "546f1f793fb20d51e5273a18e16751dfe7c1b356", "production": "docker", "fuelmain_sha": "dab17913263bbea7e9a3b55de8a0f3af5ac0e3e2", "astute_sha": "97eea90efe0a1f17b4934919d6e459d270c10372", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-10-23_00-01-06", "ostf_sha": "de177931b53fbe9655502b73d03910b8118e25f1", "build_number": "39", "api": "1.0", "nailgun_sha": "546f1f793fb20d51e5273a18e16751dfe7c1b356", "production": "docker", "fuelmain_sha": "dab17913263bbea7e9a3b55de8a0f3af5ac0e3e2", "astute_sha": "97eea90efe0a1f17b4934919d6e459d270c10372", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "d4f7d9b97ecfed33bc7ee4f628dd9e8ed3651a05"}}}, "fuellib_sha": "d4f7d9b97ecfed33bc7ee4f628dd9e8ed3651a05"