[library] Nova-compute services are in 'down' state after deployment: Timed out waiting for a reply to message

[Artem Panchenko]

api: '1.0'
astute_sha: bc60b7d027ab244039f48c505ac52ab8eb0a990c
auth_required: true
build_id: 2014-08-27_10-46-57
build_number: '481'
feature_groups:
- mirantis
fuellib_sha: 6f478caa9111e428b3c053eef44754f59ce7a2c2
fuelmain_sha: 74a97d500bb2fe9528f99771ccc2ec657ae3f76e
nailgun_sha: 04e3f9d9ad3140cd63a9b5a1a302c03ebe64fd0a
ostf_sha: 4dcd99cc4bfa19f52d4b87ed321eb84ff03844da
production: docker
release: '5.1'

Steps to reproduce:

1. Deploy new environment:
    * Ubuntu, HA, NeutronGre, Ceph for Glance, Ceilometer
    * 2 Controller+Ceph, 3 Controller+Mongo, 4 Compute+Ceph, 6 Compute+Cinder, 2 Cinder+Ceph nodes
2. Deployment is successful. Start OSTF.

Expected result:

 - health checks passed

Actual:

 - most of checks failed (some of sanity checks failed due to '504' error from haproxy(nova/glance api), compute checks failed to launch instance because all computes services are down)

Here is the part of nova-compute logs(node-24):

http://paste.openstack.org/show/101569/

Example of 504 error:

http://paste.openstack.org/show/101577/

and HaProxy stats (errors):

http://paste.openstack.org/show/101602/

Also, I discovered, that RabbitMQ has lots of messagess in queries from nova-conductor and neutron:

http://paste.openstack.org/show/101605/

and there are errors in its logs:

http://paste.openstack.org/show/101610/

[Artem Panchenko (apanchenko-8)]

I can't attach snapshot with logs (>500 MBytes), please download it here:

https://yadi.sk/d/RA09LbhxaZwcM

[Vladimir Kuklin (vkuklin)]

possible duplicate of https://bugs.launchpad.net/fuel/+bug/1362839

[Dmitry Ilyin (idv1985)]

I looks like network connectivity have failed or something.