[library] Nova-compute services are in 'down' state after deployment: Timed out waiting for a reply to message

Bug #1362527 reported by Artem Panchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Confirmed
High
Dmitry Ilyin

Bug Description

api: '1.0'
astute_sha: bc60b7d027ab244039f48c505ac52ab8eb0a990c
auth_required: true
build_id: 2014-08-27_10-46-57
build_number: '481'
feature_groups:
- mirantis
fuellib_sha: 6f478caa9111e428b3c053eef44754f59ce7a2c2
fuelmain_sha: 74a97d500bb2fe9528f99771ccc2ec657ae3f76e
nailgun_sha: 04e3f9d9ad3140cd63a9b5a1a302c03ebe64fd0a
ostf_sha: 4dcd99cc4bfa19f52d4b87ed321eb84ff03844da
production: docker
release: '5.1'

Steps to reproduce:

1. Deploy new environment:
    * Ubuntu, HA, NeutronGre, Ceph for Glance, Ceilometer
    * 2 Controller+Ceph, 3 Controller+Mongo, 4 Compute+Ceph, 6 Compute+Cinder, 2 Cinder+Ceph nodes
2. Deployment is successful. Start OSTF.

Expected result:

 - health checks passed

Actual:

 - most of checks failed (some of sanity checks failed due to '504' error from haproxy(nova/glance api), compute checks failed to launch instance because all computes services are down)

Here is the part of nova-compute logs(node-24):

http://paste.openstack.org/show/101569/

Example of 504 error:

http://paste.openstack.org/show/101577/

and HaProxy stats (errors):

http://paste.openstack.org/show/101602/

Also, I discovered, that RabbitMQ has lots of messagess in queries from nova-conductor and neutron:

http://paste.openstack.org/show/101605/

and there are errors in its logs:

http://paste.openstack.org/show/101610/

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

I can't attach snapshot with logs (>500 MBytes), please download it here:

https://yadi.sk/d/RA09LbhxaZwcM

Changed in fuel:
status: New → Confirmed
Mike Scherbakov (mihgen)
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Ilyin (idv1985)
Revision history for this message
Dmitry Ilyin (idv1985) wrote :

I looks like network connectivity have failed or something.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.