[Cinder] [Nova] Timed out waiting for a reply to message ID

[Leontii Istomin]

api: '1.0'
astute_sha: 3f1ece0318e5e93eaf48802fefabf512ca1dce40
auth_required: true
build_id: 2015-03-26_21-32-43
build_number: '233'
feature_groups:
- mirantis
fuellib_sha: 9c7716bc2ce6075065d7d9dcf96f4c94662c0b56
fuelmain_sha: 320b5f46fc1b2798f9e86ed7df51d3bda1686c10
nailgun_sha: b163f6fc77d6639aaffd9dd992e1ad96951c3bbf
ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867
production: docker
python-fuelclient_sha: e5e8389d8d481561a4d7107a99daae07c6ec5177
release: '6.1'

We applied workaround to increase heartbeats for OpenStack services:
root@node-1:~# grep rabbit_heart /etc/nova/nova.conf
rabbit_heartbeat=86400
root@node-1:~# grep rabbit_heart /etc/cinder/cinder.conf
rabbit_heartbeat=86400

deployed the following configuration
Baremetal,Ubuntu,IBP, Neutron-vlan,Сeph-all,Nova-debug,nova-quotas,6.1_233
Controllers:1 Computes:5

During rally tests we get errors from rally client like this:
rally client:
http://paste.openstack.org/show/205104/
/var/log/messages log:
http://paste.openstack.org/show/205105/
/var/log/nova-all log:
http://paste.openstack.org/show/205106/

similar behavior for cinder:
messages log:
http://paste.openstack.org/show/205107/
cinder-all log:
http://paste.openstack.org/show/205109/

Diagnostic Snapshot is here http://mos-scale-share.mirantis.com/fuel-snapshot-2015-04-22_13-32-38.tar.xz

[Leontii Istomin (listomin)]

similar with heat:
http://paste.openstack.org/show/205110/

[Leontii Istomin (listomin)]

rabbitmqctl status
http://paste.openstack.org/show/205111/

[Bogdan Dobrelya (bogdando)]

What is the point to set rabbit_heartbeat=86400 ? Wouldn't that mean what the channel failures will be detected at AMQP level only after 86400 seconds? Does it make sense at all?

[Leontii Istomin (listomin)]

reduce rabbit_heartbeat to 520 solved the issue.
This parameter should be lower then 580 (default timeout of rabbitmq-server)

[Bogdan Dobrelya (bogdando)]

Fuel should configure rabbit_heartbeat=520 for all OpenStack services in MOS

[Alexey Khivin (akhivin)]

https://bugs.launchpad.net/fuel/+bug/1447559

duplicate?