Comment 0 for bug 1576672

Revision history for this message
Michael H Wilson (geekinutah) wrote :

Diagnostic snapshot:
http://mos-scale-share.mirantis.com/fuel-snapshot-2016-04-29_08-53-26.tar.gz

For an unknown reason, OpenStack services start failing to report status via RPC.

from nova-compute: http://paste.openstack.org/show/495756/
from neutron-l3: http://paste.openstack.org/show/495757/

Rabbit broker is reporting:

=ERROR REPORT==== 29-Apr-2016::11:50:35 ===
Channel error on connection <0.3495.25> (192.168.0.174:39036 -> 192.168.0.178:5673, vhost: '/', user: 'nova'), channel 1:
operation basic.consume caused a channel exception not_found: "no queue 'reply_733637fb32a84ff5bc1c956078930a28' in vhost '/'"

=ERROR REPORT==== 29-Apr-2016::11:50:35 ===
Channel error on connection <0.3495.25> (192.168.0.174:39036 -> 192.168.0.178:5673, vhost: '/', user: 'nova'), channel 1:
operation basic.consume caused a channel exception not_found: "no queue 'reply_733637fb32a84ff5bc1c956078930a28' in vhost '/'"

=ERROR REPORT==== 29-Apr-2016::11:50:35 ===
Channel error on connection <0.3495.25> (192.168.0.174:39036 -> 192.168.0.178:5673, vhost: '/', user: 'nova'), channel 1:
operation basic.consume caused a channel exception not_found: "no queue 'reply_733637fb32a84ff5bc1c956078930a28' in vhost '/'"

Also relevant:

AMQP server on 192.168.0.138:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.

Steps to reproduce:
unknown

Workaround:
Restarting the OpenStack service in question will cause a re connection and the error should disappear. NEED TO VERIFY

Impact:
Affected services do not report status and fail various tasks.