Failed connection attempt to broker results in many services connecting to undeclared queue
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Mirantis OpenStack | Status tracked in 10.0.x | |||||
10.0.x |
Confirmed
|
High
|
MOS Oslo |
Bug Description
Diagnostic snapshot:
http://
For an unknown reason, OpenStack services start failing to report status via RPC.
from nova-compute: http://
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
2016-04-29 11:01:00.232 74189 ERROR oslo_service.
from neutron-l3: http://
<167>Apr 29 11:35:36 node-1 neutron-
281 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 0, 'removed': 0}}. Elapsed:0.114 loop_count_and_wait /usr/lib/
<163>Apr 29 11:35:37 node-1 neutron-l3-agent: 2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
2016-04-29 11:35:37.917 32638 ERROR neutron.
<164>Apr 29 11:35:37 node-1 neutron-l3-agent: 2016-04-29 11:35:37.918 32638 WARNING oslo.service.
Rabbit broker is reporting:
=ERROR REPORT==== 29-Apr-
Channel error on connection <0.3495.25> (192.168.
operation basic.consume caused a channel exception not_found: "no queue 'reply_
=ERROR REPORT==== 29-Apr-
Channel error on connection <0.3495.25> (192.168.
operation basic.consume caused a channel exception not_found: "no queue 'reply_
=ERROR REPORT==== 29-Apr-
Channel error on connection <0.3495.25> (192.168.
operation basic.consume caused a channel exception not_found: "no queue 'reply_
Also relevant:
AMQP server on 192.168.0.138:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
Steps to reproduce:
1. Deploy Fuel 9.0-244
2. Deploy environment: 3 controllers, 20 computes+ceph, 175 computes, vxlan+dvr, ceph for all, LMA plugin
3. perform rally scenario: create_
Workaround:
Restarting the OpenStack service in question will cause a re connection and the error should disappear. NEED TO VERIFY
Impact:
Affected services do not report status and fail various tasks.
description: | updated |
Added logs inline for better search engine indexing.