Reply queues are accidentally not found when services try to consume from them
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Fix Released
|
Critical
|
Dmitry Mescheryakov | ||
5.1.x |
Fix Released
|
Critical
|
Alexey Khivin | ||
6.0.x |
Fix Released
|
Critical
|
Alexey Khivin | ||
6.1.x |
Fix Released
|
Critical
|
Alexey Khivin | ||
7.0.x |
Fix Released
|
Critical
|
Dmitry Mescheryakov |
Bug Description
VERSION:
feature_groups:
- mirantis
- experimental
production: "docker"
release: "6.0"
api: "1.0"
build_number: "58"
build_id: "2014-12-
astute_sha: "16b252d93be6aa
fuellib_sha: "fde8ba5e11a1ac
ostf_sha: "a9afb68710d809
nailgun_sha: "5f91157daa6798
fuelmain_sha: "81d38d6f2903b5
Nova scheduler loses rabbit queue while booting new instance. Queue with that name exists(according to list queues). Restarting nova-scheduler solves the problem.
The same is true for other services (cinder, neutron), so this is not nova-specific.
Diagnostic snapshot: https:/
node-2 nova-api Connected to AMQP server on 127.0.0.1:5673
<182>Jan 29 13:04:45 node-2 nova-api 152.90.66.10 "POST /v2/9168b29b5b1
<179>Jan 29 13:04:45 node-2 nova-conductor Failed to consume message from queue: Basic.consume: (404) NOT_FOUND - no queue 'reply_
' in vhost '/'
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
, in ensure
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
, in _consume
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
, in consume
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
ethod
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
2015-01-29 13:04:45.355 51229 TRACE oslo.messaging.
9' in vhost '/'
node-2 nova-conductor Exception during scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
2015-01-29 13:05:45.404 51229 TRACE nova.scheduler.
<180>Jan 29 13:05:45 node-2 nova-conductor Setting instance to ERROR state.
tags: | added: nova |
tags: |
added: messaging removed: nova |
summary: |
- Nova scheduler loses rabbit queue + Reply queues are accidentally not found when services try to consume + from them |
Changed in mos: | |
status: | New → Incomplete |
assignee: | nobody → MOS Oslo (mos-oslo) |
description: | updated |
description: | updated |
Changed in mos: | |
importance: | Undecided → High |
importance: | High → Critical |
Changed in mos: | |
status: | Incomplete → In Progress |
tags: | added: sahara |
tags: | removed: sahara |
Changed in mos: | |
assignee: | MOS Oslo (mos-oslo) → Alex Khivin (akhivin) |
Changed in mos: | |
status: | In Progress → Won't Fix |
status: | Won't Fix → In Progress |
Changed in mos: | |
status: | In Progress → Fix Released |
status: | Fix Released → Fix Committed |
tags: | added: oslo.messaging |
tags: | added: scale |
tags: | added: on-verification |
tags: | removed: on-verification |
tags: | added: support |
I took a look at the diagnostic snapshot. It's still not clear what exactly happens, but must have nothing to do with nova services, as I can see the same error in neutron and cinder logs.
Updated the description accordingly. I'd like the messaging guys to take a look.
I wonder, if this may be caused by x-expires set to 3600000 on reply queues (+ those are not durable)