| 2016-08-04 12:05:30 |
Kirill Bespalov |
bug |
|
|
added bug |
| 2016-08-04 12:05:30 |
Kirill Bespalov |
attachment added |
|
nova-compute.log https://bugs.launchpad.net/bugs/1609766/+attachment/4713977/+files/nova-compute.log |
|
| 2016-08-04 12:05:56 |
Kirill Bespalov |
attachment added |
|
rabbitmq.log.3.gz https://bugs.launchpad.net/oslo.messaging/+bug/1609766/+attachment/4713978/+files/rabbitmq.log.3.gz |
|
| 2016-08-04 12:06:31 |
Kirill Bespalov |
oslo.messaging: assignee |
|
Kirill Bespalov (k-besplv) |
|
| 2016-08-04 12:56:22 |
OpenStack Infra |
oslo.messaging: status |
New |
In Progress |
|
| 2016-08-04 16:19:09 |
Dmitry Mescheryakov |
bug |
|
|
added subscriber Dmitry Mescheryakov |
| 2016-08-11 13:09:14 |
Dmitry Mescheryakov |
description |
Steps to reproduce:
1. Create a load on OpenStack, trigger restart of one of RabbitMQ nodes (exact reason unknown).
2. Observe a lot of
operation basic.publish caused a channel exception not_found: "no exchange 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/'"
entries in RabbitMQ log for various reply queues.
Looking earlier in the RabbitMQ log one can found
2016-08-03T13:33:12.945437+00:00 notice: operation queue.declare caused a channel exception not_found: "failed to perform operation on queue 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/' due to timeout"
2016-08-03T13:34:43.006600+00:00 notice: operation queue.bind caused a channel exception not_found: "no exchange 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/'"
The following stacktrace corresponds to the later message in nova-compute.log: http://paste.openstack.org/show/548803/
It seems that during RabbitMQ failover we might end up with declared queue, which is not bound to an exchange (there an exception listed in paste above is thrown). Later oslo.messaging successfully starts consuming from that queue, since the queue exists, but it is useless because it is not bound to an exchange.
You may find the whole logs containing snippets from above attached - nova-compute.log and rabbitmq.log.3.gz |
Steps to reproduce:
1. Create a load on OpenStack, trigger restart of one of RabbitMQ nodes (exact reason unknown).
2. Observe a lot of
operation basic.publish caused a channel exception not_found: "no exchange 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/'"
entries in RabbitMQ log for various reply queues.
Looking earlier in the RabbitMQ log one can found
2016-08-03T13:33:12.945437+00:00 notice: operation queue.declare caused a channel exception not_found: "failed to perform operation on queue 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/' due to timeout"
2016-08-03T13:34:43.006600+00:00 notice: operation queue.bind caused a channel exception not_found: "no exchange 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/'"
The following stacktrace corresponds to the later message in nova-compute.log: http://paste.openstack.org/show/548803/
It seems that during RabbitMQ failover we might end up with declared queue, which is not bound to an exchange (there an exception listed in paste above is thrown). Later oslo.messaging successfully starts consuming from that queue, since the queue exists, but it is useless because it is not bound to an exchange.
You may find the whole logs containing snippets from above attached - nova-compute.log and rabbitmq.log.3.gz
Versions:
oslo.messaging - stable/mitaka
RabbitMQ - 3.6.1
kombu 3.0.32
pyamqp 1.4.8 |
|
| 2016-09-06 09:02:53 |
OpenStack Infra |
oslo.messaging: assignee |
Kirill Bespalov (k-besplv) |
Mehdi Abaakouk (sileht) |
|
| 2016-09-12 10:27:59 |
OpenStack Infra |
oslo.messaging: assignee |
Mehdi Abaakouk (sileht) |
Kirill Bespalov (k-besplv) |
|
| 2016-09-12 15:46:05 |
Gustavo Randich |
bug |
|
|
added subscriber Gustavo Randich |
| 2016-09-12 17:01:38 |
OpenStack Infra |
oslo.messaging: status |
In Progress |
Fix Released |
|
| 2016-10-20 14:38:15 |
OpenStack Infra |
tags |
|
in-stable-newton |
|
| 2016-10-27 10:40:30 |
OpenStack Infra |
tags |
in-stable-newton |
in-stable-mitaka in-stable-newton |
|
