oslo.messaging does not redeclare exchange if it is missing

[Dmitry Mescheryakov]

Version: 9.0

Steps to reproduce:
1. Create a load on OpenStack, trigger restart of one of RabbitMQ nodes (exact reason unknown).
2. Observe a lot of
operation basic.publish caused a channel exception not_found: "no exchange 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/'"
   entries in RabbitMQ log for various reply queues.

Looking earlier in the RabbitMQ log one can found
2016-08-03T13:33:12.945437+00:00 notice: operation queue.declare caused a channel exception not_found: "failed to perform operation on queue 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/' due to timeout"
2016-08-03T13:34:43.006600+00:00 notice: operation queue.bind caused a channel exception not_found: "no exchange 'reply_d8786e66456a4660bebb362668a027e4' in vhost '/'"

The following stacktrace corresponds to the later message in nova-compute.log: http://paste.openstack.org/show/548803/

It seems that during RabbitMQ failover we might end up with declared queue, which is not bound to an exchange (there an exception listed in paste above is thrown). Later oslo.messaging successfully starts consuming from that queue, since the queue exists, but it is useless because it is not bound to an exchange.

You may find the whole logs containing snippets from above attached - nova-compute.log and rabbitmq.log.3.gz

[Eugene Nikanorov (enikanorov)]

I'm raising the importance.
That kind of bug will create whole-cloud outage cases.

[Dmitry Mescheryakov (dmitrymex)]

The fix itself is here - https://review.fuel-infra.org/#/c/25175/

[Alexey Galkin (agalkin)]

We can't reproduce this bug because of "1. Create a load on OpenStack, trigger restart of one of RabbitMQ nodes (exact reason unknown)." , consequently we set 'Fix released' status.