Constant exceptions "NotFound: Basic.consume: (404) NOT_FOUND - no queue abc in vhost '/'" in log

[Dmitry Mescheryakov]

Version: 9.0

Steps to reproduce:
1. Deploy environment MOS environment.
2. Run some tests on it (exact cause is unknown yet)

Expected results:
All logs are clean

Actual results:
In one of OpenStack components log you find a lot of exceptions like
NotFound: Basic.consume: (404) NOT_FOUND - no queue 'reply_4b5920a6600d4d779c61c1a82dd7b81a' in vhost '/'
(full stack trace from neutron-server logs - http://paste.openstack.org/show/494399/)

This indicates that process lost a queue it was listening on and the situation does not end by itself. Loosing a queue has an impact that server stops processing messages from it, which might be crucial to its work (depends on the queue).

In rabbit logs on node-61 with grep one can find the following entries (only several earliest are shown):
http://paste.openstack.org/show/494589/
Note the pattern - first two queue.declare operations timed out and then basic.consume fail in endless loop.

It seems that RabbitMQ failed to create the queue due to overload or something and oslo.messaging did not notice that. Unfortunately the relevant neutron-server logs were already rotated, so it is not clear what happened in oslo.messaging at the time of the queue declaration.

[Leontii Istomin (listomin)]

has been reproduced on node-119 http://paste.openstack.org/show/495774/
from this diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-04-29_08-53-26.tar.gz

[Bogdan Dobrelya (bogdando)]

Note, this must be fixed upstream, the similar way as it was done for queues https://review.openstack.org/#/c/315700/

[Bogdan Dobrelya (bogdando)]

btw, do we have https://review.openstack.org/#/c/315700/ backported?

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: Kirill Bespalov <email address hidden>
Review: https://review.fuel-infra.org/21134

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21134
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: f739aa5af33b77e3718b3e594798d02cad2850a5
Author: Kirill Bespalov <email address hidden>
Date: Mon May 23 14:04:31 2016

Fix consuming from missing queues

Redeclare queues on 'Basic.consume: (404) no queue' exception
and enable by default consumer declaration with nowait=False
in order to wait for a broker response.

Change-Id: I99f2bc858dbc7c18a2f328ee26f39105ed17cee3
Closes-Bug: #1579128
Closes-Bug: #1572085

[Dmitry Mescheryakov (dmitrymex)]

Fix for 10.0 is merged upstream here https://review.openstack.org/#/c/315700/
Fix for 9.0 was backported, see the above comment.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Kirill Bespalov <email address hidden>
Review: https://review.fuel-infra.org/28255

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/28255
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: acfb7d187c984b83a3b66716f9390cf92ed54f05
Author: Kirill Bespalov <email address hidden>
Date: Mon Nov 14 11:49:58 2016

Fix consuming from missing queues

Redeclare queues on 'Basic.consume: (404) no queue' exception
and enable by default consumer declaration with nowait=False
in order to wait for a broker response.

Change-Id: I99f2bc858dbc7c18a2f328ee26f39105ed17cee3
Closes-Bug: #1579128
Closes-Bug: #1572085

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Kirill Bespalov <email address hidden>
Review: https://review.fuel-infra.org/29003

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29003
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 250c29274fa05e4810ad470be2bb95f99e7e7524
Author: Kirill Bespalov <email address hidden>
Date: Mon May 22 15:32:27 2017

Fix consuming from missing queues

Redeclare queues on 'Basic.consume: (404) no queue' exception
and enable by default consumer declaration with nowait=False
in order to wait for a broker response.

Change-Id: I99f2bc858dbc7c18a2f328ee26f39105ed17cee3
Closes-Bug: #1579128
Closes-Bug: #1572085

[TatyanaGladysheva (tgladysheva)]

Verified on 7.0 + mu8 updates.

[Dmitry (dtsapikov)]

Verified on 8.0+mu5