Constant exceptions "NotFound: Basic.consume: (404) NOT_FOUND - no queue abc in vhost '/'" in log

[Kirill Bespalov]

Version: 9.0

Steps to reproduce:
1. Deploy environment MOS environment.
2. Run some tests on it (exact cause is unknown yet)

Expected results:
All logs are clean

Actual results:
In one of OpenStack components log you find a lot of exceptions like
NotFound: Basic.consume: (404) NOT_FOUND - no queue 'reply_4b5920a6600d4d779c61c1a82dd7b81a' in vhost '/'
(full stack trace from neutron-server logs - http://paste.openstack.org/show/494399/)

It happens due to the next HA race condition:

(1) A cluster consists of two nodes: A and B
(2) The queue 'abc' hosted on the node A.
(3) A consumer due to reconnection declare the queue on node B (not self).
(4) The node A is down and lose the queue 'abc'.
(5) The node B delete the queue metadata (because home node is down) and does not send the basic.cancel to consumers, because in this time they are not declared.
(6) The consumer trying declare self on missing queue and recieve 404.

Loosing a queue has an impact that server stops processing messages from it, which might be crucial to its work (depends on the queue).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/315700

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/315700
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=43cfc18fc9a307b44eebdd052203de8e44dbc814
Submitter: Jenkins
Branch: master

commit 43cfc18fc9a307b44eebdd052203de8e44dbc814
Author: Kirill Bespalov <email address hidden>
Date: Thu May 12 21:00:29 2016 +0300

    Fix consuming from missing queues

    Redeclare queues on 'Basic.consume: (404) no queue' exception
    and enable by default consumer declaration with nowait=False
    in order to wait for a broker response.

    Change-Id: I99f2bc858dbc7c18a2f328ee26f39105ed17cee3
    Closes-Bug: #1581148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/319320

[Bogdan Dobrelya (bogdando)]

Please note, same applies to missing exchanges

[Bogdan Dobrelya (bogdando)]

Nevermind, it seems the patch covers recreation of exchanges as well

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/oslo.messaging 5.2.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/amqp-dispatch-router
Review: https://review.openstack.org/325916

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/325916
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=1c7ff9d60ae82bc3016503c7eb3bcdc0e228fcfc
Submitter: Jenkins
Branch: feature/amqp-dispatch-router

commit 2668177256e2c09f2db69d026655e94045c20e5f
Author: Edan David <email address hidden>
Date: Thu Jun 2 01:42:01 2016 -0400

    Fix typo: 'olso' to 'oslo'

    Change-Id: Ic86ded995ecf24b094b25c140da264526eafe611

commit a620319af4583925f02b4b14d3c5106b85726dad
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Jun 2 04:08:22 2016 +0000

    Updated from global requirements

    Change-Id: Idb293b41afef83871ebfdb451f7f6a079ce10711

commit 31691745d43be5ea74772a8a2e4072616fdf0814
Author: ozamiatin <email address hidden>
Date: Mon May 16 15:43:27 2016 +0300

    [zmq] Don't skip non-direct message types

    If using the router proxy we need to be able to
    dispatch all types of messages over the routers,
    not limiting them to direct types only.

    Also added fanout possibility to do on a client-side,
    so we can reduce latency on a proxy, and not using
    pub-sub for fanout when we don't want it.

    Change-Id: Ic88c306c1d386dd582cbccfc5719fba5668a9db8

commit 8ee19159d2aa00e175bbdadac7d0d52627334cb5
Author: ozamiatin <email address hidden>
Date: Mon May 16 17:07:30 2016 +0300

    [zmq] Refactoring of zmq client

    Split ZmqClient in order to create different clients
    for different configurations (direct/proxy).

    Change-Id: Ib374f62f53f2c82278ce5bc555ea440e8eba6617

commit 4b0e247d2d9d1c8eb5b9f904dd632bda9c706670
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue May 31 03:05:21 2016 +0000

    Updated from global requirements

    Change-Id: I5e48d3f427128c6beebdd04ec9dbfc96683cee9e

commit 9d51fa4a2fa734e347be1998f5fd2b1e35dbc0a4
Author: Andrew Smith <email address hidden>
Date: Thu May 26 11:54:29 2016 -0400

    [AMQP 1.0] Randomize host list connection attempts

    Randomize the initial selection of the host list for amqp 1.0 connection
    attempts.

    This patch:
    * randomizes initial index
    * updates failover unit tests

    Change-Id: Iab5ae684a2cd8ec0c6d0c59ffffde5eb5baf883b

commit 5fb8f269e7c03fe8252cf09dccb88ad64cc43e88
Author: JiaJunsu <email address hidden>
Date: Mon May 16 06:03:07 2016 -0700

    Modify the TransportURL's docstrings

    To match the test_urls.TestParseURL multi_creds

    Change-Id: I4a37b7bce3703aa581bc46d41690bf7978f813a6
    Close-Bug: #1529310

commit 9ccfbdd3c3b19419c370b041c0f39be6b70b622d
Author: Gevorg Davoian <email address hidden>
Date: Tue May 24 12:32:11 2016 +0300

    Fix problems after refactoring RPC client

    The patch fixes some problems arised after merging the previous
    patch which this patch depends on.

    Change-Id: If38edec19f9bffc8ac54818b2536243a9edaf7e8
    Depends-On: I608c828123b196099933d16801f8ce7d9dd3a079
    Closes-Bug: #1584889

commit 4d0f7ab652416af78e1e23d62235e868c4d1e57a
Author: Mehdi Abaakouk <email address hidden>
Date: Wed May 6 09:55:23 2015 +0200

    deprecate usage ...

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/oslo.messaging 5.3.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/319320
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=eaa30a8887b33e4b3e7d4240ecef6cec1d5aa108
Submitter: Jenkins
Branch: stable/mitaka

commit eaa30a8887b33e4b3e7d4240ecef6cec1d5aa108
Author: Kirill Bespalov <email address hidden>
Date: Thu May 12 21:00:29 2016 +0300

    Fix consuming from missing queues

    Redeclare queues on 'Basic.consume: (404) no queue' exception
    and enable by default consumer declaration with nowait=False
    in order to wait for a broker response.

    Change-Id: I99f2bc858dbc7c18a2f328ee26f39105ed17cee3
    Closes-Bug: #1581148
    (cherry picked from commit 43cfc18fc9a307b44eebdd052203de8e44dbc814)