Disappeared queue may block rpc calls until timeout

[Alexey Khivin]

There was a code in oslo.messaging
file impl_rabbit.py:

for queue in queues_head:
     queue.consume(nowait=True)
queues_tail.consume(nowait=False)

https://github.com/openstack/oslo.messaging/blob/stable/juno/oslo/messaging/_drivers/impl_rabbit.py#L736-L742

If some queue was removed then exception will be raised

If rabbitmq was restarted and after reconnection queue was removed then consumption will be blocked until timeout for the first queue

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alex Khivin <email address hidden>
Review: https://review.fuel-infra.org/7925

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/7925
Submitter: Oleksii Zamiatin <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: e9a91becba156715706504a0bfb1cf587b5343ad
Author: Alex Khivin <email address hidden>
Date: Mon Jun 22 11:25:26 2015

Improve "Queue not found" exception handling

In the case queue was disapeared during reconnection process, "Queue not
found" exception may break consumption in other queues, thus rpc subsystem
may got stuck.

Added a new method _try_consume() to consume queue. If queue is not found,
this method reconnect queue (this supposes to re-create a lost queue),
and try consume it one more time.

Change-Id: I41ffe7aacbae1ac176e0063a20cdd256cef69127
Closes-bug: #1465757
Closes-bug: #1463802

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Alex Khivin <email address hidden>
Review: https://review.fuel-infra.org/8461

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/8461
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 0a4b32d018392506b10635bdffda419bb0ccec05
Author: Alex Khivin <email address hidden>
Date: Wed Jun 24 22:57:37 2015

Improve "Queue not found" exception handling

In the case queue was disapeared during reconnection process, "Queue not
found" exception may break consumption in other queues, thus rpc subsystem
may got stuck.

Added a new method _try_consume() to consume queue. If queue is not found,
this method reconnect queue (this supposes to re-create a lost queue),
and try consume it one more time.

Change-Id: I41ffe7aacbae1ac176e0063a20cdd256cef69127
Closes-bug: #1465757
Closes-bug: #1463802
(cherry picked from commit e9a91becba156715706504a0bfb1cf587b5343ad)

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Change author: Alex Khivin <email address hidden>
Review: https://review.fuel-infra.org/9335

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-5.1-updates/2014.1.1
Change author: Alex Khivin <email address hidden>
Review: https://review.fuel-infra.org/9337

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/9335
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1.1-updates/2014.1.1

Commit: 022c6c28dd73e7a76270b76c4cbe895c9750e7cb
Author: Alex Khivin <email address hidden>
Date: Tue Jul 14 15:54:34 2015

Improve "Queue not found" exception handling

In the case queue was disapeared during reconnection process, "Queue not
found" exception may break consumption in other queues, thus rpc subsystem
may got stuck.

Added a new method _try_consume() to consume queue. If queue is not found,
this method reconnect queue (this supposes to re-create a lost queue),
and try consume it one more time.

Closes-bug: #1465757
Closes-bug: #1463802

Change-Id: I41ffe7aacbae1ac176e0063a20cdd256cef69127

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alexey Khivin <email address hidden>
Review: https://review.fuel-infra.org/10420

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10420
Submitter: Oleksii Zamiatin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 750fcc8f96761b512226d696271c9d67db56240b
Author: Alexey Khivin <email address hidden>
Date: Thu Aug 13 17:55:01 2015

Improve "Queue not found" exception handling

In the case queue was disapeared during reconnection process, "Queue not
found" exception may break consumption in other queues, thus rpc subsystem
may got stuck.

Added a new method _try_consume() to consume queue. If queue is not found,
this method reconnect queue (this supposes to re-create a lost queue),
and try consume it one more time.

Also fixed a strange test in test_utils.py

Change-Id: I41ffe7aacbae1ac176e0063a20cdd256cef69127
Closes-bug: #1465757
Closes-bug: #1463802
Closes-bug: #1415932

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Alexey Khivin <email address hidden>
Review: https://review.fuel-infra.org/13463

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Dmitry Mescheryakov <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/13463
Reason: It seems like the bug, fixed by this CR, is also fixed in upstream with a different approach: https://review.openstack.org/#/c/195688/2

The upstream fix is present in 8.0 and so we are not going to merge current change. We will reconsider it only if the issue reoccurs.