rabbit/Qpid reconnection retry doesn't work as expected
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
oslo.messaging |
Fix Released
|
High
|
Mehdi Abaakouk | ||
oslo.messaging (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Trusty |
Fix Released
|
High
|
James Page | ||
Utopic |
Fix Committed
|
Undecided
|
Unassigned |
Bug Description
Since oslo.messaging 1.5.0, the executor set the polling timeout to 1:
https:/
but since: https:/
the rabbitmq driver iterconsume always honors the timeout even we lost the broker the connection.
but because the executor timeout is always 1 sec, the reconnection occurs every 1 seconds instead of respecting 'rabbit_
Cheers,
sileht
---- ---- ---- ---- ----
[Impact]
* This patch along with those from LP #1408370 and LP #1338732 fixes rabbitmq reconnects
* We are backporting this to Icehouse since oslo.messaging 1.3.0
fails to reconnect to Rabbit properly, particularly nova-compute.
* This patch alond with it's dependencies metnioend above, will ensure that
multiple reconnect attempts happen by having connections timout and retry.
[Test Case]
* Start a service that uses oslo.messaging with rabbitmq e.g. nova-compute
* Stop rabbitmq while tail-F /var/log/
* Observe that nova-compute amqp times out and it is trying to reconnect
* Restart rabbitmq
* Observe that rabbitmq connection has re-established
[Regression Potential]
* None. I have tested in my local cloud environment and it appears to be
reliable.
Changed in oslo.messaging: | |
assignee: | nobody → Mehdi Abaakouk (sileht) |
milestone: | none → next-kilo |
importance: | Undecided → High |
Changed in oslo.messaging: | |
status: | Fix Committed → Fix Released |
description: | updated |
tags: | added: icehouse-backport-potential |
Changed in oslo.messaging (Ubuntu): | |
status: | New → Fix Released |
Changed in oslo.messaging (Ubuntu Trusty): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in oslo.messaging (Ubuntu Trusty): | |
status: | Triaged → In Progress |
assignee: | nobody → James Page (james-page) |
tags: | removed: icehouse-backport-potential |
Related fix proposed to branch: master /review. openstack. org/139980
Review: https:/