rabbit: starvation of connections for reply
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Ubuntu Cloud Archive |
Fix Released
|
Undecided
|
Unassigned | |||
Juno |
Won't Fix
|
Medium
|
Unassigned | |||
Kilo |
Fix Released
|
Undecided
|
Corey Bryant | |||
Liberty |
Fix Released
|
Undecided
|
Unassigned | |||
oslo.messaging |
Fix Released
|
High
|
Mehdi Abaakouk | |||
oslo.messaging (Ubuntu) | ||||||
Trusty |
Fix Released
|
Medium
|
Corey Bryant | |||
Vivid |
Won't Fix
|
Medium
|
Corey Bryant | |||
python-oslo.messaging (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | |||
Wily |
Fix Released
|
Medium
|
Corey Bryant |
Bug Description
Hi,
When a client died/restart/stop but was waiting more replies that the rpc_connection_pool size,
the server will hold all connections from the pool during the retry logic in case of the client come back with the same reply_queue_id (that occurs only is rabbit is restart, not the client).
Cheers,
-------
[Impact]
This patch addresses an issue when multiple clients lost networks/
[Test Case]
Note steps are for trusty-icehouse, including latest oslo.messaging library (1.3.0-0ubuntu1.2 at the time of this writing).
Deploy an OpenStack cloud w/ multiple rabbit nodes and multiple nova compute hosts then cut off the network between OpenStack services and RabbitMQ. Observe that the nova-conductor is infinitely reconnecting to rabbit nodes.
[Regression Potential]
None.
Related branches
- Ubuntu Cloud Archive Team: Pending requested
-
Diff: 10493 lines (+10090/-24)32 files modified.pc/.quilt_patches (+1/-0)
.pc/.quilt_series (+1/-0)
.pc/.version (+1/-0)
.pc/0001-Fix-_poll_connection-not-timeout-issue-1-2.patch/oslo/messaging/_drivers/amqpdriver.py (+444/-0)
.pc/0001-Fix-_poll_connection-not-timeout-issue-1-2.patch/oslo/messaging/_drivers/common.py (+349/-0)
.pc/0001-Fix-_poll_connection-not-timeout-issue-1-2.patch/oslo/messaging/_drivers/impl_rabbit.py (+821/-0)
.pc/0002-Fix-_poll_connection-not-timeout-issue-2-2.patch/oslo/messaging/_drivers/common.py (+374/-0)
.pc/0002-Fix-_poll_connection-not-timeout-issue-2-2.patch/tests/test_utils.py (+49/-0)
.pc/0003-qpid-Always-auto-delete-queue-of-DirectConsumer.patch/oslo/messaging/_drivers/impl_qpid.py (+729/-0)
.pc/0004-rabbit-redeclare-consumers-when-ack-requeue-fail.patch/oslo/messaging/_drivers/impl_rabbit.py (+832/-0)
.pc/0004-rabbit-redeclare-consumers-when-ack-requeue-fail.patch/tests/drivers/test_impl_rabbit.py (+735/-0)
.pc/0005-Fix-possible-usage-of-undefined-variable.patch/oslo/messaging/_drivers/impl_rabbit.py (+839/-0)
.pc/0006-Declare-DirectPublisher-exchanges-with-passive-True.patch/oslo/messaging/_drivers/common.py (+374/-0)
.pc/0006-Declare-DirectPublisher-exchanges-with-passive-True.patch/oslo/messaging/_drivers/impl_rabbit.py (+839/-0)
.pc/0006-Declare-DirectPublisher-exchanges-with-passive-True.patch/tests/drivers/test_impl_rabbit.py (+756/-0)
.pc/0006-Declare-DirectPublisher-exchanges-with-passive-True.patch/tests/test_utils.py (+64/-0)
.pc/0007-fix-rabbit-starvation-of-connections-for-reply.patch/oslo/messaging/_drivers/amqp.py (+246/-0)
.pc/0007-fix-rabbit-starvation-of-connections-for-reply.patch/oslo/messaging/_drivers/amqpdriver.py (+452/-0)
.pc/0007-fix-rabbit-starvation-of-connections-for-reply.patch/oslo/messaging/_drivers/impl_rabbit.py (+865/-0)
.pc/applied-patches (+8/-0)
.pc/zmq-server-routing.patch/oslo/messaging/_drivers/impl_zmq.py (+995/-0)
debian/changelog (+8/-0)
debian/patches/0007-fix-rabbit-starvation-of-connections-for-reply.patch (+116/-0)
debian/patches/series (+1/-0)
oslo/messaging/_drivers/amqp.py (+4/-0)
oslo/messaging/_drivers/amqpdriver.py (+48/-11)
oslo/messaging/_drivers/common.py (+26/-0)
oslo/messaging/_drivers/impl_qpid.py (+0/-1)
oslo/messaging/_drivers/impl_rabbit.py (+38/-10)
oslo/messaging/_drivers/impl_zmq.py (+2/-0)
tests/drivers/test_impl_rabbit.py (+41/-2)
tests/test_utils.py (+32/-0)
- Michael Terry: Needs Fixing
-
Diff: 1959 lines (+1820/-27)10 files modified.pc/0005-fix-rabbit-starvation-of-connections-for-reply.patch/oslo/messaging/_drivers/amqp.py (+342/-0)
.pc/0005-fix-rabbit-starvation-of-connections-for-reply.patch/oslo/messaging/_drivers/amqpdriver.py (+467/-0)
.pc/0005-fix-rabbit-starvation-of-connections-for-reply.patch/oslo/messaging/_drivers/impl_rabbit.py (+834/-0)
.pc/applied-patches (+1/-0)
debian/changelog (+8/-0)
debian/patches/0005-fix-rabbit-starvation-of-connections-for-reply.patch (+123/-0)
debian/patches/series (+1/-0)
oslo/messaging/_drivers/amqp.py (+4/-0)
oslo/messaging/_drivers/amqpdriver.py (+32/-3)
oslo/messaging/_drivers/impl_rabbit.py (+8/-24)
Changed in oslo.messaging: | |
assignee: | nobody → Mehdi Abaakouk (sileht) |
importance: | Undecided → High |
description: | updated |
summary: |
- rabbit: starvation of connection pool + rabbit: starvation of connections for reply |
Changed in oslo.messaging: | |
milestone: | none → 3.1.0 |
status: | Fix Committed → Fix Released |
description: | updated |
Changed in oslo.messaging (Ubuntu Vivid): | |
status: | New → Fix Released |
Changed in oslo.messaging (Ubuntu): | |
importance: | Undecided → Medium |
Changed in oslo.messaging (Ubuntu Trusty): | |
importance: | Undecided → Medium |
Changed in oslo.messaging (Ubuntu Vivid): | |
importance: | Undecided → Medium |
tags: | added: sts-sru |
no longer affects: | python-oslo.messaging (Ubuntu Trusty) |
no longer affects: | python-oslo.messaging (Ubuntu Vivid) |
no longer affects: | oslo.messaging (Ubuntu Wily) |
no longer affects: | cloud-archive/juno |
no longer affects: | cloud-archive |
Changed in python-oslo.messaging (Ubuntu Wily): | |
assignee: | nobody → Corey Bryant (corey.bryant) |
Changed in oslo.messaging (Ubuntu Vivid): | |
assignee: | nobody → Corey Bryant (corey.bryant) |
Changed in oslo.messaging (Ubuntu Trusty): | |
assignee: | nobody → Corey Bryant (corey.bryant) |
Changed in oslo.messaging (Ubuntu): | |
status: | New → Invalid |
Changed in python-oslo.messaging (Ubuntu): | |
status: | New → Invalid |
Changed in cloud-archive: | |
status: | New → Invalid |
Changed in python-oslo.messaging (Ubuntu Wily): | |
status: | New → In Progress |
Changed in oslo.messaging (Ubuntu Vivid): | |
status: | Fix Released → In Progress |
Changed in python-oslo.messaging (Ubuntu Wily): | |
importance: | Undecided → Medium |
Changed in cloud-archive: | |
importance: | Undecided → Critical |
importance: | Critical → Undecided |
Changed in oslo.messaging (Ubuntu): | |
importance: | Medium → Undecided |
Changed in oslo.messaging (Ubuntu Vivid): | |
status: | In Progress → New |
no longer affects: | oslo.messaging (Ubuntu) |
Changed in oslo.messaging (Ubuntu Trusty): | |
status: | New → Triaged |
tags: | removed: verification-kilo-needed |
Changed in cloud-archive: | |
status: | Invalid → Fix Released |
Changed in python-oslo.messaging (Ubuntu): | |
status: | Invalid → Fix Released |
importance: | Undecided → Medium |
tags: | removed: sts-sru |
Fix proposed to branch: master /review. openstack. org/252351
Review: https:/