[messaging] QPID broadcast RPC requests to all servers for a given topic

[Ken Giusti]

According to the oslo.messaging documentation, when a RPC request is made to a given topic, and there are multiple servers for that topic, only _one_ server should service that RPC request. See http://docs.openstack.org/developer/oslo.messaging/target.html

"topic (str) – A name which identifies the set of interfaces exposed by a server. Multiple servers may listen on a topic and messages will be dispatched to one of the servers in a round-robin fashion."

In the case of a QPID-based deployment using topology version 2, this is not the case. Instead, each listening server gets a copy of the RPC and will process it.

For more detail, see

https://bugs.launchpad.net/oslo/+bug/1178375/comments/26

[Ken Giusti (kgiusti)]

Just to clarify:

* This bug only affects QPID deployments that:

    - have the fix for https://bugs.launchpad.net/oslo/+bug/1178375
    - AND are using qpid_topology_version=2

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/59837
Committed: http://github.com/openstack/oslo-incubator/commit/ef406a21782134aeefb944f74b3f1a47d6169318
Submitter: Jenkins
Branch: master

commit ef406a21782134aeefb944f74b3f1a47d6169318
Author: Kenneth Giusti <email address hidden>
Date: Tue Dec 3 15:59:28 2013 -0500

    Create a shared queue for QPID topic consumers

    When multiple RPC servers (consumers) are subscribed to the same RPC
    topic, a single RPC request to that topic should be received by only
    one of the consumers. A bug in the QPID driver caused every consumer
    to receive a copy of the RPC request. This bug affects only Topology
    version 2. This patch will cause a single queue to be created for
    each topic, and shared among all consumers of that topic. This
    results in each RPC request being received by only one consumer,
    in turn across all the competing consumers.

    Closes-Bug: #1257293
    Change-Id: Id6af86f7fb536f3caff50bec4d5fde8a3ac1e83a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60385
Committed: http://github.com/openstack/oslo-incubator/commit/e355fa3a9f49ff524ce98d1263a784b5a7929dfb
Submitter: Jenkins
Branch: stable/havana

commit e355fa3a9f49ff524ce98d1263a784b5a7929dfb
Author: Kenneth Giusti <email address hidden>
Date: Tue Dec 3 15:59:28 2013 -0500

    Create a shared queue for QPID topic consumers

    When multiple RPC servers (consumers) are subscribed to the same RPC
    topic, a single RPC request to that topic should be received by only
    one of the consumers. A bug in the QPID driver caused every consumer
    to receive a copy of the RPC request. This bug affects only Topology
    version 2. This patch will cause a single queue to be created for
    each topic, and shared among all consumers of that topic. This
    results in each RPC request being received by only one consumer,
    in turn across all the competing consumers.

    Closes-Bug: #1257293
    Change-Id: Id6af86f7fb536f3caff50bec4d5fde8a3ac1e83a
    (cherry picked from commit ef406a21782134aeefb944f74b3f1a47d6169318)

[Flavio Percoco (flaper87)]

Cinder Patch: https://review.openstack.org/#/c/60464/

[Flavio Percoco (flaper87)]

Nova patch: https://review.openstack.org/#/c/60468/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60475

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60478

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60479

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60484

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60487

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60488

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60501

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60514

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60524

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60525

[Flavio Percoco (flaper87)]

Neutron stable/havana: https://review.openstack.org/#/c/60525/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60533

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60475
Committed: http://github.com/openstack/ceilometer/commit/3357cc982f2c1389cd7f53e8451b061dcd604979
Submitter: Jenkins
Branch: master

commit 3357cc982f2c1389cd7f53e8451b061dcd604979
Author: Eoghan Glynn <email address hidden>
Date: Fri Dec 6 10:52:38 2013 +0000

    Create a shared queue for QPID topic consumers

    Fixes bug 1257293

    This is a cherrypick of oslo-incubator:
    ef406a21782134aeefb944f74b3f1a47d6169318

    From the oslo-incubator commit:
    When multiple RPC servers (consumers) are subscribed to the same RPC
    topic, a single RPC request to that topic should be received by only
    one of the consumers. A bug in the QPID driver caused every consumer
    to receive a copy of the RPC request. This bug affects only Topology
    version 2. This patch will cause a single queue to be created for
    each topic, and shared among all consumers of that topic. This
    results in each RPC request being received by only one consumer,
    in turn across all the competing consumers.

    Change-Id: I97c6eb5fe63322ab70716c63074818ff1b6ea82b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60524
Committed: http://github.com/openstack/neutron/commit/7afd0f4d514d25ff8469b4fc8026363120a33d57
Submitter: Jenkins
Branch: master

commit 7afd0f4d514d25ff8469b4fc8026363120a33d57
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 15:35:14 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    e227c0e Properly reconnect subscribing clients when QPID broker restarts
    ef406a2 Create a shared queue for QPID topic consumers

    Change-Id: I286edf6bc4a677aa61f60da785802c19878c79c7
    Closes-bug: #1251757
    Closes-bug: #1257293

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60501
Committed: http://github.com/openstack/keystone/commit/9cc791d73ad57c9262c50e92f3bef1dd2bba215e
Submitter: Jenkins
Branch: master

commit 9cc791d73ad57c9262c50e92f3bef1dd2bba215e
Author: Alan Pevec <email address hidden>
Date: Fri Dec 6 14:12:47 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293

    Change-Id: I45257c62168163d2d4ceda994c94ff2d16a27300

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60478
Committed: http://github.com/openstack/nova/commit/0e73913f1054a44e99b2a5a15ee8c9c32287513d
Submitter: Jenkins
Branch: stable/havana

commit 0e73913f1054a44e99b2a5a15ee8c9c32287513d
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:25:59 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: Ic21e54efa3a0263d53481ef500fb949544dae1d2

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60488
Committed: http://github.com/openstack/heat/commit/3fc572eb9ccb3f1ab948a2bff18e35d6d6b4a088
Submitter: Jenkins
Branch: stable/havana

commit 3fc572eb9ccb3f1ab948a2bff18e35d6d6b4a088
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 12:52:22 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers

    Closes-bug: #1257293
    Change-Id: Id0e1d9704ef1cb506ae68c3848f8615e7e364484

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60484
Committed: http://github.com/openstack/ceilometer/commit/4ce68fb8e30b3c4253939413dcd6b8b6a4443d2e
Submitter: Jenkins
Branch: stable/havana

commit 4ce68fb8e30b3c4253939413dcd6b8b6a4443d2e
Author: Eoghan Glynn <email address hidden>
Date: Fri Dec 6 10:52:38 2013 +0000

    Create a shared queue for QPID topic consumers

    Fixes bug 1257293

    This is a cherrypick of oslo-incubator:
    e355fa3a9f49ff524ce98d1263a784b5a7929dfb

    From the oslo-incubator commit:
    When multiple RPC servers (consumers) are subscribed to the same RPC
    topic, a single RPC request to that topic should be received by only
    one of the consumers. A bug in the QPID driver caused every consumer
    to receive a copy of the RPC request. This bug affects only Topology
    version 2. This patch will cause a single queue to be created for
    each topic, and shared among all consumers of that topic. This
    results in each RPC request being received by only one consumer,
    in turn across all the competing consumers.

    Change-Id: I97c6eb5fe63322ab70716c63074818ff1b6ea82b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60525
Committed: http://github.com/openstack/neutron/commit/9f9792972ff81f244931d830194aae979604ffa0
Submitter: Jenkins
Branch: stable/havana

commit 9f9792972ff81f244931d830194aae979604ffa0
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 15:35:14 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    e227c0e Properly reconnect subscribing clients when QPID broker restarts
    ef406a2 Create a shared queue for QPID topic consumers

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: I286edf6bc4a677aa61f60da785802c19878c79c7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60468
Committed: http://github.com/openstack/nova/commit/1af403ff66026b6616f683c6414851e51619088a
Submitter: Jenkins
Branch: master

commit 1af403ff66026b6616f683c6414851e51619088a
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:25:59 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293

    Change-Id: I917c7a6e5cb64c6383e74d0fb6e095ad17d4d478

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60533
Committed: http://github.com/openstack/cinder/commit/2b2871cd4defbad0878ceb0d1dfc7803cf7c8a83
Submitter: Jenkins
Branch: stable/havana

commit 2b2871cd4defbad0878ceb0d1dfc7803cf7c8a83
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:06:09 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: I934a6ea6e9eb510513640870750ea37e6f076df9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60856

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60464
Committed: http://github.com/openstack/cinder/commit/cbd7acfc294c4bf2ae0014bcf04646382bed469d
Submitter: Jenkins
Branch: master

commit cbd7acfc294c4bf2ae0014bcf04646382bed469d
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:06:09 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Change-Id: I934a6ea6e9eb510513640870750ea37e6f076df9
    Closes-bug: #1251757
    Closes-bug: #1257293

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60856
Committed: http://github.com/openstack/heat/commit/6dcdb2e41e780680398c23434d32e54c5f23c2f7
Submitter: Jenkins
Branch: master

commit 6dcdb2e41e780680398c23434d32e54c5f23c2f7
Author: Steven Hardy <email address hidden>
Date: Mon Dec 9 15:20:00 2013 +0000

    Sync oslo rpc

    Sync openstack/common/rpc to fbbb12ea5ff51df744d3de1f312bbcf75aea1670
    so we get the fixes required for bug 1257293

    Change-Id: Ib2f13d1af91e005c5ee74d462c3fb307f699e7ca
    Closes-Bug: #1257293

[Ihar Hrachyshka (ihar-hrachyshka)]

Hello, I'm new to qpid and oslo.messaging, trying to get more insight into those modules and verifying the bug as fixed. It seems that incorrect behaviour seen in https://bugs.launchpad.net/oslo/+bug/1178375/comments/26 is still present in the latest oslo.messaging.

Here is what I did to verify the fix:

1. Installed Fedora 20 with latest updates.
2. installed qpid-cpp-server, started service.
3. get oslo.messaging: https://github.com/openstack/oslo.messaging.git
4. installed oslo.messaging and requirements with pip, installed qpid-python.
5. get oslo server and client for testing: https://github.com/kgiusti/oslo-messaging-clients.git
6. ran two testing servers with topology=1, sent rpc request to them -> got single server action triggered (correct).
7. ran two testing servers with topology=2, sent rpc request to them -> got both server actions triggered (wrong).

In case of topology=1, it looks as follows:

[root@localhost oslo-messaging-clients]# ./my-server.py --topology=1 server01 &
[1] 2634
[root@localhost oslo-messaging-clients]# Running server, name=server01 exchange=my-exchange topic=my-topic namespace=my-namespace
Using QPID topology version 1

[root@localhost oslo-messaging-clients]# ./my-server.py --topology=1 server02 &
[2] 2656
[root@localhost oslo-messaging-clients]# Running server, name=server02 exchange=my-exchange topic=my-topic namespace=my-namespace
Using QPID topology version 1

[root@localhost oslo-messaging-clients]# ./my-client.py --topology=1 my-topic methodB arg1 arg2
Calling server on topic my-topic, server=None exchange=my-exchange namespace=my-namespace fanout=None
Method=methodB, args={'arg1': 'arg2'}
Using QPID topology version 1
server01::TestEndpoint02::methodB( ctxt={'application': u'my-client', 'cast': None, 'time': u'Mon Dec 23 14:10:04 2013'} arg={u'arg1': u'arg2'} ) called!!!
Return value={u'application': u'my-client', u'time': u'Mon Dec 23 14:10:04 2013', u'cast': None}
[root@localhost oslo-messaging-clients]# ./my-client.py --topology=1 my-topic methodB arg1 arg2
Calling server on topic my-topic, server=None exchange=my-exchange namespace=my-namespace fanout=None
Method=methodB, args={'arg1': 'arg2'}
Using QPID topology version 1
server02::TestEndpoint02::methodB( ctxt={'application': u'my-client', 'cast': None, 'time': u'Mon Dec 23 14:10:07 2013'} arg={u'arg1': u'arg2'} ) called!!!
Return value={u'application': u'my-client', u'time': u'Mon Dec 23 14:10:07 2013', u'cast': None}
[root@localhost oslo-messaging-clients]#
[root@localhost oslo-messaging-clients]# ./my-client.py --topology=1 my-topic methodB arg1 arg2
Calling server on topic my-topic, server=None exchange=my-exchange namespace=my-namespace fanout=None
Method=methodB, args={'arg1': 'arg2'}
Using QPID topology version 1
server01::TestEndpoint02::methodB( ctxt={'application': u'my-client', 'cast': None, 'time': u'Mon Dec 23 14:10:09 2013'} arg={u'arg1': u'arg2'} ) called!!!
Return value={u'application': u'my-client', u'time': u'Mon Dec 23 14:10:09 2013', u'cast': None}
[root@localhost oslo-messaging-clients]#
[root@localhost oslo-messaging-clients]# ./my-client.py --topology=1 my-topic methodB arg1 arg2
Calling server on topic my-topic, serv...

[Ihar Hrachyshka (ihar-hrachyshka)]

As per my latest comment, I still see the issue running testing servers and clients provided by the original reporter using latest oslo.messaging.

[Ken Giusti (kgiusti)]

Hi Ihar,

I don't believe that this fix has actually been merged to oslo.messaging.

Here's the version of impl_qpid.py from the github repo you are using:

https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_qpid.py

It doesn't contain the fix.

Here's a link to a code review containing the patch, just fyi:

https://review.openstack.org/#/c/59837/4/openstack/common/rpc/impl_qpid.py

[Ben Nemec (bnemec)]

This was merged here: https://review.openstack.org/64813