On restart of QPID broker, fanout no longer works

[Ken Giusti]

When the QPID broker is restarted, RPC servers attempt to re-connect. This re-connection process is not done correctly for fanout subscriptions - two subscriptions are established to the same fanout address.

This problem is compounded by the fix to bug#1178375 https://bugs.launchpad.net/oslo/+bug/1178375

With this bug fix, when topology version 2 is used, the reconnect attempt uses a malformed subscriber address.

For example, I have a simple RPC server script that attempts to service "my-topic". When it initially connects to the broker using topology-version 1, these are the subscriptions that are established:

(py27)[kgiusti@t530 work (master)]$ ./my-server.py --topology=1 --auto-delete server-02
Running server, name=server-02 exchange=my-exchange topic=my-topic namespace=my-namespace
Using QPID topology version 1
Enable auto-delete
Recevr openstack/my-topic ; {"node": {"x-declare": {"auto-delete": true, "durable": true}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": false, "durable": false}, "durable": true, "name": "my-topic"}}
Recevr openstack/my-topic.server-02 ; {"node": {"x-declare": {"auto-delete": true, "durable": true}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": false, "durable": false}, "durable": true, "name": "my-topic.server-02"}}
Recevr my-topic_fanout ; {"node": {"x-declare": {"auto-delete": true, "durable": false, "type": "fanout"}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": true, "durable": false}, "durable": true, "name": "my-topic_fanout_489a3178fc704123b0e5e2fbee125247"}}

When I restart the qpid broker, the server reconnects using the following subscriptions

Recevr my-topic_fanout ; {"node": {"x-declare": {"auto-delete": true, "durable": false, "type": "fanout"}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": true, "durable": false}, "durable": true, "name": "my-topic_fanout_b40001afd9d946a582ead3b7b858b588"}}
Recevr my-topic_fanout ; {"node": {"x-declare": {"auto-delete": true, "durable": false, "type": "fanout"}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": true, "durable": false}, "durable": true, "name": "my-topic_fanout_b40001afd9d946a582ead3b7b858b588"}}
^^^^--- Note: subscribing twice to the same exclusive address! (Bad!)
Recevr openstack/my-topic.server-02 ; {"node": {"x-declare": {"auto-delete": true, "durable": true}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": false, "durable": false}, "durable": true, "name": "my-topic.server-02"}}
Recevr openstack/my-topic ; {"node": {"x-declare": {"auto-delete": true, "durable": true}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": false, "durable": false}, "durable": true, "name": "my-topic"}}

When using topology=2, the failure case is a bit different. On reconnect, the fanout addresses are lacking proper topic names:

Recevr amq.topic/topic/openstack/my-topic ; {"link": {"x-declare": {"auto-delete": true, "durable": false}}}
Recevr amq.topic/fanout/ ; {"link": {"x-declare": {"auto-delete": true, "exclusive": true}}}
Recevr amq.topic/fanout/ ; {"link": {"x-declare": {"auto-delete": true, "exclusive": true}}}
Recevr amq.topic/topic/openstack/my-topic.server-02 ; {"link": {"x-declare": {"auto-delete": true, "durable": false}}}

Note again - two subscriptions to fanout, and 'my-topic' is missing (it should be after that trailing /)

FYI - my test RPC server and client can be accessed here: https://github.com/kgiusti/oslo-messaging-clients

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/57700

[Mark McLoughlin (markmc)]

Need this fix in oslo-incubator too since projects are still using the RPC code from there

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/59677

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/59677
Committed: http://github.com/openstack/oslo-incubator/commit/e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df
Submitter: Jenkins
Branch: master

commit e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df
Author: Kenneth Giusti <email address hidden>
Date: Tue Dec 3 10:39:49 2013 +0000

    Properly reconnect subscribing clients when QPID broker restarts

    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Closes-bug: #1251757
    Co-authored-by: Mark McLoughlin <email address hidden>
    Change-Id: I6051fb503663bb8c7c5468db6bcde10f6cf1b318

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/57700
Committed: http://github.com/openstack/oslo.messaging/commit/ffa5c077c98f5489b978f63c507793839e8f2a7d
Submitter: Jenkins
Branch: master

commit ffa5c077c98f5489b978f63c507793839e8f2a7d
Author: Kenneth Giusti <email address hidden>
Date: Fri Nov 15 15:24:37 2013 -0500

    Properly reconnect subscribing clients when QPID broker restarts

    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Closes-bug: #1251757
    Change-Id: I6051fb503663bb8c7c5468db6bcde10f6cf1b318

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60280

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60311

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60371

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60280
Committed: http://github.com/openstack/oslo-incubator/commit/55678c7ab7fbc1b2beb428d046bb4162f23b0a53
Submitter: Jenkins
Branch: stable/havana

commit 55678c7ab7fbc1b2beb428d046bb4162f23b0a53
Author: Kenneth Giusti <email address hidden>
Date: Tue Dec 3 10:39:49 2013 +0000

    Properly reconnect subscribing clients when QPID broker restarts

    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Closes-bug: #1251757
    Co-authored-by: Mark McLoughlin <email address hidden>
    Change-Id: I6051fb503663bb8c7c5468db6bcde10f6cf1b318
    (cherry picked from commit e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60311
Committed: http://github.com/openstack/heat/commit/87f22bd887b73b9d6cb08b98db9d93d067afd137
Submitter: Jenkins
Branch: master

commit 87f22bd887b73b9d6cb08b98db9d93d067afd137
Author: Steven Dake <email address hidden>
Date: Thu Dec 5 11:34:38 2013 -0700

    Properly reconnect subscribing clients when QPID broker restarts

    This is a cherrypick of oslo-incubator:
    e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df

    From the oslo-incubator bug:
    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Change-Id: I563db129e63817ad55165e318f164f06b141ee33
    Closes-Bug: 1251757

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60464

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60478

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60479

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60483

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60486

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60501

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60514

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60524

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60525

[Flavio Percoco (flaper87)]

Neutron stable/havana: https://review.openstack.org/#/c/60525/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60533

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60371
Committed: http://github.com/openstack/ceilometer/commit/ee9bd9b506e62cd673a991aecaf2c7263b71c518
Submitter: Jenkins
Branch: master

commit ee9bd9b506e62cd673a991aecaf2c7263b71c518
Author: Eoghan Glynn <email address hidden>
Date: Thu Dec 5 22:16:55 2013 +0000

    Properly reconnect subscribing clients when QPID broker restarts

    Fixes bug 1251757

    This is a cherrypick of oslo-incubator:
    e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df

    From the oslo-incubator bug:
    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Change-Id: If966bb25e49b5837293a1bb181bbb02086599d81

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60524
Committed: http://github.com/openstack/neutron/commit/7afd0f4d514d25ff8469b4fc8026363120a33d57
Submitter: Jenkins
Branch: master

commit 7afd0f4d514d25ff8469b4fc8026363120a33d57
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 15:35:14 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    e227c0e Properly reconnect subscribing clients when QPID broker restarts
    ef406a2 Create a shared queue for QPID topic consumers

    Change-Id: I286edf6bc4a677aa61f60da785802c19878c79c7
    Closes-bug: #1251757
    Closes-bug: #1257293

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60501
Committed: http://github.com/openstack/keystone/commit/9cc791d73ad57c9262c50e92f3bef1dd2bba215e
Submitter: Jenkins
Branch: master

commit 9cc791d73ad57c9262c50e92f3bef1dd2bba215e
Author: Alan Pevec <email address hidden>
Date: Fri Dec 6 14:12:47 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293

    Change-Id: I45257c62168163d2d4ceda994c94ff2d16a27300

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60478
Committed: http://github.com/openstack/nova/commit/0e73913f1054a44e99b2a5a15ee8c9c32287513d
Submitter: Jenkins
Branch: stable/havana

commit 0e73913f1054a44e99b2a5a15ee8c9c32287513d
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:25:59 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: Ic21e54efa3a0263d53481ef500fb949544dae1d2

[Alan Pevec (apevec)]

Nova master patch was not recorded here: https://review.openstack.org/60468

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60486
Committed: http://github.com/openstack/heat/commit/6ccbf6e7abbdb518599f943fb743e538b1f81708
Submitter: Jenkins
Branch: stable/havana

commit 6ccbf6e7abbdb518599f943fb743e538b1f81708
Author: Steven Dake <email address hidden>
Date: Thu Dec 5 11:34:38 2013 -0700

    Properly reconnect subscribing clients when QPID broker restarts

    This is a cherrypick of oslo-incubator:
    e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df

    From the oslo-incubator bug:
    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Change-Id: I563db129e63817ad55165e318f164f06b141ee33
    Closes-Bug: 1251757

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60483
Committed: http://github.com/openstack/ceilometer/commit/43a892138a506257992a116d2d33d3914c95e406
Submitter: Jenkins
Branch: stable/havana

commit 43a892138a506257992a116d2d33d3914c95e406
Author: Eoghan Glynn <email address hidden>
Date: Thu Dec 5 22:16:55 2013 +0000

    Properly reconnect subscribing clients when QPID broker restarts

    Fixes bug 1251757

    This is a cherrypick of oslo-incubator:
    55678c7ab7fbc1b2beb428d046bb4162f23b0a53

    From the oslo-incubator bug:
    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Change-Id: If966bb25e49b5837293a1bb181bbb02086599d81

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60525
Committed: http://github.com/openstack/neutron/commit/9f9792972ff81f244931d830194aae979604ffa0
Submitter: Jenkins
Branch: stable/havana

commit 9f9792972ff81f244931d830194aae979604ffa0
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 15:35:14 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    e227c0e Properly reconnect subscribing clients when QPID broker restarts
    ef406a2 Create a shared queue for QPID topic consumers

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: I286edf6bc4a677aa61f60da785802c19878c79c7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60468
Committed: http://github.com/openstack/nova/commit/1af403ff66026b6616f683c6414851e51619088a
Submitter: Jenkins
Branch: master

commit 1af403ff66026b6616f683c6414851e51619088a
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:25:59 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293

    Change-Id: I917c7a6e5cb64c6383e74d0fb6e095ad17d4d478

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60533
Committed: http://github.com/openstack/cinder/commit/2b2871cd4defbad0878ceb0d1dfc7803cf7c8a83
Submitter: Jenkins
Branch: stable/havana

commit 2b2871cd4defbad0878ceb0d1dfc7803cf7c8a83
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:06:09 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: I934a6ea6e9eb510513640870750ea37e6f076df9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60464
Committed: http://github.com/openstack/cinder/commit/cbd7acfc294c4bf2ae0014bcf04646382bed469d
Submitter: Jenkins
Branch: master

commit cbd7acfc294c4bf2ae0014bcf04646382bed469d
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:06:09 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Change-Id: I934a6ea6e9eb510513640870750ea37e6f076df9
    Closes-bug: #1251757
    Closes-bug: #1257293