On restart of QPID broker, fanout no longer works

Bug #1251757 reported by Ken Giusti on 2013-11-15
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
High
Eoghan Glynn
Havana
Fix Released
High
Eoghan Glynn
Cinder
High
Flavio Percoco
Havana
High
Flavio Percoco
OpenStack Compute (nova)
High
Flavio Percoco
Havana
High
Flavio Percoco
OpenStack Heat
Fix Released
High
Steven Dake
Havana
Fix Released
High
Flavio Percoco
OpenStack Identity (keystone)
High
Alan Pevec
Havana
High
Alan Pevec
neutron
High
Flavio Percoco
Havana
High
Flavio Percoco
oslo-incubator
High
Mark McLoughlin
Havana
High
Flavio Percoco
oslo.messaging
High
Ken Giusti

Bug Description

When the QPID broker is restarted, RPC servers attempt to re-connect. This re-connection process is not done correctly for fanout subscriptions - two subscriptions are established to the same fanout address.

This problem is compounded by the fix to bug#1178375 https://bugs.launchpad.net/oslo/+bug/1178375

With this bug fix, when topology version 2 is used, the reconnect attempt uses a malformed subscriber address.

For example, I have a simple RPC server script that attempts to service "my-topic". When it initially connects to the broker using topology-version 1, these are the subscriptions that are established:

(py27)[kgiusti@t530 work (master)]$ ./my-server.py --topology=1 --auto-delete server-02
Running server, name=server-02 exchange=my-exchange topic=my-topic namespace=my-namespace
Using QPID topology version 1
Enable auto-delete
Recevr openstack/my-topic ; {"node": {"x-declare": {"auto-delete": true, "durable": true}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": false, "durable": false}, "durable": true, "name": "my-topic"}}
Recevr openstack/my-topic.server-02 ; {"node": {"x-declare": {"auto-delete": true, "durable": true}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": false, "durable": false}, "durable": true, "name": "my-topic.server-02"}}
Recevr my-topic_fanout ; {"node": {"x-declare": {"auto-delete": true, "durable": false, "type": "fanout"}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": true, "durable": false}, "durable": true, "name": "my-topic_fanout_489a3178fc704123b0e5e2fbee125247"}}

When I restart the qpid broker, the server reconnects using the following subscriptions

Recevr my-topic_fanout ; {"node": {"x-declare": {"auto-delete": true, "durable": false, "type": "fanout"}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": true, "durable": false}, "durable": true, "name": "my-topic_fanout_b40001afd9d946a582ead3b7b858b588"}}
Recevr my-topic_fanout ; {"node": {"x-declare": {"auto-delete": true, "durable": false, "type": "fanout"}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": true, "durable": false}, "durable": true, "name": "my-topic_fanout_b40001afd9d946a582ead3b7b858b588"}}
^^^^--- Note: subscribing twice to the same exclusive address! (Bad!)
Recevr openstack/my-topic.server-02 ; {"node": {"x-declare": {"auto-delete": true, "durable": true}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": false, "durable": false}, "durable": true, "name": "my-topic.server-02"}}
Recevr openstack/my-topic ; {"node": {"x-declare": {"auto-delete": true, "durable": true}, "type": "topic"}, "create": "always", "link": {"x-declare": {"auto-delete": true, "exclusive": false, "durable": false}, "durable": true, "name": "my-topic"}}

When using topology=2, the failure case is a bit different. On reconnect, the fanout addresses are lacking proper topic names:

Recevr amq.topic/topic/openstack/my-topic ; {"link": {"x-declare": {"auto-delete": true, "durable": false}}}
Recevr amq.topic/fanout/ ; {"link": {"x-declare": {"auto-delete": true, "exclusive": true}}}
Recevr amq.topic/fanout/ ; {"link": {"x-declare": {"auto-delete": true, "exclusive": true}}}
Recevr amq.topic/topic/openstack/my-topic.server-02 ; {"link": {"x-declare": {"auto-delete": true, "durable": false}}}

Note again - two subscriptions to fanout, and 'my-topic' is missing (it should be after that trailing /)

FYI - my test RPC server and client can be accessed here: https://github.com/kgiusti/oslo-messaging-clients

Fix proposed to branch: master
Review: https://review.openstack.org/57700

Changed in oslo.messaging:
assignee: nobody → Ken Giusti (kgiusti)
status: New → In Progress
Mark McLoughlin (markmc) on 2013-11-29
Changed in oslo.messaging:
importance: Undecided → High
Mark McLoughlin (markmc) wrote :

Need this fix in oslo-incubator too since projects are still using the RPC code from there

Changed in oslo:
importance: Undecided → High
status: New → Triaged
Mark McLoughlin (markmc) on 2013-11-29
Changed in oslo.messaging:
milestone: none → icehouse-1

Fix proposed to branch: master
Review: https://review.openstack.org/59677

Changed in oslo:
assignee: nobody → Mark McLoughlin (markmc)
status: Triaged → In Progress

Reviewed: https://review.openstack.org/59677
Committed: http://github.com/openstack/oslo-incubator/commit/e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df
Submitter: Jenkins
Branch: master

commit e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df
Author: Kenneth Giusti <email address hidden>
Date: Tue Dec 3 10:39:49 2013 +0000

    Properly reconnect subscribing clients when QPID broker restarts

    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Closes-bug: #1251757
    Co-authored-by: Mark McLoughlin <email address hidden>
    Change-Id: I6051fb503663bb8c7c5468db6bcde10f6cf1b318

Changed in oslo:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/57700
Committed: http://github.com/openstack/oslo.messaging/commit/ffa5c077c98f5489b978f63c507793839e8f2a7d
Submitter: Jenkins
Branch: master

commit ffa5c077c98f5489b978f63c507793839e8f2a7d
Author: Kenneth Giusti <email address hidden>
Date: Fri Nov 15 15:24:37 2013 -0500

    Properly reconnect subscribing clients when QPID broker restarts

    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Closes-bug: #1251757
    Change-Id: I6051fb503663bb8c7c5468db6bcde10f6cf1b318

Changed in oslo.messaging:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-12-05
Changed in oslo.messaging:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-12-05
Changed in oslo:
milestone: none → icehouse-1
status: Fix Committed → Fix Released
Steven Dake (sdake) on 2013-12-05
Changed in heat:
milestone: none → icehouse-2
importance: Undecided → High
status: New → Triaged
Steven Dake (sdake) on 2013-12-05
Changed in heat:
assignee: nobody → Steven Dake (sdake)

Fix proposed to branch: master
Review: https://review.openstack.org/60311

Changed in heat:
status: Triaged → In Progress
Eoghan Glynn (eglynn) on 2013-12-05
Changed in ceilometer:
assignee: nobody → Eoghan Glynn (eglynn)
milestone: none → icehouse-2
importance: Undecided → High
status: New → In Progress

Reviewed: https://review.openstack.org/60280
Committed: http://github.com/openstack/oslo-incubator/commit/55678c7ab7fbc1b2beb428d046bb4162f23b0a53
Submitter: Jenkins
Branch: stable/havana

commit 55678c7ab7fbc1b2beb428d046bb4162f23b0a53
Author: Kenneth Giusti <email address hidden>
Date: Tue Dec 3 10:39:49 2013 +0000

    Properly reconnect subscribing clients when QPID broker restarts

    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Closes-bug: #1251757
    Co-authored-by: Mark McLoughlin <email address hidden>
    Change-Id: I6051fb503663bb8c7c5468db6bcde10f6cf1b318
    (cherry picked from commit e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df)

tags: added: in-stable-havana

Reviewed: https://review.openstack.org/60311
Committed: http://github.com/openstack/heat/commit/87f22bd887b73b9d6cb08b98db9d93d067afd137
Submitter: Jenkins
Branch: master

commit 87f22bd887b73b9d6cb08b98db9d93d067afd137
Author: Steven Dake <email address hidden>
Date: Thu Dec 5 11:34:38 2013 -0700

    Properly reconnect subscribing clients when QPID broker restarts

    This is a cherrypick of oslo-incubator:
    e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df

    From the oslo-incubator bug:
    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Change-Id: I563db129e63817ad55165e318f164f06b141ee33
    Closes-Bug: 1251757

Changed in heat:
status: In Progress → Fix Committed

Fix proposed to branch: master
Review: https://review.openstack.org/60464

Changed in cinder:
assignee: nobody → Flavio Percoco (flaper87)
status: New → In Progress
Changed in nova:
assignee: nobody → Flavio Percoco (flaper87)
Changed in nova:
status: New → In Progress
importance: Undecided → High
milestone: none → icehouse-2
Alan Pevec (apevec) on 2013-12-06
Changed in cinder:
importance: Undecided → High
Alan Pevec (apevec) on 2013-12-06
Changed in cinder:
milestone: none → icehouse-2
Changed in neutron:
importance: Undecided → High
Alan Pevec (apevec) on 2013-12-06
Changed in keystone:
assignee: nobody → Alan Pevec (apevec)
importance: Undecided → High
milestone: none → icehouse-2

Fix proposed to branch: master
Review: https://review.openstack.org/60501

Changed in keystone:
status: New → In Progress
Alan Pevec (apevec) on 2013-12-06
Changed in neutron:
milestone: none → icehouse-2

Fix proposed to branch: master
Review: https://review.openstack.org/60524

Changed in neutron:
assignee: nobody → Flavio Percoco (flaper87)
status: New → In Progress
Flavio Percoco (flaper87) wrote :

Reviewed: https://review.openstack.org/60371
Committed: http://github.com/openstack/ceilometer/commit/ee9bd9b506e62cd673a991aecaf2c7263b71c518
Submitter: Jenkins
Branch: master

commit ee9bd9b506e62cd673a991aecaf2c7263b71c518
Author: Eoghan Glynn <email address hidden>
Date: Thu Dec 5 22:16:55 2013 +0000

    Properly reconnect subscribing clients when QPID broker restarts

    Fixes bug 1251757

    This is a cherrypick of oslo-incubator:
    e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df

    From the oslo-incubator bug:
    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Change-Id: If966bb25e49b5837293a1bb181bbb02086599d81

Changed in ceilometer:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/60524
Committed: http://github.com/openstack/neutron/commit/7afd0f4d514d25ff8469b4fc8026363120a33d57
Submitter: Jenkins
Branch: master

commit 7afd0f4d514d25ff8469b4fc8026363120a33d57
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 15:35:14 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    e227c0e Properly reconnect subscribing clients when QPID broker restarts
    ef406a2 Create a shared queue for QPID topic consumers

    Change-Id: I286edf6bc4a677aa61f60da785802c19878c79c7
    Closes-bug: #1251757
    Closes-bug: #1257293

Changed in neutron:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/60501
Committed: http://github.com/openstack/keystone/commit/9cc791d73ad57c9262c50e92f3bef1dd2bba215e
Submitter: Jenkins
Branch: master

commit 9cc791d73ad57c9262c50e92f3bef1dd2bba215e
Author: Alan Pevec <email address hidden>
Date: Fri Dec 6 14:12:47 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293

    Change-Id: I45257c62168163d2d4ceda994c94ff2d16a27300

Changed in keystone:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/60478
Committed: http://github.com/openstack/nova/commit/0e73913f1054a44e99b2a5a15ee8c9c32287513d
Submitter: Jenkins
Branch: stable/havana

commit 0e73913f1054a44e99b2a5a15ee8c9c32287513d
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:25:59 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: Ic21e54efa3a0263d53481ef500fb949544dae1d2

Alan Pevec (apevec) wrote :

Nova master patch was not recorded here: https://review.openstack.org/60468

Reviewed: https://review.openstack.org/60486
Committed: http://github.com/openstack/heat/commit/6ccbf6e7abbdb518599f943fb743e538b1f81708
Submitter: Jenkins
Branch: stable/havana

commit 6ccbf6e7abbdb518599f943fb743e538b1f81708
Author: Steven Dake <email address hidden>
Date: Thu Dec 5 11:34:38 2013 -0700

    Properly reconnect subscribing clients when QPID broker restarts

    This is a cherrypick of oslo-incubator:
    e227c0ed7e0ed1f9b8d029336f8aeb60e38c23df

    From the oslo-incubator bug:
    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Change-Id: I563db129e63817ad55165e318f164f06b141ee33
    Closes-Bug: 1251757

Reviewed: https://review.openstack.org/60483
Committed: http://github.com/openstack/ceilometer/commit/43a892138a506257992a116d2d33d3914c95e406
Submitter: Jenkins
Branch: stable/havana

commit 43a892138a506257992a116d2d33d3914c95e406
Author: Eoghan Glynn <email address hidden>
Date: Thu Dec 5 22:16:55 2013 +0000

    Properly reconnect subscribing clients when QPID broker restarts

    Fixes bug 1251757

    This is a cherrypick of oslo-incubator:
    55678c7ab7fbc1b2beb428d046bb4162f23b0a53

    From the oslo-incubator bug:
    When the QPID broker is restarted (or fails over), subscribed clients
    will attempt to re-establish their connections. In the case of fanout
    subscriptions, this reconnection functionality is broken. For version
    1 topologies, the clients attempt to reconnect twice to the same
    exclusive address - which is illegal. In the case of version 2
    topologies, the address parsing is broken and an illegal address is
    created on reconnect. This fix avoids the problem by removing the
    special-case reconnect code that manages UUID addresses; it is
    unnecessary as the QPID broker will generate unique queue names
    automatically when the clients reconnect.

    Change-Id: If966bb25e49b5837293a1bb181bbb02086599d81

Reviewed: https://review.openstack.org/60525
Committed: http://github.com/openstack/neutron/commit/9f9792972ff81f244931d830194aae979604ffa0
Submitter: Jenkins
Branch: stable/havana

commit 9f9792972ff81f244931d830194aae979604ffa0
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 15:35:14 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    e227c0e Properly reconnect subscribing clients when QPID broker restarts
    ef406a2 Create a shared queue for QPID topic consumers

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: I286edf6bc4a677aa61f60da785802c19878c79c7

Reviewed: https://review.openstack.org/60468
Committed: http://github.com/openstack/nova/commit/1af403ff66026b6616f683c6414851e51619088a
Submitter: Jenkins
Branch: master

commit 1af403ff66026b6616f683c6414851e51619088a
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:25:59 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293

    Change-Id: I917c7a6e5cb64c6383e74d0fb6e095ad17d4d478

Changed in nova:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/60533
Committed: http://github.com/openstack/cinder/commit/2b2871cd4defbad0878ceb0d1dfc7803cf7c8a83
Submitter: Jenkins
Branch: stable/havana

commit 2b2871cd4defbad0878ceb0d1dfc7803cf7c8a83
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:06:09 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Closes-bug: #1251757
    Closes-bug: #1257293
    Change-Id: I934a6ea6e9eb510513640870750ea37e6f076df9

Alan Pevec (apevec) on 2013-12-09
tags: removed: in-stable-havana

Reviewed: https://review.openstack.org/60464
Committed: http://github.com/openstack/cinder/commit/cbd7acfc294c4bf2ae0014bcf04646382bed469d
Submitter: Jenkins
Branch: master

commit cbd7acfc294c4bf2ae0014bcf04646382bed469d
Author: Flavio Percoco <email address hidden>
Date: Fri Dec 6 11:06:09 2013 +0100

    Sync rpc fix from oslo-incubator

    Sync the following fixes from oslo-incubator:

    ef406a2 Create a shared queue for QPID topic consumers
    e227c0e Properly reconnect subscribing clients when QPID broker restarts

    Change-Id: I934a6ea6e9eb510513640870750ea37e6f076df9
    Closes-bug: #1251757
    Closes-bug: #1257293

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-01-22
Changed in keystone:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-01-22
Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-01-22
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-01-22
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-01-22
Changed in nova:
status: Fix Committed → Fix Released
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-04-17
Changed in oslo:
milestone: icehouse-1 → 2014.1
Thierry Carrez (ttx) on 2014-04-17
Changed in keystone:
milestone: icehouse-2 → 2014.1
Thierry Carrez (ttx) on 2014-04-17
Changed in oslo.messaging:
milestone: icehouse-1 → 1.3.0
Changed in ceilometer:
milestone: icehouse-2 → 2014.1
Thierry Carrez (ttx) on 2014-04-17
Changed in nova:
milestone: icehouse-2 → 2014.1
Thierry Carrez (ttx) on 2014-04-17
Changed in cinder:
milestone: icehouse-2 → 2014.1
Thierry Carrez (ttx) on 2014-04-17
Changed in heat:
milestone: icehouse-2 → 2014.1
Thierry Carrez (ttx) on 2014-04-17
Changed in neutron:
milestone: icehouse-2 → 2014.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers