Reconnect on failure for multiple servers always connects to first server

[Maru Newby]

In attempting to reconnect to an AMQP server when a communication failure occurs, both the qpid and rabbit drivers target the configured servers in the order in which they were provided. If a connection to the first server had failed, the subsequent reconnection attempt would be made to that same server instead of trying one that had not yet failed. This could increase the time to failover to a working server.

A plausible workaround for qpid would be to decrease the value for qpid_timeout, but since the problem only occurs if the failed server is the first configured, the results of the workaround would depend on the order that the failed server appears in the configuration.

[Ihar Hrachyshka (ihar-hrachyshka)]

Somehow Openstack-Infra failed to add comments with links to code reviews here. Adding them below:

rabbitmq part: https://review.openstack.org/#/c/67450/
qpid patt: https://review.openstack.org/#/c/67449/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/68958

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/68959

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/68959
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=3fdd52e0d48dab3b5cf6623d58a4ac53530b20e8
Submitter: Jenkins
Branch: master

commit 3fdd52e0d48dab3b5cf6623d58a4ac53530b20e8
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Jan 24 17:22:27 2014 +0100

    Qpid: advance thru the list of brokers on reconnect

    In Qpid implementation, when using multiple qpid_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Change-Id: I257c2ad6d7ebead356c0239134340975da6dbc07
    Partial-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/68958
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=71c6866471d628b207d7c5b84bfd37cc9fed0898
Submitter: Jenkins
Branch: master

commit 71c6866471d628b207d7c5b84bfd37cc9fed0898
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Jan 24 17:31:46 2014 +0100

    RabbitMQ: advance thru the list of brokers on reconnect

    In RabbitMQ implementation, when using multiple rabbit_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Change-Id: I76965ea06c1fb6fd0cd384a5b3edc14703470b8f
    Partial-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/81261

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/81962

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/67450
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=abe3e5ee994fddd5141e2cae3f4785ffe62805ea
Submitter: Jenkins
Branch: master

commit abe3e5ee994fddd5141e2cae3f4785ffe62805ea
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Jan 17 13:44:28 2014 +0100

    RabbitMQ: advance thru the list of brokers on reconnect

    In RabbitMQ implementation, when using multiple rabbit_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    This is a backport of 71c6866471d628b207d7c5b84bfd37cc9fed0898 from
    oslo.messaging.

    Change-Id: I67923cb024bbd143edc8edccf35b9b400df31eb3
    Partial-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/67449
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=7fe95de20ccdfa5aa523ccb8a9ca3afdde5eddbb
Submitter: Jenkins
Branch: master

commit 7fe95de20ccdfa5aa523ccb8a9ca3afdde5eddbb
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Jan 16 18:22:49 2014 +0100

    Qpid: advance thru the list of brokers on reconnect

    In Qpid implementation, when using multiple qpid_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    This is a backport of the following changes from oslo.messaging:
    * I257c2ad6d7ebead356c0239134340975da6dbc07 (code change)
    * I32e20c0e747e6489dab4b0358d422b8a9c6b982e (unit test).

    Change-Id: Ia148baa6e1ec632789ac3621c85173c2c16f3918
    Partial-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/83085

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/83086

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/81261
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=27e4a0da8b0c6191145953bd7639b7318b48b6e8
Submitter: Jenkins
Branch: master

commit 27e4a0da8b0c6191145953bd7639b7318b48b6e8
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Mar 18 14:22:53 2014 +0100

    Add unit test to check the order of Qpid hosts on reconnect

    Change-Id: I32e20c0e747e6489dab4b0358d422b8a9c6b982e
    Partial-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83086
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=2887edbdba8e139aaf0fb2b50b2d94e24d10a963
Submitter: Jenkins
Branch: stable/havana

commit 2887edbdba8e139aaf0fb2b50b2d94e24d10a963
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Jan 17 13:44:28 2014 +0100

    RabbitMQ: advance thru the list of brokers on reconnect

    In RabbitMQ implementation, when using multiple rabbit_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    This is a backport of 71c6866471d628b207d7c5b84bfd37cc9fed0898 from
    oslo.messaging.

    Change-Id: I67923cb024bbd143edc8edccf35b9b400df31eb3
    Partial-Bug: 1261631
    (cherry picked from commit abe3e5ee994fddd5141e2cae3f4785ffe62805ea)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83085
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=877f1cab328fe8a9f8214687ea7e1287862de87d
Submitter: Jenkins
Branch: stable/havana

commit 877f1cab328fe8a9f8214687ea7e1287862de87d
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Jan 16 18:22:49 2014 +0100

    Qpid: advance thru the list of brokers on reconnect

    In Qpid implementation, when using multiple qpid_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    This is a backport of the following changes from oslo.messaging:
    * I257c2ad6d7ebead356c0239134340975da6dbc07 (code change)
    * I32e20c0e747e6489dab4b0358d422b8a9c6b982e (unit test).

    Change-Id: Ia148baa6e1ec632789ac3621c85173c2c16f3918
    Partial-Bug: 1261631
    (cherry picked from commit 7fe95de20ccdfa5aa523ccb8a9ca3afdde5eddbb)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/83460

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83460
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=eb4c0458136139561e6286fce0ad81d542dd0d9d
Submitter: Jenkins
Branch: stable/havana

commit eb4c0458136139561e6286fce0ad81d542dd0d9d
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 27 17:19:32 2014 +0100

    Qpid, RabbitMQ: advance thru the list of brokers on reconnect

    When using multiple brokers, we don't want to immediately retry failed
    connection for the same failed broker. This was not the case in
    existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So
    if the first broker failed, we initiated reconnect to the same failed
    broker.

    This change makes reconnect() implementation to select the next broker
    in the list. This also means that non-failure reconnect attempts will
    also switch the current broker. All in all, users should not rely on
    any particular order to use brokers from the list, so this should not
    constitute an issue.

    This sync includes the following fixes for HA deployments:
    * Ia148baa6e1ec632789ac3621c85173c2c16f3918
    * I67923cb024bbd143edc8edccf35b9b400df31eb3

    Change-Id: I77be95dd2db4546297d597032055459a12b52cfa
    Closes-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/81183
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=af72ad9a924340a48651813f7156bc3d4a37f8d5
Submitter: Jenkins
Branch: master

commit af72ad9a924340a48651813f7156bc3d4a37f8d5
Author: ChangBo Guo(gcb) <email address hidden>
Date: Tue Mar 18 21:42:14 2014 +0800

    Refactor AMQP message broker selection

    Drop redundant variable, make code more clean.

    Partial-Bug: #1261631

    Change-Id: I5fd71ce3bcd3a82839204ee1fb130de3a3e1d3ac

[Ihar Hrachyshka (ihar-hrachyshka)]

FYI: this bug affects all the projects using rpc module from oslo-incubator. Those projects/branches that have already migrated to oslo.messaging may close the bug as Invalid. For those projects, a backport to stable/havana is still needed though.

[Ihar Hrachyshka (ihar-hrachyshka)]

the fixes to sync to stable branches to fix the bug are:
- https://review.openstack.org/83086 (rabbitmq)
- https://review.openstack.org/83085 (qpid)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/80998
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=665222b38b7580a30f50c1fef78feebcae666f40
Submitter: Jenkins
Branch: master

commit 665222b38b7580a30f50c1fef78feebcae666f40
Author: Ihar Hrachyshka <email address hidden>
Date: Mon Mar 17 14:18:28 2014 +0100

    Synced rpc and gettextutils modules from oslo-incubator

    The main reason for sync is to get the following oslo-rpc fixes in Neutron:
    * I537015f452eb770acba41fdedfe221628f52a920 (reduces delays when reconnecting
      to Qpid in HA deployments)
    * Ia148baa6e1ec632789ac3621c85173c2c16f3918 (fixed HA failover, Qpid part)
    * I67923cb024bbd143edc8edccf35b9b400df31eb3 (fixed HA failover, RabbitMQ part)

    Latest oslo-incubator commit at the moment of sync:
    * 2eab986ef3c43f8d1e25065e3cbc1307860c25c7

    Change-Id: I2f5bb0d195e050f755ecdbf06a6bbed587a04fbe
    Closes-Bug: 1281148
    Closes-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84609

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84612

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/85791

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/85791
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=04e24eeb8a8de02831a7e60c7ec6161c364add3a
Submitter: Jenkins
Branch: master

commit 04e24eeb8a8de02831a7e60c7ec6161c364add3a
Author: Thomas Herve <email address hidden>
Date: Mon Apr 7 21:17:28 2014 +0200

    Cherry pick oslo rpc HA fixes

    Get 2 commits from oslo-rpc fixing HA failover:

    * Ia148baa6e1ec632789ac3621c85173c2c16f3918 (fixed HA failover, Qpid
     part)
    * I67923cb024bbd143edc8edccf35b9b400df31eb3 (fixed HA failover, RabbitMQ
     part)

    Change-Id: I45f679f3da720a0c28fb552d9f4cfb3d8bd21c20
    Closes-Bug: #1261631

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/85902

[Ildiko Vancsa (ildiko-vancsa)]

The fix is contained by this commit: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=5f953e2765f8a20eebd20779316944e2a4221c94

[Steve Baker (steve-stevebaker)]

This is Fix Released for heat, launchpad times out when attempting to set this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/86745

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/86745
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=bbc49d3fcf98384df9b211f1f451ab64255dd698
Submitter: Jenkins
Branch: milestone-proposed

commit bbc49d3fcf98384df9b211f1f451ab64255dd698
Author: Devananda van der Veen <email address hidden>
Date: Thu Apr 10 16:46:18 2014 -0700

    Cherry pick oslo rpc HA fixes

    Get 2 commits from oslo-rpc fixing HA failover:

    * Ia148baa6e1ec632789ac3621c85173c2c16f3918 (fixed HA failover, Qpid part)
    * I67923cb024bbd143edc8edccf35b9b400df31eb3 (fixed HA failover, RabbitMQ part)

    Change-Id: I8353288560c0c6a9ebe1d143690609765e59df1a
    Closes-Bug: #1261631

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/87314

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/87314
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=b1e971c9b19409b8acbee0b2cb2287d7221cbafb
Submitter: Jenkins
Branch: master

commit b1e971c9b19409b8acbee0b2cb2287d7221cbafb
Author: Devananda van der Veen <email address hidden>
Date: Thu Apr 10 16:46:18 2014 -0700

    Cherry pick oslo rpc HA fixes

    Get 2 commits from oslo-rpc fixing HA failover:

    * Ia148baa6e1ec632789ac3621c85173c2c16f3918 (fixed HA failover, Qpid part)
    * I67923cb024bbd143edc8edccf35b9b400df31eb3 (fixed HA failover, RabbitMQ part)

    Change-Id: I8353288560c0c6a9ebe1d143690609765e59df1a
    Closes-Bug: #1261631

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/81962
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=8ae1880f7a31f6e7c985b99b2015db792d5cca46
Submitter: Jenkins
Branch: master

commit 8ae1880f7a31f6e7c985b99b2015db792d5cca46
Author: ChangBo Guo(gcb) <email address hidden>
Date: Fri Mar 21 09:49:32 2014 +0800

    Select AMQP message broker at random

    The rule of choosing AMQP message broker is that chose first
    available one in order now. The order depends on what we set in
    configuration file. That means all the connections will flock
    to same message broker and that may lead out performance issue.
    This patch randomizes the order of choosing message broker for
    each connection to leverage broker cluster.

    Change-Id: Ib5098e574d4ef81428065885e2295d0f87aba715
    Partial-Bug: #1261631

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/99754

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/99822

[OpenStack Infra (hudson-openstack)]

Change abandoned by Russell Bryant (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/99754

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/107789

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/107789
Reason: Abandoning as stable/havana is now in security and critical fixes only mode

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/109533

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/109536

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109536
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=af1eb64f6541517d8f0a94b96bbd1cae56953396
Submitter: Jenkins
Branch: stable/icehouse

commit af1eb64f6541517d8f0a94b96bbd1cae56953396
Author: Nejc Saje <email address hidden>
Date: Fri Jul 25 12:35:01 2014 +0200

    Sync RPC module from oslo, fixing reconnect failures

    This change syncs the following patches from oslo-incubator RPC:
    - I67923cb024bbd143edc8edccf35b9b400df31eb3
    - Ia148baa6e1ec632789ac3621c85173c2c16f3918

    RabbitMQ: advance thru the list of brokers on reconnect

    In RabbitMQ implementation, when using multiple rabbit_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Qpid: advance thru the list of brokers on reconnect

    In Qpid implementation, when using multiple qpid_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Partial-Bug: #1261631
    Change-Id: Ie9c4de2c094a5d420acca5e4a49ee6a36279ed87

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109533
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=18772ea860e6d6248b244e7076cfaa479e35df9e
Submitter: Jenkins
Branch: stable/havana

commit 18772ea860e6d6248b244e7076cfaa479e35df9e
Author: Nejc Saje <email address hidden>
Date: Fri Jul 25 11:51:46 2014 +0200

    Sync RPC module from oslo

    This change syncs the following patches from stable/havana oslo RPC:
    - I67923cb024bbd143edc8edccf35b9b400df31eb3
    - Ia148baa6e1ec632789ac3621c85173c2c16f3918

    RabbitMQ: advance thru the list of brokers on reconnect

    In RabbitMQ implementation, when using multiple rabbit_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Qpid: advance thru the list of brokers on reconnect

    In Qpid implementation, when using multiple qpid_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Partial-Bug: #1261631
    Change-Id: Icde6b3a1fa173be2a3ccc729b7ed739dd4f4aa11

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/99754
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=512c0e43efa2748d26556214ee59b86f2b4b061d
Submitter: Jenkins
Branch: stable/havana

commit 512c0e43efa2748d26556214ee59b86f2b4b061d
Author: Russell Bryant <email address hidden>
Date: Thu Jun 12 16:13:58 2014 -0400

    Qpid: advance thru the list of brokers on reconnect

    Sync 877f1cab328fe8a9f8214687ea7e1287862de87d from oslo-incubator's
    stable/havana branch.

    In Qpid implementation, when using multiple qpid_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Change-Id: Ia148baa6e1ec632789ac3621c85173c2c16f3918
    Partial-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/99822
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e1329080a4566848c31f077392542ec0cae42ae1
Submitter: Jenkins
Branch: stable/havana

commit e1329080a4566848c31f077392542ec0cae42ae1
Author: Nathan Kinder <email address hidden>
Date: Thu Jun 12 19:04:08 2014 -0700

    Qpid: advance thru the list of brokers on reconnect

    Sync 877f1cab328fe8a9f8214687ea7e1287862de87d from oslo-incubator's
    stable/havana branch.

    In Qpid implementation, when using multiple qpid_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Change-Id: Ia148baa6e1ec632789ac3621c85173c2c16f3918
    Partial-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/107789
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2a5b4a78f3adb2b180b157532247ae0a87159569
Submitter: Jenkins
Branch: stable/havana

commit 2a5b4a78f3adb2b180b157532247ae0a87159569
Author: Steven Hardy <email address hidden>
Date: Thu Jul 17 19:24:03 2014 +0100

    Qpid: advance thru the list of brokers on reconnect

    Sync 877f1cab328fe8a9f8214687ea7e1287862de87d from oslo-incubator's
    stable/havana branch.

    In Qpid implementation, when using multiple qpid_hosts, we don't want to
    immediately retry failed connection for the same failed broker. This was not
    the case in existing implementation though, where we've always attempted to
    reconnect starting from the first broker in the list of candidates. So if the
    first broker failed, we initiated reconnect to the same failed broker.

    This change makes reconnect() implementation to select the next broker in the
    list. This also means that non-failure reconnect attempts will also switch the
    current broker. All in all, users should not rely on any particular order to
    use brokers from the list, so this should not constitute an issue.

    Change-Id: I5a630e76880c0d4219c4a8a2b3354e41899d9695
    Partial-Bug: 1261631

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/115305

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115305
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=776fcb59e07f976f542b930f17e3416d281204e8
Submitter: Jenkins
Branch: stable/havana

commit 776fcb59e07f976f542b930f17e3416d281204e8
Author: Ivan Kolodyazhny <email address hidden>
Date: Tue Aug 19 17:40:12 2014 +0300

    Cherry pick oslo rpc HA fixes

    Get 2 commits from oslo-rpc fixing HA failover:
    * Ia148baa6e1ec632789ac3621c85173c2c16f3918 (fixed HA failover, Qpid
     part)
    * I67923cb024bbd143edc8edccf35b9b400df31eb3 (fixed HA failover, RabbitMQ
     part)
    Closes-Bug: #1261631

    Change-Id: I154a1e4230cf956e0d6a71a8e717866d0cd17a12

[Alan Pevec (apevec)]

Keystone >= Icehouse uses oslo.messaging

[Ian Cordasco (icordasc)]

Glance has been using oslo.messaging 1.3.0 or greater since Icehouse.

[Sean McGinnis (sean-mcginnis)]

Automatically unassigning due to inactivity.