Race condition with failover logic in BigSwitch plugin

[Kevin Benton]

The failover logic modifies a list of servers as it iterates through them.

https://github.com/openstack/neutron/blob/ee3fe4e836ca1c81e50a8324a9b5f982de4fa97f/neutron/plugins/bigswitch/plugin.py#L300

If a thread fails for an unexpected exception, a server could get permanently removed from the failover pool.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/36581
Committed: http://github.com/openstack/neutron/commit/3732f022778a7c4a47c58029ae77e32a0a4a6ee9
Submitter: Jenkins
Branch: master

commit 3732f022778a7c4a47c58029ae77e32a0a4a6ee9
Author: Kevin Benton <email address hidden>
Date: Wed Jul 10 16:28:12 2013 -0700

    Prevent possible server list damage in BigSwitch plugin

    The old failover logic for the BigSwitch plugin modified a
    list of controllers as it tested them. If the code for one
    thread unexpectedly raised an exception or died, the global
    server list could lose a server permanently.
    This patch addresses that by flagging servers as failed instead
    so the global server list is never modified.

    Fixes: bug #1200022
    Change-Id: Id2dcb820ef9f62fd03e3215bff3345e56c78afe2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/38491

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/38491
Committed: http://github.com/openstack/neutron/commit/49278f22b64cf8eb1b99b12e55ff33d88b62a9fc
Submitter: Jenkins
Branch: stable/grizzly

commit 49278f22b64cf8eb1b99b12e55ff33d88b62a9fc
Author: Kevin Benton <email address hidden>
Date: Wed Jul 10 16:28:12 2013 -0700

    Prevent possible server list damage in BigSwitch plugin

    The old failover logic for the BigSwitch plugin modified a
    list of controllers as it tested them. If the code for one
    thread unexpectedly raised an exception or died, the global
    server list could lose a server permanently.
    This patch addresses that by flagging servers as failed instead
    so the global server list is never modified.

    Fixes: bug #1200022

    (cherry picked from commit 3732f022778a7c4a47c58029ae77e32a0a4a6ee9)

    Conflicts:

     quantum/plugins/bigswitch/plugin.py

    Change-Id: Id2dcb820ef9f62fd03e3215bff3345e56c78afe2