Fullstack linux bridge tests sometimes fail because an agent wouldn't come up as it cannot connect to RabbitMQ

[Assaf Muller]

Here's a couple of examples:

http://logs.openstack.org/78/292178/10/check/gate-neutron-dsvm-fullstack/b54b61b/logs/TestLinuxBridgeConnectivitySameNetwork.test_connectivity_VXLAN_/neutron-linuxbridge-agent--2016-03-23--13-54-07-458571.log.txt.gz

http://logs.openstack.org/07/296507/2/check/gate-neutron-dsvm-fullstack/1a24251/logs/TestLinuxBridgeConnectivitySameNetwork.test_connectivity_VXLAN_/neutron-linuxbridge-agent--2016-03-23--21-05-41-093902.log.txt.gz

Note that in both cases the other two agents in the same test were able to connect successfully. The commonality between agents that cannot connect to rabbit is that they use a local_ip that is the *broadcast address* of the subnet they belong to.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/296859

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/296859
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9466e7765a51074a75eceb514d5d22cae01daec4
Submitter: Jenkins
Branch: master

commit 9466e7765a51074a75eceb514d5d22cae01daec4
Author: Assaf Muller <email address hidden>
Date: Wed Mar 23 21:46:11 2016 -0400

    Fix fullstack LB agent not connecting to rabbit

    Using the broadcast address of a network is not going to work
    very well.

    It's still possible for different LB agents (From the same test)
    to pick the same address, and that will be handled in a follow up
    via some sort of atomic resource allocation mechanism.

    Change-Id: I666cf7dbe5dd6a0eaf4130ee1e52cf29edb2ddca
    Related-Bug: #1561248

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/297397

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/298056

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/297397
Reason: It will be fixed by https://review.openstack.org/#/c/298056/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/298056
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=03999961ac620249950d8bca628719e9c14c4382
Submitter: Jenkins
Branch: master

commit 03999961ac620249950d8bca628719e9c14c4382
Author: Assaf Muller <email address hidden>
Date: Thu Mar 24 22:14:07 2016 -0400

    Add fullstack cross-process port/ip address fixtures

    We've had a series of bugs with resources that need
    to be unique on the system across test runner
    processes. Ports are used by neutron-server and the
    OVS agent when run in native openflow mode. The function
    that generates ports looks up random unused ports and
    starts the service. However, it is raceful: By the time the
    port is found to be unused and the service is started,
    another test runner can pick the same random port.
    With close to 65536 ports to choose from, the chance
    for collision is low, but given enough test runs, it's
    happened a non-trivial amount of times, and given that
    a voting job needs a very low false-negative rate, we
    need a more robust solution. The same applies to IP
    addresses that are used by the OVS agent in tunneling
    mode, and for the LB agent in all modes. With IP addresses,
    we don't check if the IP address is used, we simply
    pick a random address from a large pool, and again
    we've seen a non-trivial amount of test failures.

    The bugs referenced below had simple, short term solutions
    applied but the bugs remain remain. This patch is a correct,
    long term solution that doesn't rely on chance.

    This patch adds a resource allocator that uses the disk
    to persist allocations. Access to the disk is guarded
    via a file lock. IP address, networks and ports fixtures
    use an allocator internally.

    Closes-Bug: #1551288
    Closes-Bug: #1561248
    Closes-Bug: #1560277
    Change-Id: I46c0ca138b806759128462f8d44a5fab96a106d3

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 9.0.0.0b1 development milestone.