Intermittent failure in BGP API tests

[Ryan Tidwell]

Seeing the following failure intermittently in API test jobs:

http://paste.openstack.org/show/489400/

In the 2 failed jobs [1][2] I've analyzed, test_get_advertised_routes_floating_ips() runs before this failed test. test_get_advertised_routes_null_address_scope() assumes no floating IP associations and no address scopes so that it can assert that no routes are being announced by a bgp_speaker. My theory is that the floating IP created in test_get_advertised_routes_floating_ips() doesn't get cleaned up fast enough, and test_get_advertised_routes_null_address_scope() runs before the floating IP from the previous test has actually been cleaned up.

[1] http://logs.openstack.org/85/267985/9/check/gate-neutron-dsvm-api/b09db1c/console.html
[2] http://logs.openstack.org/76/282876/17/check/gate-neutron-dsvm-api/b49cae9/console.html

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/288818

[Armando Migliaccio (armando-migliaccio)]

On closer inspection there are a bunch of bgp tests that are not designed to be isolated. These needs more work, because as they are they are a time bomb waiting to happen irrespective of the lockfixture. In fact, the lockfixture is ineffective to synchronize server side operations, so please remove them

[Armando Migliaccio (armando-migliaccio)]

I meant to add, with test execution, across multiple tests

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/288825

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288818
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3f556074c0219bab373535964888677da09dd27c
Submitter: Jenkins
Branch: master

commit 3f556074c0219bab373535964888677da09dd27c
Author: Carl Baldwin <email address hidden>
Date: Fri Mar 4 17:40:50 2016 -0700

    Skip racey BGP tests

    Change-Id: I58e6056a9fe6460352855461e74700f4934e6e3f
    Related-bug: #1553374

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/288921

[Ryan Tidwell (ryan-tidwell)]

The tests that are problematic are tests that assert functionality outside of an address scope. This includes tests that assert floating IP host route announcements and 1 test that asserts behavior when no address scope or floating IP's are involved. When not asserting floating IP annouincements, these tests can be isolated by simply creating a BGP speaker with advertise_floating_ip_host_routes=False. I have proposed https://review.openstack.org/288921 which does exactly this for a couple of tests.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/288825
Reason: Abandoning in favor of a more solid approach from Ryan.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ryan Tidwell (<email address hidden>) on branch: master
Review: https://review.openstack.org/288921

[Carl Baldwin (carl-baldwin)]

Just talked to Ryan. Since we're not running dsvm-api jobs on the neutron-dynamic-routing repo yet, we're not seeing this. We'll keep this open until we do and then check it. Ryan suspects that the contention with floating ips will just be gone by running these tests in isolation.

[Armando Migliaccio (armando-migliaccio)]

@Carl, I agree that's the reason. Ideally we would kill the contention altogether if every test would have its own external network but that is blocked until we can get rid of the external_network_id config option.

[Armando Migliaccio (armando-migliaccio)]

No longer applies.