Race conditions in neutron_tempest_plugin/scenario/test_security_groups.py

[Federico Ressi]

I found a couple of race conditions in neutron-tempest-plugin affecting recular CI build when executing test jobs with neutron-tempest-plugin test suite. These race conditions are mostly because two test case wrongly modify default security group causing random failures in other scenario test cases or also failing themself.

1) Test case test_default_sec_grp_scenarios shouldn't change default security group rule at below lines:
https://github.com/openstack/neutron-tempest-plugin/blob/cf38b77328dbf94f1323f96f68aa77124a6f4a7b/neutron_tempest_plugin/scenario/test_security_groups.py#L109-L112
This could caouse other tests to fail. Modifying default resources should be forbidden. There is no difference between default security group and another group, therefore the test makes no sense at all. Tests that do some changes on resources should create its own target resources.

2) Test case test_protocol_number_rule list for security groups and takes the first one, then it changes it at line
https://github.com/openstack/neutron-tempest-plugin/blob/cf38b77328dbf94f1323f96f68aa77124a6f4a7b/neutron_tempest_plugin/scenario/test_security_groups.py#L147-L149
The problem here is that the first security group that is taken could be the default security group instead of the one created by the test. This would cause a failure in the same test and even worst in another test later.

[Federico Ressi (fressi-redhat)]

I started fixing this problem here: https://review.openstack.org/#/c/612733/5

[Federico Ressi (fressi-redhat)]

There are two patches in progress to fix this bug:

- https://review.openstack.org/#/c/612733 - to cleanup CI status
- https://review.openstack.org/#/c/615813 - complete test case refactory to improve code quality and readybility

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/612733
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=6f9bb77d9872a1cac4e1815ce13d97b8a143b3a8
Submitter: Zuul
Branch: master

commit 6f9bb77d9872a1cac4e1815ce13d97b8a143b3a8
Author: Federico Ressi <email address hidden>
Date: Tue Oct 23 17:35:47 2018 +0200

    Fix race condition in test_security_groups

    Listing security groups and taking the first one to change
    rules doesn't give any warranty listed security group is
    exactly the one used to create servers ports.

    This avoids this problem by creating a security group and
    passing it to server instead of having to search for it
    after the server has been created.

    Despite a test cases require a general refactory this should
    at least fix the CI job intermittent failures related to
    this test.

    Partial-Bug: #1801306
    Change-Id: I5ecb3c8711e7455e7f1588943a2ffd08ce154578

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/615813
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Brian Haley (brian-haley)]

The bug is fixed, the additional change to cleanup the code was never merged. For that reason I will close this but please reopen if necessary.