Fullstack tests neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork fails

[Slawek Kaplonski]

Fullstack test neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork. test_securitygroup again fails quite often for iptables based firewall driver.

Example of failure: http://logs.openstack.org/46/571546/7/gate/neutron-fullstack/be437ac/logs/testr_results.html.gz

There is at least 15 hits between 26.06 and 29.06: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20_test_using_multiple_security_groups%5C%22

In log stash there are no such failures before 26.06 so maybe something merged then broke it?

[Slawek Kaplonski (slaweq)]

Recently it found it failing sometimes again. It's always same failure:

Traceback (most recent call last):
  File "neutron/tests/base.py", line 140, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/fullstack/test_securitygroup.py", line 282, in test_securitygroup
    self._test_using_multiple_security_groups()
  File "neutron/tests/fullstack/test_securitygroup.py", line 371, in _test_using_multiple_security_groups
    vms[0].block_until_ping(vms[2].ip)
  File "neutron/tests/common/machine_fixtures.py", line 70, in block_until_ping
    "No ICMP reply obtained from IP address %s" % dst_ip)
  File "neutron/common/utils.py", line 646, in wait_until_true
    raise exception
neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 30.0.0.18

It happens more often in neutron-fullstack-python35 job than in job which uses python 2.7.
Also what I found is that it was always failed in linuxbridge-iptables scenario.

Example of failures:
Python 2.7:
    * http://logs.openstack.org/27/534227/16/check/neutron-fullstack/2a475f6/logs/testr_results.html.gz

Python 3.5:
    * http://logs.openstack.org/27/534227/16/check/neutron-fullstack-python35/f9e25c1/logs/testr_results.html.gz
    * http://logs.openstack.org/14/529814/27/check/neutron-fullstack-python35/19aef6b/logs/testr_results.html.gz
    * http://logs.openstack.org/52/585752/2/check/neutron-fullstack-python35/208d4a6/logs/testr_results.html.gz
    * http://logs.openstack.org/65/588465/1/check/neutron-fullstack-python35/4b236ad/logs/testr_results.html.gz
    * http://logs.openstack.org/41/586341/4/check/neutron-fullstack-python35/49711a5/logs/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/591645

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/591645
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0bd1cec6768c9748cb415261fd8e4d4da0ee7ddb
Submitter: Zuul
Branch: master

commit 0bd1cec6768c9748cb415261fd8e4d4da0ee7ddb
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 14 14:21:57 2018 +0200

    [Fullstack] Add debug_iptables=True in security groups tests

    In security groups tests with "iptables" or "iptables_hybrid"
    driver it will be useful to have debug_iptables_rules enabled
    to check what rules are applied by L2 agent in case of test
    failures.

    Change-Id: Ib7d12b2e589019bc6043affe371ef5aa5425945b
    Related-Bug: #1779328

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/597426

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/597426
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f9dc83eb9576ca01d732bf6a3ff3b3e9d01647e7
Submitter: Zuul
Branch: master

commit f9dc83eb9576ca01d732bf6a3ff3b3e9d01647e7
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 29 10:33:53 2018 +0200

    [Fullstack] Mark securitygroups tests as unstable

    It agains fails quite often in iptables scenarios.
    We need to debug it more to find out what cause this issue
    but for now let's mark this test as unstable to make
    life of other people easier.

    Change-Id: I7bf6f9b346c6c853193cb045fb364b97375e9d93
    Related-bug: #1779328

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/603073

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603073
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0cf75ecb57c1d7583c8e55a7b127be247188646d
Submitter: Zuul
Branch: stable/rocky

commit 0cf75ecb57c1d7583c8e55a7b127be247188646d
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 29 10:33:53 2018 +0200

    [Fullstack] Mark securitygroups tests as unstable

    It agains fails quite often in iptables scenarios.
    We need to debug it more to find out what cause this issue
    but for now let's mark this test as unstable to make
    life of other people easier.

    Change-Id: I7bf6f9b346c6c853193cb045fb364b97375e9d93
    Related-bug: #1779328
    (cherry picked from commit f9dc83eb9576ca01d732bf6a3ff3b3e9d01647e7)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/710782

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/710964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/710964
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8456240e7da79f0409259597d796b7d6cab3298d
Submitter: Zuul
Branch: master

commit 8456240e7da79f0409259597d796b7d6cab3298d
Author: Slawek Kaplonski <email address hidden>
Date: Tue Mar 3 11:27:34 2020 +0100

    Run fullstack security group test always serially

    Security groups are based on iptables in many cases and can interfere
    between other tests. So it means that if some other test will manipulate
    with iptables on host during SG related test is run, this SG related
    test may fail without any obvious reason.
    So lets try to run those test serially to be sure that no other agents
    will manipulate iptables/openflow rules in same time.

    Change-Id: I4d19f2a457e004306fdf40980a943073f1b8704a
    Closes-Bug: #1779328

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/710782
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7f925249a598aba88ecd58bb4759fc1767954e5
Submitter: Zuul
Branch: master

commit d7f925249a598aba88ecd58bb4759fc1767954e5
Author: Nate Johnston <email address hidden>
Date: Mon Mar 2 09:22:43 2020 -0500

    Mark security group fullstack tests as stable

    It has been some time since we marked the security group tests in
    fullstack as unstable. Now that mark is impeding the fullstack tests
    for stateless security groups.

    Based on testing it appears that the unstable tests are specific to the
    linuxbridge driver, which is not really very much supported these days.
    Testing for that driver is dropped, since it is better to at least test
    for the other scenarios than nothing at all.

    Change-Id: Ib1ef0763496aa536ba80f1c34c64d8a6c313fd0b
    Needed-By: https://review.opendev.org/710252
    Related-bug: #1779328