[stable/wallaby] test_established_tcp_session_after_re_attachinging_sg is unstable on ML2/OVS iptables_hybrid job

[Bernard Cafarelli]

In recent weeks this job started to fail regularly on stable/wallaby (not 100% some backports are passing)
neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest.test_established_tcp_session_after_re_attachinging_sg

Most of the time this was seen in neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby job

Two recent backports affected by issue:
https://review.opendev.org/c/openstack/neutron/+/871759
https://review.opendev.org/c/openstack/neutron/+/868087

And some sample logs:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_26b/871759/2/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/26b64ec/testr_results.html
https://f450bef156b424c8c132-a0541882d2023eca9a1cc07087449de0.ssl.cf1.rackcdn.com/868087/3/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/55a6ead/testr_results.html

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 80, in wait_until_true
    eventlet.sleep(sleep)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/eventlet/greenthread.py", line 36, in sleep
    hub.switch()
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 320, in test_established_tcp_session_after_re_attachinging_sg
    con.test_connection(should_pass=False)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 202, in test_connection
    wait_until_true(
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 85, in wait_until_true
    raise WaitTimeout("Timed out after %d seconds" % timeout)
neutron_tempest_plugin.common.utils.WaitTimeout: Timed out after 10 seconds

I wonder if this is similar to https://bugs.launchpad.net/neutron/+bug/1936911 where the test was unstable on linuxbridge backend

[yatin (yatinkarel)]

Marking as Critical as it's consistent failed ~15 runs, will check what's wrong.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

New occurrences:
* https://c4c1e3fa2f7be1ec5ca2-4728a33d3e305b86424370fe8e72ce6f.ssl.cf5.rackcdn.com/873627/1/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/c4638f2/testr_results.html
* https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b4b/873627/1/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/b4bd787/testr_results.html

[yatin (yatinkarel)]

So [1] pinned neutron-tempest-plugin to 1.8.0 and since then it's consistently failing[2]. The test has to be excluded for iptables hybrid job[3] but it's not due to name change in[4], it was indeed caught in wallaby pin tests too[5] but somehow missed. Will push a patch to skip the test as before.

[1] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/871793
[2] https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby
[3] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/793196
[4] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/867802
[5] https://review.opendev.org/c/openstack/neutron/+/871797

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/873708

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/873708
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/28337975b76fce8e575a12576563428ce4cef6cd
Submitter: "Zuul (22348)"
Branch: master

commit 28337975b76fce8e575a12576563428ce4cef6cd
Author: yatinkarel <email address hidden>
Date: Tue Feb 14 19:19:50 2023 +0530

    [Wallaby] Fix tempest_exclude for ovs iptables_hybrid

    It was missed in [1] while pinning neutron-tempest-plugin
    to 1.8.0 where test name is different.

    The test is required to be exluded as per [2] in
    iptables hybrid ovs job, this patch fixes it for
    wallaby job.

    [1] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/871793
    [2] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/793196

    Closes-Bug: #2006953
    Change-Id: I6a50611ea828223510dfef8290458c19a65f3048

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 2.2.0 release.