stable: Neutron unit tests timeout on stable/ussuri and stable/victoria (perhaps wallaby also)

[Lajos Katona]

On stable branches the unit tests fail with time out on Ussuri and Victoria branches (on Wallaby there's also a timeout but it has different log, so perhaps just an accident, see [3])

The periodic jobs show the issue and my test patch also (see [1] and [2])

The timeout happens with this pattern, always after the test test_two_member_trailing_chain:
2022-07-19 02:48:57.591352 | ubuntu-focal | {1} neutron.tests.unit.test_wsgi.TestWSGIServer.test_disable_ssl [0.039549s] ... ok
2022-07-19 02:48:57.613094 | ubuntu-focal | {1} neutron.tests.unit.tests.test_post_mortem_debug.TestGetIgnoredTraceback.test_two_member_trailing_chain [0.021062s] ... ok
2022-07-19 03:05:57.910628 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/zuul/zuul-jobs/playbooks/tox/run.yaml@master]
2022-07-19 03:05:57.926082 | POST-RUN START: [untrusted : opendev.org/zuul/zuul-jobs/playbooks/tox/post.yaml@master]

[1]: https://lists.openstack.org/pipermail/openstack-stable-maint/2022-July/094216.html
[2]: https://72340191a2149eb73d49-2d27e805ff456f7d0a585c8eee091a7d.ssl.cf2.rackcdn.com/850391/1/check/openstack-tox-py37/9ab5c3a/job-output.txt
[3]: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f2a/periodic-stable/opendev.org/openstack/neutron/stable/wallaby/openstack-tox-py38/f2a3976/job-output.txt

[Lajos Katona (lajos-katona)]

I can't reproduce the timeout locally, so I tend to believe something changed in the gate

[Lajos Katona (lajos-katona)]

Update: wallaby seems to be free of this error, and the timeout happens after different tests

[Lajos Katona (lajos-katona)]

infra (fungi and elodilles from release team actually) was so kind to help debugging the issue, and there suspicious pkg upgrades between passing and failing runs (actually between 11 and 12 July), see the diff from elodilles:
https://paste.opendev.org/show/b8JwAKL4wMqdxCGOZpp8/

IRC log:
https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2022-07-20.log.html#t2022-07-20T12:59:32

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/850612

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/850612
Reason: Instead of this, we are going to mock this method in UTs

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/850670

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Lajos Katona <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/850544

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Lajos Katona <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/850391
Reason: https://review.opendev.org/c/openstack/neutron/+/850670 can be backported to mock out the problematic sysctl calls

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/850670
Committed: https://opendev.org/openstack/neutron/commit/119b82f1b10df50a696e936e80672c7ddb00436f
Submitter: "Zuul (22348)"
Branch: master

commit 119b82f1b10df50a696e936e80672c7ddb00436f
Author: elajkat <email address hidden>
Date: Thu Jul 21 16:53:45 2022 +0200

    Test: mock out _check_netfilter_for_bridges in unit tests

    _check_netfilter_for_bridges do some calls which are not for unit tests.
    Specifically the call for sysctl -N net.bridge seems to be broken with
    a recent kernel upgrade on bionic (see the irc log for this discussion
    in [1]).
    Add OS_TEST_TIMEOUT:180 to tox testenv, this can help to debug such
    issues.
    During the rootwrap to privsep change some of these mocks were added,
    (see [2]) so backporting (before Wallaby mostly) can be done with
    caution.

    [1]: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2022-07-20.log.html#t2022-07-20T12:59:32
    [2]: https://review.opendev.org/c/openstack/neutron/+/764015/12/neutron/tests/unit/agent/linux/openvswitch_firewall/test_iptables.py

    Change-Id: Ic366d67770cc6e19a4412eaf2f1ab53780e98ee8
    Closes-Bug: #1982206

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/850920

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/851063

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/851064

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/851212

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/851065

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/850920
Committed: https://opendev.org/openstack/neutron/commit/81809eac06116361149bd13424251fff531aa7db
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 81809eac06116361149bd13424251fff531aa7db
Author: elajkat <email address hidden>
Date: Thu Jul 21 16:53:45 2022 +0200

    Test: mock out _check_netfilter_for_bridges in unit tests

    _check_netfilter_for_bridges do some calls which are not for unit tests.
    Specifically the call for sysctl -N net.bridge seems to be broken with
    a recent kernel upgrade on bionic (see the irc log for this discussion
    in [1]).
    Add OS_TEST_TIMEOUT:180 to tox testenv, this can help to debug such
    issues.
    During the rootwrap to privsep change some of these mocks were added,
    (see [2]) so backporting (before Wallaby mostly) can be done with
    caution.

    [1]: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2022-07-20.log.html#t2022-07-20T12:59:32
    [2]: https://review.opendev.org/c/openstack/neutron/+/764015/12/neutron/tests/unit/agent/linux/openvswitch_firewall/test_iptables.py

    Change-Id: Ic366d67770cc6e19a4412eaf2f1ab53780e98ee8
    Closes-Bug: #1982206
    (cherry picked from commit 119b82f1b10df50a696e936e80672c7ddb00436f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/851063
Committed: https://opendev.org/openstack/neutron/commit/ff9c7cf1a28736c431e5d8571012f3512e21ba7c
Submitter: "Zuul (22348)"
Branch: stable/xena

commit ff9c7cf1a28736c431e5d8571012f3512e21ba7c
Author: elajkat <email address hidden>
Date: Thu Jul 21 16:53:45 2022 +0200

    Test: mock out _check_netfilter_for_bridges in unit tests

    _check_netfilter_for_bridges do some calls which are not for unit tests.
    Specifically the call for sysctl -N net.bridge seems to be broken with
    a recent kernel upgrade on bionic (see the irc log for this discussion
    in [1]).
    Add OS_TEST_TIMEOUT:180 to tox testenv, this can help to debug such
    issues.
    During the rootwrap to privsep change some of these mocks were added,
    (see [2]) so backporting (before Wallaby mostly) can be done with
    caution.

    [1]: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2022-07-20.log.html#t2022-07-20T12:59:32
    [2]: https://review.opendev.org/c/openstack/neutron/+/764015/12/neutron/tests/unit/agent/linux/openvswitch_firewall/test_iptables.py

    Change-Id: Ic366d67770cc6e19a4412eaf2f1ab53780e98ee8
    Closes-Bug: #1982206
    (cherry picked from commit 119b82f1b10df50a696e936e80672c7ddb00436f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/851065
Committed: https://opendev.org/openstack/neutron/commit/dedac9ff4e4690a5b0d0d1213c9be614d347c034
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit dedac9ff4e4690a5b0d0d1213c9be614d347c034
Author: elajkat <email address hidden>
Date: Thu Jul 21 16:53:45 2022 +0200

    Test: mock out _check_netfilter_for_bridges in unit tests

    _check_netfilter_for_bridges do some calls which are not for unit tests.
    Specifically the call for sysctl -N net.bridge seems to be broken with
    a recent kernel upgrade on bionic (see the irc log for this discussion
    in [1]).
    Add OS_TEST_TIMEOUT:180 to tox testenv, this can help to debug such
    issues.
    During the rootwrap to privsep change some of these mocks were added,
    (see [2]) so backporting (before Wallaby mostly) can be done with
    caution.
    From stable/victoria and backward
    TestHybridIptablesHelper.test_overloaded_remove_conntrack need the extra
    mock as [2] was not backported to older branches than stable/wallaby.

    Conflicts:
        neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/test_ovs_neutron_agent.py

    [1]: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2022-07-20.log.html#t2022-07-20T12:59:32
    [2]: https://review.opendev.org/c/openstack/neutron/+/764015/12/neutron/tests/unit/agent/linux/openvswitch_firewall/test_iptables.py

    Change-Id: Ic366d67770cc6e19a4412eaf2f1ab53780e98ee8
    Closes-Bug: #1982206
    (cherry picked from commit 119b82f1b10df50a696e936e80672c7ddb00436f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/851212
Committed: https://opendev.org/openstack/neutron/commit/7b87139fa09348987f5380e2a06b5abc32a26390
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 7b87139fa09348987f5380e2a06b5abc32a26390
Author: elajkat <email address hidden>
Date: Thu Jul 21 16:53:45 2022 +0200

    Test: mock out _check_netfilter_for_bridges in unit tests

    _check_netfilter_for_bridges do some calls which are not for unit tests.
    Specifically the call for sysctl -N net.bridge seems to be broken with
    a recent kernel upgrade on bionic (see the irc log for this discussion
    in [1]).
    Add OS_TEST_TIMEOUT:180 to tox testenv, this can help to debug such
    issues.
    During the rootwrap to privsep change some of these mocks were added,
    (see [2]) so backporting (before Wallaby mostly) can be done with
    caution.
    From stable/victoria and backward
    TestHybridIptablesHelper.test_overloaded_remove_conntrack need the extra
    mock as [2] was not backported to older branches than stable/wallaby.

    [1]: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2022-07-20.log.html#t2022-07-20T12:59:32
    [2]: https://review.opendev.org/c/openstack/neutron/+/764015/12/neutron/tests/unit/agent/linux/openvswitch_firewall/test_iptables.py

    Change-Id: Ic366d67770cc6e19a4412eaf2f1ab53780e98ee8
    Closes-Bug: #1982206
    (cherry picked from commit 119b82f1b10df50a696e936e80672c7ddb00436f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/851064
Committed: https://opendev.org/openstack/neutron/commit/868427c104f9ad7cb996143646d587b0f43079eb
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 868427c104f9ad7cb996143646d587b0f43079eb
Author: elajkat <email address hidden>
Date: Thu Jul 21 16:53:45 2022 +0200

    Test: mock out _check_netfilter_for_bridges in unit tests

    _check_netfilter_for_bridges do some calls which are not for unit tests.
    Specifically the call for sysctl -N net.bridge seems to be broken with
    a recent kernel upgrade on bionic (see the irc log for this discussion
    in [1]).
    Add OS_TEST_TIMEOUT:180 to tox testenv, this can help to debug such
    issues.
    During the rootwrap to privsep change some of these mocks were added,
    (see [2]) so backporting (before Wallaby mostly) can be done with
    caution.

    [1]: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2022-07-20.log.html#t2022-07-20T12:59:32
    [2]: https://review.opendev.org/c/openstack/neutron/+/764015/12/neutron/tests/unit/agent/linux/openvswitch_firewall/test_iptables.py

    Change-Id: Ic366d67770cc6e19a4412eaf2f1ab53780e98ee8
    Closes-Bug: #1982206
    (cherry picked from commit 119b82f1b10df50a696e936e80672c7ddb00436f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron victoria-eom release.