[skip-level] OVN tests constantly failing

[Rodolfo Alonso]

In the new Zed-Bobcat skip-level jobs [1], the OVN job has 4 tests constantly failing (1 fail is actually a setup class method):
*tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
*tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesUnderV243Test.test_add_remove_fixed_ip
*setUpClass (tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON)
*tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops

Logs:
*https://fd50651997fbb0337883-282d0b18354725863279cd3ebda4ab44.ssl.cf5.rackcdn.com/878632/6/experimental/neutron-ovn-grenade-multinode-skip-level/baf4ed5/controller/logs/grenade.sh_log.txt
*https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_607/878632/6/experimental/neutron-ovn-grenade-multinode-skip-level/6072d85/controller/logs/grenade.sh_log.txt

[1]https://review.opendev.org/c/openstack/neutron/+/878632

[Bence Romsics (bence-romsics)]

In Bobcat (since it's not a slurp release) we do not have this job in the check queue, so this is not an immediate gate blocker, but as we agreed on the PTG we should keep this job clean, so it doesn't become a blocker when we move to release C. That's why I set the importance to high and not critical.

[yatin (yatinkarel)]

Not only skip level jobs but also normal ovn grenade multinode jobs are also impacted[1]. Basically any multinode job running kernel>=5.5 and deploying ovs/ovn from source will be impacted.

The issue got visible after grenade jobs switched to jammy[2]. Since jammy have kernel >=5.5 openvswitch modules doesn't get's reloaded(already loaded as part of multi-node-bridge execution) after ovs is installed from source[3]. And until it get's reloaded the traffic to/from ovs ports do not work.

So we need to reload the modules always irrespective of build_modules is True or False. Also build modules is deprecated since OVS2.15 and dropped with OVS3.0[4]. This can additionally handled as seperate patch.

As it's part of devstack install reloading modules always shouldn't hurt as we need not worry about dataplane downtime. Will push a patch to devstack(as [3] now lives there) and backport to stable branches in order to fix it. Verified in [5].

[1] https://zuul.openstack.org/builds?job_name=neutron-ovn-grenade-multinode
[2] https://review.opendev.org/c/openstack/grenade/+/877998
[3] https://review.opendev.org/c/openstack/neutron/+/726759
[4] https://github.com/openvswitch/ovs/commit/422e90437
[5] https://review.opendev.org/c/openstack/neutron/+/878761

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/devstack/+/880533

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/devstack/+/880494

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/devstack/+/880495

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/880582

[Lajos Katona (lajos-katona)]

Is this also related to the other grenade job failure bug: https://bugs.launchpad.net/neutron/+bug/2015065 ?

[yatin (yatinkarel)]

<<< Is this also related to the other grenade job failure bug: https://bugs.launchpad.net/neutron/+bug/2015065 ?

No this is different and OVN/OVS source deploy specific.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/880582
Committed: https://opendev.org/openstack/neutron/commit/73ac4510c7c76fed8ec0cd0a00be50d29549c902
Submitter: "Zuul (22348)"
Branch: master

commit 73ac4510c7c76fed8ec0cd0a00be50d29549c902
Author: yatinkarel <email address hidden>
Date: Mon Apr 17 12:26:36 2023 +0530

    [grenade] Collect ovn services logs

    Would be useful investigating issues.

    Related-Bug: #2015364
    Change-Id: I5ef00b1150ebc4bbf03e1c9f192cf0cb884d0c08

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/880533
Committed: https://opendev.org/openstack/devstack/commit/42517968ff7bdced07c5bc08b6cb2b8d10d246cc
Submitter: "Zuul (22348)"
Branch: master

commit 42517968ff7bdced07c5bc08b6cb2b8d10d246cc
Author: yatinkarel <email address hidden>
Date: Fri Apr 14 19:06:03 2023 +0530

    [ovs] Reload ovs kernel module always

    Irrespective of build_modules is True
    or False reload ovs modules always.

    If ovs is installed from package before(like
    with multi-node-bridge role), then installing
    ovs from source requires openvswitch kernel
    module to be reloaded.

    The issue was not seen before jammy as there
    module was reloaded when build_modules was set
    to True.

    Closes-Bug: #2015364
    Change-Id: I1785b49b2ef72ca1f817f504d5ea56021410c052

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/880494
Committed: https://opendev.org/openstack/devstack/commit/f1d555d1e7af0fc5097a89c3df044bc4a35ec53f
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit f1d555d1e7af0fc5097a89c3df044bc4a35ec53f
Author: yatinkarel <email address hidden>
Date: Fri Apr 14 19:06:03 2023 +0530

    [ovs] Reload ovs kernel module always

    Irrespective of build_modules is True
    or False reload ovs modules always.

    If ovs is installed from package before(like
    with multi-node-bridge role), then installing
    ovs from source requires openvswitch kernel
    module to be reloaded.

    The issue was not seen before jammy as there
    module was reloaded when build_modules was set
    to True.

    Closes-Bug: #2015364
    Change-Id: I1785b49b2ef72ca1f817f504d5ea56021410c052
    (cherry picked from commit 42517968ff7bdced07c5bc08b6cb2b8d10d246cc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/880495
Committed: https://opendev.org/openstack/devstack/commit/4181791e065c54d396748f79367673309f84acb6
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 4181791e065c54d396748f79367673309f84acb6
Author: yatinkarel <email address hidden>
Date: Fri Apr 14 19:06:03 2023 +0530

    [ovs] Reload ovs kernel module always

    Irrespective of build_modules is True
    or False reload ovs modules always.

    If ovs is installed from package before(like
    with multi-node-bridge role), then installing
    ovs from source requires openvswitch kernel
    module to be reloaded.

    The issue was not seen before jammy as there
    module was reloaded when build_modules was set
    to True.

    Closes-Bug: #2015364
    Change-Id: I1785b49b2ef72ca1f817f504d5ea56021410c052
    (cherry picked from commit 42517968ff7bdced07c5bc08b6cb2b8d10d246cc)
    (cherry picked from commit f1d555d1e7af0fc5097a89c3df044bc4a35ec53f)

[Brian Haley (brian-haley)]

Since the skip-level job is now passing and voting in our gate I am going to close this bug.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "yatin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/878761
Reason: fix merged

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/devstack/+/880481
Reason: stable/zed branch of openstack/devstack is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/zed if you want to further work on this patch.