[OVN] neutron-ovn-tempest-ovs-master-fedora periodic job is failing

[Maciej Jozefczyk]

https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master-fedora

Last success:
https://zuul.openstack.org/build/3816a88272ea413995408846a52b5366

First failure:
https://zuul.openstack.org/build/b12ec3ab38e6418d9580829f0e98bfd2

Failure is on installation of kernel-devel for OVS module compilation:

2020-04-26 06:28:30.004 | No match for argument: kernel-devel-5.5.17
2020-04-26 06:28:30.004 | Error: Unable to find a match: kernel-devel-5.5.17
2020-04-26 06:28:30.004 | YUM_FAILED 1

Strange is based on the logs from last success run, the package kernel-devel-5.5.17 has been installed properly.

[Maciej Jozefczyk (maciejjozefczyk)]

Based on packages list:
https://ftp.icm.edu.pl/pub/Linux/fedora/linux/updates/31/Everything/x86_64/Packages/k/

Looks like the kernel-5.5 has been deleted from it? I think we're using old image on gates.

[Maciej Jozefczyk (maciejjozefczyk)]

Can't find it also there:
https://ftp.icm.edu.pl/pub/Linux/fedora/linux/releases/31/Everything/x86_64/os/Packages/k/

[Maciej Jozefczyk (maciejjozefczyk)]

Last failures are different:

https://zuul.openstack.org/build/d7872c4f83c448f0aca6041a3571ca89

checking for kernel version... 5.6.8
configure: error: Linux kernel in /lib/modules/5.6.8-200.fc31.x86_64/build is version 5.6.8, but version newer than 5.5.x is not supported (please refer to the FAQ for advice)
+ /opt/stack/neutron/devstack/lib/ovn_agent:_compile_ovs:1 : exit_trap

[Maciej Jozefczyk (maciejjozefczyk)]

Looks like we can't compile kernel module in version higher than 5.5.x:

https://github.com/openvswitch/ovs/commit/59e994426645358a271a0c9f485e9defafffd474

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/726759

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726759
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0ba08d7eb488f1b7bd66448514f468c30f402a15
Submitter: Zuul
Branch: master

commit 0ba08d7eb488f1b7bd66448514f468c30f402a15
Author: Maciej Józefczyk <email address hidden>
Date: Mon May 11 08:54:08 2020 +0000

    Skip OVS module compilation for kernels newer than 5.5

    For now OVS doesn't support compilation of kernel module
    for kernels newer than 5.5 [1].
    Recently Fedora 31 upgraded the kernel to 5.6.8 and that is
    why the periodic jobs started to fail [2].
    By default we enabled OVS module compilation because Ubuntu
    Bionic one doesn't have support for Openflow Meter actions.

    In case flag OVN_BUILD_MODULES is set to True and kernel is
    newer than 5.5 lets skip the module compilation with proper log.

    Closes-Bug: #1877377

    [1] https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master-fedora
    [2] https://github.com/openvswitch/ovs/commit/59e994426645358a271a0c9f485e9defafffd474

    Change-Id: I63b2f198468c9e30af8284c8033fa31f2af2518a

[Maciej Jozefczyk (maciejjozefczyk)]

We have the same bug in different stage of ovs/ovn installation.

I believe we should unify the code in that area (devstack lib), because some parts of it are duplicating each other.

I'll propose a patch to unify OVS/OVN compilation for both devstack libs.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737984
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fb2806f8088dccafd6616a68f5b296914d192150
Submitter: Zuul
Branch: master

commit fb2806f8088dccafd6616a68f5b296914d192150
Author: Maciej Jozefczyk <email address hidden>
Date: Thu Jun 25 11:24:48 2020 +0000

    [OVN] Unify OVN/OVS compilation

    There were duplicated methods doing almost the same in terms
    of OVS/OVN compilation.

    This change:
     * move of OVS related compilation code to devstack/lib/ovs
     * delete of OVS related compilation code from devstack/lib/ovn_agent
     * source unified functions in devstack/lib/ovn_agent from
       devstack/lib/ovs
     * Unify NEUTRON_PATH variable to NEUTRON_DIR

    Closes-Bug: #1877377

    Change-Id: Ia012a8e116a276a6674f86366c803e0e2d8ff704

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/740255

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/741606

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740255
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=61851345cededd906517cea783cd86f8abd3a01a
Submitter: Zuul
Branch: stable/ussuri

commit 61851345cededd906517cea783cd86f8abd3a01a
Author: Maciej Jozefczyk <email address hidden>
Date: Thu Jun 25 11:24:48 2020 +0000

    [OVN] Unify OVN/OVS compilation

    There were duplicated methods doing almost the same in terms
    of OVS/OVN compilation.

    This change:
     * move of OVS related compilation code to devstack/lib/ovs
     * delete of OVS related compilation code from devstack/lib/ovn_agent
     * source unified functions in devstack/lib/ovn_agent from
       devstack/lib/ovs
     * Unify NEUTRON_PATH variable to NEUTRON_DIR

    Stable-specific change:
     * Sets default variable for OVN_BRANCH to point latest
       stable release

    Conflicts:
       devstack/lib/ovn_agent

    Closes-Bug: #1877377

    Change-Id: Ia012a8e116a276a6674f86366c803e0e2d8ff704
    (cherry picked from commit fb2806f8088dccafd6616a68f5b296914d192150)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741606
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=294720fcd28485326e62eb38de5d6ba9683120a2
Submitter: Zuul
Branch: master

commit 294720fcd28485326e62eb38de5d6ba9683120a2
Author: Maciej Józefczyk <email address hidden>
Date: Fri Jul 17 11:38:38 2020 +0200

    Fix neutron-ovn-tempest-ovs-master-fedora job

    The job should use OVN_BRANCH = master and OVS_BRANCH = master,
    because based on its definition, it should use master code.

    Change-Id: I0a708ae63d6a2c8933324157a4bbbd229fc43073
    Related-Bug: #1877377

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/742519

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742519
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2b00041f4dcf9b36355037beb6d352526ae67e1a
Submitter: Zuul
Branch: stable/ussuri

commit 2b00041f4dcf9b36355037beb6d352526ae67e1a
Author: Maciej Józefczyk <email address hidden>
Date: Fri Jul 17 11:38:38 2020 +0200

    Fix neutron-ovn-tempest-ovs-master-fedora job

    The job should use OVN_BRANCH = master and OVS_BRANCH = master,
    because based on its definition, it should use master code.

    Change-Id: I0a708ae63d6a2c8933324157a4bbbd229fc43073
    Related-Bug: #1877377
    (cherry picked from commit 294720fcd28485326e62eb38de5d6ba9683120a2)