ovn based neutron gate jobs failing 100% of times

[Slawek Kaplonski]

It seems that since last few days our tempest jobs with ovn are failing all the time. Example of failure:

https://zuul.opendev.org/t/openstack/build/d3a28cb7a40c46dea20fb96a4d116c98

Error:

2020-06-15 16:43:44.093113 | controller | CC [M] /opt/stack/ovs/datapath/linux/exthdrs_core.o
2020-06-15 16:43:44.132270 | controller | CC [M] /opt/stack/ovs/datapath/linux/geneve.o
2020-06-15 16:43:44.145219 | controller | CC [M] /opt/stack/ovs/datapath/linux/gre.o
2020-06-15 16:43:44.520322 | controller | CC [M] /opt/stack/ovs/datapath/linux/gso.o
2020-06-15 16:43:44.613210 | controller | CC [M] /opt/stack/ovs/datapath/linux/genetlink-openvswitch.o
2020-06-15 16:43:44.837038 | controller | CC [M] /opt/stack/ovs/datapath/linux/inet_fragment.o
2020-06-15 16:43:44.871982 | controller | CC [M] /opt/stack/ovs/datapath/linux/ip_gre.o
2020-06-15 16:43:44.929289 | controller | /opt/stack/ovs/datapath/linux/geneve.c: In function 'geneve_get_v6_dst':
2020-06-15 16:43:44.929366 | controller | CC [M] /opt/stack/ovs/datapath/linux/ip_fragment.o
2020-06-15 16:43:44.929479 | controller | /opt/stack/ovs/datapath/linux/geneve.c:966:17: error: 'const struct ipv6_stub' has no member named 'ipv6_dst_lookup'; did you mean 'ipv6_dst_lookup_flow'?
2020-06-15 16:43:44.929520 | controller | if (ipv6_stub->ipv6_dst_lookup(geneve->net, gs6->sock->sk, &dst, fl6)) {
2020-06-15 16:43:44.929549 | controller | ^~~~~~~~~~~~~~~
2020-06-15 16:43:44.930026 | controller | ipv6_dst_lookup_flow
2020-06-15 16:43:44.961658 | controller | CC [M] /opt/stack/ovs/datapath/linux/ip_tunnel.o
2020-06-15 16:43:44.964932 | controller | CC [M] /opt/stack/ovs/datapath/linux/ip_output.o
2020-06-15 16:43:44.983199 | controller | scripts/Makefile.build:330: recipe for target '/opt/stack/ovs/datapath/linux/geneve.o' failed
2020-06-15 16:43:44.983267 | controller | make[5]: *** [/opt/stack/ovs/datapath/linux/geneve.o] Error 1
2020-06-15 16:43:44.983296 | controller | make[5]: *** Waiting for unfinished jobs....
2020-06-15 16:43:46.142816 | controller | Makefile:1577: recipe for target '_module_/opt/stack/ovs/datapath/linux' failed
2020-06-15 16:43:46.142890 | controller | make[4]: *** [_module_/opt/stack/ovs/datapath/linux] Error 2
2020-06-15 16:43:46.142915 | controller | make[4]: Leaving directory '/usr/src/linux-headers-4.15.0-106-generic'
2020-06-15 16:43:46.142946 | controller | Makefile.main:71: recipe for target 'default' failed
2020-06-15 16:43:46.142968 | controller | make[3]: *** [default] Error 2
2020-06-15 16:43:46.142988 | controller | make[3]: Leaving directory '/opt/stack/ovs/datapath/linux'
2020-06-15 16:43:46.143818 | controller | Makefile:579: recipe for target 'all-recursive' failed
2020-06-15 16:43:46.143846 | controller | make[2]: *** [all-recursive] Error 1
2020-06-15 16:43:46.143867 | controller | make[2]: Leaving directory '/opt/stack/ovs/datapath'
2020-06-15 16:43:46.144917 | controller | Makefile:4990: recipe for target 'all-recursive' failed
2020-06-15 16:43:46.144967 | controller | make[1]: *** [all-recursive] Error 1
2020-06-15 16:43:46.144990 | controller | make[1]: Leaving directory '/opt/stack/ovs'
2020-06-15 16:43:46.145010 | controller | Makefile:2750: recipe for target 'all' failed
2020-06-15 16:43:46.145030 | controller | make: *** [all] Error 2

[Maciej Jozefczyk (maciejjozefczyk)]

We can ask Core OVN folks to cherry-pick that patch:
https://github.com/openvswitch/ovs/commit/5519e384f6a17f564fef4c5eb39e471e16c77235

To OVN, at least to branch20.03, and ask for its release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/735799

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/736863

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: master
Review: https://review.opendev.org/736863
Reason: .

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/736951

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/735799
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d0860588957730b97fe3ebbefa77c6421da45adc
Submitter: Zuul
Branch: master

commit d0860588957730b97fe3ebbefa77c6421da45adc
Author: Jakub Libosvar <email address hidden>
Date: Tue Jun 16 10:07:49 2020 +0200

    ovn: Use newer OVS

    OVS tag v2.13.0 is not compatible with current used kernel on upstream
    Bionic Ubuntu kernel 4.15.0. This patch sticks to commit hash to unblock
    the gate. We can either change to newer 2.13 tag once released or better
    stop compiling OVS and use one packaged by Ubuntu.

    Run functional/fullstack tests with selected OVN and OVS versions

    Previously for functional tests OVS version was hardcoded - v2.12.0,
    and OVN was installed from OVS repository.

    After we merged OVN driver to Neutron tree we run both
    Neutron/OVS and Neutron/OVN functional tests in one job.

    This patch adds possibility to specify from which tag/branch OVS and OVN
    should be checkout.

    Change-Id: I83688031951b97bfe64f3aaa761ad7afc1d5ea55
    Closes-Bug: #1883601
    Closes-Bug: #1878160

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lucas Alvares Gomes (<email address hidden>) on branch: stable/ussuri
Review: https://review.opendev.org/736951

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736286
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=080123851cc7b004e386fe213697b6a8f0ebebd5
Submitter: Zuul
Branch: stable/ussuri

commit 080123851cc7b004e386fe213697b6a8f0ebebd5
Author: Bernard Cafarelli <email address hidden>
Date: Wed Jun 17 17:54:22 2020 +0200

    Fix ussuri gates

    This is a combination of 3 commits.
    This is the 1st commit message:

    Temporarily disable grenade-multinode jobs

    We need to backport multiple gate fixes, disable the relevant jobs.
    Backports should re-enable the matching jobs

    This is the commit message #2:

    Add ensure-tox in functional job

    neutron functional jobs run tests using the tox env.
    Even tempest jobs need tox to be present to run the tempest
    tests in venv. But functional job derived from devstack-minimal
    jobs does not make sure tox is present or not. devstack jobs does not
    mak sure that as those are not only used for tox run but also for
    other integration testing.

    Current neutorn functional job failure-
    - https://zuul.opendev.org/t/openstack/build/59865004855c404ab18f06fc0ec1d005

    let's call ensure-tox role in job to make sure tox is present.

    Closes-Bug: 1884256
    (cherry picked from commit bda80e8e2a94ff3297ec750589cd879896f12c70)

    This is the commit message #3:

    ovn: Use newer OVS

    OVS tag v2.13.0 is not compatible with current used kernel on upstream
    Bionic Ubuntu kernel 4.15.0. This patch sticks to commit hash to unblock
    the gate. We can either change to newer 2.13 tag once released or better
    stop compiling OVS and use one packaged by Ubuntu.

    Run functional/fullstack tests with selected OVN and OVS versions

    Previously for functional tests OVS version was hardcoded - v2.12.0,
    and OVN was installed from OVS repository.

    After we merged OVN driver to Neutron tree we run both
    Neutron/OVS and Neutron/OVN functional tests in one job.

    This patch adds possibility to specify from which tag/branch OVS and OVN
    should be checkout.

    Closes-Bug: #1883601
    Closes-Bug: #1878160
    (cherry picked from commit d0860588957730b97fe3ebbefa77c6421da45adc)

    Change-Id: I8c59e209f3a4573c3f94da6c2acd7d6f57e30b31

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/751882

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751882
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e10c0dedf43673554eff2670c259e8234b0c65fd
Submitter: Zuul
Branch: master

commit e10c0dedf43673554eff2670c259e8234b0c65fd
Author: Brian Haley <email address hidden>
Date: Mon Sep 14 15:03:54 2020 -0400

    ovn: Use new OVS commit hash

    We have to pick OVS patches compatible with kernel 4.15.0-118
    after switching to it.

    Change-Id: I1ff47fc3398760d5b631bf89e6cfcc7ae339f0e3
    Related-bug: #1883601
    Closes-bug: #1896766

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/753963

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/753963
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=93459c1c8859c7782c490be1ee721c7324321e03
Submitter: Zuul
Branch: stable/ussuri

commit 93459c1c8859c7782c490be1ee721c7324321e03
Author: Brian Haley <email address hidden>
Date: Mon Sep 14 15:03:54 2020 -0400

    ovn: Use new OVS commit hash

    We have to pick OVS patches compatible with kernel 4.15.0-118
    after switching to it.

    Conflicts:
            zuul.d/base.yaml
            zuul.d/tempest-singlenode.yaml

    Change-Id: I1ff47fc3398760d5b631bf89e6cfcc7ae339f0e3
    Related-bug: #1883601
    Closes-bug: #1896766
    (cherry picked from commit e10c0dedf43673554eff2670c259e8234b0c65fd)