dpdk 18.11-5 ADT test failure with linux 5.0.0-3.4

[Seth Forshee]

[Impact]

 * DKMS fails to build the module with Linux 5.0 kernels

 * Add upstream fix (will be in next stable release anyway) to be ready
   for 5.0 based HWE kernel for Bionic

[Test Case]

 * Install a linux 5.0 kernel
 * Install dpdk-rte-kni-dkms and check if it DKMS-built fine
   Without the fix it will fail as outlined below, with the fix it will
   work.

[Regression Potential]

 * The change is minimal and should not have a regression potential.
   If anything then build on some awkward kernel versions (e.g. 5.0 with
   related changes reverted) would fail on dkms rebuild.

[Other Info]

 * n/a

Testing failed on:
    amd64: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco-canonical-kernel-team-unstable/disco/amd64/d/dpdk/20190206_134022_c55a8@/log.gz
    arm64: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco-canonical-kernel-team-unstable/disco/arm64/d/dpdk/20190206_145936_c55a8@/log.gz
    i386: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco-canonical-kernel-team-unstable/disco/i386/d/dpdk/20190206_134552_c55a8@/log.gz
    ppc64el: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco-canonical-kernel-team-unstable/disco/ppc64el/d/dpdk/20190206_134652_c55a8@/log.gz

[Christian Ehrhardt  (paelzer)]

For reference last good build on 4.19.0-12.13 kernel

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco/disco/amd64/d/dpdk/20190205_000700_24c9f@/log.gz

We have two dkms modules
1. dpdk-igb-uio (still working)
2. dpdk-rte-kni-dkms (the failing one)

From the log:
ERROR (dkms apport): kernel package linux-headers-5.0.0-3-generic is not supported
Error! Bad return status for module build on kernel: 5.0.0-3-generic (x86_64)
Consult /var/lib/dkms/dpdk-rte-kni/18.11/build/make.log for more information.

I ran the same in a local KVM autopkgtest with disco against ppa:canonical-kernel-team/unstable.
From there I have the detailed failure of the dkms build:
DKMS make.log for dpdk-rte-kni-18.11 for kernel 5.0.0-3-generic (x86_64)
Thu Feb 7 10:57:36 CET 2019
make: Entering directory '/usr/src/linux-headers-5.0.0-3-generic'
  CC [M] /var/lib/dkms/dpdk-rte-kni/18.11/build/kni_net.o
  CC [M] /var/lib/dkms/dpdk-rte-kni/18.11/build/kni_misc.o
  CC [M] /var/lib/dkms/dpdk-rte-kni/18.11/build/kni_ethtool.o
  CC [M] /var/lib/dkms/dpdk-rte-kni/18.11/build/ethtool/ixgbe/ixgbe_ethtool.o
  CC [M] /var/lib/dkms/dpdk-rte-kni/18.11/build/ethtool/ixgbe/ixgbe_82598.o
/usr/src/dpdk-rte-kni-18.11/ethtool/ixgbe/ixgbe_ethtool.c: In function ‘ixgbe_diag_test’:
/usr/src/dpdk-rte-kni-18.11/ethtool/ixgbe/ixgbe_ethtool.c:1746:4: error: too few arguments to function ‘dev_open’
    dev_open(netdev);
    ^~~~~~~~
In file included from /usr/src/dpdk-rte-kni-18.11/ethtool/ixgbe/ixgbe_ethtool.c:18:
./include/linux/netdevice.h:2620:5: note: declared here
 int dev_open(struct net_device *dev, struct netlink_ext_ack *extack);
     ^~~~~~~~
make[1]: *** [scripts/Makefile.build:286: /var/lib/dkms/dpdk-rte-kni/18.11/build/ethtool/ixgbe/ixgbe_ethtool.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1582: _module_/var/lib/dkms/dpdk-rte-kni/18.11/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.0.0-3-generic'

I wanted to take a look at 18.11.1 anyway - it is not out yet - but maybe there is a fix in it.

[Christian Ehrhardt  (paelzer)]

Even latest DPDK master nor the WIP 18.11.1 branch has a fix for that.
The still have:
1741 skip_loopback:
1742 »···»···ixgbe_reset(adapter);
1743
1744 »···»···clear_bit(__IXGBE_TESTING, &adapter->state);
1745 »···»···if (if_running)
1746 »···»···»···dev_open(netdev);

The include is from linux-libc-dev:amd64 which is built by the kernel.

The upstream kernel change which caused this is:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=00f54e68924eaf075f3f24be18557899d347bc4a

Following the message there most likely we'd just set NULL - but since that breaks backports we at least need a kernel-version separator to have old&new code.

[Christian Ehrhardt  (paelzer)]

On the dpdk mailing list a discussion for that exists:
http://mails.dpdk.org/archives/dev/2019-January/123909.html

This ended in:
https://git.dpdk.org/dpdk/commit/?id=d01c18ff821b81baa3b99329ebf051c6a1bfd4ee
https://git.dpdk.org/dpdk/commit/?id=18cb8293e312bc3bd67211a5a540676f1c667d88

I think we can pick those two, but since 18.11.1 is really soon I'd ping the maintainer if he'd consider adding those. Then we could use that instead of adding delta.

[Christian Ehrhardt  (paelzer)]

Pinged dpdk stable here: http://mails.dpdk.org/archives/stable/2019-February/012722.html
I also pinged them for 17.11.5 which would appear soon'ish as well.
That would get into Ubuntu 18.04 / 18.10 which would solve your issues for the 5.0 based HWE kernel right?

Overall what is left is the timing question.
@Seth - are you ok waiting ~2 weeks until upstream released the new stable branches so that I then can pick and push those to B/C/D releases?
Or are you hard-blocked with Linux 5.0 into Disco on that?
Then we could add an interim Delta to resolve the issue for the time being.

[Seth Forshee (sforshee)]

A couple of weeks should be okay. That's 5.0-rc7 time frame and it's unlikely that we'll be trying to get 5.0 into disco-release before then.

[Christian Ehrhardt  (paelzer)]

FYi pushed changes to deb_dpdk to end up in Debian soon and be synced for us

[Christian Ehrhardt  (paelzer)]

Arr ignore the timelines, it was so easy to do and there are some other constraints that Luca and I fixed it in Debian. It is already synced and currently in Disoc-proposed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpdk - 18.11-6

---------------
dpdk (18.11-6) unstable; urgency=medium

  [ Luca Boccassi ]
  * Add NEWS entry to list changes to dpdk-dev. (Closes: #921043)

  [ Christian Ehrhardt ]
  * d/control: ensure break relationships are backportable (Closes: #921042)
  * d/p/*kni-fix-build*: fix build with kernel 5.0 (LP: #1814919)

 -- Luca Boccassi <email address hidden> Thu, 07 Feb 2019 16:34:57 +0000

[Christian Ehrhardt  (paelzer)]

I'll evaluate if we want/need to make this part of the 17.11.x MRE updates that are ongoing.
The result is either "we don't need it" (status will become invalid) or "we need it" (status will become triaged)

And while yet untestable for now this will be the best prep we can do for 5.0 based HWE kernel appearing in Bionic.

[Christian Ehrhardt  (paelzer)]

FYI - all prechecks complete - uploaded to -unapproved

[Brian Murray (brian-murray)]

Hello Seth, or anyone else affected,

Accepted dpdk into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpdk/17.11.5-0~ubuntu18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Seth, or anyone else affected,

Accepted dpdk into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpdk/17.11.5-0~ubuntu18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Christian Ehrhardt  (paelzer)]

Using 5.0 kernel from Mainline at [1] in containers of Bionic and Cosmic I re-installed the dkms module for rte-kni.

# apt install --reinstall dpdk-rte-kni-dkms
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 302 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 dpdk-rte-kni-dkms amd64 17.11.5-0~ubuntu18.04.1 [302 kB]
Fetched 302 kB in 0s (983 kB/s)
(Reading database ... 152174 files and directories currently installed.)
Preparing to unpack .../dpdk-rte-kni-dkms_17.11.5-0~ubuntu18.04.1_amd64.deb ...

-------- Uninstall Beginning --------
Module: dpdk-rte-kni
Version: 17.11.5
Kernel: 4.15.0-47-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

rte_kni.ko:
 - Uninstallation
   - Deleting from: /lib/modules/4.15.0-47-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.

-------- Uninstall Beginning --------
Module: dpdk-rte-kni
Version: 17.11.5
Kernel: 5.0.0-050000-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

rte_kni.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.0.0-050000-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.

------------------------------
Deleting module version: 17.11.5
completely from the DKMS tree.
------------------------------
Done.
Unpacking dpdk-rte-kni-dkms (17.11.5-0~ubuntu18.04.1) over (17.11.5-0~ubuntu18.04.1) ...
Setting up dpdk-rte-kni-dkms (17.11.5-0~ubuntu18.04.1) ...
Loading new dpdk-rte-kni-17.11.5 DKMS files...
It is likely that 4.18.0-15-generic belongs to a chroot's host
Building for 5.0.0-050000-generic
Building initial module for 5.0.0-050000-generic
Done.

rte_kni:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.0.0-050000-generic/updates/dkms/

depmod...

DKMS: install completed.

Thereby kernel 5.0.0-050000.201903032031 vs DPDK 17.11.5-0~ubuntu18.04.1/17.11.5-0~ubuntu18.10.1 working. No build error with kernel 5.0 anymore - setting verified

[1]: https://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D

[Brian Murray (brian-murray)]

Thank you for taking the time to verify this stable release fix. We have noticed that you have used the verification-done tag for marking the bug as verified and would like to point out that due to a recent change in SRU bug verification policy fixes now have to be marked with per-release tags (i.e. verification-done-$RELEASE). Please remove the verification-done tag and add one for the release you have tested the package in. Thank you!

https://wiki.ubuntu.com/StableReleaseUpdates#Verification

[Christian Ehrhardt  (paelzer)]

I guess this was a bot right?
I just forgot to also switch the second tag - Thanks Bot or Brian (whoever it was) for reminding me :-)

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpdk - 17.11.5-0~ubuntu18.10.1

---------------
dpdk (17.11.5-0~ubuntu18.10.1) cosmic; urgency=medium

  * New upstream release 17.11.5; for a full list of changes see:
    https://doc.dpdk.org/guides-17.11/rel_notes/release_17_11.html#id4
    https://doc.dpdk.org/guides-17.11/rel_notes/release_17_11.html#id5
    Among many other fixes this closes the following bugs:
    - request to merge 17.11.5 (LP: #1817675)
    - issues with -mavx512f on recent Skylake chips (LP: #1799397)
    - Drop d/p/net-mlx5-fix-build-with-rdma-core-v19.patch which is part of
      17.11.4
  * d/p/*kni-fix-build*: fix build with kernel 5.0 (LP: #1814919)
    as preparation for a HWE kernel based on the 5.0 version of 19.04

 -- Christian Ehrhardt <email address hidden> Tue, 26 Feb 2019 12:34:12 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for dpdk has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpdk - 17.11.5-0~ubuntu18.04.1

---------------
dpdk (17.11.5-0~ubuntu18.04.1) bionic; urgency=medium

  * New upstream release 17.11.5; for a full list of changes see:
    https://doc.dpdk.org/guides-17.11/rel_notes/release_17_11.html#id4
    https://doc.dpdk.org/guides-17.11/rel_notes/release_17_11.html#id5
    Among many other fixes this closes the following bugs:
    - request to merge 17.11.5 (LP: #1817675)
    - issues with -mavx512f on recent Skylake chips (LP: #1799397)
    - Drop d/p/net-mlx5-fix-build-with-rdma-core-v19.patch which is part of
      17.11.4
  * d/p/*kni-fix-build*: fix build with kernel 5.0 (LP: #1814919)
    as preparation for a HWE kernel based on the 5.0 version of 19.04

 -- Christian Ehrhardt <email address hidden> Tue, 26 Feb 2019 12:34:12 +0100