dkms: add module build ordering for ZFS on Linux

Bug #1491729 reported by Colin Ian King
32
This bug affects 3 people
Affects Status Importance Assigned to Milestone
dkms (Ubuntu)
Fix Released
High
Colin Ian King
Trusty
Fix Released
High
Colin Ian King
Wily
Fix Released
High
Unassigned

Bug Description

updated: SRU request, trusty (24th June, 2016)

(updated now from a FFe to a SRU for Wily)(old)
SRU request, Wily.

[IMPACT]
Without this, we can have incorrect ordering which causes DKMS build failures for multiple kernel module projects, such as ZFS (where ZFS requires the ZFS support module spl to be built first).

[TEST CASE]
Install Wily zfs, occasionally the build ordering will be zfs and the spl, and the zfs DKMS build will fail because spl is not yet built. With the fix, build ordering for zfs will always occur in spl, then zfs order because it uses the new BUILD_DEPENDS directive.
Note: The upload will trigger the testing of all DKMS modules, so it'll get an adequate beating by britney (according to Martin Pitt).

[REGRESSION POTENTIAL]
DKMS package builds will fail.

-------------------
Feature Freeze Exception Request:

build ordering to build ZFS on Linux (ZoL) is required, the motivating case is for ordering such as modules such as zfs -> spl

Without this, we can have incorrect ordering which causes DKMS build failures for multiple kernel module projects, such as ZFS (where ZFS requires the ZFS support module spl to be built first).

Pull in fixes from the ZFS on Linux project that enable this ordering to DKMS.

Changed in dkms:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Colin Ian King (colin-king)
Revision history for this message
Colin Ian King (colin-king) wrote :
Revision history for this message
Colin Ian King (colin-king) wrote :
Revision history for this message
Colin Ian King (colin-king) wrote :
description: updated
summary: - dkms: add module build ordering for ZFS on Linux
+ [FFE]: dkms: add module build ordering for ZFS on Linux
Revision history for this message
Martin Pitt (pitti) wrote : Re: [FFE]: dkms: add module build ordering for ZFS on Linux

https://launchpadlibrarian.net/216237731/dkms_2.2.0.3-2ubuntu3_2.2.0.3-2ubuntu4.diff.gz looks okay to me. A new dkms will now automatically trigger tests of all our DKMS packages, so britney will complain at you if this introduces any regressions. So +1 to the FFE.

affects: dkms → dkms (Ubuntu)
Revision history for this message
Colin Ian King (colin-king) wrote :

Additional change log info:

* Add POST_BUILD to the dkms_conf_variables list
   The POST_BUILD directive is ignored by the read_conf() function
   because it not in dkms_conf_variables list
* Add BUILD_DEPENDS configuration option
   Implement a BUILD_DEPENDS directive that instructs `dkms autoinstall`
   to build Linux modules in a specific order.

Revision history for this message
Colin Ian King (colin-king) wrote :

Forget the above, the following change log items are probably better:

* Add-POST_BUILD-to-the-dkms_conf_variables-list.patch: Add POST_BUILD directive, and implement
  a BUILD_DEPENDS directive that instructs `dkms autoinstall` to build Linux modules in a specific order to
  ensure ZFS to build after SPL.
* Add-support-for-forcing-modules-installation.patch: Force the installation of a module if it is listed in
   /usr/share/dkms/modules_to_force_install

Revision history for this message
Martin Pitt (pitti) wrote :

Asked on IRC, but didn't get a response:

 - What are the changes to Add-support-for-forcing-modules-installation.patch? That patch already existed, and the debdiff here completely changes it. A lot of it just seems to be reformatting, but the diff-of-diff makes it really hard to see what changed and whether that was intended.

 - This also changes the already existing apport_python3.patch. There's also an awful amount of noise, but one effective change:

+- for directory in "$dkms_tree/$1/$2/"${3:-+([0-9]).*}/${4:-*}; do
++ for directory in "$dkms_tree/$1/$2/"${3:-[0-9].*}/${4:-*}; do

  which is adjusting the py3 port for the new 0003-Add_BUILD_DEPENDS_configuration_option.patch. Thus the new patches in this debdiff need to be applied *before* at least apport_python3.patch, otherwise the patch dependencies are the wrong way around.

Have these been submitted upstream anywhere, as they are rather intrusive? Thanks!

Revision history for this message
Colin Ian King (colin-king) wrote :

Martin, I've re-worked the patches, new dkms in https://launchpad.net/~colin-king/+archive/ubuntu/zfs-dkms, the deb diff should be easier to grok now.

I've not submitted these upstream yet.

Revision history for this message
Martin Pitt (pitti) wrote :

Thanks for cleaning up! Uploaded. This will trigger the testing of all DKMS modules, so it'll get an adequate beating by britney.

summary: - [FFE]: dkms: add module build ordering for ZFS on Linux
+ dkms: add module build ordering for ZFS on Linux
Changed in dkms (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 2.2.0.3-2ubuntu7

---------------
dkms (2.2.0.3-2ubuntu7) xenial; urgency=medium

  * Build Linux modules in a specific orfer (LP: #1491729)
    - Add POST_BUILD directive, and implement a BUILD_DEPENDS directive that
      instructs `dkms autoinstall` to build Linux modules in a specific order to
      ensure ZFS to build after SPL.
    - Force the installation of a module if it is listed in
      /usr/share/dkms/modules_to_force_install

 -- Colin Ian King <email address hidden> Tue, 10 Nov 2015 11:02:00 +0000

Changed in dkms (Ubuntu):
status: Fix Committed → Fix Released
description: updated
Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

Please SRU this fix for Wily. @danwest (https://launchpad.net/~danwest) was running into this very issue trying to install the ZFS dkms package in Wily, ie.

ProblemType: Package
DKMSBuildLog:
 DKMS make.log for zfs-0.6.4.2 for kernel 4.2.0-18-generic (x86_64)
 Tue Nov 10 15:33:06 UTC 2015
 make: *** No targets specified and no makefile found. Stop.
DKMSKernelVersion: 4.2.0-18-generic
Date: Tue Nov 10 15:33:10 2015
Package: zfs-dkms 0.6.4.2-0ubuntu1
PackageVersion: 0.6.4.2-0ubuntu1
SourcePackage: zfs-linux
Title: zfs-dkms 0.6.4.2-0ubuntu1: zfs kernel module failed to build

However, after installing the fixed dkms-2.2.0.3-2ubuntu7 from cking's PPA, he is unable to reproduce.

Revision history for this message
Colin Ian King (colin-king) wrote :
Revision history for this message
Chris J Arges (arges) wrote :

uploaded for wily, with version ubuntu6.1 and minor spelling fix

Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Colin, or anyone else affected,

Accepted dkms into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.2.0.3-2ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in dkms (Ubuntu Wily):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Colin Ian King (colin-king) wrote :

Tested dkms-2.2.0.3-2ubuntu6.1 from -proposed: iterated on clean install, install dkms, install zfs with no build ordering issues. Repeat tested this 14 times, no problems now observed.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote :

http://people.canonical.com/~ubuntu-archive/proposed-migration/wily/update_excuses.html#dkms is looking good. openafs/amd64 is a cloud glitch and I'll restart it. There are still a lot of outstanding ones, and we should let these finish to be sure. (Queues are super-long at the moment though)

Revision history for this message
Martin Pitt (pitti) wrote :

Good enough now, so double verification-done. :-)

Revision history for this message
Colin Ian King (colin-king) wrote :

Thanks Martin!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 2.2.0.3-2ubuntu6.1

---------------
dkms (2.2.0.3-2ubuntu6.1) wily; urgency=medium

  * Build Linux modules in a specific order (LP: #1491729)
    - Add POST_BUILD directive, and implement a BUILD_DEPENDS directive that
      instructs `dkms autoinstall` to build Linux modules in a specific order to
      ensure ZFS to build after SPL.
    - Force the installation of a module if it is listed in
      /usr/share/dkms/modules_to_force_install

 -- Colin Ian King <email address hidden> Tue, 10 Nov 2015 11:02:00 +0000

Changed in dkms (Ubuntu Wily):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for dkms has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Nicolas Dichtel (nicolas-dichtel) wrote :

Will this patch be backported in Trusty version also?

Revision history for this message
Colin Ian King (colin-king) wrote :

The intention was just to for this fix to just support ZFS in Wily only.

tags: added: 6wind
Revision history for this message
Vincent JARDIN (vincent-jardin) wrote :

We have seen this issue with Trusty for other modules than ZFS too.

Revision history for this message
Colin Ian King (colin-king) wrote :

Sounds like enough reason to justify it for an SRU for Trusty. Any specific module examples that need this?

Revision history for this message
Vincent JARDIN (vincent-jardin) wrote :

Sorry, I do not have a handy module to be compiled for this case, but once you integrate this fix/patch, we can check it works using our framework.

Revision history for this message
Colin Ian King (colin-king) wrote :

Hi Vincent,

I've applied the fixes to dkms for trusty and I've got the package built in my ppa for testing:

sudo add-apt-repository ppa:colin-king/dkms-sru-1491729
sudo apt-get update && sudo apt-get dist-upgrade

Do you mind checking to see if this solves the issue for you?

Revision history for this message
Colin Ian King (colin-king) wrote :
description: updated
description: updated
Revision history for this message
Julien Floret (julien-floret) wrote :

Hi Colin,

Tested, your package solves the issue for us.
Thanks!

Revision history for this message
Vincent JARDIN (vincent-jardin) wrote :

Thanks Julien and Colin. Looking to get it into the next updates for 14.04.

Revision history for this message
Colin Ian King (colin-king) wrote :

Ah, looks like there was a 14.04.6 in -proposed. I've re-sync'd with that and re-spun the package for a .7 version, so ignore the debdiff in comment #27

Updated debdiff attached below:

Revision history for this message
Julien Floret (julien-floret) wrote :

The .7 version works for us, thanks.

Tim Gardner (timg-tpi)
Changed in dkms (Ubuntu Trusty):
assignee: nobody → Colin Ian King (colin-king)
status: New → In Progress
Revision history for this message
Andy Whitcroft (apw) wrote : Please test proposed package

Hello Colin, or anyone else affected,

Accepted dkms into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.2.0.3-1.1ubuntu5.14.04.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in dkms (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Colin Ian King (colin-king) wrote :

@Julien, do you mind re-testing against the -proposed version? Thanks!

Revision history for this message
Nicolas Dichtel (nicolas-dichtel) wrote :

> @Julien, do you mind re-testing against the -proposed version? Thanks!

He will be back on Monday the 1st August.

Revision history for this message
Julien Floret (julien-floret) wrote :

Colin,

Still took the time to test the -proposed version.
It fixes the problem for us, thanks!

Revision history for this message
Colin Ian King (colin-king) wrote :

Thanks Julien!

tags: added: verification-done
removed: verification-needed
Mathew Hodson (mhodson)
Changed in dkms (Ubuntu Trusty):
importance: Undecided → High
Changed in dkms (Ubuntu Wily):
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 2.2.0.3-1.1ubuntu5.14.04.7

---------------
dkms (2.2.0.3-1.1ubuntu5.14.04.7) trusty; urgency=medium

  * Build Linux modules in a specific order (LP: #1491729)
    - Add POST_BUILD directive, and implement a BUILD_DEPENDS directive that
      instructs `dkms autoinstall` to build Linux modules in a specific order to
      ensure ZFS to build after SPL.
    - Force the installation of a module if it is listed in
      /usr/share/dkms/modules_to_force_install

 -- Colin Ian King <email address hidden> Fri, 24 Jun 2016 16:43:12 +0100

Changed in dkms (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.