dkms key enrolled in mok, but dkms module fails to load

[Dan Watkins]

[Impact]
All Ubuntu users for whom Secure Boot is enabled.

[Test cases]
1) install dkms module (use virtualbox-dkms for example)
2) Upgrade kernel (for example, install 4.15.0-22-generic on top of 4.15.0-20-generic).
3) Verify that the generated module for the new kernel (4.15.0-22-generic in this example) is built and signed by verifying that the file in /lib/modules/$kernel/updates/dkms/$module.ko ends in ~Module signature appended~:

$ hexdump -Cv /lib/modules/4.15.0-22-generic/updates/dkms/vboxdrv.ko | tail -n 100
[...]
~Module signature appended~

4) Reboot
5) modprobe -v the module.
It should not respond "Required key not available", and should return with no error.
6) Verify that dkms does not contain PKCS#7 errors.

[Regression potential]
Possible regressions involve failure to sign and/or be able to load modules after updates: failure to sign leading to a module being built but unsigned after a new kernel is installed or after a new DKMS module is installed, failure to load modules after reboot (usually caused by module being unsigned); failure to sign due to missing keys, signature key not being automatically slated for enrollment. All these potential regression scenarios present as failure to load a DKMS module after a reboot when it should be loaded successfully.

---

At my last reboot, I was prompted to enable SecureBoot, so I did.

When I booted, however, I noticed that the virtualbox service failed to start because it couldn't load its kernel module. If I attempt the same thing, I see that there's an issue with keys:

$ sudo modprobe vboxdrv
modprobe: ERROR: could not insert 'vboxdrv': Required key not available

I do have keys enrolled; `mokutil --list-enrolled` produces http://paste.ubuntu.com/p/rntTQr5XJV/

[Dan Watkins (oddbloke)]

term.log for installation of my current kernel: https://paste.ubuntu.com/p/3TVVFpFSNX/

term.log from the last time I see virtualbox DKMS stuff happening: https://paste.ubuntu.com/p/7f7p6t48pn/

[Steve Langasek (vorlon)]

The logs show the new kernel being installed, but show no dkms module building at time of kernel install. That seems strange to me. We should figure out what generated /lib/modules/4.15.0-22-generic/updates/dkms/vboxdrv.ko and when and why it's not correctly signed.

[Steve Langasek (vorlon)]

Based on timestamp info provided out of band, /lib/modules/4.15.0-22-generic/updates/dkms/vboxdrv.ko was generated as part of the kernel install via /etc/kernel/postinst.d/dkms, despite the lack of verbosity.

[Steve Langasek (vorlon)]

The dkms package's shim integration only happens in /usr/lib/dkms/common.postinst. It appears this code is only triggered on installation of a dkms package; this code path is not used as part of the kernel postinst hook when building modules for a newly-installed kernel - that hook only calls /usr/lib/dkms/dkms_autoinstaller .

Marking this critical, since this means users will lose their dkms modules on kernel upgrade.

[Dan Watkins (oddbloke)]

I can confirm that the new module isn't signed at all:

$ hexdump -Cv /lib/modules/4.15.0-22-generic/updates/dkms/vboxdrv.ko | tail -n 100 | pastebinit
http://paste.ubuntu.com/p/BFSg9DsqR8/

Contrast with a previous kernel that was installed when virtualbox was last upgraded:

$ hexdump -Cv /lib/modules/4.15.0-15-generic/updates/dkms/vboxdrv.ko | tail -n 100 | pastebinit
http://paste.ubuntu.com/p/W8WyVTd2zc/

[Launchpad Janitor (janitor)]

This bug was fixed in the package dkms - 2.3-3ubuntu10

---------------
dkms (2.3-3ubuntu10) cosmic; urgency=medium

  * 0009-Add-support-for-UEFI-Secure-Boot-validation-toggling.patch: move sign
    code to dkms script itself, so it also applies on kernel upgrades.
    (LP: #1772950)

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 23 May 2018 13:15:53 -0400

[Steve Langasek (vorlon)]

Hello Dan, or anyone else affected,

Accepted dkms into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.3-3ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mathieu Trudel-Lapierre (cyphermox)]

Verification-done on bionic:

ii dkms 2.3-3ubuntu9.1 all Dynamic Kernel Module Support Framework
ii virtualbox-dkms 5.2.10-dfsg-6 all x86 virtualization solution - kernel mod

I have verified that with the old dkms, kernel upgrades lead to an unsigned vboxdrv module; and with the new dkms, kernel upgrades do have signed modules that load correctly with SecureBoot enabled.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dkms - 2.3-3ubuntu9.1

---------------
dkms (2.3-3ubuntu9.1) bionic; urgency=medium

  * 0009-Add-support-for-UEFI-Secure-Boot-validation-toggling.patch: move sign
    code to dkms script itself, so it also applies on kernel upgrades.
    (LP: #1772950)

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 23 May 2018 13:15:53 -0400

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for dkms has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Brian Murray (brian-murray)]

Hello Dan, or anyone else affected,

Accepted dkms into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.2.0.3-2ubuntu11.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Dan, or anyone else affected,

Accepted dkms into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.2.0.3-1.1ubuntu5.14.04.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mathieu Trudel-Lapierre (cyphermox)]

Verification-done on trusty:

dkms/2.2.0.3-1.1ubuntu5.14.04.10

I've installed bbswitch on a test UEFI system, upgraded the kernel to a newer version (ie. linux-image-hwe-trusty-generic) and was still able to load the module in; the module in the updates/dkms directory for the kernel version is clearly a signed copy.

ubuntu@ubuntu:~$ dpkg -l dkms shim-signed | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=============================================-============-==============================================================
ii dkms 2.2.0.3-2ubuntu11.6 all Dynamic Kernel Module Support Framework
ii shim-signed 1.33.1~16.04.4+15+1533136590.3beb971-0ubuntu1 amd64 Secure Boot chain-loading bootloader (Microsoft-signed binary)

[ 173.890220] usbcore: registered new interface driver asic0x
[ 356.605416] bbswitch: version 0.7
[ 356.605431] bbswitch: Found integrated VGA device 0000:00:02.0: \_SB_.PCI0.VID_
[ 356.605443] bbswitch: No discrete VGA device found

[Mathieu Trudel-Lapierre (cyphermox)]

Verification-done on xenial:

dkms 2.2.0.3-2ubuntu11.6

Upgraded kernel to hwe kernel, drivers can still be loaded from the right versioned directory for the kernel and loads succesfully -- signature is validated fined as the kernel module is signed.

ubuntu@ubuntu:~$ dpkg -l shim-signed dkms | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=============================================-============-==============================================================
ii dkms 2.2.0.3-2ubuntu11.6 all Dynamic Kernel Module Support Framework
ii shim-signed 1.33.1~16.04.4+15+1533136590.3beb971-0ubuntu1 amd64 Secure Boot chain-loading bootloader (Microsoft-signed binary)
ubuntu@ubuntu:~$ sudo modprobe bbswitch
[sudo] password for ubuntu:
modprobe: ERROR: could not insert 'bbswitch': No such device
ubuntu@ubuntu:~$ dmesg | tail
[ 7.551980] wlp3s0: waiting for beacon from fc:ec:da:3c:dd:85
[ 7.654548] wlp3s0: associate with fc:ec:da:3c:dd:85 (try 1/3)
[ 7.656500] wlp3s0: RX AssocResp from fc:ec:da:3c:dd:85 (capab=0x411 status=0 aid=3)
[ 7.676864] wlp3s0: associated
[ 7.676917] IPv6: ADDRCONF(NETDEV_CHANGE): wlp3s0: link becomes ready
[ 17.687856] random: nonblocking pool is initialized
[ 122.752094] bbswitch: loading out-of-tree module taints kernel.
[ 122.752723] bbswitch: version 0.8
[ 122.752745] bbswitch: Found integrated VGA device 0000:00:02.0: \_SB_.PCI0.VID_
[ 122.752767] bbswitch: No discrete VGA device found

ubuntu@ubuntu:~$ cat /proc/version_signature
Ubuntu 4.4.0-143.169-generic 4.4.170
ubuntu@ubuntu:~$ sudo insmod /lib/modules/4.4.0-143-generic/updates/dkms/bbswitch.ko
insmod: ERROR: could not insert module /lib/modules/4.4.0-143-generic/updates/dkms/bbswitch.ko: No such device
ubuntu@ubuntu:~$ dmesg |tail
[ 7.676864] wlp3s0: associated
[ 7.676917] IPv6: ADDRCONF(NETDEV_CHANGE): wlp3s0: link becomes ready
[ 17.687856] random: nonblocking pool is initialized
[ 122.752094] bbswitch: loading out-of-tree module taints kernel.
[ 122.752723] bbswitch: version 0.8
[ 122.752745] bbswitch: Found integrated VGA device 0000:00:02.0: \_SB_.PCI0.VID_
[ 122.752767] bbswitch: No discrete VGA device found
[ 221.958525] bbswitch: version 0.8
[ 221.958540] bbswitch: Found integrated VGA device 0000:00:02.0: \_SB_.PCI0.VID_
[ 221.958554] bbswitch: No discrete VGA device found
ubuntu@ubuntu:~$ sudo hexdump -Cv /lib/modules/4.4.0-143-generic/updates/dkms/bbswitch.ko | tail
00005740 40 ac 93 85 cb 5f 1e 3e 6b 7b db 62 86 66 ea 81 |@...._.>k{.b.f..|
00005750 1e 9a 9a 1e a6 05 dc e1 18 dd 27 40 27 42 31 9f |..........'@'B1.|
00005760 fd 54 ac 4a f6 26 21 32 f3 b4 52 70 f4 79 a6 0d |.T.J.&!2..Rp.y..|
00005770 c9 75 93 46 a5 2b ed fe ef a1 68 97 c0 e0 67 c7 |.u.F.+....h...g.|
00005780 32 f7 4c c9 6d 0a 00 29 ce 87 a0 0a 95 be f1 4b |2.L.m..).......K|
00005790 c3 2e 6b df 7f a5 b7 67 55 27 cb bf a8 ea 51 7b |..k.....

[Mathieu Trudel-Lapierre (cyphermox)]

Re-verified trusty since the previous trusty comment was imprecise:

dkms 2.2.0.3-1.1ubuntu5.14.04.10

Upgrading kernel and headers follows with a loadable, properly signed module using the MOK generated previously.

ubuntu@ubuntu:~$ dpkg -l shim-signed dkms | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================-====================================================-============-===============================================================================
ii dkms 2.2.0.3-1.1ubuntu5.14.04.10 all Dynamic Kernel Module Support Framework
ii shim-signed 1.33.1~14.04.4+13-0ubuntu2 amd64 Secure Boot chain-loading bootloader (Microsoft-signed binary)

[...]

Unpacking linux-headers-4.4.0-142-generic (4.4.0-142.168~14.04.1) ...
Setting up linux-headers-4.4.0-142 (4.4.0-142.168~14.04.1) ...
Setting up linux-headers-4.4.0-142-generic (4.4.0-142.168~14.04.1) ...
Examining /etc/kernel/header_postinst.d.
run-parts: executing /etc/kernel/header_postinst.d/dkms 4.4.0-142-generic /boot/vmlinuz-4.4.0-142-generic
Nothing to do.
Nothing to do.
ubuntu@ubuntu:/lib/modules/4.4.0-142-generic$ cat /proc/version_signature
Ubuntu 4.4.0-142.168~14.04.1-generic 4.4.167
ubuntu@ubuntu:/lib/modules/4.4.0-142-generic$ sudo modprobe bbswitch
modprobe: ERROR: could not insert 'bbswitch': No such device
ubuntu@ubuntu:/lib/modules/4.4.0-142-generic$ dmesg | tail
[ 15.036233] audit: type=1400 audit(1550095748.630:15): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=1004 comm="apparmor_parser"
[ 15.036504] audit: type=1400 audit(1550095748.630:16): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=1004 comm="apparmor_parser"
[ 15.118903] audit: type=1400 audit(1550095748.714:17): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=1006 comm="apparmor_parser"
[ 15.273612] init: plymouth-upstart-bridge main process ended, respawning
[ 16.272167] random: nonblocking pool is initialized
[ 219.644638] bbswitch: loading out-of-tree module taints kernel.
[ 219.644704] bbswitch: module verification failed: signature and/or required key missing - tainting kernel
[ 219.645133] bbswitch: version 0.7
[ 219.645146] bbswitch: Found integrated VGA device 0000:00:02.0: \_SB_.PCI0.VID_
[ 219.645159] bbswitch: No discrete VGA device found

[Launchpad Janitor (janitor)]

This bug was fixed in the package dkms - 2.2.0.3-1.1ubuntu5.14.04.10

---------------
dkms (2.2.0.3-1.1ubuntu5.14.04.10) trusty; urgency=medium

  * debian/patches/shim_secureboot_support.patch:
    - Move to signing just after module build to ensure it correctly applies
      at kernel update times. (LP: #1772950)
    - Generate a new MOK if there isn't one yet, and use that so sign
      newly-built kernel modules. (LP: #1748983)
  * debian/control: Breaks: shim-signed (<< 1.33.1~14.04.4) to ensure both
    are updated in lock-step since the changes above require a new version of
    update-secureboot-policy to correctly generate the new MOK and enroll it
    in firmware.

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 28 Jan 2019 11:05:49 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package dkms - 2.2.0.3-2ubuntu11.6

---------------
dkms (2.2.0.3-2ubuntu11.6) xenial; urgency=medium

  * debian/patches/shim_secureboot_support.patch:
    - Move to signing just after module build to ensure it correctly applies
      at kernel update times. (LP: #1772950)
    - Generate a new MOK if there isn't one yet, and use that so sign
      newly-built kernel modules. (LP: #1748983)

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 28 Jan 2019 10:21:09 -0500