grub2 signed kernel enforcement doesn't check on upgrade that signatures are from trusted keys
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | grub2 (Ubuntu) |
High
|
Mathieu Trudel-Lapierre | ||
| | Trusty |
Undecided
|
Unassigned | ||
| | Bionic |
Undecided
|
Unassigned | ||
| | Cosmic |
Undecided
|
Unassigned | ||
| | mokutil (Ubuntu) |
High
|
Mathieu Trudel-Lapierre | ||
Bug Description
[Impact]
This affects UEFI users upgrading grub, especially when upgrading from an earlier release or when using custom kernels (signed by PPA keys, or unsigned).
[Test case]
1) Install a custom / PPA kernel, or copy an existing kernel into an unsigned version of it:
sudo cp /boot/vmlinuz-
2) Make sure the kernel is unsigned or signed with an unknown key (in this case, remove signature for convenience):
sudo sbattach --remove /boot/vmlinuz-
3) Upgrade grub2.
4) Validate that the upgrade fails, and complains about incorrectly signed kernels for the new "vmlinuz-
5) Run /usr/share/
[Regression potential]
Relatively low risk of failure. This only affects the upgrade process maintainer scripts, which run an additional script to fail the upgrade if it is detected that a newer kernel than the one currently running would be invalid for Secure Boot. This already catches invalid signatures and unsigned kernels, and mitigates against broken firmwares by shipping the certificate for the most common kernel signatures (the Canonical cert). Upgrades should already have been migrating users to forcing the signed version of kernels to be installed when the official pacakges are being used.
Watch for upgrade failures due to false positives (correcly signed kernels that are detected as signed by unknown keys) or other failure modes of the upgrade of the grub packages. Grub functionality at boot has been unchanged.
[Background information]
Secure Boot will soon enforce that kernels are properly signed to be able to boot them. Catch systems early where this upgrade would break boot for our users, to ensure they can correct the situation while the system is running.
---
This is on a cosmic system. I wanted to test the 4.18 kernel in the kernel teams unstable ppa. I enabled that ppa, then ran "sudo apt-get update; sudo apt-get dist-upgrade" and then rebooted. Upon boot grub started reporting that none of the kernels I have installed have valid signatures. These were working just fine before this update. The only remedy was to disable secure boot in my bios.
---
ProblemType: Bug
ApportVersion: 2.20.10-0ubuntu9
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.10
EcryptfsInUse: Yes
InstallationDate: Installed on 2017-08-14 (380 days ago)
InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Alpha amd64 (20170812)
Package: grub2 (not installed)
ProcEnviron:
TERM=tmux-256color
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSign
Tags: wayland-session cosmic
Uname: Linux 4.18.0-7-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip kvm libvirt lpadmin plugdev sambashare sudo
_MarkForUpload: True
| tags: | added: apport-collected cosmic wayland-session |
| description: | updated |
What does 'dpkg -S /boot/vmlinuz-*' show on your system?
| Brad Figg (brad-figg) wrote : | #3 |
$ dpkg -S /boot/vmlinuz-*
linux-image-
linux-image-
linux-image-
linux-image-
linux-image-
linux-image-
linux-image-
| Steve Langasek (vorlon) wrote : | #4 |
These are the correct packages to be providing the signed versions of these files from the Canonical archive, and https:/
Can you attach /boot/vmlinuz-
| Adam Conrad (adconrad) wrote : | #5 |
My guess is that Brad's been getting all his kernels from the ckt PPA, which means they'd all have snakeoil sigs on them instead of the archive sig. In this case, "linux-
Disabling the ckt PPA and doing an "apt-get --reinstall install <list of packages above>" will probably fix it.
In future, I imagine kernel team folks might want to add their PPA's EFI signing key to MOK on systems where they're likely to run PPA kernels.
| Adam Conrad (adconrad) wrote : | #6 |
Err, I'll note the reinstall trick won't work for the old kernels we've removed from the archive, but testing with 4.17-9, which is still published, should prove the theory.
On Thu, Aug 30, 2018 at 05:21:16PM -0000, Adam Conrad wrote:
> My guess is that Brad's been getting all his kernels from the ckt PPA,
> which means they'd all have snakeoil sigs on them instead of the archive
> sig. In this case, "linux-
> image-4.
> binaries are rebuilt when we copy to the archive.
> Disabling the ckt PPA and doing an "apt-get --reinstall install <list of
> packages above>" will probably fix it.
> In future, I imagine kernel team folks might want to add their PPA's EFI
> signing key to MOK on systems where they're likely to run PPA kernels.
Yes. Having the exact vmlinuz binary attached to this bug report will let
us confirm this.
I removed the pointer to the ckt/unstable ppa and purged all of the 4.18 kernels. I then installed the latest 4.17 kernel from -updates. That kernel, 4.17.0-9.10 now boots just fine with secure boot enabled. It looks like I did have more kernels installed from the ckt ppas than I thought I had.
| Brad Figg (brad-figg) wrote : | #9 |
| Steve Langasek (vorlon) wrote : | #10 |
$ sbattach --detach detached-sig ./vmlinuz-
$ openssl pkcs7 -in detached-sig -inform DER -print_certs
subject=/CN=PPA canonical-
issuer=/CN=PPA canonical-
-----BEGIN CERTIFICATE-----
MIIDLzCCAhegAwI
BAMMI1BQQSBjYW5
MDE3MjI0NVoXDTI
bC1rZXJuZWwtdGV
ggEKAoIBAQDHr2a
7mcfJiASEal6I2X
+T/riNheiGgnhFs
I56wOgkV83+
nLdV8uhVkOBiaM7
pFw4RAEgwTujRI1
/VAHgSo/
ujAMBgNVHRMEBTA
4GPEDGMUZrgaUmu
YDHe/mY8n/
+XwDAxqryCzHIpN
vtrIfIhWUny41pX
pWa9vwIpDqtVA3s
uVoh
-----END CERTIFICATE-----
$
Thanks, this confirms that the kernel you have installed came from the ckt ppa, and not from the archive.
So it is not a bug that grub fails to boot this kernel; though we should revisit whether we could have detected this case at the time grub was upgraded and avoid installing the new bootloader in the case that all your kernels are signed but with a key not trusted by the firmware.
This was discussed in https:/
| Steve Langasek (vorlon) wrote : | #11 |
Note that 'mokutil --list-enrolled' may only present keys as text output unsuitable for feeding into sbverify, but 'mokutil --export' will export each key as a separate file.
| summary: |
- /boot/vmlinux-4.17 has invalid signature + grub2 signed kernel enforcement doesn't check on upgrade that signatures + are from trusted keys |
| tags: | added: id-5b88580cdfde156e4d27cc95 |
| Changed in grub2 (Ubuntu): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in mokutil (Ubuntu): | |
| status: | New → In Progress |
| importance: | Undecided → High |
| assignee: | nobody → Mathieu Trudel-Lapierre (cyphermox) |
mokutil now allows exporting the various keys, we'll still need to integrate this with grub to check for what signs the kernels (and check for blacklists)
| Changed in mokutil (Ubuntu): | |
| status: | In Progress → Fix Released |
I've been working on fixing this; code is here:
https:/
I'll finish testing that it all works correctly, installing unstable kernels, and then upload to disco and proceed with preparing the SRUs.
| Changed in grub2 (Ubuntu): | |
| status: | Triaged → In Progress |
| assignee: | nobody → Mathieu Trudel-Lapierre (cyphermox) |
Adding block-proposed for one last test run in -proposed.
| tags: | added: block-proposed |
Testing looks good; removing block-proposed.
Kernels are checked as expected, a custom kernel signed with a custom but known key is let through.
| tags: | removed: block-proposed |
| Launchpad Janitor (janitor) wrote : | #16 |
This bug was fixed in the package grub2 - 2.02+dfsg1-
---------------
grub2 (2.02+dfsg1-
* debian/
in firmware, in case a kernel is signed but not using a key that will pass
validation, such as when using kernels coming from a PPA. (LP: #1789918)
-- Mathieu Trudel-Lapierre <email address hidden> Mon, 21 Jan 2019 09:34:36 -0500
| Changed in grub2 (Ubuntu): | |
| status: | In Progress → Fix Released |
| description: | updated |
Hello Brad, or anyone else affected,
Accepted grub2 into cosmic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
| Changed in grub2 (Ubuntu Cosmic): | |
| status: | New → Fix Committed |
| tags: | added: verification-needed verification-needed-cosmic |
| Brian Murray (brian-murray) wrote : | #18 |
Hello Brad, or anyone else affected,
Accepted grub2-signed into cosmic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
| Changed in grub2 (Ubuntu Bionic): | |
| status: | New → Fix Committed |
| tags: | added: verification-needed-bionic |
| Brian Murray (brian-murray) wrote : | #19 |
Hello Brad, or anyone else affected,
Accepted grub2 into bionic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
| Brian Murray (brian-murray) wrote : | #20 |
Hello Brad, or anyone else affected,
Accepted grub2-signed into bionic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
Verification-done on bionic with grub2 / grub2-signed:
iF grub-efi-amd64 2.02-2ubuntu8.10 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 version)
ii grub-efi-amd64-bin 2.02-2ubuntu8.10 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 binaries)
ii grub-efi-
Forcing an unsigned copy of the kernel, or one signed by an unknown key leads to the system failing to upgrade, as expected:
ubuntu@
ubuntu@
sbattach sbkeysync sbsiglist sbsign sbvarsign sbverify
ubuntu@
ubuntu@
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 47.0 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://
Fetched 47.0 kB in 0s (112 kB/s)
Preconfiguring packages ...
(Reading database ... 66920 files and directories currently installed.)
Preparing to unpack .../grub-
Unpacking grub-efi-amd64 (2.02-2ubuntu8.10) over (2.02-2ubuntu8.10) ...
Setting up grub-efi-amd64 (2.02-2ubuntu8.10) ...
/boot/vmlinuz-
E: Your kernels are not signed with a key known to your firmware. This system will fail to boot in a Secure Boot environment.
dpkg: error processing package grub-efi-amd64 (--configure):
installed grub-efi-amd64 package post-installation script subprocess returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)
ubuntu@ubuntu:~$ sudo apt install --reinstall grub-efi-amd64
[sudo] password for ubuntu:
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/47.0 kB of archives.
After this operation, 0 B of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 66920 files and directories currently installed.)
Preparing to unpack .../grub-
Unpacking grub-efi-amd64 (2.02-2ubuntu8.10) over (2.02-2ubuntu8.10) ...
Setting up grub-efi-amd64 (2.02-2ubuntu8.10) ...
/boot/vmlinuz-
Subject: CN = PPA cyphermox efi
E: Your kernels are not signed with a key known to your firmware. This system will fail to boot in a Secure Boot environment.
dpkg: error processing package grub-efi-amd64 (--configure):
installed grub-efi-amd64 package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
grub-efi-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
And a properly signed kernel obviously passes validation with no issues;...
| tags: |
added: verification-done-bionic removed: verification-needed-bionic |
Verification-done on cosmic with grub2 / grub2-signed.
Forcing an unsigned copy of the kernel, or one signed by an unknown key leads to the system failing to upgrade, as expected:
ubuntu@ubuntu:~$ dpkg -l grub-efi\* | grep ii | awk '{ print $2" "$3 }'
grub-efi-amd64 2.02+dfsg1-
grub-efi-amd64-bin 2.02+dfsg1-
grub-efi-
ubuntu@ubuntu:~$ sudo apt install --reinstall grub-efi-
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 295 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://
Fetched 295 kB in 0s (742 kB/s)
(Reading database ... 106062 files and directories currently installed.)
Preparing to unpack .../grub-
Unpacking grub-efi-
Setting up grub-efi-
/boot/vmlinuz-
E: Your kernels are not signed with a key known to your firmware. This system will fail to boot in a Secure Boot environment.
dpkg: error processing package grub-efi-
installed grub-efi-
Errors were encountered while processing:
grub-efi-
E: Sub-process /usr/bin/dpkg returned an error code (1)
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo sbsign --key ~/uefi-keys/uefi.
uefi.crt uefi.key
ubuntu@ubuntu:~$ sudo sbsign --key ~/uefi-
ubuntu@ubuntu:~$ sudo apt install grub-efi-
Reading package lists... Done
Building dependency tree
Reading state information... Done
grub-efi-
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up grub-efi-
/boot/vmlinuz-
Subject: CN = PPA cyphermox efi
/boot/vmlinuz-
E: Your kernels are not signed with a key known to your firmware. This system will fail to boot in a Secure Boot environment.
dpkg: error processing package grub-efi-
installed grub-efi-
Errors were encountered while processing:
grub-efi-
E: Sub-process /usr/bin/dpkg returned an error code (1)
| tags: |
added: verification-done-cosmic removed: verification-needed verification-needed-cosmic |
The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
| Launchpad Janitor (janitor) wrote : | #24 |
This bug was fixed in the package grub2 - 2.02+dfsg1-
---------------
grub2 (2.02+dfsg1-
[ Mathieu Trudel-Lapierre ]
* debian/
in firmware, in case a kernel is signed but not using a key that will pass
validation, such as when using kernels coming from a PPA. (LP: #1789918)
* debian/
leaves a trace of what files were sourced to help generate the config
we're building. (LP: #1812863)
[ Steve Langasek ]
* debian/
grubenv and we're on EFI, always show the menu. Closes LP: #1800722.
-- Mathieu Trudel-Lapierre <email address hidden> Tue, 22 Jan 2019 09:57:07 -0500
| Changed in grub2 (Ubuntu Cosmic): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote : | #25 |
This bug was fixed in the package grub2 - 2.02-2ubuntu8.10
---------------
grub2 (2.02-2ubuntu8.10) bionic; urgency=medium
[ Mathieu Trudel-Lapierre ]
* debian/
in firmware, in case a kernel is signed but not using a key that will pass
validation, such as when using kernels coming from a PPA. (LP: #1789918)
* debian/
leaves a trace of what files were sourced to help generate the config
we're building. (LP: #1812863)
[ Steve Langasek ]
* debian/
grubenv and we're on EFI, always show the menu. Closes LP: #1800722.
-- Mathieu Trudel-Lapierre <email address hidden> Wed, 09 Jan 2019 14:04:09 -0500
| Changed in grub2 (Ubuntu Bionic): | |
| status: | Fix Committed → Fix Released |
Hello Brad, or anyone else affected,
Accepted grub2 into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
| Changed in grub2 (Ubuntu Trusty): | |
| status: | New → Fix Committed |
| tags: | added: verification-needed verification-needed-trusty |
| Steve Langasek (vorlon) wrote : | #27 |
Hello Brad, or anyone else affected,
Accepted grub2-signed into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
Verification-done on trusty:
ii grub-common 2.02~beta2-
ii grub-efi-amd64 2.02~beta2-
ii grub-efi-amd64-bin 2.02~beta2-
ii grub-efi-
ii grub-legacy-ec2 0.7.5-0ubuntu1.23 all Handles update-grub for ec2 instances
ii grub-pc-bin 2.02~beta2-
ii grub2-common 2.02~beta2-
Kernel signatures are correctly enforced, but a postinst check is missing in grub2-signed to check signatures before applying the next update (missing in grub2-signed 1.34.19)
| tags: |
added: verification-done-trusty removed: verification-needed-trusty |
| Steve Langasek (vorlon) wrote : | #29 |
Hello Brad, or anyone else affected,
Accepted grub2-signed into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
| tags: |
added: verification-needed-trusty removed: verification-done-trusty |
Verification-done on trusty:
ubuntu@
grub-efi-
Installed: 1.34.20+
Candidate: 1.34.20+
Package pin: 1.34.20+
Version table:
*** 1.34.20+
-1 http://
100 /var/lib/
1.
500 http://
1.
500 http://
1.
500 http://
ubuntu@
grub-efi-amd64:
Installed: 2.02~beta2-
Candidate: 2.02~beta2-
Package pin: 2.02~beta2-
Version table:
*** 2.02~beta2-
-1 http://
100 /var/lib/
2.
500 http://
2.
500 http://
2.02~beta2-9 500
500 http://
Verified that now the kernel signature is correctly enforced by grub, and if no kernel is signed / signed by a trusted key, the upgrade will correctly be failed to avoid leaving the system unbootable.
| tags: |
added: verification-done-trusty removed: verification-needed verification-needed-trusty |
| Launchpad Janitor (janitor) wrote : | #31 |
This bug was fixed in the package grub2 - 2.02~beta2-
---------------
grub2 (2.02~beta2-
* debian/
in firmware, in case a kernel is signed but not using a key that will pass
validation, such as when using kernels coming from a PPA. (LP: #1789918)
* debian/
kernels if UEFI Secure Boot is enabled. If UEFI Secure Boot is enabled
and kernel signature verification fails, do not boot the kernel. Patch
from Linn Crosetto. (LP: #1401532)
-- Mathieu Trudel-Lapierre <email address hidden> Fri, 22 Mar 2019 11:36:54 -0400
| Changed in grub2 (Ubuntu Trusty): | |
| status: | Fix Committed → Fix Released |
apport information