unattended-upgrades + nvidia stack upgrade == dkms fail (package shim-signed 1.32~16.04.1+0.9+1474479173.6c180c6-1ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1)

Bug #1726803 reported by Patrick Thomas on 2017-10-24
94
This bug affects 11 people
Affects Status Importance Assigned to Milestone
shim-signed (Ubuntu)
High
Balint Reczey
Bionic
High
Unassigned
unattended-upgrades (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned

Bug Description

[Impact]

 * When Secure Boot is enabled and MOK is not set and a new dkms module is installed sim-signed asks for a Secure Boot MOK, or aborts package installation in non-interactive mode. When unattended-upgrades performed the upgrade the aborted installation leaves an unconfigured system behind that may even fail to boot. In nvdidia's special case the new module is actually just a new version of the nvidia module which should be fine to install.

 * The fix in shim-signed now handles nvidia dkms module directory renames as simple upgrades and also does not handle module removals as a reason to abort installation.

[Test Case (shim-signed)]

1. Set up Bionic system with Secure Boot enabled.

2. Install packagages to trigger MOK enrollment and enroll the key:

 apt install dkms shim-signed r8168-dkms

3. Create a new key to be enrolled again:

 rm /var/lib/shim-signed/mok/MOK.der
 update-secureboot-policy --new-key

4. Simulate module removal and test that the command returns 0:

# echo /var/lib/dkms/zzz >> /var/lib/shim-signed/dkms-list
# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
0

5. Simulate nvidia module rname and test that the command returns 0:

# cat > /var/lib/shim-signed/dkms-list <<EOF
/var/lib/dkms
/var/lib/dkms/nvidia-1
/var/lib/dkms/r8168
EOF
# mkdir /var/lib/dkms/nvidia
# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
0

6. Simulate addition of nvidia driver, it should still return 1:
# cat > /var/lib/shim-signed/dkms-list <<EOF
/var/lib/dkms
/var/lib/dkms/r8168
EOF

# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
--- /var/lib/shim-signed/dkms-list 2018-10-26 19:19:19.526697542 +0200
+++ /var/lib/shim-signed/dkms-list.new 2018-10-26 19:19:19.530697604 +0200
@@ -1,2 +1,3 @@
 /var/lib/dkms
+/var/lib/dkms/nvidia
 /var/lib/dkms/r8168
1

[Test Case (unattended-upgrades)]

1. Set up a fully - or almost fully updated Bionic system.

2. Install packagages to trigger the block:
apt install dkms shim-signed r8168-dkms

3. Fake enabled secure boot:
echo "shim-signed shim/enable_secureboot boolean true" | debconf-set-selections

4. Add and enable PPA hosting updated dkms package pulling in a new dkms-like dependency:
add-apt-repository ppa:rbalint/scratch
echo 'Unattended-Upgrade::Allowed-Origins {"LP-PPA-rbalint-scratch:${distro_codename}";}' > /etc/apt/apt.conf.d/51unattended-upgrades-all

5. Observe u-u keeping back the new package:
unattended-upgrade --verbose --dry-run --debug
...
Checking: r8168-dkms ([<Origin component:'main' archive:'bionic' origin:'LP-PPA-rbalint-scratch' label:"Scratch space, don't use" site:'ppa.launchpad.net' isTrusted:True>])
pkg new-dkms-dep may trigger secure boot key prompt
sanity check failed
...

[Regression Potential (shim-signed)]

* The fix lets installation of nvidia dkms module upgrades continue and also lets dkms module removals continue when MOK is not set and those should not cause regressions themselves. In case of an implementation mistake a new module installation could go undetected and could cause the system not load a dkms module on next boot.
In practice not loading new modules rarely cause regressions, but if a module is converted from being in the kernel to a dkms module upon an upgrade this is possible.
 * I tested the module addition, removal, nvidia module upgrade and not module change cases with stubs pretending that the system is secure-boot capable an found the changed script working properly.

[Regression Potential (unattended-upgrades)]

* Since the fix is holding back packages from installation it is expected that systems that would have otherwise broke during the installation would not receive all updates. Since exact detection of the installation failure reported here does not seem possible u-u holds back more packages than it would be absolutely necessary.

* Administrators are expected to set up email notifications about the updates performed by u-u and act on held back packages.

* Since updates pulling in new packages are fairly rare especially in the -security pocket which u-u installs from by default unwanted regressions are unlikely to show up.

[Original Bug Text]

Occurred a minute after logging in

ProblemType: Package
DistroRelease: Ubuntu 16.04
Package: shim-signed 1.32~16.04.1+0.9+1474479173.6c180c6-1ubuntu1
ProcVersionSignature: Ubuntu 4.10.0-37.41~16.04.1-generic 4.10.17
Uname: Linux 4.10.0-37-generic x86_64
.proc.sys.kernel.moksbstate_disabled: 0
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
Date: Tue Oct 24 11:35:53 2017
EFITables:
 Oct 24 11:33:04 paddy-laptop kernel: efi: EFI v2.40 by American Megatrends
 Oct 24 11:33:04 paddy-laptop kernel: efi: ACPI=0x78660000 ACPI 2.0=0x78660000 SMBIOS=0xf0000 SMBIOS 3.0=0xf0020 ESRT=0x79360598
 Oct 24 11:33:04 paddy-laptop kernel: esrt: Reserving ESRT space from 0x0000000079360598 to 0x00000000793605d0.
 Oct 24 11:33:04 paddy-laptop kernel: Secure boot enabled
ErrorMessage: subprocess installed post-installation script returned error exit status 1
InstallationDate: Installed on 2017-09-11 (42 days ago)
InstallationMedia: Ubuntu 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
RelatedPackageVersions:
 dpkg 1.18.4ubuntu1.2
 apt 1.2.24
SecureBoot: 6 0 0 0 1
SourcePackage: shim-signed
Title: package shim-signed 1.32~16.04.1+0.9+1474479173.6c180c6-1ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1
UpgradeStatus: No upgrade log present (probably fresh install)

Patrick Thomas (padman) wrote :
tags: removed: need-duplicate-check
Steve Langasek (vorlon) wrote :

I'm surprised to see that your nvidia stack is being switched as part of an unattended upgrade. Reassigning to the unattended-upgrades package for further investigation.

In order to fix the particular upgrade failure, you should run 'sudo apt -f install' from a terminal. Note that you must complete the process of disabling SecureBoot, which requires a reboot, in order to use the nvidia driver on a SecureBoot-enabled system.

affects: shim-signed (Ubuntu) → unattended-upgrades (Ubuntu)
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed
Steve Langasek (vorlon) wrote :

It appears this is normal that the nvidia packages will upgrade to pull in a newer version as a dependency. This means we need to special-case nvidia in /var/lib/dkms/nvidia-* to avoid triggering errors in unattended upgrades.

affects: unattended-upgrades (Ubuntu) → shim-signed (Ubuntu)
Changed in shim-signed (Ubuntu):
importance: Undecided → High
status: Confirmed → Triaged
Steve Langasek (vorlon) on 2017-10-25
summary: - package shim-signed 1.32~16.04.1+0.9+1474479173.6c180c6-1ubuntu1 failed
- to install/upgrade: subprocess installed post-installation script
- returned error exit status 1
+ unattended-upgrades + nvidia stack upgrade == dkms fail (package shim-
+ signed 1.32~16.04.1+0.9+1474479173.6c180c6-1ubuntu1 failed to
+ install/upgrade: subprocess installed post-installation script returned
+ error exit status 1)
tags: added: rls-bb-incoming
Steve Langasek (vorlon) on 2017-11-09
tags: removed: rls-bb-incoming
tags: added: id-5ab94d1375e8d544f030e3fa
Balint Reczey (rbalint) on 2018-10-08
Changed in shim-signed (Ubuntu):
status: Triaged → In Progress
assignee: nobody → Balint Reczey (rbalint)
Balint Reczey (rbalint) on 2018-10-09
description: updated
Balint Reczey (rbalint) on 2018-10-09
description: updated
Changed in shim-signed (Ubuntu):
status: In Progress → Confirmed
status: Confirmed → New
Changed in unattended-upgrades (Ubuntu):
status: New → In Progress
Steve Langasek (vorlon) wrote :

Balint, I wrote in comment #4 that changing the nvidia stack is expected on upgrades, and that we should special-case this in shim-signed to suppress the error. Why are you proposing to change unattended-upgrades instead?

Changed in unattended-upgrades (Ubuntu Bionic):
status: New → Incomplete
Changed in unattended-upgrades (Ubuntu):
status: In Progress → Incomplete
Balint Reczey (rbalint) wrote :

@vorlon My bad, thought that a generic solution would be useful in u-u, but now I'm implementing now the special case.

Balint Reczey (rbalint) on 2018-10-10
Changed in shim-signed (Ubuntu):
status: New → In Progress
Balint Reczey (rbalint) on 2018-10-11
Changed in shim-signed (Ubuntu):
status: In Progress → Fix Committed
Balint Reczey (rbalint) on 2018-10-12
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package shim-signed - 1.38

---------------
shim-signed (1.38) cosmic; urgency=medium

  * Don't fail non-interactive upgrade of nvidia module and module removals
    (LP: #1726803)

 -- Balint Reczey <email address hidden> Thu, 11 Oct 2018 18:12:37 +0200

Changed in shim-signed (Ubuntu):
status: Fix Committed → Fix Released
Balint Reczey (rbalint) on 2018-10-26
description: updated

Hello Patrick, or anyone else affected,

Accepted shim-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/shim-signed/1.37~18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in shim-signed (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
Balint Reczey (rbalint) on 2018-11-06
description: updated
description: updated
Balint Reczey (rbalint) wrote :

Verified with shim-signed 1.37~18.04.3+15+1533136590.3beb971-0ubuntu1:
...
ubuntu@autopkgtest:/var/cache/apt/archives$ sudo rm /var/lib/shim-signed/mok/MOK.der
ubuntu@autopkgtest:/var/cache/apt/archives$ sudo update-secureboot-policy --new-key
Generating a new Secure Boot signing key:
Generating a 2048 bit RSA private key
...........................................................................................................+++
...................+++
writing new private key to '/var/lib/shim-signed/mok/MOK.priv'
-----
root@autopkgtest:/var/cache/apt/archives# echo /var/lib/dkms/zzz >> /var/lib/shim-signed/dkms-list
root@autopkgtest:/var/cache/apt/archives# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
0
root@autopkgtest:/var/cache/apt/archives# cat > /var/lib/shim-signed/dkms-list <<EOF
> /var/lib/dkms
> /var/lib/dkms/nvidia-1
> /var/lib/dkms/r8168
> EOF
root@autopkgtest:/var/cache/apt/archives# mkdir /var/lib/dkms/nvidia
root@autopkgtest:/var/cache/apt/archives# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
0
root@autopkgtest:/var/cache/apt/archives# cat > /var/lib/shim-signed/dkms-list <<EOF
> /var/lib/dkms
> /var/lib/dkms/r8168
> EOF
root@autopkgtest:/var/cache/apt/archives# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
--- /var/lib/shim-signed/dkms-list 2018-11-06 17:47:43.490071095 +0100
+++ /var/lib/shim-signed/dkms-list.new 2018-11-06 17:47:43.490071095 +0100
@@ -1,2 +1,3 @@
 /var/lib/dkms
+/var/lib/dkms/nvidia
 /var/lib/dkms/r8168
1

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Balint Reczey (rbalint) wrote :

For the record the shim-signed fix alone is enough to fix the reported issue, i.e. installation failing when the nvidia module changes dirs.

The u-u fix is an attempt to fix the generic case, when an update brings in a new dkms module, but this is not happening in the archive at the moment and is not expected to happen in stable releases.

The verification of the Stable Release Update for shim-signed has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package shim-signed - 1.37~18.04.3

---------------
shim-signed (1.37~18.04.3) bionic; urgency=medium

  * Don't fail non-interactive upgrade of nvidia module and module removals
    (LP: #1726803)

 -- Balint Reczey <email address hidden> Thu, 25 Oct 2018 20:55:38 +0200

Changed in shim-signed (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers