/boot/initrd.img-*.old-dkms files left behind

Bug #1515513 reported by Robin Green on 2015-11-12
34
This bug affects 5 people
Affects Status Importance Assigned to Milestone
dkms (Debian)
New
Unknown
dkms (Ubuntu)
High
Brian Murray
Xenial
High
Brian Murray
Zesty
High
Brian Murray
initramfs-tools (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Zesty
Undecided
Unassigned

Bug Description

[Impact]
If a dkms package is installed which has REMAKE_INITRD or the same setting has be manually configured by a user then when a kernel is removed its possible for an ".old-dkms" file to be left in /boot with no associated kernel.

[Test Case]
On a system with two old kernels and one new kernel available in -updates:
1) install r8168-dkms
2) install the dkms module for the old kernel e.g. 'sudo dkms install -m r8168 -v 8.041.00 -k 4.4.0-31-generic'
3) upgrade your kernel e.g. "sudo apt install linux-image-generic'
4) sudo apt autoremove
5) observe something like "initrd.img-4.4.0-31-generic.old-dkms" in /boot without a corresponding "initrd.img-4.4.0-31-generic"

With the version of dkms in -proposed, the .old-dkms file will be removed when the kernel is auto removed.

[Regression Potential]
Somebody out there might expect the .old-dkms file to be kept, but that seems like an odd expectation.

One notices *.old-dkms files being left behind still sitting on the disk after purging the related kernel. This can cause /boot to become full, and when it gets really bad, even sudo apt-get autoremove won't fix the problem - only deleting the old-dkms files manually solves the problem.

Note: Filling up the /boot partition causes updates to fail.

ProblemType: BugDistroRelease: Ubuntu 15.04
Package: dkms 2.2.0.3-2ubuntu3.3
ProcVersionSignature: Ubuntu 3.19.0-28.30-generic 3.19.8-ckt5
Uname: Linux 3.19.0-28-generic x86_64
ApportVersion: 2.17.2-0ubuntu1.7
Architecture: amd64
CurrentDesktop: KDE
Date: Thu Nov 12 08:17:10 2015
InstallationDate: Installed on 2015-05-05 (190 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Release amd64 (20150422)
PackageArchitecture: allSourcePackage: dkms
UpgradeStatus: No upgrade log present (probably fresh install)

Robin Green (greenrd) wrote :
Changed in dkms (Debian):
status: Unknown → New
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in dkms (Ubuntu):
status: New → Confirmed
spike speigel (frail-knight) wrote :

Might be causing/aggravating this issue with full /boot:

https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1357093

spike speigel (frail-knight) wrote :

This affects 16.04 LTS as well:

$ dkms --version
dkms: 2.2.0.3

Jarno Suni (jarnos) wrote :

The Bourne shell script code to remove the backups could be added in kernel_prerm.d_dkms (or currently non-existing kernel_postrm.d_dkms) in dkms package source, but if the package was removed, there would be no code to remove the backups in conjunction with kernel removing. Thus, the code could be added to kernel/postrm.d/initramfs-tools (or currently non-existing kernel/prerm.d/initramfs-tools) in initramfs-tools package source, too:

# This is applied from make_initrd function in dkms command, which
# creates the possible initrd backup file.
remove_initrd_backup() {
 for initrd in "initrd-$1.img" "initramfs-$1.img" "initrd.img-$1" "initrd-$1"; do
  rm -fv /boot/"${initrd}".old-dkms \
   /boot/efi/efi/redhat/"${initrd}".old-dkms
 done
}

remove_initrd_backup "$1"

Jarno Suni (jarnos) wrote :

Here is the code as a file to retain text formatting.

Jarno Suni (jarnos) wrote :

Or update-initramfs could do more comprehensive deletion in its delete function:

rm -f "${initramfs}" "${initramfs}".*

instead of

rm -f "${initramfs}" "${initramfs}.bak"

spike speigel (frail-knight) wrote :

Also found this:

https://github.com/dell/dkms/issues/2

It's known upstream in debian and dkms, but whether dkms or initramfs-tools should be responsible for cleaning up files left behind by dkms...I dunno. What you posted sounded like it made sense to me.

I just wanted to note at least dkms has an upstream bug report (which was opened May 2016).

Jarno Suni (jarnos) wrote :

Why not adding removing to both? That way update-initramfs would remove the extra ones even if dkms was uninstalled.

tags: added: xenial
spike speigel (frail-knight) wrote :

Merged upstream :D

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in initramfs-tools (Ubuntu):
status: New → Confirmed
tags: added: zesty
spike speigel (frail-knight) wrote :

Will the fix be included in Zesty or Artful?

tags: added: artful yakkety
description: updated
affects: initramfs-tools → dell-sputnik

DKMS is a Dell project but not Project Sputnik. Marking as invalid for dell-sputnik.

Changed in dell-sputnik:
status: New → Invalid
no longer affects: dell-sputnik
Jarno Suni (jarnos) wrote :

According to commant #8
patch against current https://git.launchpad.net/~usd-import-team/ubuntu/+source/initramfs-tools/tree/update-initramfs
Removes all leftover initramfs files, even after dkms is uninstalled.

The attachment "patch against update-imitramfs" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Jarno Suni (jarnos) wrote :

Note that the patch in #15 removes the leftovers related to the kernel version being removed currently. Removing leftovers related to already purged kernels still have to be done manually, for example by linux-purge https://www.bountysource.com/issues/38300038-feature-request-the-command-should-work-like-this together with other extra kernel related stuff.

Changed in dkms (Ubuntu):
status: Confirmed → Triaged
importance: Undecided → High
Changed in dkms (Ubuntu):
assignee: nobody → Brian Murray (brian-murray)
tags: added: full-boot
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 2.3-3ubuntu2

---------------
dkms (2.3-3ubuntu2) artful; urgency=medium

  * 0014-remove-initrd-backup.patch: When removing a kernel also remove
    related .old-dkms file from /boot. (LP: #1515513)

 -- Brian Murray <email address hidden> Fri, 15 Sep 2017 15:33:17 -0700

Changed in dkms (Ubuntu):
status: Triaged → Fix Released
Brian Murray (brian-murray) wrote :

While I've uploaded this fix to Artful for dkms, its not quite clear to me how the extra files end up getting created. I've a couple of systems using dkms and nvidia and I don't see any .old-dkms files on them. For this to SRU'ed, which I'm happy to do, we'd need a test case which describes how to recreate the issue. Could someone provide some information on how to recreate the issue of .old-dkms files filling up /boot? Thanks in advance!

Serhiy Zahoriya (xintx-ua) wrote :

It's in the DKMS itself. Not sure what the main branch is, but here is an example from some branch:
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/dkms/vivid-updates/view/head:/dkms#L277

Brian Murray (brian-murray) wrote :

I'm looking for specific steps to recreate the problem on an Ubuntu system. What steps do I need to take for dkms to leave to a bunch of .old-dkms files in boot?

Serhiy Zahoriya (xintx-ua) wrote :

1. Have nvidia-3## and/or virtualbox installed.
2. Update kernel.

Brian Murray (brian-murray) wrote :

My testing indicates that the .old-dkms files are only created if "remake_initrd" is yes in /usr/sbin/dkms and that is controlled by "REMAKE_INITRD" in a dkms.conf file. From what I can tell most dkms.conf files do not contain the REMAKE_INITRD option. So I don't think this effects a lot of people.

Brian Murray (brian-murray) wrote :

I found some commonly used dkms packages that do set REMAKE_INITRD.

broadcom-sta-6.30.223.271/debian/broadcom-sta-dkms.dkms:REMAKE_INITRD="YES"
flashcache-3.1.3+git20150701/src/dkms.conf:REMAKE_INITRD=yes
r8168-8.044.02/debian/r8168-dkms.dkms:REMAKE_INITRD="YES"

And zfs sets REMAKE_INITRD sometimes.

description: updated
Changed in dkms (Ubuntu Xenial):
status: New → In Progress
Changed in dkms (Ubuntu Zesty):
status: New → In Progress
Changed in dkms (Ubuntu Xenial):
importance: Undecided → High
Changed in dkms (Ubuntu Zesty):
importance: Undecided → High
Changed in dkms (Ubuntu Xenial):
assignee: nobody → Brian Murray (brian-murray)
Changed in dkms (Ubuntu Zesty):
assignee: nobody → Brian Murray (brian-murray)
Serhiy Zahoriya (xintx-ua) wrote :

Sorry I didn't test it. Turns out in my case it's Intel drivers

/var/lib/dkms $ grep REMAKE_INITRD * -rF
i915-4.6.3-4.4.0/1/build/dkms.conf:REMAKE_INITRD=yes

tags: added: id-59d2b866ce8150327afa2464

Hello Robin, or anyone else affected,

Accepted dkms into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.2.0.3-2ubuntu11.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in dkms (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Changed in dkms (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed-zesty
Chris J Arges (arges) wrote :

Hello Robin, or anyone else affected,

Accepted dkms into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.3-3ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Brian Murray (brian-murray) wrote :

The verification of this has failed as the fix was incorrect.

Removing linux-image-4.10.0-19-generic (4.10.0-19.21) ...
Examining /etc/kernel/prerm.d.
run-parts: executing /etc/kernel/prerm.d/dkms 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic
removed '/boot/initrd.img-4.10.0-19-generic.old-dkms'
dkms: removing: r8168 8.043.02 (4.10.0-19-generic) (x86_64)

-------- Uninstall Beginning --------
Module: r8168
Version: 8.043.02
Kernel: 4.10.0-19-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

r8168.ko:
 - Uninstallation
   - Deleting from: /lib/modules/4.10.0-19-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

Backing up initrd.img-4.10.0-19-generic to /boot/initrd.img-4.10.0-19-generic.old-dkms
Making new initrd.img-4.10.0-19-generic
(If next boot fails, revert to initrd.img-4.10.0-19-generic.old-dkms image)
update-initramfs...

DKMS: uninstall completed.
Examining /etc/kernel/postrm.d .

The .old-dkms file is removed before a new one is created (see the Backing up bits). So the remove_initrd_backup run needs to be reordered for after the dkms call.

Changed in dkms (Ubuntu):
status: Fix Released → In Progress
tags: added: verification-failed-xenial verification-failed-zesty
removed: verification-needed-xenial verification-needed-zesty
Brian Murray (brian-murray) wrote :

I've uploaded fixes to all releases previously identified as worth fixing which remove the .old-dkms file after its been created.

Jarno Suni (jarnos) wrote :

Brian, Where you have uploaded the code? The code at https://code.launchpad.net/ubuntu/+source/dkms is old.

Łukasz Zemczak (sil2100) wrote :

Hello Robin, or anyone else affected,

Accepted dkms into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.3-3ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed-zesty
removed: verification-failed-zesty
tags: added: verification-needed-xenial
removed: verification-failed-xenial
Łukasz Zemczak (sil2100) wrote :

Hello Robin, or anyone else affected,

Accepted dkms into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.2.0.3-2ubuntu11.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Brian Murray (brian-murray) wrote :

I've verified that the version of dkms, 2.3-3ubuntu1.2, in zesty-proposed now does the right thing.

Backing up initrd.img-4.10.0-19-generic to /boot/initrd.img-4.10.0-19-generic.old-dkms
Making new initrd.img-4.10.0-19-generic
(If next boot fails, revert to initrd.img-4.10.0-19-generic.old-dkms image)
update-initramfs...

DKMS: uninstall completed.
removed '/boot/initrd.img-4.10.0-19-generic.old-dkms'
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic
update-initramfs: Deleting /boot/initrd.img-4.10.0-19-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.10.0-37-generic
Found initrd image: /boot/initrd.img-4.10.0-37-generic
Found linux image: /boot/vmlinuz-4.10.0-28-generic
Found initrd image: /boot/initrd.img-4.10.0-28-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
bdmurray@clean-zesty-amd64:~$ ls -lh /boot/
total 149M
-rw-r--r-- 1 root root 1.4M Jun 29 02:14 abi-4.10.0-28-generic
-rw-r--r-- 1 root root 1.4M Oct 6 12:45 abi-4.10.0-37-generic
-rw-r--r-- 1 root root 201K Jun 29 02:14 config-4.10.0-28-generic
-rw-r--r-- 1 root root 201K Oct 6 12:45 config-4.10.0-37-generic
drwxr-xr-x 5 root root 4.0K Oct 10 09:33 grub
-rw-r--r-- 1 root root 42M Oct 10 09:26 initrd.img-4.10.0-28-generic
-rw-r--r-- 1 root root 41M Oct 10 09:26 initrd.img-4.10.0-28-generic.old-dkms
-rw-r--r-- 1 root root 42M Oct 10 09:29 initrd.img-4.10.0-37-generic
-rw-r--r-- 1 root root 179K Jan 28 2016 memtest86+.bin
-rw-r--r-- 1 root root 181K Jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 181K Jan 28 2016 memtest86+_multiboot.bin
-rw------- 1 root root 3.6M Jun 29 02:14 System.map-4.10.0-28-generic
-rw------- 1 root root 3.6M Oct 6 12:45 System.map-4.10.0-37-generic
-rw------- 1 root root 7.3M Jun 29 02:14 vmlinuz-4.10.0-28-generic
-rw------- 1 root root 7.3M Oct 6 12:45 vmlinuz-4.10.0-37-generic

tags: added: verification-done-zesty
removed: verification-needed-zesty
Brian Murray (brian-murray) wrote :

The verification of this also passed for Xenial, Ubuntu 16.04.

run-parts: executing /etc/kernel/prerm.d/dkms 4.4.0-31-generic /boot/vmlinuz-4.4.0-31-generic
dkms: removing: r8168 8.041.00 (4.4.0-31-generic) (x86_64)

-------- Uninstall Beginning --------
Module: r8168
Version: 8.041.00
Kernel: 4.4.0-31-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

r8168.ko:
 - Uninstallation
   - Deleting from: /lib/modules/4.4.0-31-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod....

Backing up initrd.img-4.4.0-31-generic to /boot/initrd.img-4.4.0-31-generic.old-dkms
Making new initrd.img-4.4.0-31-generic
(If next boot fails, revert to initrd.img-4.4.0-31-generic.old-dkms image)
update-initramfs....

DKMS: uninstall completed.
removed '/boot/initrd.img-4.4.0-31-generic.old-dkms'
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.4.0-31-generic /boot/vmlinuz-4.4.0-31-generic
update-initramfs: Deleting /boot/initrd.img-4.4.0-31-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.4.0-31-generic /boot/vmlinuz-4.4.0-31-generic
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.4.0-97-generic
Found initrd image: /boot/initrd.img-4.4.0-97-generic
Found linux image: /boot/vmlinuz-4.4.0-38-generic
Found initrd image: /boot/initrd.img-4.4.0-38-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
bdmurray@clean-xenial-amd64:~$ ls -lh /boot/
total 132M
-rw-r--r-- 1 root root 1.2M Sep 6 2016 abi-4.4.0-38-generic
-rw-r--r-- 1 root root 1.2M Sep 19 13:29 abi-4.4.0-97-generic
-rw-r--r-- 1 root root 186K Sep 6 2016 config-4.4.0-38-generic
-rw-r--r-- 1 root root 187K Sep 19 13:29 config-4.4.0-97-generic
drwxr-xr-x 5 root root 4.0K Oct 10 09:42 grub
-rw-r--r-- 1 root root 36M Oct 10 09:39 initrd.img-4.4.0-38-generic
-rw-r--r-- 1 root root 36M Oct 10 09:38 initrd.img-4.4.0-38-generic.old-dkms
-rw-r--r-- 1 root root 37M Oct 10 09:41 initrd.img-4.4.0-97-generic
-rw-r--r-- 1 root root 179K Jan 28 2016 memtest86+.bin
-rw-r--r-- 1 root root 181K Jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 181K Jan 28 2016 memtest86+_multiboot.bin
-rw------- 1 root root 3.7M Sep 6 2016 System.map-4.4.0-38-generic
-rw------- 1 root root 3.8M Sep 19 13:29 System.map-4.4.0-97-generic
-rw------- 1 root root 6.8M Sep 6 2016 vmlinuz-4.4.0-38-generic
-rw------- 1 root root 6.8M Sep 19 13:29 vmlinuz-4.4.0-97-generic

tags: added: verification-done-xenial
removed: verification-needed-xenial
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 2.3-3ubuntu3

---------------
dkms (2.3-3ubuntu3) artful; urgency=medium

  * 0014-remove-initrd-backup.patch: When removing a kernel also remove
    related .old-dkms file from /boot after its been created not before.
    (LP: #1515513)

 -- Brian Murray <email address hidden> Thu, 05 Oct 2017 13:10:42 -0700

Changed in dkms (Ubuntu):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 2.3-3ubuntu1.2

---------------
dkms (2.3-3ubuntu1.2) zesty; urgency=medium

  * 0014-remove-initrd-backup.patch: When removing a kernel also remove
    related .old-dkms file from /boot after its been created not before.
    (LP: #1515513)

 -- Brian Murray <email address hidden> Thu, 05 Oct 2017 11:45:16 -0700

Changed in dkms (Ubuntu Zesty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for dkms has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 2.2.0.3-2ubuntu11.5

---------------
dkms (2.2.0.3-2ubuntu11.5) xenial; urgency=medium

  * 0014-remove-initrd-backup.patch: When removing a kernel also remove
    related .old-dkms file from /boot after its been created not before.
    (LP: #1515513)

 -- Brian Murray <email address hidden> Thu, 05 Oct 2017 13:14:20 -0700

Changed in dkms (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.