Removing a linux-image-extra package fails, if /boot is about full

[Jarno Suni]

System calls /etc/kernel/postinst.d/initramfs-tools when purging/removing a linux-image-extra package. That calls "update-initramfs -c" which needs significant amount of additional disk space in /boot temporarily. But there is no space left, if /boot is full.

Likewise /etc/kernel/postinst.d/dkms may call "update-initramfs -u".

The fix could be to create the new initrg.img file in different partition before replacing the old one by it in update-initramfs. So the update-initramfs script should be fixed. But there may not be such a partition..
Anyway the likely case when space runs out is when there is a separate /boot partition.

Alternatively the init scripts should remove the respective /boot/initrd.img-<version> file when removing/installing the linux-image-extra package. That could also be done by
update-initramfs -d -k <version>
That may be worse way, as then initrd.img file is missing for longer period of time and system integrity may suffer in case of e.g. power cut.

Related question: http://askubuntu.com/q/898499/21005
The output of 'dpkg --purge' presented there shows that corresponding linux-image package may get successfully removed while the linux-image-extra is left broken. If the linux-image-extra package will be removed later, the post-installation script will create an initrd.img file for a non-installed kernel! Same would happen, if removing would be done by apt-get purge.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: initramfs-tools 0.103ubuntu4.7
ProcVersionSignature: Ubuntu 4.4.0-71.92~14.04.1-generic 4.4.49
Uname: Linux 4.4.0-71-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.23
Architecture: amd64
CurrentDesktop: XFCE
Date: Fri Mar 31 17:42:35 2017
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-09-21 (922 days ago)
InstallationMedia: Ubuntu-Studio 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.1)
PackageArchitecture: all
SourcePackage: initramfs-tools
UpgradeStatus: No upgrade log present (probably fresh install)

[Jarno Suni (jarnos)]

What triggers a postinst script when removing a linux-image-extra package by dpkg?

[Jarno Suni (jarnos)]

dpkg may also fail to install a linux-image-extra package, because "update-initramfs -c" needs more space for initrd.img in /boot temporarily, even if the new initrd.img would fit when the old one has been removed.

[Jarno Suni (jarnos)]

I managed to change update-initramfs in my computer so that it uses a temporary location (temporarily) for new initrd.img file thus behaving better in case of separate /boot partition and low disk space there.

[Jarno Suni (jarnos)]

Maybe also dkms may need more space on /boot when updating and backing up initrd.img?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[spike speigel (frail-knight)]

Not sure if using dkms, but if so may be related to bug 1515513

[Jarno Suni (jarnos)]

Patch against current https://git.launchpad.net/~usd-import-team/ubuntu/+source/initramfs-tools/tree/update-initramfs

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "patch concerning generate_initramfs function in update_initramfs" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Jarno Suni (jarnos)]

This patch is better, because it takes into account the size of possible existing initramfs file when calculating the needed available free space.

[Jarno Suni (jarnos)]

patch version 2 is also against (then) current
https://git.launchpad.net/~usd-import-team/ubuntu/+source/initramfs-tools/tree/update-initramfs

[Brian Murray (brian-murray)]

I tried recreating this in Ubuntu 17.04 by having three kernels installed on a /boot partition with only 11M of free space left. I then used 'sudo apt autoremove' which removed my oldest kernel, including linux-image-extra-4.10.0-19-generic, and did not run into any issues. Is this issue specific to only removing the linux-image-extra package or does it only affect Ubuntu 16.04.?

Could you please provide some detailed steps to recreate this bug? Thanks in advance.

[Brian Murray (brian-murray)]

Using the same procedure I was unable to recreate this on Ubuntu 16.04.

[Jarno Suni (jarnos)]

Tested in 16.04

Lower free space at /boot to 10MB by the following one-liner:

sudo dd if=/dev/zero of=/boot/tmpfile bs=1MB count=$(($(stat -f --format="%f*%S" /boot)/1000000-10))

Then remove a linux-image-extra package by apt, e.g.

sudo apt purge linux-image-extra-4.10.0-19-generic

Check status of the package:

dpkg -l linux-image-extra-4.10.0-19-generic

(If you run

sudo apt purge linux-image-4.10.0-19-generic

There is an error, too, but the requested package will be removed. Running 'sudo apt install' thereafter finishes removing the linux-image-extra package.)

Remember to remove the created temporary file:

sudo rm /boot/tmpfile

[Brian Murray (brian-murray)]

I've recreated this on Ubuntu 16.04 now, it seems that the missing bit was creating a sufficiently small enough amount of free space on my /boot partition.

While running 'sudo apt install' did finish the removal and I was able to continue using apt, I did notice that I had a leftover / abandoned initrd.img-4.4.0-31-generic.

[Jarno Suni (jarnos)]

Then not incomplete anymore?

[Brian Murray (brian-murray)]

I think source of the problem here is bug 1501844 and removing a linux-image-extra package causes update-initramfs to be run. If the kernel package maintainer scripts were written differently then removing linux-image and linux-image-extra would not have a problem if /boot was full, but removing linux-image-extra by itself would still be an issue as update-initramfs would be run.

[Jarno Suni (jarnos)]

There is no bug with that number.

[Jarno Suni (jarnos)]

For some reason with my patch mv may fail even if I try to make sure mv will not be run, if there is not enough space on /boot:

Setting up linux-generic-lts-xenial (4.4.0.97.81) ...
Processing triggers for initramfs-tools (0.103ubuntu4.8) ...
update-initramfs: Generating /boot/initrd.img-4.4.0-97-generic
W: TMPDIR is mounted noexec, will not cache run scripts.
mv: error writing '/boot/initrd.img-4.4.0-97-generic': No space left on device
mv: failed to extend '/boot/initrd.img-4.4.0-97-generic': No space left on device
update-initramfs: failed for /boot/initrd.img-4.4.0-97-generic with mv returning 1.
Backup file /boot/initrd.img-4.4.0-97-generic.dpkg-bak remains.
dpkg: error processing package initramfs-tools (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Jarno Suni (jarnos)]

As for the patch, the following test shows that a file stored in /boot may require significantly more available space than the same file in /var/tmp, even if blocksize in (partition containing) /var/tmp is bigger. I don't know what is the best way to estimate how much available space is needed for storing a file in /boot.

$ echo $TMPDIR

$ echo $(( $(stat -f --format='%S' /boot ) ))
1024
$ echo $(( $(stat -f --format='%S' /var/tmp/ ) ))
4096
sudo dd if=/dev/zero of=/boot/file bs=K count=35553
35553+0 records in
35553+0 records out
36406272 bytes (36 MB) copied, 0,334945 s, 109 MB/s
$ sudo dd if=/dev/zero of=/var/tmp/file bs=K count=35553
35553+0 records in
35553+0 records out
36406272 bytes (36 MB) copied, 0,171743 s, 212 MB/s
$ echo $(( $(stat --format='%b*%B' /var/tmp/file ) ))
36409344
$ echo $(( $(stat --format='%b*%B' /boot/file ) ))
36549632

[Jarno Suni (jarnos)]

A new attempt to estimate required space in /boot.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Hayden Clark (missing.semicolon)]

Just ran in to this on Ubuntu 20.04, with an encrypted root partition. The partition layout is the one automatically generated when you install and ask for root partition encryption - I didn't mess with it.
Drive is 512Gb NVME
Machine is a Lenovo Thinkbook 15
Kernel 5.8.0-59-generic

The /boot partition is 732Mb, but is filled with initrd.img* files.

[Jarno Suni (jarnos)]

Did you upgrade from older release or are the files generated when you were using 20.04? Can you provide any logs concerning the issue? Do the extra initrd.img* files match /boot/initrd.img-*.old-dkms, see https://bugs.launchpad.net/ubuntu/+source/dkms/+bug/1515513 ? Or do you still have the respective kernels (vmlinuz-* etc.) there, too?

It seems that my modified generate_initramfs function has not been applied in /usr/sbin/update-initramfs in 20.04, so you can not remove a kernel successfully. The modified version should do better job not leaving extra files around IIRC. Anyway, it should be possible with my linux-purge utility (https://launchpad.net/linux-purge).

[Hayden Clark (missing.semicolon)]

This was a fresh install of 20.04. I enabled encryption as it's a work laptop.
I don't think I have logs - just the failure due to /boot being full on the update UI.
I then followed the instructions at https://help.ubuntu.com/community/RemoveOldKernels under "Safely remove old kernels" so I could complete the update.

There were no /boot/initrd.img-*.old-dkms files, just kernels and initrds corresponding to the kernels.

/boot is about 700Mb. More space is taken as both the -generic and -oem kernels are being installed, even though I'd prefer the -oem ones.

[Hayden Clark (missing.semicolon)]

Ok, more information. Just got warned that /boot is full again.
I have attached a directory listing for /boot.
Where do I find the logs for the update process that's doing this?

[Hayden Clark (missing.semicolon)]

I have now run linux-purge. It only freed up 50Mb or so, and no kernels got removed

[Hayden Clark (missing.semicolon)]

I managed to get some free space back on /boot.
I'm posting this in the hopes that what I did may inform any fixing process.
1) List kernels with `dpkg -l | tail -n +6 | grep -E 'linux-image-[0-9]+'`
2) Delete the oldest initrd with update-initramfs -d -k 5.8.0-59-generic. There was now 111Mb free on /boot
3) Run `linux-purge -f`. This did nothing, but it output (presumably from running apt) a "The following packages were automatically installed and are no longer required" message
4) Run `apt autoremove`. This removed some kernels and initrd files.

Now I have 256Mb free.

This would be easier if /boot was a little larger. The install process gives you the choice of "totally automatic" and "you're on your own, buddy!" when partitioning. One neat trick the old Centos install has was to allow you to pre-load the manual partitioning scheme with the default, which you could then modify.

[Jarno Suni (jarnos)]

Did you run "sudo linux-purge --keep=1"? (or replace "1" by "0" to purge even more)

[Jarno Suni (jarnos)]

Could you attach output of "linux-purge --info"?

[Hayden Clark (missing.semicolon)]

Here is the result of linux-purge --info

My /boot is like this:
# du -sh /boot/*
248K /boot/config-5.10.0-1038-oem
248K /boot/config-5.10.0-1044-oem
248K /boot/config-5.11.0-25-generic
248K /boot/config-5.11.0-27-generic
244K /boot/config-5.8.0-63-generic
48M /boot/efi
8.0M /boot/grub
0 /boot/initrd.img
113M /boot/initrd.img-5.10.0-1038-oem
113M /boot/initrd.img-5.10.0-1044-oem
112M /boot/initrd.img-5.11.0-25-generic
112M /boot/initrd.img-5.11.0-27-generic
113M /boot/initrd.img-5.8.0-63-generic
0 /boot/initrd.img.old
16K /boot/lost+found
180K /boot/memtest86+.bin
184K /boot/memtest86+.elf
184K /boot/memtest86+_multiboot.bin
5.6M /boot/System.map-5.10.0-1038-oem
5.6M /boot/System.map-5.10.0-1044-oem
5.6M /boot/System.map-5.11.0-25-generic
5.6M /boot/System.map-5.11.0-27-generic
5.3M /boot/System.map-5.8.0-63-generic
0 /boot/vmlinuz
12M /boot/vmlinuz-5.10.0-1038-oem
12M /boot/vmlinuz-5.10.0-1044-oem
9.7M /boot/vmlinuz-5.11.0-25-generic
9.7M /boot/vmlinuz-5.11.0-27-generic
9.4M /boot/vmlinuz-5.8.0-63-generic
0 /boot/vmlinuz.old

[Hayden Clark (missing.semicolon)]

And then I tried "sudo linux-purge --keep=1".
That has tidied up a bit, but there's still not much space?field.comment=And then I tried "sudo linux-purge --keep=1".
That has tidied up a bit, but there's still not much space.

Now /boot contains:
# du -sh /boot/*
248K /boot/config-5.10.0-1038-oem
248K /boot/config-5.10.0-1044-oem
248K /boot/config-5.11.0-25-generic
248K /boot/config-5.11.0-27-generic
48M /boot/efi
8.0M /boot/grub
0 /boot/initrd.img
113M /boot/initrd.img-5.10.0-1038-oem
113M /boot/initrd.img-5.10.0-1044-oem
112M /boot/initrd.img-5.11.0-25-generic
112M /boot/initrd.img-5.11.0-27-generic
0 /boot/initrd.img.old
16K /boot/lost+found
180K /boot/memtest86+.bin
184K /boot/memtest86+.elf
184K /boot/memtest86+_multiboot.bin
5.6M /boot/System.map-5.10.0-1038-oem
5.6M /boot/System.map-5.10.0-1044-oem
5.6M /boot/System.map-5.11.0-25-generic
5.6M /boot/System.map-5.11.0-27-generic
0 /boot/vmlinuz
12M /boot/vmlinuz-5.10.0-1038-oem
12M /boot/vmlinuz-5.10.0-1044-oem
9.7M /boot/vmlinuz-5.11.0-25-generic
9.7M /boot/vmlinuz-5.11.0-27-generic
0 /boot/vmlinuz.old

# df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p2 705M 526M 129M 81% /boot

[Jarno Suni (jarnos)]

Hayden Clark, so you use "oem" kernels. If you do not need "generic" kernels, you can save a lot of space in /boot by deleting all "generic" kernels. Probably the easiest way to do it in your case is "sudo apt autoremove linux-generic-hwe-20.04". Check that it does not remove any "oem" kernels. If that does not work, you can use "sudo linux-purge --choose --optimize" and choose all "generic" kernels for purging.

[Hayden Clark (missing.semicolon)]

Thanks for that advice.
I'm a bit in trepidation about removing the default HWE. How do I protect myself from having an unbootable system?

[Jarno Suni (jarnos)]

Remove the older generic kernel by linux-purge and then try booting the respective oem kernel via advanced grub menu.

[Jarno Suni (jarnos)]

Oh, the versions do not match. I do not see how removing the hwe kernels would make it unbootable.

[Jarno Suni (jarnos)]

Hayden, did you try?

[Hayden Clark (missing.semicolon)]

I've not yet found a way to make a USB boot that would allow me to start the machine in the event of something going wrong. Without a plan to revert the change, I'm a bit nervous. I know it *should* work fine - but I don't want to suddenly have an unbootable laptop I can't rescue.

[Jarno Suni (jarnos)]

Well, you could search for support in askubuntu.com. I think it may be a bug that you have two flavors of kernels installed, if you did not install the other manually. You might need to make a separate bug report about that; removing the other flavor might be a workaround.

[Hayden Clark (missing.semicolon)]

Ok, did it.
That's freed up a load of space.
Where would I file the bug about having two sets of kernels?

[Jarno Suni (jarnos)]

Please check out https://help.ubuntu.com/community/ReportingBugs#Filing_a_general_bug_against_no_particular_package (and the document in general). Please tell me the URL of the report thereafter.

You told in #23 that you "Just ran in to this on Ubuntu 20.04", but to be precise it is not even possible, as Ubuntu 20.04 does not have linux-image-extra packages. Do linux-modules packages have some similar effect? Did you even try to remove a kernel by dpkg without running "update-initramfs -d"? Or did you even try removing a kernel by apt before that? It would be helpful if you provided the output of such commands that fail. You also told "The /boot partition is 732Mb, but is filled with initrd.img* files." Still there were no more of those than e.g. vmlinuz-* files in the listings you gave.

I have contributed to https://help.ubuntu.com/community/RemoveOldKernels, authored the linux-purge tool, made this bug report and provided the patch. Please follow some accuracy and relevancy in comments.

[Hayden Clark (missing.semicolon)]

Sorry I was not clear enough.
When I said "Filled with" I meant that they were the files occupying space. I'm sorry if that gave the idea that the corresponding vmlinuz-* files weren't present - it was just that in terms of bytes occupied, the initrd files are the big ones.

As you can see from the directory listings, this has been a problem, and, since I'm using a Lenovo Thinkbook 15, I used the OEM kernel install. I probably followed https://wiki.ubuntu.com/Kernel/OEMKernel to install it and it fixed my audio problems.

When removing kernels, I followed the instructions on the RemoveOldKernels page once I found them.

What this bug is about is that when you somehow get a full /boot (from whatever cause), the updates stop working. Since /boot is a finite resource, it would be nice if the system could recover either automatically, or in an easy-to-initiate way.

Thank you for my free operating system :-)

[Jarno Suni (jarnos)]

Actually this bug report is not about updates stopping working. This is about not being able to remove a kernel with related packages completely by usual ways due to a specific reason, if /boot becomes full enough. There are other bug reports that call for bigger /boot partition or an automatic way of removing unused kernels.

You are welcome. Feel free to donate at https://launchpad.net/linux-purge