Bump minimum /boot size in ubuntu-release-upgrader

[Bump]

System information:
ubuntu 23.04 x86_64
6.2.0-24-generic

Upgrading the system:

$ sudo dpkg --configure -a
Setting up linux-firmware (20230323.gitbcdcfbcf-0ubuntu1.2) ...
update-initramfs: Generating /boot/initrd.img-6.2.0-24-generic
I: The initramfs will attempt to resume from /dev/dm-2
I: (/dev/mapper/ubuntu--vg-swap_1)
I: Set the RESUME variable to override this.
zstd: error 70 : Write error : cannot write block : No space left on device
E: mkinitramfs failure zstd -q -1 -T0 70

Check space left (none):
$ df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/sdb2 707M 693M 0 100% /boot

What is required is to remove older kernels (and purge them most likely). List all the installed kernels except the current one:
$ dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d'
linux-headers-6.2.0-23
linux-headers-6.2.0-23-generic
linux-headers-6.2.0-25
linux-headers-6.2.0-25-generic
linux-image-6.2.0-23-generic
linux-libc-dev:amd64
linux-libc-dev:i386
linux-modules-6.2.0-23-generic
linux-modules-6.2.0-25-generic
linux-modules-extra-6.2.0-23-generic
linux-modules-extra-6.2.0-25-generic

Is not possible to uninstall any kernel because fails:
$ sudo apt purge linux-headers-6.2.0-23
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

Reconfigure fails again so some kernel have to be manually removed to free up space:
$ sudo rm /boot/initrd.img-6.2.0-23-generic
$ sudo rm -rf /boot/initrd.img-6.2.0-25-generic.new

Now you can reconfigure:
$

Remove kernels:
$ sudo apt purge linux-headers-6.2.0-23 linux-headers-6.2.0-23-generic linux-headers-6.2.0-25 linux-headers-6.2.0-25-generic linux-image-6.2.0-23-generic linux-modules-6.2.0-23-generic linux-modules-6.2.0-25-generic linux-modules-extra-6.2.0-23-generic linux-modules-extra-6.2.0-25-generic

Reboot:
$ sudo reboot

Upgrade as usual:
$ sudo apt update && sudo apt upgrade

Before upgrading new kernels I need to check for free space and remove older kernels manually. Fix this so there's only one kernel left.

[Julian Andres Klode (juliank)]

We generally don't know how much space is going to be needed. APT only keeps 2 kernel versions around, and the /boot partition is sized accordingly for 3.

If you override this by marking kernels as manually installed, or manually installing other versions you need to ensure that there is enough space for those.

I only see two kernels installed though, so it's not clear to me how you could be running out of space in /boot in the first place.

Are there additional files in there? The kernels shouldn't fill that up.

[Bump (bump55)]

There are 3 kernels installed:
6.2.0-24-generic (running)
linux-image-6.2.0-23-generic
/boot/initrd.img-6.2.0-25-generic.new (file 225MB placed by apt upgrade but remember the upgrade broke).

The /boot partition is sized for 3, but maybe today in 2023. This ubuntu system has been being upgraded for several years, so maybe back in the day a kernel image didn't weight 200+ MB but 100MB.
So now I have the problem that I must resize the partition, because kernels are only getting bigger.

With time, my 707M /boot partition will only be fit for one kernel, maybe ubuntu 26.10 or whatever it be.

[Bump (bump55)]

> APT only keeps 2 kernel versions around
Yes, but when installing a new kernel, it does not remove beforehand the oldest kernel, so there's a situation where there are 3 kernels installed at the same time, until you reboot and run upgrade again.

[Julian Andres Klode (juliank)]

Right so it seems we forgot to bump the size requirements in ubuntu-release-upgrader when bumping them in the installer in bug 1959971, you probably should not have been able to upgrade. Albeit the size it seems is determined from the initramfs that is in /boot.

We also don't seem to acccount for system.map which is about 10MB per kernel

Unfortunately there isn't much else we can do here. You can of course opt to

1) set MODULES=dep in /etc/initramfs-tools/initramfs.conf - this will mean that you can't replace your mainboard or swap your disk into a different machine as the initrd will only include modules relevant to the device the system is running on

2) specify COMPRESS=xz or preferably if available a higher COMPRESSLEVEL=1 in the same file such that compression is higher.

I don't know why your initramfs are that big, presumably you are using nvidia gpus? With 3 kernels on my /boot and no out-of-kernel drivers I use 438MB.

[Bump (bump55)]

I think this install was performed mid-2018, judging by my bug report: https://bugs.launchpad.net/ubuntu/+bug/1773633

[Bump (bump55)]

I was able to upgrade because left only 1 installed kernel just before "update-manager -d". Probably I should have opened the issue by then...

[Bump (bump55)]

These are the 'linux-' packages which I have installed:

binutils-x86-64-linux-gnu/lunar-updates,lunar-security,now 2.40-2ubuntu4.1 amd64 [installed,automatic]
linux-base/lunar,lunar,lunar,lunar,lunar,lunar,now 4.5ubuntu9 all [installed,automatic]
linux-firmware/lunar-updates,lunar-updates,lunar-security,lunar-security,now 20230323.gitbcdcfbcf-0ubuntu1.2 all [installed,automatic]
linux-headers-6.2.0-24-generic/lunar-updates,lunar-security,now 6.2.0-24.24 amd64 [installed,automatic]
linux-headers-6.2.0-24/lunar-updates,lunar-updates,lunar-security,lunar-security,now 6.2.0-24.24 all [installed,automatic]
linux-image-6.2.0-24-generic/lunar-updates,lunar-security,now 6.2.0-24.24 amd64 [installed,automatic]
linux-libc-dev/lunar-updates,lunar-security,now 6.2.0-25.25 amd64 [installed,automatic]
linux-libc-dev/lunar-updates,lunar-security,now 6.2.0-25.25 i386 [installed,automatic]
linux-modules-6.2.0-24-generic/lunar-updates,lunar-security,now 6.2.0-24.24 amd64 [installed,automatic]
linux-modules-extra-6.2.0-24-generic/lunar-updates,lunar-security,now 6.2.0-24.24 amd64 [installed,automatic]
linux-sound-base/lunar,lunar,lunar,lunar,lunar,lunar,now 1.0.25+dfsg-0ubuntu7 all [installed,automatic]
selinux-utils/lunar,lunar,lunar,now 3.4-1build4 amd64 [installed,automatic]
syslinux-common/lunar,lunar,lunar,lunar,lunar,lunar,now 3:6.04~git20190206.bf6db5b4+dfsg1-3ubuntu1 all [installed,automatic]
util-linux-extra/lunar,lunar,lunar,now 2.38.1-4ubuntu1 amd64 [installed,automatic]

[Bump (bump55)]

I'm using nvidia drivers.
According to stat, /boot was birth in 2018-11

sudo fdisk -l /dev/sdb2 (/boot mount point)
Disk /dev/sdb2: 735,99 MiB, 771740160 bytes, 1507305 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes

df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/sdb2 707M 252M 405M 39% /boot

[Bump (bump55)]

ls -RSs1pq --block-size=1 /boot/
/boot/:
total 254656512
231763968 initrd.img-6.2.0-24-generic
 13856768 vmlinuz-6.2.0-24-generic
  8155136 System.map-6.2.0-24-generic
   278528 config-6.2.0-24-generic
   147456 memtest86+x64.efi
   147456 memtest86+x64.bin
   143360 memtest86+ia32.efi
   139264 memtest86+ia32.bin
    16384 lost+found/
     4096 efi/
     4096 grub/
        0 initrd.img
        0 initrd.img.old
        0 vmlinuz
        0 vmlinuz.old
ls: cannot open directory '/boot/lost+found': Permission denied
ls: cannot open directory '/boot/efi': Permission denied

/boot/grub:
total 2445312
2396160 unicode.pf2
  20480 x86_64-efi/
  12288 grub.cfg
   4096 fonts/
   4096 locale/
   4096 grubenv
   4096 gfxblacklist.txt

/boot/grub/x86_64-efi:
total 3584000
180224 normal.mod
147456 core.efi
147456 grub.efi
135168 net.mod
 81920 zstd.mod
 81920 regexp.mod
 61440 gfxmenu.mod
 61440 zfs.mod
 49152 bsd.mod
 49152 functional_test.mod
 49152 legacycfg.mod
 45056 mpi.mod
 45056 xnu.mod
 36864 gcry_twofish.mod
 32768 syslinuxcfg.mod
 28672 video_fb.mod
 28672 gcry_camellia.mod
 28672 btrfs.mod
 28672 relocator.mod
 28672 file.mod
 28672 ehci.mod
 24576 multiboot2.mod
 24576 ahci.mod
 24576 gcry_whirlpool.mod
 24576 multiboot.mod
 24576 linux.mod
 24576 terminfo.mod
 24576 luks2.mod
 24576 gcry_rijndael.mod
 20480 xzio.mod
 20480 font.mod
 20480 pgp.mod
 20480 chain.mod
 20480 cryptodisk.mod
 20480 gfxterm.mod
 20480 gcry_serpent.mod
 20480 gcry_des.mod
 16384 legacy_password_test.mod
 16384 ohci.mod
 16384 usb.mod
 16384 acpi.mod
 16384 ntfs.mod
 16384 gcry_cast5.mod
 16384 serial.mod
 16384 reiserfs.mod
 16384 linuxefi.mod
 16384 gcry_seed.mod
 16384 gcry_tiger.mod
 16384 diskfilter.mod
 16384 efi_gop.mod
 16384 gzio.mod
 16384 iso9660.mod
 16384 udf.mod
 12288 gcry_crc.mod
 12288 lvm.mod
 12288 hfsplus.mod
 12288 usbms.mod
 12288 macho.mod
 12288 xfs.mod
 12288 png.mod
 12288 zfsinfo.mod
 12288 jpeg.mod
 12288 efinet.mod
 12288 hdparm.mod
 12288 hfs.mod
 12288 uhci.mod
 12288 f2fs.mod
 12288 video_colors.mod
 12288 squash4.mod
 12288 nilfs2.mod
 12288 mmap.mod
 12288 http.mod
 12288 bfs.mod
 12288 geli.mod
 12288 loadenv.mod
 12288 video_cirrus.mod
 12288 gcry_blowfish.mod
 12288 jfs.mod
 12288 ata.mod
 12288 video.mod
 12288 linux16.mod
 12288 ext2.mod
 12288 fat.mod
 12288 signature_test.mod
 12288 exfat.mod
 12288 gettext.mod
 12288 smbios.mod
 12288 zfscrypt.mod
 12288 hashsum.mod
 12288 video_bochs.mod
 12288 afs.mod
 12288 setpci.mod
 12288 ldm.mod
  8192 affs.mod
  8192 div_test.mod
  8192 gcry_rmd160.mod
  8192 sfs.mod
  8192 ufs1_be.mod
  8192 ufs1.mod
  8192 ufs2.mod
  8192 gfxterm_menu.mod
  8192 extcmd.mod
  8192 gcry_sha1.mod
  8192 tftp.mod
  8192 pata.mod
  8192 test.mod
  8192 halt.mod
  8192 nativedisk.mod
  8192 parttool.mod
  8192 tpm.mod
  8192 lspci.mod
  8192 efi_uga.mod
  8192 lsacpi.mod
  8192 scsi.mod
  8192 crypto.mod
  8192 tga.mod
  8192 lzopio.mod
  8192 at_keyboard.mod
  8192 terminal.mod
  8192 luks.mod
  8192 cmp_test.mod
  8192 keylayouts.mod
  8192 ls.mod
  8192 elf.mod
  8192 gcry_sha512.mod
  8192 minicmd.mod
  8192 minix3_be.mod
  8192 usb_keyboard.mod
  8192 minix2_be.mod
  8192 minix3.mod
  8192 cbfs.mod
  ...

[Steve Langasek (vorlon)]

The purpose of the size check in u-r-u is to ensure that we don't fail mid-upgrade due to lack of disk space on /boot.

But AIUI that's not what happened here to the bug submitter; this is an upgrade within an Ubuntu series between individual kernel packages.

We should NOT have a check in u-r-u that refuses upgrades when /boot is less than the "ideal" size, only when we expect with a reasonable degree of confidence that /boot is too small to let the upgrade proceed.

[Brian Murray (brian-murray)]

You can confirm the release you originally installed on the system by looking at /var/log/installer/media-info. An example:

 $ cat /var/log/installer/media-info
Ubuntu-Server 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)

[Bump (bump55)]

$ cat /var/log/installer/media-info
Ubuntu 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.3)

[Benjamin Drung (bdrung)]

We have a bunch of bugs to decrease initramfs size and to make it faster:

* bug #2028567: initramfs-tools: Do not re-compress compressed kernel modules and firmware files
* bug #2028568: Ship kernel modules compressed
* bug #1942260: compress firmware in /lib/firmware
* bug #2028571: klibc-utils contains identical binaries
* bug #2028574: Duplicate firmware files in /lib/firmware

This started on ubuntu-devel: https://lists.ubuntu.com/archives/ubuntu-devel/2023-July/042652.html