Default /boot size is too small on xgene/uboot systems

[Po-Hsu Lin]

Issue found on ARM64 moonshot nodes

Steps to reproduce:
 1. Deploy Focal on a moonshot ARM64 node
 2. Enable proposed, install linux-generic-hwe-20.04 and reboot
 3. Install udev package from proposed

It will fail with:
$ sudo apt install libudev1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  udev
The following packages will be upgraded:
  libudev1 udev
2 upgraded, 0 newly installed, 0 to remove and 43 not upgraded.
Need to get 1390 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 udev arm64 245.4-4ubuntu3.6 [1319 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libudev1 arm64 245.4-4ubuntu3.6 [70.4 kB]
Fetched 1390 kB in 2s (872 kB/s)
(Reading database ... 112340 files and directories currently installed.)
Preparing to unpack .../udev_245.4-4ubuntu3.6_arm64.deb ...
Unpacking udev (245.4-4ubuntu3.6) over (245.4-4ubuntu3.5) ...
Preparing to unpack .../libudev1_245.4-4ubuntu3.6_arm64.deb ...
Unpacking libudev1:arm64 (245.4-4ubuntu3.6) over (245.4-4ubuntu3.5) ...
Setting up libudev1:arm64 (245.4-4ubuntu3.6) ...
Setting up udev (245.4-4ubuntu3.6) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for systemd (245.4-4ubuntu3.5) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for initramfs-tools (0.136ubuntu6.4) ...
update-initramfs: Generating /boot/initrd.img-5.8.0-51-generic
flash-kernel: installing version 5.8.0-51-generic
Generating kernel u-boot image... done.
Taking backup of uImage.
Installing new uImage.
WARNING: e4defrag of file //boot/uImage failed.
WARNING: Try freeing up disk space in /boot and retrying flash-kernel.
Generating initramfs u-boot image... done.
Taking backup of uInitrd.
Installing new uInitrd.
mv: error writing '//boot/uInitrd': No space left on device
run-parts: /etc/initramfs/post-update.d//flash-kernel exited with return code 1
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned er
ror exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

If you watch the df -h output closely, you will see the /boot filled up gradually until it's full:
/dev/sda1 488M 460M 0 100% /boot

This issue does not exist on other ARM64 nodes (wright / starmie) but just Moonshots. Nor on AMD64 KVM.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.8.0-50-generic 5.8.0-50.56~20.04.1
ProcVersionSignature: User Name 5.8.0-50.56~20.04.1-generic 5.8.18
Uname: Linux 5.8.0-50-generic aarch64
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: arm64
CasperMD5CheckResult: skip
Date: Wed Apr 28 11:52:50 2021
SourcePackage: linux-signed-hwe-5.8
UpgradeStatus: No upgrade log present (probably fresh install)

[dann frazier (dannf)]

I don't see any evidence that this is an issue with udev itself. Rather, udev upgrades cause update-initramfs to run, which triggers flash-kernel to generate u-boot-bootable files (uImage, uInitrd), which will also create a backup of the existing boot files (uImage->uImage.bak and uInitrd->uInitrd.bak) for recovery purposes. I'm guessing you have enough kernels installed that there just isn't room for these .bak files in /boot.

Some suggestions:
  1) Assuming you don't require a fallback kernel, remove /boot/uImage.bak and /boot/uInitrd.bak before installing new kernels. This will give flash-kernel more space to work with.
  2) If you have more than 2 kernels installed (current + next), try removing older kernels before installing new ones.
  3) Create a larger /boot partition.

[Po-Hsu Lin (cypressyew)]

There are only two kernels on the system:
1. 5.4.0-72.80
2. 5.8.0-51.57~20.04.1

And after the kernel installation everything is fine, there is no problem to boot the 5.8.0-51.57~20.04.1.
Processing triggers for linux-image-5.8.0-51-generic (5.8.0-51.57~20.04.1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.8.0-51-generic
flash-kernel: deferring update (trigger activated)
/etc/kernel/postinst.d/zz-flash-kernel:
flash-kernel: deferring update (trigger activated)
Processing triggers for flash-kernel (3.103ubuntu1~20.04.1) ...
flash-kernel: installing version 5.8.0-51-generic
Generating kernel u-boot image... done.
Taking backup of uImage.
Installing new uImage.
Generating initramfs u-boot image... done.
Taking backup of uInitrd.
Installing new uInitrd.
Generating boot script u-boot image... done.
Taking backup of boot.scr.
Installing new boot.scr.

However install the udev package that triggers update-initramfs and it will cause this failure immediately.
Looks like it's the initrd.img-5.8.0-51-generic.new that's growing.

Maybe the solution is to create a larger /boot partition as you mentioned.

[Po-Hsu Lin (cypressyew)]

Also as mentioned in MM earlier today, as we just need to test it for scalingstack, we're going to test it just for Xenial (X-4.4 and X-4.15). I am not sure if it's worth it to debug this.
Thanks

[dann frazier (dannf)]

The only real bug I see here is that curtin's default partitioning scheme does not create a large enough /boot for this platform. I'll therefore mark this as impacting curtin. IMO, the minimum /boot size for this platform should be able to hold 2 concurrent kernel installs - the current one, and one update. In order to do this, it would need space to hold 4 kernel images and 4 initrds: current and next vmlinuz/initrd.img + uImage/uInitrd and uImage.bak/uInitrd.bak. Also, given that files are first copied into /boot under a temp name then renamed, we also need space to hold the temporary file - the largest of which will be an initrd. Current focal kernels are 11M, and initrds are 82M, and there's about 12M of other content in /boot (System.maps, configs), so we'd need at least:

4 * 11 + 5 + 82 + MAX(11, 82) + 12 = 548M.

@cypressyew: For our testing purposes, I've preseeded a 2G /boot for all of the m400s in our MAAS.