vmlinuz is very large in arm64 -raspi2

[Michael Hudson-Doyle]

The debs are close to the same size:

mwhudson@ringil:~/Downloads$ ls -l linux-image-4.15.0-1030-raspi2_4.15.0-1030.32_arm64.deb
-rw-rw-r-- 1 mwhudson mwhudson 5398652 Dec 12 10:42 linux-image-4.15.0-1030-raspi2_4.15.0-1030.32_arm64.deb
mwhudson@ringil:~/Downloads$ ls -l linux-image-4.15.0-1030-raspi2_4.15.0-1030.32_armhf.deb
-rw-rw-r-- 1 mwhudson mwhudson 6640960 Dec 12 10:42 linux-image-4.15.0-1030-raspi2_4.15.0-1030.32_armhf.deb

But the vmlinuz on arm64 is several times larger:

$ dpkg-deb -c linux-image-4.15.0-1030-raspi2_4.15.0-1030.32_armhf.deb
-rw------- root/root 6633048 2018-12-07 06:38 ./boot/vmlinuz-4.15.0-1030-raspi2
$ dpkg-deb -c linux-image-4.15.0-1030-raspi2_4.15.0-1030.32_arm64.deb
-rw------- root/root 23914504 2018-12-07 06:38 ./boot/vmlinuz-4.15.0-1030-raspi2

This forced us to make the boot partition larger for the raspi3 b+ arm64 image we made.

[Michael Hudson-Doyle (mwhudson)]

It seems the vmlinuz is not compressed on arm64 which would explain the difference. If u-boot on arm64/rpi3b+ supports a compressed image that would suggest the fix is fairly simple.

[dann frazier (dannf)]

fyi, -generic ships an Image.gz as vmlinuz, and this boots fine on X-Gene/u-boot based systems. (Trivia: the uncompressed Image grew too large for these systems, which is what precipitated the switch to Image.gz in -generic)

[Michael Hudson-Doyle (mwhudson)]

So it seems after some testing that if one changes the boot.scr to this:

fdt addr ${fdt_addr_r}
fdt get value bootargs /chosen bootargs
setenv kernel_addr_r 0x01000000
setenv ramdisk_addr_r 0x03100000
fatload mmc 0:1 ${ramdisk_addr_r} vmlinuz
unzip ${ramdisk_addr_r} ${kernel_addr_r} ${filesize} || cp {ramdisk_addr_r} ${kernel_addr_r} ${filesize}
fatload mmc 0:1 ${ramdisk_addr_r} initrd.img
setenv initrdsize $filesize
booti ${kernel_addr_r} ${ramdisk_addr_r}:${initrdsize} ${fdt_addr_r}

then you can simply compress the vmlinuz in place with gzip and everything continues to work.

[Michael Hudson-Doyle (mwhudson)]

That said we just shipped a bunch of images with the original boot.scr to a bunch of people and I don't think we have a mechanism for updating the boot.scr on these images, although I would love to be wrong about that.

[Łukasz Zemczak (sil2100)]

@mwhudson I think those can get updated by flash-kernel actually. Not sure if that works though as I did not have any occasion to test that. But in theory the flash-kernel triggers, besides updating the kernel on the boot partition, should also update the boot.scr script every time.

That being said, this would then require us to also handle the gzipping of the kernel on upgrades, which is why I said all this would be relatively simple but would require some changes. We'd need to modify flash-kernel to gzip the new kernel before performing the upgrade on the boot partition. Sounds like a trivial addition, but might need some testing.

[Dimitri John Ledkov (xnox)]

Can kernel team elaborate on this ticket?

Cause as far as I can tell, we can change flash-kernel / boot.scr in eoan, and kernels do support compressed images on armhf.... And rewrite boot.scr on upgrade to eoan+

Is there something that I'm missing from shipping compressed kernels here? Something specific to armhf? arm64?

[Paolo Pisati (p-pisati)]

Armhf is already using zImage (and the kernel decompress itself once we jump to it), so this is an arm64-only issue and no, there's no reason to have vmlinuz uncompressed as long as the bootloader can handle it - i'll send a fix to switch to Image.gz in Eoan.

[Łukasz Zemczak (sil2100)]

Paolo - so last time we talked about this, you mentioned that the arm64 kernel doesn't support decompressing the kernel on boot like the armhf one. Is that still the case? Since I have marked the linux-raspi2 task as 'Won't Fix' because of this. If yes, is there any way of having the same functionality of uncompressing the kernel on boot for arm64 as we have for armhf?

I mean, I guess we could still switch compression on and try to gunzip the kernel in u-boot, sure. But I'm worried that without a built-in decompression mechanism we might be breaking some user edge-cases.

I left the linux-raspi2 task as Won't Fix also because I originally thought that we'd be doing the compression of the kernels manually on boot-partition content upgrades. But maybe this way is better indeed. Of course it would be best if we just had the same situation here as with armhf...

[Paolo Pisati (p-pisati)]

Yes, arm64 kernels don't have any decompressor build-in (and they'll never get one) - i'm fine with your idea of compressing / decompressing kernels upon installation (to avoid regression in corner cases) but i would like (at some point in the future) to switch to compressed kernels though (next LTS? F?).

[Łukasz Zemczak (sil2100)]

@p-pisati After discussing it with Steve and Dave, we have decided that we'd like to switch to compressed images by default in linux-raspi2. Yes, we might break some edge cases here, but in overall we only officially support our own raspi images which we will be making sure are still working correctly after this change.

That being said, this will require a bit of coordination. The biggest challenge of such a change is making sure our ubuntu-core arm64 images still work with the new kernel. We officially introduced the arm64 raspi arch with 18.04.2 for core, so we'd have to find a way to not 'brick' existing images in the field. With classic images it's quite trivial, as we just need to set the dependencies right, but for core - there are no 'snap dependencies' we can use.

[Paolo Pisati (p-pisati)]

On the kernel side, the switch to a compressed kernel is a trivial change:

diff --git a/debian.raspi2/rules.d/arm64.mk b/debian.raspi2/rules.d/arm64.mk
index ada61b8..9f82433 100644
--- a/debian.raspi2/rules.d/arm64.mk
+++ b/debian.raspi2/rules.d/arm64.mk
@@ -3,7 +3,7 @@ build_arch = arm64
 header_arch = arm64
 defconfig = defconfig
 flavours = raspi2
-build_image = Image
+build_image = Image.gz
 kernel_file = arch/$(build_arch)/boot/Image
 install_file = vmlinuz
 no_dumpfile = true

Feel free to send such a change to the kernel ml once userspace issues (classic and core) are sorted out.

[Łukasz Zemczak (sil2100)]

Thanks Paolo! We'll do that once we have everything in place. For now Dave (the person responsible for Pi images) will do some tests to make sure we're able to handle all the upgrade cases. But what we think will have to happen is that we'd probably ask not to backport this change into bionic and xenial. With snaps lacking dependency handling, it would be too easy to actually cause machines to stop working - even if in our gadget snaps we support loading of both compressed and uncompressed kernels (with determining which case is currently needed), there's still the case where someone could update only the kernel and leave an old gadget snap. So we'd probably have to stick to uncompressed kernels for core18, only switching to .gz in core20.

[Dave Jones (waveform)]

I've put together an update for flash-kernel that handles booting an uncompressed image (to support existing arm64 installs), a self-compressed image (to support existing armhf installs), and a gzip'd image (to support the proposed format). It assumes the kernel is called "vmlinuz" in all cases (because that's currently the case, and I presume will continue to be the case in future).

Just finishing up testing today, but I've already confirmed it works on pi2, pi3, and pi3+ with all (supported) kernel cases - just cm3 and cm3+ to go. The update also switches the pi2 case to using the same boot-script as all the rest. I'll ping Lukasz for a review when everything's confirmed as working.

[Łukasz Zemczak (sil2100)]

The needed flash-kernel changes landed in 3.98ubuntu2, setting to Fix Released. Should we start looking into building compressed linux-raspi2 kernels for eoan?

[Dave Jones (waveform)]

It would be nice to start building compressed arm64 kernels for eoan, but it may have to wait a bit as the core boot-env won't yet handle this correctly (that doesn't matter for core-16, but core-18 has arm64 images).