build a fully generic initrd.img without modules

[Oliver Grawert]

currently our initrd.img on snappy contains modules and is not very generic, this makes porting and maintaining the device tarball harder than it should be, adds massively to the size of the initrd (which the bootloader needs to load into ram during boot, requiring more ram and causing slower boot times if the size is bigger by several megabytes due to the modules) and duplicates the existing modules in the rootfs under /lib/modules.

our initrd should be generic enough that a porter or maintainer of a port does not need to make changes to it for updates of kernel or modules but can just use the binary initrd.img file without modifications.

to not duplicate the modules we should instead ship a "modules.img" file in /boot/$bootloader/a|b/ that the initrd can loop mount on startup.
this has the advantage of having all modules readily available immediately without duplicating any of them and without having to maintain a list of which of them have to be shipped inside the initrd.
once the rootfs partition is mounted the loop mounted img file gets relocated to ${rootmount}/lib/modules via a "mount --move" command call and gets used as usual in the running rootfs.

in case we consider making squashfs built into the kernel a requirement, the modules.img file could be a squashfs which could additionally reduce our disk footprint.

implementing the above should only require a few minimal changes to the existing initramfs scripts and some re-consideration of the partition size for /boot (plus some changes to the device-tarball/kernel-snap build instructions how to generate the modules.img)

[Steve Langasek (vorlon)]

On Sat, Aug 29, 2015 at 08:30:13AM -0000, Oliver Grawert wrote:

> to not duplicate the modules we should instead ship a "modules.img" file
> in /boot/$bootloader/a|b/ that the initrd can loop mount on startup.

The modules that are shipped in the initramfs are, by definition, those
modules that may be required in order for the kernel to access the root
filesystem. If the modules are not required in order to access the root
filesystem, they should not be included in the initramfs in the first place.
If they *are* required in order to access the root filesystem (in one or
more configurations), then providing these modules in a separate file on the
/boot partition which will be accessed *from the initramfs* is not viable
because the /boot partition is on the same device as the root filesystem and
will require the same drivers in order to access it.

The kernel does support concatenating multiple initramfs into a single
image. This may be a workable option for what you're trying to accomplish.
In any case, the filesystem containing the modules for the initramfs needs
to be loaded by the bootloader, not by the kernel.

[Oliver Grawert (ogra)]

would the concaternating approach allow the move mount when switching to the rootfs ?
our readonly filesystems will soon be img files under /boot as well (as i understood one of the short term targets is currently to achieve a 2 partition setup) so i dont see an issue with loop mounting just another img if that helps droppping the duplication of (compressed) 10-15MB of modules inside the initrd and allows us to maintain the modules completely separated from the rootfs snap.

i also understand that we will soon start to use squashfs everywhere for snap content, so a squashfs img feels kind of logical for the modules as well.

i don't understand "In any case, the filesystem containing the modules for the initramfs needs
to be loaded by the bootloader, not by the kernel." is there a technical issue i am missing or do you refer to "best habit" with this ?

if i load the modules.img from inside the running initramfs i'm far past the bootloader, the system-boot partition is currently vfat everywhere on snappy and vfat is compiled into all our kernels, as is loop device and ext3/4 support.
so this would technically be enough to loop mount a modules.img from the initrd.

[Oliver Grawert (ogra)]

hmm, and re-reading all the above, i actually meant system-boot everywhere i wrote /boot indeed (i.e. the partition that gets later mounted under /boot/$bootloader/ ), not the /boot directory/mountpoint ...

[Steve Langasek (vorlon)]

On Sun, Aug 30, 2015 at 08:43:23AM -0000, Oliver Grawert wrote:
> would the concaternating approach allow the move mount when switching to
> the rootfs ?

I'm afraid I don't know the answer to this. I'm aware this functionality
exists in the kernel but we've never used it in practice.

> i don't understand "In any case, the filesystem containing the modules for
> the initramfs needs to be loaded by the bootloader, not by the kernel." is
> there a technical issue i am missing or do you refer to "best habit" with
> this ?

> if i load the modules.img from inside the running initramfs i'm far past
> the bootloader, the system-boot partition is currently vfat everywhere on
> snappy and vfat is compiled into all our kernels, as is loop device and
> ext3/4 support. so this would technically be enough to loop mount a
> modules.img from the initrd.

I don't understand the disconnect here. The only reason for putting kernel
modules in an initramfs is if they're needed to access the rootfs. If you
don't have these modules in the initramfs, you don't have the driver support
you need to access the rootfs. That's the same driver support you need to
access this extra loopback filesystem that you are proposing to put on the
boot partition, because that boot partition is accessed using those same
drivers.

Which means you *can't* mount this image from the initramfs. You have to
have it passed to the kernel from the bootloader.

If you could guarantee that this filesystem was always accessible from the
initramfs then there would be no need to load it in the initramfs, because
they would not be on the critical path and could be loaded from the rootfs
instead.

[Oliver Grawert (ogra)]

"I don't understand the disconnect here. The only reason for putting kernel
modules in an initramfs is if they're needed to access the rootfs."

correct, except this should read "any rootfs" ... we ship all possible filesystem modules as well as all network card modules (for possible netboot) etc ... there is a lot more in a -generic initrd than you need today (my amd64 initrd here shrinks from 19M to 5.8M if i remove the modules)

note that i'm not proposing this change for non snappy installs where we actually might want the ability to boot any kind of any generic rootfs.

on snappy we have a predefined partition and filesystem setup, with all needed filesystems already in the kernel by default. porters with own kernels currently have to open the initrd and replace or remove the modules to get a sane sized file without (for them) useless bloat.
on the phone where we have a similar situation we produce a fully generic binary initrd from a deb package (without any modules) which massively reduces size and maintenance cost (and support for ports that have breakage due to the porter manually tinkering with the initrd).

we could indeed go with a completely module-less initrd for the above, the module.img approach only comes into play if we ever want to offer netboot installs or early access to specific hardware (security hardware for decrypting an encrypted writeable partition during boot comes to mind here) without making the initrd non-generic.

additionally the img format for /lib/modules makes deployment inside a snap a lot easier (you only need to verify the checksum of a single file against the store etc).

the modules.img bit is not cruical for this bug and should probably be a separate wishlist bug, the main focus here should be on producing an actually generic initrd binary that is re-usable everywhere without modification and optimized for size and boot speed.

[Oliver Grawert (ogra)]

the initamfs-tools-ubuntu-core side as well as livecd-rootfs changes have landed for this
if we want to go with split-initrd from thw bootloader only the snappy task remains