Installed packages in squasfs are not registered in final installed OS - Ubuntu 22.04

[Roman Spiak]

In the final installed OS the packages that are installed in chroot of unsquashed fs of casper/ubuntu-server-minimal.squashfs are not registered to dpkg. This means that the binary files and libraries are in fact present on the filesystem, however if I run dpkg -l | grep <package> it yields no results. Using mlocate for example throws error message: unknown group: plocate - which suggests binary is present however group under which command runs is not.

[Dan Bungert (dbungert)]

Hi there, thanks for the report.

I'm not sure how plocate ended up on the system in question - it's not part of the seeded packages, and not there when I run the install as suggested. I can't reproduce the above claim using the 22.04.2 live-server ISO. I think you should look into how plocate ended up on the installed system. Is plocate the example package above?

Note that `dpkg -l|grep ^ii|wc -l` yields and answer of 430, so this is certainly not categorically true.

What packages/binary files/libraries are present on the system but not registered to dpkg?

[Roman Spiak (roman-spiak)]

It seems we have a miss-understanding of what exactly am I reporting.

I want to do Install ISO customisation. The way I do the customisation is:

1) extract the ISO image (via 7z)
2) unsquash the file casper/ubuntu-server-minimal.squashfs
3) bind mount /proc /dev /sys to unsqhashed rootfs
4) chroot into unsqhashed rootfs
5) apt-get install mlocate
6) exit
7) mksquash
8) xorriso back to ISO file
9) install OS
10) there's a mlocate binary present however dpkg -l|grep mlocate sees no package

is there a squashfs layout documented for 22.04 server that I can have a link to ? I need to reverse engineer how to unsquash and bind mount these FS parts into a working whole.

Thanks.

[Michael Hudson-Doyle (mwhudson)]

Are you doing a minimal install when you install the resulting system though? If not what I suspect is going on is that there is a /var/lib/dpkg/status file in the ubuntu-server-minimal.ubuntu-server.squashfs file that is masking the changes you are making to the status file in the lower layer. In general modifying the base layer and then overlaying it with a layer that was based on the unmodified base is going to be skating on the edge of reasonable behaviour...

[Roman Spiak (roman-spiak)]

Hi Michael,

please know that in this step:

2) unsquash the file casper/ubuntu-server-minimal.squashfs

I'm unsquashing only ubuntu-server-minimal.squashfs and making changes in this FS layer by installing packages using apt-get command.

I'm installing "ubuntu-server-minimal" option, however I feel ubuntu subiquity installer makes some undocumented (or I was unable to find docs concerning this) magic with these files:

isodir/casper/ubuntu-server-minimal.squashfs
isodir/casper/ubuntu-server-minimal.ubuntu-server.installer.squashfs
isodir/casper/ubuntu-server-minimal.ubuntu-server.installer.generic-hwe.squashfs
isodir/casper/ubuntu-server-minimal.ubuntu-server.squashfs
isodir/casper/ubuntu-server-minimal.ubuntu-server.installer.generic.squashfs

and overlaying them during the actual installation of the resulting OS.

This makes changes in /var/lib/dpkg/status file not present due to this file coming from different squashfs file (other then ubuntu-server-minimal.squashfs).

There's a nice wiki writeup for Ubuntu desktop (that does not follow server FS layout) here: https://help.ubuntu.com/community/LiveCDCustomization

There's also a nice docs dealing with cloud-init for ubuntu here: https://ubuntu.com/server/docs/install/autoinstall however there's no docs I could find how is squasfs over-layered during installation of the actual OS.

Can me point the to a docs that deals with squashfs customisation that ends up in the final OS.

[Michael Hudson-Doyle (mwhudson)]

If you are selecting the minimal install option, then the /casper/ubuntu-server-minimal.squashfs is what is copied to the target system (well, an in memory overlay over that that has had "apt update" run in it). The other layers are stacked for the environment the installer runs in, but that is not what is copied. Certainly nothing in subiquity explicitly touches /var/lib/dpkg/status/

There is not really any documentation on how the squashfs layers are handled, it's mostly an implementation detail. You're going to need to read the source if you want to know all the gory details.

[Roman Spiak (roman-spiak)]

I guess you can close this bug as I was able to work around the problem by having all the customisation happening in cloud-init post-steps section.

There's certainly an overlay of /var/lib/dpkg happening from some other squashfs prior to OS install or there's no clear way to select installation type (minimal/standard) via cloud init or it just did not work for me..

Thanks anyway for the effort to help - appreciated.

[Launchpad Janitor (janitor)]

[Expired for subiquity because there has been no activity for 60 days.]