layered images result in not rebuilding initrd with uuid set from lb_chroot_hacks

[Sebastien Bacher]

Copying the description from Dimitri wrote via email about the issue

'Here is analysis of the canary image build log

$ grep -e lb_chroot_hacks -e 'skipping chroot_hacks' -e 'installing'
-e 'update-initramfs: Genera' *.txt
P: Begin installing packages (minimal pass)...
update-initramfs: Generating /boot/initrd.img-5.0.0-15-generic

Do minimal pass and generate initrd.

[2019-06-07 08:03:30] lb_chroot_hacks
update-initramfs: Generating /boot/initrd.img-5.0.0-15-generic

lb_chroot_hacks has a lot of code, most of which is not executed as
far as i can tell.
The bits that are executed are ->
1) regenerate initrd with CASPER_UUID set => but at this point casper
is not installed yet!
(with UPDATE_INITRAMFS_OPTIONS set to have casper UUID thing)
Chroot chroot "${UPDATE_INITRAMFS_OPTIONS} update-initramfs -k all -t -u"

2) do some cleanups of files

find chroot/boot -name 'initrd*' -print0 | xargs -r -0 chmod go+r
# Remove cruft
rm -f chroot/boot/initrd*bak*
rm -f chroot/etc/apt/trusted.gpg~
rm -f chroot/etc/group- chroot/etc/passwd-
rm -f chroot/etc/gshadow- chroot/etc/shadow-
rm -f chroot/var/cache/debconf/*-old
rm -f chroot/var/lib/dpkg/*-old
rm -f chroot/var/log/apt/term.log
rm -f chroot/etc/adjtime

...

Now after this, you can see that lb_chroot_hacks are not executed on
any further layers!

P: Begin installing packages (minimal.desktop-minimal-de pass)...
[2019-06-07 08:03:52] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.desktop-minimal-en pass)...
[2019-06-07 08:04:01] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.desktop-minimal-es pass)...
[2019-06-07 08:04:10] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.desktop-minimal-fr pass)...
[2019-06-07 08:04:19] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.desktop-minimal-it pass)...
[2019-06-07 08:04:28] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.desktop-minimal-pt pass)...
[2019-06-07 08:04:37] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.desktop-minimal-ru pass)...
[2019-06-07 08:04:47] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.desktop-minimal-zh pass)...
[2019-06-07 08:04:55] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages
(minimal.no-desktop-minimal-default-languages pass)...
[2019-06-07 08:05:04] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard pass)...
[2019-06-07 08:05:58] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.desktop-de pass)...
[2019-06-07 08:06:14] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.desktop-en pass)...
[2019-06-07 08:06:28] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.desktop-es pass)...
[2019-06-07 08:06:43] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.desktop-fr pass)...
[2019-06-07 08:06:58] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.desktop-it pass)...
[2019-06-07 08:07:13] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.desktop-pt pass)...
[2019-06-07 08:07:29] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.desktop-ru pass)...
[2019-06-07 08:07:44] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.desktop-zh pass)...
[2019-06-07 08:07:59] lb_chroot_hacks
W: skipping chroot_hacks, already done
P: Begin installing packages (minimal.standard.live pass)...
update-initramfs: Generating /boot/initrd.img-5.0.0-15-generic

Here in live layer casper is installed and initrd is regenerated without a uuid!

[2019-06-07 08:08:38] lb_chroot_hacks
W: skipping chroot_hacks, already done

And lb_chroot_hacks is not executed here.

P: Begin installing packages
(minimal.standard.no-desktop-default-languages pass)...
[2019-06-07 08:08:54] lb_chroot_hacks
W: skipping chroot_hacks, already done''

[Sebastien Bacher (seb128)]

Dimitri also wrote that post the hangout discussing the issue

'We should make kernel/linux install be done in the is_live_layer only. Including regenerating the initrd as done in hacks, and rip it out.

However, we also need to run chroot_hacks in every layer, to ensure they are squeaky clean:
- empty /var/log
- empty /run/*
- empty /dev/*
- no mlocate database'

[Dimitri John Ledkov (xnox)]

I've provided coaching to describe the problem and how to fix it.

The code fixes are to be done in live-build, rather than in casper, given that layered images incorrectly generate layers without executing the right hooks at the right layer...

Didier and jibel had complete description of what is happening with interactive debugging in a hangout.

@WillCooke @PatGaughen => can you please schedule live-build work to fix layers generation?