Subiquity silent on "Installing system" view on Jammy

[Paride Legovini]

When installing Jammy with either:

 subiquity 21.10.3+git136.cde71504 latest/stable/ubuntu-22.04
 subiquity 21.12.2 latest/edge

the "Installing system" view remains blank, even if the installation actually proceeds in background (see the attached screenshot). This isn't (can't) be spotted by our current CI.

Steps to reproduce: try to install from an amd64 ISO.

[Michael Hudson-Doyle (mwhudson)]

Hmm. I wonder if the systemd journal format has changed?

[Michael Hudson-Doyle (mwhudson)]

Yes, I think the removal of patches/debian/Keep-journal-files-compatible-with-older-versions.patch in https://salsa.debian.org/systemd-team/systemd/-/commit/7ce044098c120b25cd3a5a0f3173fb446ad71873 is to blame for this.

We could see about getting that patch reinstated, or we could configure systemd-journald to run with SYSTEMD_JOURNAL_KEYED_HASH=0 in the live session...

[Michael Hudson-Doyle (mwhudson)]

Ah so it's not only that, it's that systemd in jammy now defaults to zstd compression in the journal, which is not supported by the focal libsystemd that python3-systemd is linked too. If I disable both keyed hashes and compression in the journal in the live session, it works again... but that's a bit of a hack. Not sure what the alternative is though.

[Lukas Märdian (slyon)]

Zstd compression by default and hash table hardening has been introduced in
systemd v246, up to now this has been disabled via a systemd distro patch.

I guess we could reinstate this patch, at least in 22.04 (then drop again in
22.10). This would keep the compatibility for the LTS -> LTS upgrade.

But IIUC as long as subiquity is using the core20 base snap this will break
again in the future as soon as the patch is dropped.

I am in favor of fixing this at runtime inside the live session by using some
drop-in configuration that you described above, as this would only affect the
live session and keep compatibility as long as needed, independently of systemd
itself. Would be the best place/package to put this config?

/etc/systemd/journald.conf.d/no-zstd.conf:
```
[Journal]
Compress=false
```

/etc/systemd/system/systemd-journald.service.d/no-hardening.conf:
```
[Service]
Environment="SYSTEMD_JOURNAL_KEYED_HASH=0"
```

A 3rd option would be to backport zstd compression support to Focal's systemd
v245 (and still disabling hash table hardening at runtime). But that sounds
risky and is the least favourable option IMO.

[Michael Hudson-Doyle (mwhudson)]

Yeah I guess we should disable the compression / keyed hash in the live session. Its a bit unfortunate that we can't configure the journal to use xz compression or something compatible with focal but oh well.

On a side note, maybe we should disable the persistent journal in the live session too -- journald saving the journal to "disk" is just going to waste memory I assume ...

[Michael Hudson-Doyle (mwhudson)]

Whoops dropped the ball here. Fix here, hopefully https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/414846