Kubuntu 24.04 daily builds - Install randomly hangs/fails/crashes on systems with less than ~3GB of RAM

[Eugene San]

Since around introduction of Kernel 6.8 to Daily builds, installation of Kubuntu / Ubuntu / Ubuntu-Mate builds randomly hangs(mostly), crashes or fails on systems with less than 3GB of RAM.

When the system hangs, VM indicates heavy CPU and CDROM usage. My bet is on Kernel killing itself trying to "juggle" data between OverlayFS and Filesystem cache without a way to swap-out since someone decided that removing zram-config from livefs was a good idea.

When the system doesn't hang, I can see kernel OOM killing random processes like calamares, cryptsetup etc.

I had 100% success of installing Kubuntu 24.04 daily on VMs with 3GB, 75% with 2.5GB and almost 0% on 2GB VMs.
Lvecd installers prior to Linux 6.8 had no issues with 2GB VMs.

Once the system is installed it functions satisfactory with 2GB.

I believe adding zram-config to livefs/casper would allow installation on systems with only 2GB of RAM.

Tested over a couple of months with many snapshots of 24.04 Daily builds for Ubuntu, Kubutu and Ubuntu-Mate on VirtualBox and KVM/Qemu with default settings except enabled EFI.

PS,
The issue is not unique to calamares but I am filing it here since that's what I was asked to do by people on #kubuntu-devel and to be honest at the moment I don't care about anything but being able to test Kubuntu images on small VMs.

P.P.S
Specific example #1 (bad):
* Kubuntu snapshot from http://cdimage.ubuntu.com/kubuntu/daily-live/20240418.1/
* VirtualBox 7.1.16 - VM - Default Ubuntu Profile, Enable EFI, Adjust RAM to 2.5GB
* Boot LiveCD, Choose Try option
* Start Install with default settings
* Observe Calamares existing mid install without any visual indication
* Logs show OOM-Killer killing cryptsetup and Calamares

Specific example #2 (Good):
* Kubuntu snapshot from http://cdimage.ubuntu.com/kubuntu/daily-live/20240418.1/
* VirtualBox 7.1.16 - VM - Default Ubuntu Profile, Enable EFI, Adjust RAM to 2.5GB
* Boot LiveCD, Choose Try option
* >> Install zram-config, start zram-config service <<
* Start Install with default settings
* Observe Calamares finishing install

[Chris Guiver (guiverc)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 2062428

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

No clues as to product are provided (three flavors use calamares, though Kubuntu is used as example so it maybe Kubuntu but we don't know; is this 24.04? as its not said though mentioned too), nor clues as to what ISO was used, nor package version details. It's likely nothing can be done without this detail; as package versions have changed multiple times in the last few days & no useful data (no logs etc) were provided. Please in future use `ubuntu-bug`... which can be run even from terminal even during operation (or calamares-hang).

If your system has already gone (ie. I assume it was LIVE and thus after failure you shutdown box), some detail can be added to the report if you BOOT the same ISO on the SAME HARDWARE and run `apport-collect 2062428` as requested there. This will populate most missing fields, even if no calamares-log will be attached (you'd need to re-run the install & have the same issue to get that).

As currently little detail is provided, I'll mark this bug as incomplete. If you do run `apport-collect`, please change the status back to "New". Thank you for filing the bug report in attempts to make Ubuntu better.

[Eugene San (eugenesan)]

@guiverc

Thank you for pointing deficiencies in my reporting.
I've added additional context.

Unfortunately, I probably won't be able to provide apport reporting since (1) Most of the times the system hard locks. (2) Since I've seen it fail dozens of times, in order to save time I either increase VM's RAM or install Zram before starting install.

[ԜаӀtеr Ⅼарсһуnѕkі (wxl)]

If this happens in Ubuntu, it’s not Calamares. Are you suggesting that the new kernel may have some part to play?

[Steve Langasek (vorlon)]

Walter, you assigned this to livecd-rootfs, but that's a dead end. The memory consumption of the livecd is largely a property of the software it runs. Ubuntu and Kubuntu might both fail now on a system with 3GiB of RAM, but any fix to get under that threshold is likely to be flavor-specific and not a function of the image build system.

This is not necessarily a calamares issue, but I'm reopening a task there so that it's somewhat on the radar of Kubuntu folks.

[ԜаӀtеr Ⅼарсһуnѕkі (wxl)]

Steve, since zram-config solves the issue, my thinking is we could make it a default for livecd-rootfs.

Anyways, like I said, it's not a Calamares issue, so pointing it at kubuntu-meta so they can choose to add it to their seed if they like.

FWIW Lubuntu has seeded it for years.

[Eugene San (eugenesan)]

@wxl
I am not sure if the Kernel change is the sole reason, but I first encountered the issue when Kernel switched from 6.6 to 6.8.

Maybe it was just a tipping point when memory consumption became too high for 2GB systems and later for 2.5GB.

Also I am not sure when zram-config was removed from manifest but it for sure didn't help.

[ԜаӀtеr Ⅼарсһуnѕkі (wxl)]

What's curious is that at least by commit message, there seems to be no mention of zram at all in the Kubuntu seeds:
https://git.launchpad.net/~kubuntu-dev/ubuntu-seeds/+git/kubuntu/log/?qt=grep&q=zram

[Eugene San (eugenesan)]

@wxl
People on #kubuntu-dev decided to skip zram for now since they wanted for issue to be recognized as "release critical " for that change.

You can see more details at the end of https://irclogs.ubuntu.com/2024/04/18/%23kubuntu-devel.html

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.