Can't boot: "error: out of memory." immediately after the grub menu

[Gordon Mckeown]

[Workaround]

Today, 2023-02-14, the following link works to find the workarounds:

https://bugs.launchpad.net/oem-priority/+bug/1842320/comments/125

[Impact]

 * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2.

 * Some real cases from OEM projects:

In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed.

In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file:

```
#0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376
#1 0x000000007dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408
#2 0x000000007dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076)
    at ../../../grub-core/kern/verifiers.c:150
#3 0x000000007dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem",
    type=131076) at ../../../grub-core/kern/file.c:121
#4 0x000000007bcd5a30 in ?? ()
#5 0x000000007fe21247 in ?? ()
#6 0x000000007bc030c8 in ?? ()
#7 0x000000017fe21238 in ?? ()
#8 0x000000007bcd5320 in ?? ()
#9 0x000000007fe21250 in ?? ()
#10 0x0000000000000000 in ?? ()
```

Based on grub_mm_dump, we can see the memory region starvation in <1G addresses:

Type Start End # Pages Attributes
Available 0000000000000000-0000000000086FFF 0000000000000087 000000000000000F
BS_Data 0000000000087000-0000000000087FFF 0000000000000001 000000000000000F
Available 0000000000088000-000000000009EFFF 0000000000000017 000000000000000F
Reserved 000000000009F000-000000000009FFFF 0000000000000001 000000000000000F
Available 0000000000100000-0000000000FFFFFF 0000000000000F00 000000000000000F
LoaderCode 0000000001000000-0000000001021FFF 0000000000000022 000000000000000F
Available 0000000001022000-00000000238A7FFF 0000000000022886 000000000000000F
BS_Data 00000000238A8000-0000000023927FFF 0000000000000080 000000000000000F
Available 0000000023928000-0000000028860FFF 0000000000004F39 000000000000000F
BS_Data 0000000028861000-000000002AB09FFF 00000000000022A9 000000000000000F
LoaderCode 000000002AB0A000-000000002ACF8FFF 00000000000001EF 000000000000000F
BS_Data 000000002ACF9000-000000002B2FAFFF 0000000000000602 000000000000000F
Available 000000002B2FB000-000000002B611FFF 0000000000000317 000000000000000F
BS_Data 000000002B612000-000000002B630FFF 000000000000001F 000000000000000F
Available 000000002B631000-000000002B632FFF 0000000000000002 000000000000000F
BS_Data 000000002B633000-000000002B63CFFF 000000000000000A 000000000000000F
Available 000000002B63D000-000000002B649FFF 000000000000000D 000000000000000F
BS_Data 000000002B64A000-000000002B64EFFF 0000000000000005 000000000000000F
Available 000000002B64F000-000000002B666FFF 0000000000000018 000000000000000F
BS_Data 000000002B667000-000000002D8D5FFF 000000000000226F 000000000000000F
LoaderCode 000000002D8D6000-000000002D8E9FFF 0000000000000014 000000000000000F
BS_Data 000000002D8EA000-000000002D925FFF 000000000000003C 000000000000000F
LoaderCode 000000002D926000-000000002D932FFF 000000000000000D 000000000000000F
BS_Data 000000002D933000-000000002D969FFF 0000000000000037 000000000000000F
BS_Code 000000002D96A000-000000002D973FFF 000000000000000A 000000000000000F
BS_Data 000000002D974000-000000002E377FFF 0000000000000A04 000000000000000F
Available 000000002E378000-000000002E37AFFF 0000000000000003 000000000000000F
...
Reserved 000000003C08B000-000000004192FFFF 00000000000058A5 000000000000000F
ACPI_NVS 0000000041930000-0000000041B2FFFF 0000000000000200 000000000000000F
ACPI_Recl 0000000041B30000-0000000041BFEFFF 00000000000000CF 000000000000000F
BS_Data 0000000041BFF000-0000000041BFFFFF 0000000000000001 000000000000000F
Available 0000000100000000-00000002AB7FFFFF 00000000001AB800 000000000000000F
Reserved 00000000000A0000-00000000000FFFFF 0000000000000060 0000000000000000
Reserved 0000000041C00000-0000000043FFFFFF 0000000000002400 0000000000000000
Reserved 0000000044000000-0000000047FFFFFF 0000000000004000 000000000000000F
Reserved 0000000049400000-00000000495FFFFF 0000000000000200 000000000000000F
Reserved 000000004C000000-000000004FFFFFFF 0000000000004000 0000000000000009
Reserved 0000000050000000-00000000547FFFFF 0000000000004800 0000000000000000
MMIO 00000000C0000000-00000000CFFFFFFF 0000000000010000 8000000000000001
Reserved 00000000FED20000-00000000FED7FFFF 0000000000000060 0000000000000000
MMIO 00000000FF800000-00000000FFFFFFFF 0000000000000800 8000000000001000
...
  LoaderCode: 562 Pages (2,301,952 Bytes)
  LoaderData: 0 Pages (0 Bytes)
...
  Available : 1,917,598 Pages (7,854,481,408 Bytes)

Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages().

As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory.

 * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot).

[Test Plan]
tl;dr: Create an initrd that fails to boot with the old grub and boot it successfully with the new grub.

 * detailed instructions how to reproduce the bug:

1. Any method to grow the initramfs, such as install nvidia-driver.

2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like:

```
$ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file
#!/bin/sh

PREREQ=""

prereqs()
{
        echo "$PREREQ"
}

case $1 in
# get pre-requisites
prereqs)
        prereqs
        exit 0
        ;;
esac

. /usr/share/initramfs-tools/hook-functions
dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500
```

And then update-initramfs

 * After cloning grub2 (2.06-2ubuntu16) from lunar-proposed and built, the results as following:

421M /boot/initrd.img-5.17.0-1021-oem # pass
453M /boot/initrd.img-5.17.0-1024-oem # pass
471M /boot/initrd.img-5.17.0-1024-oem # fail

Only 453M is because verifier will consume the same memory size of initrd in <1G memories.

The loadable initrd will locate at >4G if machine/kernel support it.

[Where problems could occur]

* The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system.
The other change is to modify the “grub-core/kern/efi/mm.c” but it use the original addressing for “arm/arm64/ia64/riscv32/riscv64”.
Thus it should not impact them.

* There are some “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system.

If everything works as expected, then i386 should working good.

If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access.

[Other Info]

 * The package grub2 (2.06-2ubuntu16) is now in lunar-proposed.

---

Upgraded from 19.04 to current 19.10 using "do-release-upgrade -d". Can still boot using the previous 5.0.0-25-generic kernel, but the 5.2.0-15-generic fails to start.

On selecting Ubuntu from Grub, the message "error: out of memory." is immediately shown. Pressing a key attempts to start boot-up but fails to mount root fs.

Machine is HP Spectre X360 with 8GB RAM. Under kernel 5.0.0, free shows the following (run from Gnome terminal):

              total used free shared buff/cache available
Mem: 7906564 1761196 3833240 1020216 2312128 4849224
Swap: 1003516 0 1003516

Kernel packages installed:

linux-generic 5.2.0.15.16 amd64
linux-headers-5.2.0-15 5.2.0-15.16 all
linux-headers-5.2.0-15-generic 5.2.0-15.16 amd64
linux-headers-generic 5.2.0.15.16 amd64
linux-image-5.0.0-25-generic 5.0.0-25.26 amd64
linux-image-5.2.0-15-generic 5.2.0-15.16+signed1 amd64
linux-image-generic 5.2.0.15.16 amd64
linux-modules-5.0.0-25-generic 5.0.0-25.26 amd64
linux-modules-5.2.0-15-generic 5.2.0-15.16 amd64
linux-modules-extra-5.0.0-25-generic 5.0.0-25.26 amd64
linux-modules-extra-5.2.0-15-generic 5.2.0-15.16 amd64

Photo of kernel panic attached.

NVMe drive partition layout (GPT):

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 1050623 1048576 512M EFI System
/dev/nvme0n1p2 1050624 2549759 1499136 732M Linux filesystem
/dev/nvme0n1p3 2549760 1000214527 997664768 475.7G Linux filesystem

$ sudo pvs
  PV VG Fmt Attr PSize PFree
  /dev/mapper/nvme0n1p3_crypt ubuntu-vg lvm2 a-- <475.71g 0

$ sudo lvs
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  root ubuntu-vg -wi-ao---- 474.75g
  swap_1 ubuntu-vg -wi-ao---- 980.00m

Partition 3 is LUKS encrypted. Root LV is ext4.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu7
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: gmckeown 1647 F.... pulseaudio
CurrentDesktop: ubuntu:GNOMEDistroRelease: Ubuntu 19.10
InstallationDate: Installed on 2019-08-15 (18 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 003: ID 8087:0a2b Intel Corp.
 Bus 001 Device 002: ID 04f2:b593 Chicony Electronics Co., Ltd HP Wide Vision FHD Camera
 Bus 001 Device 004: ID 046d:c52b Logitech, Inc. Unifying Receiver
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: HP HP Spectre x360 Convertible 13-ae0xx
Package: linux (not installed)
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.0.0-25-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash
ProcVersionSignature: Ubuntu 5.0.0-25.26-generic 5.0.18
RelatedPackageVersions:
 linux-restricted-modules-5.0.0-25-generic N/A
 linux-backports-modules-5.0.0-25-generic N/A
 linux-firmware 1.181
Tags: eoan
Uname: Linux 5.0.0-25-generic x86_64
UpgradeStatus: Upgraded to eoan on 2019-09-02 (0 days ago)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 05/17/2019
dmi.bios.vendor: AMI
dmi.bios.version: F.25
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: 83B9
dmi.board.vendor: HP
dmi.board.version: 56.43
dmi.chassis.type: 31
dmi.chassis.vendor: HP
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAMI:bvrF.25:bd05/17/2019:svnHP:pnHPSpectrex360Convertible13-ae0xx:pvr:rvnHP:rn83B9:rvr56.43:cvnHP:ct31:cvrChassisVersion:
dmi.product.family: 103C_5335KV HP Spectre
dmi.product.name: HP Spectre x360 Convertible 13-ae0xx
dmi.product.sku: 2QH38EA#ABU
dmi.sys.vendor: HP

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1842320

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

Issue appears to be related to display resolution. Grub was booting into 3840x2160 mode, with the resulting tiny, near-unreadable font. So I set GFXMODE=800x600 and now - unexpectedly - kernel 5.2.0 is able to boot without the memory errors.

Not sure why the high resolution causes an out-of-memory error; a 3840x2160x32 framebuffer should only require ~32MB of RAM.

[MrMEEE (mj-casalogic)]

Same issue here.. workaround works..

[Kai-Heng Feng (kaihengfeng)]

Can you please perform a kernel bisection?

[Steven Mackenzie (z-steven)]

I have a similar issue. I upgraded to 19.10 from 19.04 today.

After grub, I get this message:

"error: out of memory.
Press any key to continue..."

Then kernel panic message (image attached).

I have been unable to successfully boot using the workaround above.

The kernel is shown in grub is 5.3.0-18.
My computer is an Intel NUC (Skull Canyon), 16GB RAM, Intel NVMe, 2 x 2560 x 1440 displays.

[Steven Mackenzie (z-steven)]

Update: My machine boots normally from the USB install media.

[Terry Dawson (tjd-animats)]

ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu8.2
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
DistroRelease: Ubuntu 19.10
HibernationDevice: RESUME=UUID=21a8b99d-8777-4e3e-8fd7-2ad7bb66b24e
InstallationDate: Installed on 2017-01-24 (1068 days ago)
InstallationMedia: Ubuntu 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
IwConfig:
 lo no wireless extensions.

 eno1 no wireless extensions.
Package: linux (not installed)
ProcEnviron:
 LANGUAGE=en_AU:en
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_AU.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-131-generic root=/dev/mapper/ubuntu--vg-root ro plymouth:debug
ProcVersionSignature: Ubuntu 4.4.0-131.157-generic 4.4.134
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-131-generic N/A
 linux-backports-modules-4.4.0-131-generic N/A
 linux-firmware 1.183.3
RfKill:

Tags: eoan
Uname: Linux 4.4.0-131-generic x86_64
UpgradeStatus: Upgraded to eoan on 2019-12-28 (0 days ago)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 08/17/2016
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: KYSKLi70.86A.0041.2016.0817.1130
dmi.board.name: NUC6i7KYB
dmi.board.vendor: Intel Corporation
dmi.board.version: H90766-406
dmi.chassis.type: 3
dmi.chassis.vendor: Intel Corporation
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrKYSKLi70.86A.0041.2016.0817.1130:bd08/17/2016:svn:pn:pvr:rvnIntelCorporation:rnNUC6i7KYB:rvrH90766-406:cvnIntelCorporation:ct3:cvr1.0:

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

I'd like to echo Steven Mackenzie's report.

The workaround of explicitly configuring GFXMODE does mean that I can hit a key through the "Hit any key to continue" prompt, but the boot still fails. Without the workaround, grub is unresponsive at the "Hit any key.." prompt.

[Terry Dawson (tjd-animats)]

I've opened 1857786 indepdently for my boot issue.

[Ben Hoyt (benhoyt)]

Just noting that I had this same issue on my Dell XPS 9550 laptop, which has a 3840x2160 display. The workaround of setting GFXMODE=800x600 in the grub config worked for me too, thanks! And of course had the nice side effect of actually making the grub menu readable. :-)

[Gordon Mckeown (thefluffyone)]

Just had the problem again after upgrading 20.10 to 21.04.

Kernel 5.8.0-55 boots OK.

Kernel 5.11.0-18 fails with the memory error.

I'm just about to re-add the GFXMODE line to grub, which I hope will (again) work around the problem.

[H (bluesforte)]

I had this issue as well, through ALL of the 5.x kernels. Tried everything mentioned in the thread here, but nothing worked other than going to 4.15 kernel.
This gave me suspicions it was related to something with the UEFI handoff to OS, so I looked for BIOS updates and sure enough there was one from earlier this year. Updated to the latest BIOS from Intel, and everything boots fine and working now.

[Callum L (little-king-john)]

I have this issue on my Dell XPS 9380. I can boot 5.13.0.35, but 0.37 and 0.39 simply show the error: out of memory.

I have tried changing my GFXMODE to 800x600x24, which hwinfo lists as a supported resolution, but grub.... just ignores it? Same with timeout? It just shows in the 4K native res every time, almost as if it's taunting me.

Eventually the 0.35 kernel will be deleted by a system update, and my system will be bricked. I guess I'm going to need to reinstall, aren't I?

[Paulo (ptsneves)]

This is still valid and the issue is more pernicious because with the latest livecd one cannot change the GFXMODE to 800x600. Re-imaging a new ISO with new grub settings is outside most people's abilities nor is it the intention. This means that Ubuntu's livecd lead to a panic in a very popular laptop model.

[rustyx (rustyx2)]

Had the same issue with Ubuntu 22.04 install ISO flashed onto a USB.

Even tried booting "manually" (press "c" to enter GRUB command prompt)

grub> ls
(proc) (hd0) (hd0,gpt1) (hd0,gpt2) (hd0,gpt3) (hd1) . . .
grub> ls (hd0,gpt1)
        Partition hd0,gpt1: Filesystem type ... - Label 'UBUNTU 22_0' ...
grub> set root=(hd0,gpt1)
grub> linux /casper/vmlinuz
grub> initrd /casper/initrd
error: out of memory.
grub>

Couldn't find any way out of this condition (is there?).

Fortunately I already had GRUB installed on the machine so I could proceed from that one.

[m1st0 (m1st0)]

I had the grub "error: out of memory" issue after upgrading Ubuntu to 22.04 from 21.10. I used a Live USB of Ubuntu to chroot into my system and after many trials and failures, I could get past the issue by changing line 36 in /etc/initramfs-tools/initramfs.conf by using a different compression choice

    #
    # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
    #

    COMPRESS=zstd

However, this was still not a fix as zstd provided a quicker boot previously than any other but was failing now. Yet from there it seemed it was a compression issue. Changed it back to zstd and went to another fix.

I further learned that zstd had defaulted to a lower compression value in /usr/sbin/mkinitramfs at 1 instead of 19 in Ubuntu 22.04. I changed line 196 to 19 compression

    zstd) compress="zstd -q -19 -T0" ;;

Instead of

    zstd) compress="zstd -q -1 -T0" ;;

Problem solved from proper chroot and updating grub2 after changing line 196 of /usr/sbin/mkinitramfs for my system.

[Chris Guiver (guiverc)]

Thank you for reporting this bug to Ubuntu.

Ubuntu 19.10 (eoan) reached end-of-life on July 17, 2020.
Ubuntu 19.04 (disco) reached end-of-life on January 23, 2020.

See this document for currently supported Ubuntu releases:
https://wiki.ubuntu.com/Releases

We appreciate that this bug may be old and you might not be interested in discussing it any more. But if you are then please upgrade to the latest Ubuntu version and re-test. If you then find the bug is still present in the newer Ubuntu version, please add a comment here telling us which new version it is in.

[Christian Mauduit (ufoot)]

@guiverc this is not related to Ubuntu 19.10 or 19.04.

It was initially reported at that time, but it is still open.

I just stumbled on it, ugrading from 21 to 22.04.

Same behavior:

"error: out of memory.
Press any key to continue..."

Then kernel panic message.

I created a bootable USB image -> same problem. Now downloading a 21 image, and see if I can boot and try and tweak grub configuration.

But the problem, despite old, is still happening on the latest version of Ubuntu. And it is pretty bad, your computer is *really* unuable once you're hit.

My hardware is a MSI Prestige 15 A10SC, has a 4k screen, which some users also have apparently.

[Christian Mauduit (ufoot)]

I tried the fix with GFXMODE=800x600 -> it did nothing. I had bigger letters but the same error.

I tried the fix with editing mkinitramfs and replacing 1 by 19 on line 196 to restore the high compression level. Then regenerated the image -> it worked.

For readers, this is not so easy, because when you get there your system is stuck. What I did is download an Ubuntu image 21.10. From there, boot on the USB stick. Open a terminal, and:

--------8<-------------------------------------------
$ sudo su -
# mkdir root
# mkdir boot
# mkdir efi
# cryptsetup luksOpen /dev/nvme0n1p3 nvme0n1p3_crypt
# mount /dev/mapper/vgubuntu-root ./root
# mount /dev/nvme01p1 ./efi
# mount /dev/nvme01p2 ./boot
# mount -B ./boot ./root/boot
# mount -B ./efi ./root/boot/efi
# mount -B ./dev ./root/boot/dev
# mount -B ./dev/pts ./root/boot/dev/pts
# mount -B ./proc ./root/proc
# mount -B ./sys ./root/sys
# chroot ./root
# update-initramfs -u -k all
# update-grub
# exit
# reboot
--------8<-------------------------------------------

^ in the above examples, the logical volume names and crypt devices need to be adapted to your setup, obviously.

I do not know what is the gain of switching that compression level from 19 to 1, but the side effect (not being able to boot at all...) is quite annoying.

Thanks very much @ybdjkfd for the fix, that saved my day (if not my week & month).

[m1st0 (m1st0)]

Thanks @ufoot . It took quite a while to find that issue, but the solution was hidden among many people reporting other solutions and bugs. Remember after the

# chroot ./root

and before the

# update-initramfs -u -k all

You have to edit the file

# /etc/initramfs-tools/initramfs.conf

as noted above. I submitted the bug, but it was marked as duplicate of this anyway. In a discussion elsewhere which didn't seem as related, the issue of speeding boot times was being discussed for Raspberri Pi systems, and the decision seemed to come down to not needing such a high compression ratio. I am unsure if it is limited to MSI hardware, but an unintended consequence of it all was making our MSI systems not boot regardless of not being important to Raspberry Pi systems.

Mine is a MSI GS63VR 6RF Stealth Pro with a 4K screen.

Related: https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-Initramfs-Zstd-Too-High
Related: https://lists.ubuntu.com/archives/ubuntu-devel/2021-December/041726.html

[Dominik Ptaszek (sarquazz)]

Exactly the same problem with GRUB occurs on LG Gram 14 (2021) with Ubuntu 22.04.

In my case it means that after upgrading Ubuntu from 21.10 to 22.04, GRUB no longer boots, showing "error: out of memory.". I had to boot another OS from USB which is not Ubuntu 22.04, because live image have the same problem. I`ve used the image of Ubuntu 20.04 LTS.

CSM/UEFI and Secure boot enabled/disable - tried all and thats not related to this in any case.

After chrooting and regenerating GRUB with GFXMODE set to 800x600, I could finally reach the stage of loading kernel and decrypting the disk. :) It still not booting due to another problems, but thats another case, not related to this problem.

The real problem is that I cannot perform clean installation due to not booting USB live image of Ubuntu 22.04 with "error: out of memory.". For now I've tried to repack squashfs with GFXMODE=800x600 change applied and write that to USB stick, but for now without luck (still the same error) which is strange, because it helped in already installed Ubuntu 22.04. Maybe I missed something - I will dig into this later and let you know. I'm gonna also try @ybdjkfd fix.

Also, its a little bit frustrating, because if something happens in the future with OS on my PC storage and it stops working, I can't revive it with USB stick with latest Ubuntu, because it's also not working. :) Keeping additional old OS versions on pendrives or specially prepared (with repacked squashfs) is not an ideal solution when sometimes "on the go" you just need to download&write the ISO. It generates unnecessary extra work, mounting/chrooting and systems juggling, so thats a real problem.

[zapphh (zapphh)]

I had the same problem after updating from Kubuntu 21.10 to 22.04 on a Dell XPS 15 9550 with 4k display. Changing the compress parameter fixed the problem.

[sudodus (nio-wiklund)]

If you have problems to boot live systems (from USB), I think it can be solved with mkusb

https://help.ubuntu.com/community/mkusb

because it makes the live system use GFXMODE set to 800x600 except when cloning. So select 'live-only' and in the next menu 'dus-iso2usb' ...

Please report the result here, whatever it is (success or failure), if you try with mkusb.

-o-

But of course, mkusb does not edit /etc/initramfs-tools/initramfs.conf, so that will still be necessary in some computers until this bug is squashed.

[jeremyszu (os369510)]

the problem seems to me that memory allocation issue with higher resolution monitor.

When issue happening, the grub shell not able to read some files (depends on current memory usage).

For instance, I create 20M, 30M, 40M, 50M ... 80M, 90M, 100M, 101M, 102M ... 110M, 120M, 130M, etc... files and try to read them in grub shell.

The "ls -al" only list the part of files.

sometimes:
"20M, 30M, 40M, 50M ... 80M, 90M, 100M, 101M, 102M"
sometimes:
"20M, 30M, 40M, 50M ... 80M, 90M"
sometimes:
"20M, 30M, 40M, 50M ... 80M"

when force "ls -al test-130M", it will show "out-of-memory"

it looks to me that depends the current memory usage.

Based on my test, this issue is likely relate to EFI or grub (or initramfs config) instead of kernel.

For the people be blocked by this issue, the best practice should change the initramfs compression method to reduce the file size as what ybdjkfd shared in comment#41.

btw, a initramfs compression thread is here
https://www.phoronix.com/forums/forum/software/distributions/1295285-ubuntu-rethinking-its-initramfs-compression-strategy

a related upstream discussion:
https://savannah.gnu.org/bugs/index.php?61058

[jeremyszu (os369510)]

#0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376
#1 0x000000007dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408
#2 0x000000007dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076)
    at ../../../grub-core/kern/verifiers.c:150
#3 0x000000007dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem",
    type=131076) at ../../../grub-core/kern/file.c:121
#4 0x000000007bcd5a30 in ?? ()
#5 0x000000007fe21247 in ?? ()
#6 0x000000007bc030c8 in ?? ()
#7 0x000000017fe21238 in ?? ()
#8 0x000000007bcd5320 in ?? ()
#9 0x000000007fe21250 in ?? ()
#10 0x0000000000000000 in ?? ()

...

Thus, it likely same as my assumption, grub not able to read a lager file.

[jeremyszu (os369510)]

Updates:

There are many available memory blocks but they are fragment.
When grub_real_malloc(), it requires contiguous free memory blocks for a single request.

grub_real_malloc (grub_mm_header_t *first, grub_size_t n, grub_size_t align) { ...
if (p->size >= n + extra) { ...

(gdb) p n
$1 = 18506690
# n has been shifted by GRUB_MM_ALIGN_LOG2 (5)[1]

from grub_mm_dump():
...
F:0x5cac1020:509111712:0x7be5f920
...

592214080 (18506690<<5) > 509111712

It seems to me that the grub doesn't have GC mechanism so the memory fragments are expected..

It's quite risky if implements GC in grub_memalign() if allocation failed.

Based on grub_mm_dump(), it seems has the limited memory addressing.
The next action will check if it's possible to access higher memory address.

(gdb) p total_pages
$1 = 512502

no matter how many RAM assigned to qemu. there is only 2GB memory on total_pages, no sure whether related to TOLUD

[1] ea8: c1 e0 05 shl $0x5,%eax

[jeremyszu (os369510)]

Based on UEFI Spec v2.9

>>EFI_BOOT_SERVICES.AllocatePages()
>>..
>>MemoryType
>>values in the range 0x80000000..0xFFFFFFFF are reserved for use by
>>UEFI OS loaders that are provided by operating system vendors.

efi_call_4 (b->allocate_pages, alloctype, memtype, pages, &address);

thus, grub only get 2GB from EFI.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[jeremyszu (os369510)]

--- a/debian/patches/series
+++ b/debian/patches/series
@@ -126,3 +126,8 @@ linuxefi-do-not-validate-kernels-twice.patch
 fdt-add-debug-output-to-devicetree-command.patch
 efi-EFI-Device-Tree-Fixup-Protocol.patch
 ubuntu-disable-LOAD-FILE2-protocol-for-initrd-on-ARM.patch
+0129-Replace-the-memory-allocation-by-using-the-macro.patch
+0130-x86-efi-Make-our-own-allocator-for-kernel-stuff.patch
+0131-x86-efi-Allow-initrd-params-cmdline-allocations-abov.patch
+0132-x86-efi-Use-bounce-buffers-for-reading-to-addresses-.patch
+0133-Fix-wrong-print-of-types.patch

will update the SRU template later

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "0129-Replace-the-memory-allocation-by-using-the-macro.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Julian Andres Klode (juliank)]

So I'd like us to avoid diverging even further from upstream (whether upstream upstream or rhboot upstream), this patch series is creating too much technical debt.

Does this affect rhboot too? If not, why not, and what are their fixes, and why do we not use them? If yes, please submit them to the rhboot/grub repo on GitHub.

The patches should not be numbered.

Also of course this is blocked by the security update being committed and released first. Please do not commit or upload that.

[Julian Andres Klode (juliank)]

Notably use

 Gbp-Pq: Name foo

in the commit message in the Patch-Queue branch when preparing changes so that it gets the right name.

[jeremyszu (os369510)]

the patches 130-132 are from rhboot already (I did find them to replace the simular patch I made).
let me try to find the alternative/related patch from rhboot.

btw,
>Does this affect rhboot too? If not, why not, and what are their fixes, and why do we not use them? If yes, please submit them to the rhboot/grub repo on GitHub.

I tried the fedora 36 from https://getfedora.org/en/workstation/download/ in my qemu.
The issue is reproducible but I think it could be a regression or something I don't know how they generated the grubx64.efi.

I believe there are some patches since 2019 could solve this issue.
I'll backport them to give it a try.

[jeremyszu (os369510)]

MP submit https://github.com/rhboot/grub2/pull/102

which works with https://github.com/os369510/grub2/commits/patch-queue/lp1842320v2

[fractaluser (fractaluser)]

Found this after doing a lot of research and testing both with ubuntu, popOs and Zorin. I'm also on MSI Prestige 15 A10SC and get either a black screen or the memory leak (happened only with ubuntu) when trying to boot the live usb. The only workaround I've found is to change from UEFI boot to Legacy in my bios, that seems to work to boot the usb, but on the other hand I get error after installing the OS (pop os) since (I believe) i'm in legacy and not UEFI boot.

This might be relevant for this investigation, the logs are from PopOS when I'm in Legacy mode in Bios, successfully booted the usb and full install, this fails in the end:

[INFO distinst:src/installer/state.rs:33] starting configuring bootloader step
[INFO distinst:src/installer/steps/bootloader.rs:35] /dev/nvme0n1: installing bootloader for Bios
[INFO distinst:crates/chroot/src/command.rs:109] running "chroot" "/tmp/distinst.5v9VjAvHYBqC" "grub-install" "--recheck" "--target=i386-pc" "/dev/nvme0n1"
[WARN distinst:crates/chroot/src/command.rs:99] Installing for i386-pc platform.
[ERROR distinst:src/installer/state.rs:37] configuring bootloader error: command failed with exit status: exit status: 1
[ERROR distinst:src/installer/mod.rs:300] errored while installing system: command failed with exit status: exit status: 1
[INFO distinst:ffi/src/installer.rs:190] Install error: command failed with exit status: exit status: 1

[jeremyszu (os369510)]

Hi fractaluser,

Please see comment#1, #20, #40.
The symptom of this bug is point to somehow kernel not able to find/mount initramfs and a workaround from comment#41.
You can have a similar steps to see whether related or report the other bug.

[fractaluser (fractaluser)]

Hi jeremyszu,

Thanks for the reply, I already saw them and thought I'd post here. I have now also found my solution to this and it was to install ubuntu 21.10 which didn't have this issue at all. I could boot from UEFI, got to the install, managed to create a second boot partition (since im dual boot) and successfully boot the system after install. Then upgraded the OS and everything is working fine. Leaving this if anyone else have this issue and want to try an easier approach. Cheers!

[m1st0 (m1st0)]

As with comment #41, #44, and #45, has the decision been made to fix grub2's memory issues or just the compression level change that was with initramfs (as I believe it was a choice made to increase Raspberry Pie boot times)?

[Daniel van Vugt (vanvugt)]

In case it's relevant...

I bricked my old PC while trying to work around this bug(!). After putting the same RAM and plugging the same monitor into a new machine, the bug does not occur. So the trigger for the bug was more to do with the old machine's firmware.

[Julian Andres Klode (juliank)]

Yes, grub will be fixed eventually, but we are blocked by the security update not being out yet. This is not a blocker for enabling upgrades to 22.04.1, as it only affects a small number of systems and grub fixes itself by using the previous kernel version if the new one fails.

[jeremyszu (os369510)]

As my MP from comment#69 doesn't have response from rhboot team.

Shared the debdiff here for review to consider to carry it in Ubuntu.

[jeremyszu (os369510)]

This patch is not from upstream or rhboot but which has been submitted as a MP to rhboot
https://github.com/rhboot/grub2/pull/102

+0129-Try-to-pick-better-locations-for-kernel-and-initrd.patch
+0130-x86-efi-Use-bounce-buffers-for-reading-to-addresses-.patch
+0131-x86-efi-Re-arrange-grub_cmd_linux-a-little-bit.patch
+0132-x86-efi-Make-our-own-allocator-for-kernel-stuff.patch
+0133-x86-efi-Allow-initrd-params-cmdline-allocations-abov.patch
+0134-linuxefi-fail-kernel-validation-without-shim-protoco.patch
+0135-Fix-4GB-memory-be-filtered-out-by-filter_memory_map.patch

0129-0134 are from rhboot.

[Joseph Price (pricechild)]

@juliank: I was affected by this bug when upgrading to 22.04.1 from 20.04 at the weekend. (originally installed as 18.04)

Unfortunately the system did *not* auto-recover.

Manually picking the previous kernel options from grub gave the same "error: out of memory."

I was able to recover the system through live cd -> decrypt -> chroot -> MODULES=dep.

This got me to initramfs on next boot, where I was able to `cryptsetup luksOpen` and mounting /root manually before continuing the boot.

After getting back into the system, a final `update-initramfs` gave me a cleanly booting system.

I've seen various other reports in Google of similar problems on Dell XPS 13 hardware like my own with 22.04.

[Ben Hoyt (benhoyt)]

Joseph, out of interest, what model of XPS 13 do you have? I'm getting new XPS 13 soon and wondering if it'll have the same issue (I guess it's likely given the 4k screen).

[Daniel van Vugt (vanvugt)]

Ben, XPS 13 should have no problems with the factory-installed certified (by us) images:
https://ubuntu.com/certified?q=XPS+13&limit=20&release=22.04+LTS

Certainly when we were certifying the latest XPS 13 Plus (with 4K), the only blocking issue I saw was a multi-monitor bug that we fixed before launch.

[John Wiggins (jcwiggi)]

This is also affecting the MSI MEG Z690I Unify motherboard; can not boot any live disk images of Ubuntu 22.04

[robinkooli (robinkooli)]

Confirmed affecting Samsung Galaxy Book2 (i7-1255U, Intel Iris Xe Graphics, 16GB LPDDR4x / NP750XED-KC4SE).

[Rolf Koch (rolfkoch)]

Confirmed affecting GIGABYTE Z690 AORUS ELITE DDR4 (LGA1700, USB 3.2, PCIe 5.0) after installing nvidia-driver-515

[Ben Hoyt (benhoyt)]

A couple of days ago I upgraded my 21.10 system to 22.04 and got this "out of memory" issue (Dell XPS 15 9550 with 3840x2160 display).

The fix in comment #41 worked (changing the compression to 19), and the sequence of commands in #44 helped. However, just noting that there are several typos in #44 that had me stumped for a while (it's the first time I've used chroot to update grub).

So here are (I think!) the corrected commands to update grub:

$ sudo su -
# mkdir root
# mkdir boot
# mkdir efi
# cryptsetup luksOpen /dev/nvme0n1p3 nvme0n1p3_crypt
# mount /dev/mapper/vgubuntu-root ./root
# mount /dev/nvme0n1p1 ./efi # was "nvme01p1"
# mount /dev/nvme0n1p2 ./boot # was "nvme01p2"
# mount -B ./boot ./root/boot
# mount -B ./efi ./root/boot/efi
# mount -B /dev ./root/dev # was: mount -B ./dev ./root/boot/dev
# mount -B /dev/pts ./root/dev/pts # was: mount -B ./dev/pts ./root/boot/dev/pts
# mount -B /proc ./root/proc # was: mount -B ./proc ./root/proc
# mount -B /sys ./root/sys # was: mount -B ./sys ./root/sys
# chroot ./root
# ### Now edit line 196 of /usr/sbin/mkinitramfs to to use -19 instead of -1
# update-initramfs -u -k all
# update-grub
# exit
# reboot

[Mauricio Faria de Oliveira (mfo)]

Hi Jeremy,

It seems the PR in rhboot's grub2 hasn't been reviewed yet;
just mentions from other folks who also found it helpful.

Would you consider sending this to the grub-devel list? (i.e., upstream gnu's grub2)
It looks like it wasn't yet (search results only cover rhboot).

By the way, this type of change had an interesting discussion
about the implementation approach a few years ago [1], but it
seems it ceased as the sender didn't provide further replies.

Your understanding of the tech details seems very deep, so it
might hopefully be a good opportunity to propose that again :)

Thanks!

[1] https://lists.gnu.org/archive/html/grub-devel/2017-03/msg00032.html

[renton (renton)]

Hi, @jeremyszu

How can I build grub from your repository https://github.com/os369510/grub2 with your patches for testing my linux box?

Thanks in advance!

[Nicholas Stommel (nstommel)]

I am getting this error and cannot even boot Ubuntu 22.04.1 from a live USB. I am using an HP Elite Dragonfly G2 with 16GB of RAM. I get an error: out of memory message then a kernel panic. Please fix this, I would very much like to try running Ubuntu but I have to run Fedora 36 instead due to the fact that I can't even boot Ubuntu because of this bug.

[jeremyszu (os369510)]

Hi renton,

If you want to build it by yourself, then you should clone this:
https://people.canonical.com/~jeremysu/lp1842320/

and built it locally and don't clear the build environment, you can found the efi binary on "obj/monolithic/grub-efi-amd64/grubx64.efi". I personally use pbuilder with some hooks when debugging.

Alternatively, I upload the binary built by me on
https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320v2
and for sure, the efi binary relates to security, then build it by yourself makes sense (but TBH it's not easy as other packages.)

If you want to give it a try, then I suggest don't replace the original efi binary.
Instead, you can use something like

efibootmgr -c -L 'lp1842320' -d ${u-r-boot-dev} -p ${partition} -l '\EFI\ubuntu\grubx64.efi.lp1842320v2'

for example:
efibootmgr -c -L 'lp1842320' -d /dev/vda -p 2 -l '\EFI\ubuntu\grubx64.efi.lp1842320v2'

and use something like
'efibootmgr -n ${your-boot-order-of-lp1842320}'

or press F9 on HP machine
or press F12 on Lenovo machine

Please only do that if you know what you are doing.

[John Wiggins (jcwiggi)]

I found if the Max TOLUD is set to 1GB or higher in the BIOS (anything other than dynamic) then Ubuntu 22.04 ISO will not get this out of memory error

[IMarvinTPA (imarvintpa-y)]

Earlier today, I upgraded my Mint 20.3 system to Mint 21.0 and got this "out of memory" issue (Asus ROG G752VT laptop.).

The fix in comment #41 and #89 worked (changing the compression to 19), but a lot of the steps were not needed.

This is the minimal steps needed, (maybe fewer possible, anybody up for some code golf?):

$ sudo su -
# mkdir root
He probably had his disk encrypted, I do not: # cryptsetup luksOpen /dev/nvme0n1p3 nvme0n1p3_crypt

This one was very different for me due to mint vs Ubuntu.
# mount /dev/mapper/vgmint-root ./root was # mount /dev/mapper/vgubuntu-root ./root

This one makes the computer think the normal root is really root to fake out update-initramfs:
# chroot ./root

Yes, do this:
# ### Now edit line 196 of /usr/sbin/mkinitramfs to to use -19 instead of -1
This wasn't completely happy with me, but it worked out ok for me.
# update-initramfs -u -k all

This command completely failed for me, but it didn't seem to hate me afterwards:
# update-grub

# exit
# reboot

I hope you have a happy reboot too.

Then re-run
# update-initramfs -u -k all
and
# update-grub

To clean up any oddities.

I wish there were a way to recompress this from grub though.

[jeremyszu (os369510)]

Hi Mauricio,

From the patch set in this ticket:
```
+0129-Try-to-pick-better-locations-for-kernel-and-initrd.patch
+0130-x86-efi-Use-bounce-buffers-for-reading-to-addresses-.patch
+0131-x86-efi-Re-arrange-grub_cmd_linux-a-little-bit.patch
+0132-x86-efi-Make-our-own-allocator-for-kernel-stuff.patch
+0133-x86-efi-Allow-initrd-params-cmdline-allocations-abov.patch
+0134-linuxefi-fail-kernel-validation-without-shim-protoco.patch
+0135-Fix-4GB-memory-be-filtered-out-by-filter_memory_map.patch
```
0129-0133 are the key patches to support the >4G memory allocation which made by vathpela and I didn't see they are upstreamed.
From my point of view, it won't upstream because upstream has different approach to allocate memory for kernel and initramfs.

Therefore, the last time I debug this issue is based on ubuntu code base.

Indeed, I should take a look to see if upstream version whether has this issue.
However, I learned from Julian that upstream refactored the MM part. (says, the patches will likely be dropped after 2.12 grub)

I can give it a try but it takes time because I'm a newbie in the grub.
Even if there is a patch on upstream, it need to based on the refactored MM which might not easy to land in LTS version as users expected.
I suggest to consider to carry these patches in ubuntu to unblock the ubuntu users first.

[Mauricio Faria de Oliveira (mfo)]

Hey Jeremy,

Understood; thanks for clarifying!

I thought that maybe having upstream input could be helpful
(as Julian mentioned to avoid diverging further), but if it
is so much different nowadays, as you mentioned, it likely
won't be that helpful anyway.

By the way, appreciate that "newbie" statement.. after all
this work/analysis/patches, I couldn't think that of you :)

[gerald.yang (gerald-yang-tw)]

Add one more case caused by this issue:

When SEV is enabled on the host, the guest also enables SEV and running 5.15 kernel
with this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12

The SWIOTLB driver in the guest could allocate 1G contiguous memory and fail:
[ 0.005854] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.042923] software IO TLB: Cannot allocate buffer
[ 0.821973] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822542] software IO TLB: No low mem

With Try-to-pick-better-locations-for-kernel-and-initrd.patch, this will be solved

Test procedure:
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1

create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole

Make sure the running kernel in VM is 5.15

Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4000-0x0000000054fb4000] (1024MB)
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers

more details are in lp:1989446

[Henri Cook (henricook)]

The steps in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/89 didn't _seem_ to be working for me

I looked at my /etc/initramfs-tools/initramfs.conf and noticed my COMPRESS was set to lz4. Casting my mind back that might have been attempts to solve a similar issue weeks ago. Setting it back to zstd, and following the steps in #89 sorted the problem for me.

[Mar Kus (ubuntukus)]

I tested latest daily build https://cdimage.ubuntu.com/jammy/daily-live/20220920/jammy-desktop-amd64.iso and out of memory stil exist. I hope someone can provide an iso for download with a livesystem and install for jammy with changed compression.
Ubuntu 21.10 (out of support) and Fedora 36 works fine.

[Julian Andres Klode (juliank)]

@jeremyszu (os369510) Did you see #90, it seems your patch would cause boot failures on older system unable to handle this memory range.

[jeremyszu (os369510)]

Hi Julian,

I didn't see the 'boot failures on older system' from comment#90.
Would you please point it out more specifically?

[Seth Arnold (seth-arnold)]

I've been asked to prepare a summary of the current status of this bug:

- there's a grub2 security update that's been published and then pulled:
  https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu10/+publishinghistory
  https://bugs.launchpad.net/ubuntu/+source/grub2-signed/+bug/1990684

- apt's dependency solver is being modified to handle updates that need to phase identically

- once the apt update is released, then the grub security fixes can be republished

- once the grub security fixes are republished, then this issue can be addressed.

It appears there's an open question about the risks of jeremyszu's changes possibly causing problems for older systems. The closest thing I found in the linked thread was on this message:

https://lists.gnu.org/archive/html/grub-devel/2017-03/msg00033.html

> I seem to recall that the x86_64 port was being restricted due to
> known bad firmware encountered in the past. It could be that it would
> be worth adding an option to configure for enabling access to higher
> addresses, alternatively for retaining compatibility with the broken
> systems.

I haven't read through the patches nor the upstream issue tracker to find out if these are recent problems or not, but this sounds like the usual warning that grub is difficult to test, lives in firmwares that may be ignored or otherwise horrible, etc. I hope we have a representative sample of machines to test in our labs, as well as our home offices, and in our wider community.

Was there a more specific problem that I missed?

Are there outstanding tasks that need doing that could be done before the apt+security update steps are complete? Refreshing patches, or skimming through issue trackers to find regressions from the patches, etc?

Thanks

[jeremyszu (os369510)]

I don't have such "there is allways that machine (or bad firmware) that nobody heard of that will fail" case from my hand but it's frustrated if it becomes a blocker for fixing the bug for new devices from market.
However, I understand how users feel helpless if a SRU introduces a regression especially the bootloader.

I think it's worth to mention that, here are many users reported their certain real cases (as opposed to "there is allways that machine (or bad firmware) that nobody heard of that will fail") which causes their system not able to boot.

As we all known, Fedora and Arch linux are using higher compression rate (for initramfs) to avoid this issue.

I could try the upstream grub as soon as I can and to see if I can prepare such scheme as what the mailing thread mentioned but I afraid it'll definitely take times.

Anyway, if we can have some solutions for existing impacted user, then it'll very helpful.

[Julian Andres Klode (juliank)]

Our goal should be to merge the entire patch series into kinetic, worst case, kinetic will not be installable for some users. This means we will have decent results from people trying that in the next 4 weeks (by Oct 27 the release has been out 1 week).

In the meantime, next week we should push out the security update again. Then we can take dannf's patches and the first patch here for that SRU to unblock the other OEM thing which certainly is less controversial. And on like Oct 27 if we don't have serious regressions do a final SRU with all the patches.

I'll see what I can do to get those into kinetic so that we get that bit moving but I'll be travelling or meeting people most of next week.

[Julian Andres Klode (juliank)]

OK the patch set here is broken, we gotta do this from scratch properly. So I'm going to start cherry picking the rhboot patches for memory management. I have applied so far from bug 1989446 the backport of "Try to pick better locations for kernel and initrd" and cherry-picked from rhboot the "x86-efi: Use bounce buffers for reading to addresses > 4GB" patch and will continue picking up the memory changes from the rhboot branch and then we can address any gaps remaining.

Now the problem with the patch set is clearly that the rhboot patches miss half their content and are applied after a custom patch that changes most of the memory management, which does not help.

0129 I thought was "Try to pick better locations for kernel and initrd" but it's something custom that's deep into the memory management code and just looks like it. I thought only increasing the memory limit in the last patch was the custom bit. I do not want to change general memory allocation in grub vs our upstreams. Scary.

[Peter-and-Crazybear (crazybear)]

When is this going to be fixed and is there a work-around. I have tried
both updating from 20.04 and direct installation of 22.04 and have found
this problem preventing me from using 22.04 [it never boots]. I have two
computers. One is a Dell with nvidia drivers [the perfect target machine
to show this problem] - and it fails, but I also have a desktop I built
myself that should not have problems [and obviously some having
installed 22.04 do NOT have problems, but I do - and can not tell you
what component is the problem [hardware or software]. Somewhere I read
that making some spaces larger for the possible 'expansion' to not cause
problems, but can not find it now with all of the many entries. I have
spent MONTHS trying to install 22.04....without success on either
machine. I do not think you are correct that it only effects a small
number of machines. Anyway, for those of us who it does, the problem is
IMMENSE and total.

On 03/08/2022 10:44, Julian Andres Klode wrote:
> Yes, grub will be fixed eventually, but we are blocked by the security
> update not being out yet. This is not a blocker for enabling upgrades to
> 22.04.1, as it only affects a small number of systems and grub fixes
> itself by using the previous kernel version if the new one fails.
>

[Daniel van Vugt (vanvugt)]

Workaround:

You can usually install Ubuntu on a drive in one machine and then move the drive into another machine.

At least I think that's what I did on the affected machine with 22.04, which is why I never discovered this bug until 22.10 development started.

[sudodus (nio-wiklund)]

Another workaround might be to use Ubuntu's own compressed image with a preinstalled Ubuntu Server (although not officially released) according to the following link

https://ubuntuforums.org/showthread.php?t=2474692

Scroll down to read also the following posts, that describe how to install Lubuntu. In the same way you can install Ubuntu Desktop as well as all the community flavours.

[Daniel van Vugt (vanvugt)]

Duplicate bug 1970402 mentions that:

> Workaround: Disable Secure Boot and SGX.

works for multiple people.

[Stefan Akatyschew (fried-gluttony)]

I can replicate this on my Dell XPS13 9380 (no encryption). Occurs as well with Ubuntu 20.04 LTS, but not with 18.04 LTS, so not sure if it is related to the same reason?
-> Disabling Secure Boot and SGX did not resolve the issue

[Julian Andres Klode (juliank)]

Really the workaround is to set MODULES=dep in /etc/initramfs-tools/initramfs.conf, this yields a much smaller initramfs (but you can't take out the disk and boot it in another machine), disabling secure boot or sgx should not be doing anything.

[Adrien Nader (adrien)]

As I've said elsewhere, if we dedup firmware files through symlinks, we can save 10MB in initrds. Compression does not help because the compressors have very small compression windows and cannot see redundancy in practice (this applies to xz to a lesser extent but even for xz there is an improvement to be gained).

I don't know if it will be enough but it could tip us just below some threshold and it's easy to implement. It should also save memory at runtime.

And for reference, there's a working but weird work-around that only touches initramfs.conf, keeps MODULES=most and is reasonably fast. Use 'xz' as the compress program and then add the following on a new line:

  export XZ_OPT=--lzma2=preset=0,dict=16M

(yes, it relies on implementation details of mkinitramfs)

[m1st0 (m1st0)]

I upgraded from 22.04.1 to 22.10. Issue returned as /usr/sbin/mkinitramfs was changed to default. Had to follow my previous steps shown after #41 . . . zstd had defaulted to a lower compression value in /usr/sbin/mkinitramfs at 1 instead of 19. I changed line 196 to 19 compression

    zstd) compress="zstd -q -19 -T0" ;;

Instead of

    zstd) compress="zstd -q -1 -T0" ;;

Problem solved from proper chroot and updating grub2 after changing line 196 of /usr/sbin/mkinitramfs for my system.

I can boot normally again and any upgrades to initramfs don't cause this failure.

[Benjamin Drung (bdrung)]

initramfs-tools 0.142 added support for specifying the compression level. Ubuntu will probably merge this version for Ubuntu 23.04.

[jeremyszu (os369510)]

If possible, I still expect we can make grub supports to boot from bigger initramfs.

There will still have users to meet this issue in the future if their initramfs somehow bigger as long as we don't choose use higher compression as workaround (for ubuntu desktop / server).

[Andy Chi (andch)]

Download ubuntu-20.04.5-desktop-amd64.iso and try to install on a HP laptop, I can't boot into installation menu. If initrd.img lower than 100MB then everything is fine.

[iMac (imac-netstatz)]

My XPS 13 9380 (no luks) also triggers this on 5.19 after the 22.10 upgrade; I can see the initrd is about 7% bigger in 22.10 over 22.04.

 -rw-r--r-- 1 root root 111203941 Oct 30 18:16 initrd.img-5.15.0-52-generic
 -rw-r--r-- 1 root root 120222346 Oct 31 21:39 initrd.img-5.19.0-23-generic

#103 lays out some interesting blockers for grub2, so as of right now there is no simple way to get upstream through ubuntu repositories as grub2 2.06-2ubuntu12 does not have a fix.

I think it is interesting that in https://lists.gnu.org/archive/html/grub-devel/2017-03/msg00037.html there was pushback on autodetection to fix this, citing "that machine that nobody heard of that will fail" however it ended up being a bunch of not-yet-built systems that failed

Looking at initramfs-tools in Jammy, the very last changelog entry was the change to lower compression, that is triggering this issue:

 http://changelogs.ubuntu.com/changelogs/pool/main/i/initramfs-tools/initramfs-tools_0.140ubuntu13/changelog

 initramfs-tools (0.140ubuntu13) jammy; urgency=medium

  * Lower the compression levels for zstd and lz4 (LP: #1958148)
    Following the discussion on the mailing list, we have reached
    a conclusion to lower the default compression levels:
    - For lz4, the compression level is lowered to 2 from 9
    - For zstd, the compression level is lowered to 1 from 19

Simple fix IMHO is to just go back one version. Ubuntu main archive doesn't include all old versions, but other mirrors do. Here are some quick steps that let you run official binaries from Jammy under Kinetic to work around this issue until grub2 catches up and/or they tweak compression, or make compression more easily tunable.

wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools-bin_0.140ubuntu12_amd64.deb
wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools-core_0.140ubuntu12_all.deb
wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools_0.140ubuntu12_all.deb
dpkg -i initramfs-tools-bin_0.140ubuntu12_amd64.deb
dpkg -i initramfs-tools-core_0.140ubuntu12_all.deb
dpkg -i initramfs-tools_0.140ubuntu12_all.deb
apt-mark hold initramfs-tools-bin
apt-mark hold initramfs-tools-core
apt-mark hold initramfs-tools
update-initramfs -u

Afterwards, big drop in initrd size. 34% smaller. I am definately getting those fractions of a second back on NVME boot.
-rw-r--r-- 1 root root 111203941 Oct 30 18:16 initrd.img-5.15.0-52-generic
-rw-r--r-- 1 root root 79034965 Oct 31 22:50 initrd.img-5.19.0-23-generic

Sometime down the road, when this bug is closed, you will want to un-hold these to restore the lastest packages

 apt-mark unhold initramfs-tools-bin
 apt-mark unhold initramfs-tools-core
 apt-mark unhold initramfs-tools

[jeremyszu (os369510)]

I don't think "MODULES=dep" is the proper workaround for this.

If an user somehow met this issue, then the system is not able to boot.
Users need to do many extra works to find out this workaround and do apply unless we make "MODULES=dep" as default setting in initramfs configs (but I don't think it's doable).

Based on #117, we also confirm the stock ubuntu not able to boot on some machines.
The fix needs to be included in the packages from ubuntu-archive as common solution.

Julian,

Can we go back to consider #105, #106 to try to land it in Lunar or Kinetic first?

[iMac (imac-netstatz)]

-> If an user somehow met this issue, then the system is not able to boot.

It can boot, if it did previously, and this issue will get fixed pretty quick IMHO in the updates repository given it impacts current 20.04, 22.04 and 22.10.

Here is just the workaround part, for current 22.10 or 22.04 to help people stuck on boot that used to have a working system.

1) Hold the SHIFT key on boot, and select the 2nd newest kernel from the advanced list in the grub menu: i.e. for myself, 5.19 broke, so using the older 5.15 allowed me to boot.

2) Open a terminal, sudo -i to root, and run these commands to download, install, and prevent upgrades to last version of initramfs-tools before the compression rates were changed, which allows you to regenerate your initrd image with the previously working smaller size.

Download:
 wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools-bin_0.140ubuntu12_amd64.deb
 wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools-core_0.140ubuntu12_all.deb
 wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools_0.140ubuntu12_all.deb

Install:
 dpkg -i initramfs-tools-bin_0.140ubuntu12_amd64.deb
 dpkg -i initramfs-tools-core_0.140ubuntu12_all.deb
 dpkg -i initramfs-tools_0.140ubuntu12_all.deb

Hold Upgrades:
 apt-mark hold initramfs-tools-bin
 apt-mark hold initramfs-tools-core
 apt-mark hold initramfs-tools

Re-generate current initrd image:
 update-initramfs -u

A reboot should just bring you up in 5.19. To restore the system to a normal state once the problem has been fixed in the current versions, unhold those same packages using the steps below, and the new ones will automatically be downloaded and installed.

 apt-mark unhold initramfs-tools-bin
 apt-mark unhold initramfs-tools-core
 apt-mark unhold initramfs-tools

Some additional notes, not for the workaround above follow.

Do NOT issue the "update-initramfs -k all" command, if you have been thinking about it as a first triage step. The reason being, if you have not applied some kind of fix or workaround prior, this will rebuild all your old previously working kernel initrd images with the new low compression directive, which might render them all unbootable, further complicating matters. If you get to this state, and you need a working live ISO, to chroot and fix, make sure to choose one prior with the older, increased initrd compresssion which must be 20.04.4 LTS or older. Current 20.05.5, and all 22.04 and 22.10 releases use the lower compression that can cause this issue and may not work. Impatient new installers may find this info useful as well.

Another simpler workaround for more advanced users with 4K screens is to pass GFXMODE=640x480 in grub defaults and update-grub as mentioned here: http://savannah.gnu.org/bugs/?61058

[Yohn Chrichton (yokrin)]

The issue happened for me after installing newer nvidia proprietary drivers (515 and also 520).
The only way to recover was to boot using a previous installed kernel (I used rescue mode. The rescue mode in current kernel still gives the same error)

Not sure how this helps this bug investigation but was real pain knowing that I couldn't install the nvidia drivers.

The only thing that worked was installing the nvidia drivers 470 version.

https://forums.developer.nvidia.com/t/ubuntu-22-04-1-nvidia-driver-open-kernel-nvidia-driver-515-open-issue/231356/12?u=yohn.krichton

[Craig Hesling (hesling-craig)]

Confirmed that reducing the size of the initrd.img slightly allowed my machine to boot. I just enabled `COMPRESSLEVEL=19` with `COMPRESS=zstd` (default on debian) in initramfs.conf. That reduced my initrd.img size from 72MB to 62MB, which allowed my machine to boot.

Ultimately, I set `MODULES=dep` (instead of `MODULES=most`) and reverted the compression to defaults. This reduced my initrd.img to 22MB, which includes Intel microcode (specific binary), nvme, crypto, and lvm support, among other things.

---

For the record:

My machine is running Debian bookworm with the Nvidia proprietary drivers with 32GB of RAM. I first noticed the issue when, coincidentally, upgrading from kernel 5.19 to 6.0.

```
Loading Linux 6.0.0-4amd64 ...
Loading initial ramdisk ...
error: out of memory.

Press any key to continue...
```

[ali nourzad (anourzad)]

can this be a solution ? https://stackoverflow.com/questions/2895816/how-do-i-strip-local-symbols-from-linux-kernel-module-without-breaking-it

[Julian Andres Klode (juliank)]

@Craig The limit in Debian is actually much lower than in Ubuntu even, but fixing it there is even harder as that misses a lot more patches.

[Adrien Nader (adrien)]

I put together some notes and work-arounds in order to provide a simpler reference for people hitting this issue. I didn't test everything below but nothing should be risky.

# Summary

Grub attempts to read the initrd into a memory location that is too small.

This issue is caused by a combination of several factors:
- Grub not setting aside enough space for the initrd in memory and not at the right location,
- Initrds having grown due to holding more modules and more firmware. This is especially the case with the nvidia proprietary driver.
- Typical screen resolutions and their associated buffers in memory having increased.

A related issue is not having enough space in /boot to hold enough initrds.

# Solutions

There are four ways to tackle these issues:
- Patch grub,
- Have fewer modules,
- Compress initrds more,
- Lower screen resolution at boot.

## Patch grub
This is being worked on. It’s the trickier option so I’m not going to provide details but you can find that in the discussion above.

## Have fewer modules
The default configuration creates initrds that are meant to be universal in order to accommodate as many hardware variants as possible. Unsurprisingly this makes initrds much bigger.

It is possible to change the value of MODULES to ‘dep’ in /etc/initramfs-tools/initramfs.conf . This will drastically reduce the initrd size in virtually every configuration at the cost of not supporting during boot hardware that is not plugged when the initrd generation takes place.

## Compress initrds more
Change the value of COMPRESS in /etc/initramfs-tools/initramfs.conf .
In terms of compression level, you will have something along the lines of the following:
    lz4: 74066711
    zstd: 54260379
    gzip: 53829310
    xz: 29665544

NB: these values are only valid on one specific machine and configuration; they are only meant to give an idea of compression ratio that can be obtained but the initrd uses MODULES=dep as outlined above.

With a small hack, it is also possible to make xz compression much faster at a small cost in compression by adding the following at the end of initramfs.conf:
    export XZ_OPT='--lzma2=preset=0,dict=8M'

In the future, it will be possible to directly set compression levels for every compression method.

## Lower screen resolution at boot
You can use a lower resolution screen when booting.

You can also edit /etc/default/grub and use the following:
    GRUB_GFXMODE=640x480
Remember to run update-grub afterwards.
If you need to do it at boot-time, you will use GFXMODE instead (no ‘GRUB_’ prefix).

[jeremyszu (os369510)]

Is it possible if we have an overwrite in debian/rule by using something like DEB_BUILD_ARCH to specify higher compression rate for amd64?

[Craig Hesling (hesling-craig)]

Ah, didn't realize Debian had a different limit. Thanks @juliank.

@anourzad, I'm not sure I would want to start adding custom steps to the kernel update path. I'm much more inclined to add an /etc option for a supported automatic mechanism, like compression or module selection.

@adrien-n, great notes in #125, I'm sure this will help others.

[Adrien Nader (adrien)]

Jeremy, there are duplicate firmware files. Replacing duplicates with symlinks is probably the easiest and most efficient way to improve the situation.

I get the following:

> % jdupes -mrS /lib/firmware
> Scanning: 2830 files, 286 items (in 1 specified)
> 405 duplicate files (in 212 sets), occupying 37 MB

With "jdupes -rl", "tar c /lib/firmware | zstd -1" drops from 428MB to 411MB. The effect on initrds is maybe slightly lower iirc but it's still noticeable and it should be very easy and safe for us.

I can't tell exactly how many more people will be able to boot with that change but I think the cost is small enough that it's worth trying. Worst case, every kernel update is faster.

[Julian Andres Klode (juliank)]

I have now picked all the rhboot patches in 2.06-2ubuntu16~ppa1 in https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/ppa/+packages and will boot that before uploading to the archive.

This *should* allow initrds over 4GB but obviously bugs could be there. The important bit is that we are now close to the rhboot branch again in that file such that the whole thing is maintainable to some extent at least.

Once this is uploaded to lunar people can test this easily on secure boot systems without having to install PPA signing keys in MOK.

I'm gonna see if I can run some tests myself and reproduce this.

Also it turns out it fails to build the i386-efi target due to pointer size mismatches.

[NM64 (nm64)]

I'm insanely late on this, but I couldn't help but notice that nobody mentioned Ventoy (at least ctrl+f doesn't return any results in the full activity log for "ventoy")

Since installing via Rufus set to ISO mode has been a suggested work-around, wouldn't this by definition mean that using Ventoy would "just work"?

Normally I would test this myself and report back, but I'm not actually running into this issue myself, so...

Link for those that aren't aware of what Ventoy is:
https://github.com/ventoy/Ventoy

[m1st0 (m1st0)]

Introduction: The original workaround fixes I posted in #41 and #45 (i.e., the workarounds I found to get things to boot), after upgrading a kernon on 22.10, I now have an updated issue.

Issue now: I still have the workarounds active (after the upgrade) so that the compression and module dependency setups are fixed to make a smaller initrd. After an upgrade to the newest kernel 5.19.0-26-generic, I get on grub:

error: out of memory
Press any key to boot . . .

Pressing a key will normally boot forward this time thankfully. My /boot/initrd.img-5.19.0-26-generic size is 131 Mb. This happens only with this new kernel. The previous 5.19.0-23 one did not have this similar error show and is 49 Mb per the workaround setups above.

[Julian Andres Klode (juliank)]

Hi ybdjkfd I don't really understand what you are doing and how it's relevant. This issue should be fixed in lunar-proposed 2.06-2ubuntu16, please only add comments if you have meaningful insights regarding the verification of that update.

[m1st0 (m1st0)]

Hi juliank, I understand completely you want me to have insights related to verification of the update in lunar-proposed and that I am confusing you with my #131 post.

I have edited it for clarity. I hope it helps people who need the workarounds until lunar or backported fixes are released

[Julian Andres Klode (juliank)]

The term you're looking for is workaround, not triage or fixes. Triage is the process of understanding the bug and marking it the correct things.

Not using the same language as everyone else and writing very long comments makes it hard to understand.

First I thought you reported an out of memory issue on Pis. Then I thought you were saying that the fix doesn't help. It took me a long while to realize it was completely irrelevant for the process of resolving that bug.

If the initrd grows significantly please open a separate bug against linux so that is tracked - maybe something went wrong - and leave a comment here.

[m1st0 (m1st0)]

juliank, edited and removed more notes as you advised. I hope my edits per your advice help clarify things and help anyone to see if a new bug is necessary to report. Thanks for your input.

[Chaim Eliyah (chaimeliyah)]

Can confirm on 22.10, Lenovo Yoga 720 (4k display). Edit: looks like #125 has a comprehensive list of workarounds. Edit2: Holy moly,

-rw-r--r-- 1 root root 115M Dec 18 03:33 initrd.img-5.19.0-26-generic
-rw-r--r-- 1 root root 22M Dec 18 05:08 initrd.img-5.19.0-26-generic.new

Edit3: This definitely got me to (initramfs), however the root volume wasn't available. (crypt+LVM) To boot, I had to:

exit

(because it never shows you the error until you do this)

cryptsetup open /dev/nvme0n1p3 root
lvm lvscan

and voila. I don't know if this is related to "MODULES=dep" (would this keep it from opening the crypt?)... but it's been a day, will test and report back later. :greenbeer:

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu16

---------------
grub2-unsigned (2.06-2ubuntu16) lunar; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Fri, 09 Dec 2022 17:11:44 +0100

[Rex Tsai (chihchun)]

Hi, There are a few OEM platforms that also requires these memory patches in grub2 on jammy, could we also port 2.06-2ubuntu16 back to jammy?

[Aaron Honeycutt (aaronhoneycutt)]

I second backporting this to jammy at least as it is an LTS.

[iMac (imac-netstatz)]

Backporting to LTS is a good idea IMHO. A few additional use cases to consider

- Users on the threshold who experience slight initrd image growth due to any module update

- Users who thought they might be able to continue just using older images may fail to recognize grub will eventually remove those older images during regular updates.

- Users that take their laptops on the road, but then plug them into their dock with larger displays to get the real work done only to have a boot failure. And yet when they take it into Best Buy it just works somehow.

[Julian Andres Klode (juliank)]

The entire upload will be uploaded to all older releases but we do have a security upload already in the unapproved queues that should go out first, but that needs approving those first, them passing the SRU verification, a resigning against the new signing key the new shim needs, and finally someone to release them. So that the security update can be released to security pocket.

This won't be ready before Monday the week after next week at the earliest (7 day testing period in proposed).

People that can approve are still out and only return next week, so I don't think that's entirely realistic we get the existing uploads into proposed on Monday.

If we could get them in by Thursday they could go out Monday the 23rd and we may be able to then upload this fix and release it on like 31st or so.

We're strictly limited by the point release - it needs to be ready for that.

[Madpower (madzohan)]

Hi guys, I want to share my success story:
It happened after I've updated to CUDA repo's nvidia drivers :) my `initrd.img-5.15.0-58-generic` (`ls -lah /boot`) was about 192M and using tuning option of the `/etc/initramfs-tools/initramfs.conf` file I've decreased it (`sudo update-initramfs -c -k 5.15.0-58-generic`) to 116M and now it works =)

my `/etc/initramfs-tools/initramfs.conf`

```
#
# initramfs.conf
# Configuration file for mkinitramfs(8). See initramfs.conf(5).
#
# Note that configuration options from this file can be overridden
# by config files in the /etc/initramfs-tools/conf.d directory.

#
# MODULES: [ most | netboot | dep | list ]
#
# most - Add most filesystem and all harddrive drivers.
#
# dep - Try and guess which modules to load.
#
# netboot - Add the base modules, network modules, but skip block devices.
#
# list - Only include modules from the 'additional modules' list
#

# original
#MODULES=most

# patch
MODULES=dep

#
# BUSYBOX: [ y | n | auto ]
#
# Use busybox shell and utilities. If set to n, klibc utilities will be used.
# If set to auto (or unset), busybox will be used if installed and klibc will
# be used otherwise.
#

BUSYBOX=auto

#
# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
#

# original
# COMPRESS=zstd

COMPRESS=xz

#
# DEVICE: ...
#
# Specify a specific network interface, like eth0
# Overridden by optional ip= or BOOTIF= bootarg
#

DEVICE=

#
# NFSROOT: [ auto | HOST:MOUNT ]
#

NFSROOT=auto

#
# RUNSIZE: ...
#
# The size of the /run tmpfs mount point, like 256M or 10%
# Overridden by optional initramfs.runsize= bootarg
#

RUNSIZE=10%

#
# FSTYPE: ...
#
# The filesystem type(s) to support, or "auto" to use the current root
# filesystem type
#

FSTYPE=auto

# patch
export XZ_OPT='--lzma2=preset=0,dict=8M'
```

Hope it will help you (and don't forget about `sudo update-initramfs -c -k YOUR_VERSION-generic`). Keep yourself safe (from kernel panics too) and Glory to Ukraine!

[m1st0 (m1st0)]

@madzohan Congrats to you. It is nice to get things working for now.
 From my comment around #41 above and related workarounds from there downward, it seems you tried a different compression scheme and found a smaller size in this case from zstd to xz.

I looked over your settings and options for xz compression, but also reading the man pages for XZ it shows that it is already at a -6 setting by default that makes the dictionary size 8M and uses lzma2 compression. XZ settings are already at default in /etc/sbin/mkinitramfs .

With your options it was a few MB larger than from xz default in my setup. I tried both xz settings. Finally, I switched back to zstd and my compression changes I made in /etc/sbin/mkinitramfs as my workaround. I got a good compression size, but people note online that zstd has great read speeds over xz no matter the compression level. So I decided to stay at zstd with only a 7 MB difference: from 125 MB on default xz to 131 MB on zstd with -q 22 compression in /etc/sbin/mkinitramfs .

[Adrien Nader (adrien)]

The terrible thing with compression is how we know of no universal rule. I'm sure you can even find non-pathological cases where lz4 compresses better than zpaq (and does so 1000000 times faster). And that's without taking I/O into account (or filters).

An important thing to keep in mind here is that zstd currently uses its -1 preset while xz uses its default of -6. That means that these two tools which are often in the same ballpark are configured at vastly different compression/speed setting. What's important is that you get something that works for your setup and as I said above: we don't have a rule for optimal settings (and it's probably impossible to find one).

One last note: xz decompression speed depends on the compressed size, not uncompressed size. This means that a more-compressed file decompresses faster. This might be the same with zstd since they're related compressors but I'm not 100% sure.

[Julian Andres Klode (juliank)]

This fix will be included in grub2-unsigned/2.06-2ubuntu14.1 for kinetic, jammy, focal, bionic. Binaries have been built in https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/ppa/+packages for kinetic.

Signing request: https://answers.launchpad.net/canonical-signing-jobs/+question/704589

[Julian Andres Klode (juliank)]

All grubs uploaded to proposed unapproved queues.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-unsigned into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-signed into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-unsigned into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-signed into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.3~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-unsigned into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-signed into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.3~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-unsigned into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.3~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Antoine M (antmfra)]

Hello,

I'm affected by this problem but on my side, it's when I try to install Ubuntu (tested with 22.04 and 23.04dev) with a USB stick. How can I test the resolution with an ISO?

[Matthew L. Dailey (matthew-l-dailey)]

I can confirm that the jammy-proposed package fixes the bug for me on our one affected system at 22.04.1. After installing nvidia-525 drivers, the system would not boot - grub out of memory error and kernel panic.

Updated to grub-efi-amd64_2.06-2ubuntu14.1, grub-efi-amd64-bin_2.06-2ubuntu14.1 and grub-efi-amd64-signed_1.187.3~22.04.1+2.06-2ubuntu14.1 and the system boots normally.

I will change the tag from verification-needed-jammy to verification-done-jammy.

Thanks to everyone for their work on this bug!

[m1st0 (m1st0)]

TESTED
• Reinstalled initramfs-tools-core to undo changes of compression on /sbin/mkinitramfs .
• Confirmed compression set to default values.
• Restored /etc/initramfs-tools/initramfs.conf to default values, with "MODULES=most" .
• Installed grub-efi-amd64/kinetic,now 2.06-2ubuntu14.1 amd64 and related versions of packages.
• Ran the following
  sudo update-initramfs -v -u
  sudo grub-mkconfig -o /boot/grub/grub.cfg
  sudo grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
  sudo update-grub2
• Rebooted

VERIFICATION DONE
• Confirmed fix on MSI GS63VR 6RF laptop. No "out of memory" error on grub boot.
• (For some who mentioned nvidia drivers, I have 525.78.01-0ubuntu0.22.10.1 amd64 installed.)
• My current initrd.img size is larger of course, so if you have multiple versions and your /boot partition is small you may have to remove older versions so you don't run out of room---not in any way meaning the "out of memory" error.

[Kochin (kochinc)]

I also confirm that the proposed update fixed the out of memory error when booting on my computer. The packages I installed and tested:

grub-efi-amd64-bin/jammy-proposed,now 2.06-2ubuntu14.1 amd64 [installed,automatic]
grub-efi-amd64-signed/jammy-proposed,now 1.187.3~22.04.1+2.06-2ubuntu14.1 amd64 [installed]
grub-efi-amd64/jammy-proposed,now 2.06-2ubuntu14.1 amd64 [installed]

The kernel and Nvidia driver installed on the system are:

linux-image-5.15.0-58-generic/jammy-updates,jammy-security,now 5.15.0-58.64 amd64 [installed,automatic]
nvidia-driver-525/jammy-updates,jammy-security,now 525.78.01-0ubuntu0.22.04.1 amd64 [installed]

Since my system hadn't been working because of the error, I had to boot up my computer with a Live CD, mount the filesystem, chroot, add the jammy-proposed repository, and install those proposed packages. My system booted up successfully afterward.

My sincere thanks for the speedy fix.

[Stavros Moiras (wirespecter)]

I get the same error. Is there any ISO image already built that includes the latest fix?

[Julian Andres Klode (juliank)]

Grub update for images is tracked in

[Bug 2004164] 2022v1 signed cd-boot-images

[Julian Andres Klode (juliank)]

The updates for bionic and focal still need verification, though I guess they're the same binary (just different signature due to time of signing) so it's a bit pointless.

I have added blocked tags for those uploads as the changes are a bit larger and we know there are issues with it (three more patches have been posted upstream to avoid OOM and performance regressions), so let's do release jammy and kinetic for now only.

We have decided last week that while we know there are these bugs we tested the existing updates for a while now, and don't want to introduce regression potential by picking up those fixes before the point release.

[Adam (adambzh)]

Is there an expected release date for Kinetic?
And will there be a new version of Kinetic live CD with this patch? since current version just won't boot.

[Julian Andres Klode (juliank)]

No there won't be an updated image for kinetic.

[Olaf Seibert (oseibert-sys11)]

I had a server that suffered from the same problem when netbooting a live system (which has a fairly large initrd).

I tried with the file "grubnetx64.efi.signed" of "grub-efi-amd64-signed 1.187.3~22.04.1+2.06-2ubuntu14.1" from jammy-proposed/main, and the problem was gone.

However I would need this version for focal (our automatic deployment of grub is based on focal at the moment).

[Marietto (marietto2008)]

Hello to everyone. I'm using kinetic and I have this problem. At the very top of this page I read that there is a workaround here :

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125

but that link does not work. So,I would like to ask what should I do to fix this bug. Thanks.

[sudodus (nio-wiklund)]

@ Marietto,

Try this link:

https://bugs.launchpad.net/oem-priority/+bug/1842320/comments/125

[Marietto (marietto2008)]

Sorry,my mistake,I have two ubuntu installations. The first one runs kinetic,the second one jammy. I see the error on both the installations,but kinetic is able to ignore it,jammy can't. I gave a look at the link that you have provided,but the solutions proposed aren't detailed. It's not explained well what should I do technically to fix the error. Sorry If I'm a newbie and that's because I'm looking for the easiest method. For example,is there an alternative to grub to boot jammy ? Maybe if I install rEFIt,will it be able to boot my jammy ? thanks.

[Julian Andres Klode (juliank)]

For focal and bionic I started lxd vms with 4-8GB each (the allocate over 4GB parts I don't think are actually needed, lol), rebuilt initrd with the initrams tools hook, upgraded grub to the ubuntu14.1 and made sure it booted.

Then I downgraded grubs again to ubuntu14 and made sure it failed with out of memory again.

[fantaz (jskific)]

I just updated the the following packages:
firmware-sof-signed:amd64 (2.0-1ubuntu4, 2.0-1ubuntu4.1),
grub-efi-amd64-signed:amd64 (1.182~22.04.1+2.06-2ubuntu10, 1.187.2+2.06-2ubuntu14),
grub-efi-amd64-bin:amd64 (2.06-2ubuntu10, 2.06-2ubuntu14)
and after the restart I get the same message
For what it's worth, I0m on kubuntu 22.04 and have a nvidia-driver-525 installed

[fantaz (jskific)]

I just updated the the following packages:
firmware-sof-signed:amd64 (2.0-1ubuntu4, 2.0-1ubuntu4.1),
grub-efi-amd64-signed:amd64 (1.182~22.04.1+2.06-2ubuntu10, 1.187.2+2.06-2ubuntu14),
grub-efi-amd64-bin:amd64 (2.06-2ubuntu10, 2.06-2ubuntu14)
I'm on kubuntu 22.04, have a 5.15-60-generic kernel, nvidia-driver-525 installed and after the restart I get the same message: "out of memory"

[Julian Andres Klode (juliank)]

@fantaz yes, the version with the fix is the one after that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.1

---------------
grub2-unsigned (2.06-2ubuntu14.1) kinetic; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 11:51:57 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.3

---------------
grub2-signed (1.187.3) kinetic; urgency=medium

  * Build against grub2 2.06-2ubuntu14.1 (LP: #1842320)
  * Add check that we are not signed with a revoked key
  * Incorporate changes from the bionic branch:
    - Bump grub2-common dependency to 2.02~beta2-36ubuntu3.33 in xenial and
      2.02-2ubuntu8.25 in bionic to fix LP #1995751

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 12:09:44 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for grub2-unsigned has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.1

---------------
grub2-unsigned (2.06-2ubuntu14.1) kinetic; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 11:51:57 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.3~22.04.1

---------------
grub2-signed (1.187.3~22.04.1) jammy; urgency=medium

  * Build against grub2 2.06-2ubuntu14.1 (LP: #1842320)
  * Add check that we are not signed with a revoked key
  * Incorporate changes from the bionic branch:
    - Bump grub2-common dependency to 2.02~beta2-36ubuntu3.33 in xenial and
      2.02-2ubuntu8.25 in bionic to fix LP #1995751

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 12:09:44 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.1

---------------
grub2-unsigned (2.06-2ubuntu14.1) kinetic; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 11:51:57 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.3~20.04.1

---------------
grub2-signed (1.187.3~20.04.1) focal; urgency=medium

  * Build against grub2 2.06-2ubuntu14.1 (LP: #1842320)
  * Add check that we are not signed with a revoked key
  * Incorporate changes from the bionic branch:
    - Bump grub2-common dependency to 2.02~beta2-36ubuntu3.33 in xenial and
      2.02-2ubuntu8.25 in bionic to fix LP #1995751

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 12:09:44 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.1

---------------
grub2-unsigned (2.06-2ubuntu14.1) kinetic; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 11:51:57 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.3~18.04.1

---------------
grub2-signed (1.187.3~18.04.1) bionic; urgency=medium

  * Build against grub2 2.06-2ubuntu14.1 (LP: #1842320)
  * Add check that we are not signed with a revoked key
  * Incorporate changes from the bionic branch:
    - Bump grub2-common dependency to 2.02~beta2-36ubuntu3.33 in xenial and
      2.02-2ubuntu8.25 in bionic to fix LP #1995751

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 12:09:44 +0100

[Marietto (marietto2008)]

Excuse me for my ignorance. I see that the bug has been fixed. Do you confirm that,to fix it,I just need to install the package "grub-efi-amd64_2.06-2ubuntu14.1_amd64.deb" with the command "dpkg -i grub-efi-amd64_2.06-2ubuntu14.1_amd64.deb" or I just need to do something else ? And installing this version of grub is supposed to work with what version of ubuntu ? And it should work even on Debian 11 ? yes,because even on Debian I see the same bug.

[Aaron Rainbolt (arraybolt3)]

@marietto: The bug has a fix released for Ubuntu 18.04, 20.04, 22.04, and 22.10. No user intervention should be needed - the update should be installed in the course of normal system updates. You should be able to make it install with "sudo apt update && sudo apt full-upgrade".

This bug fix does not affect Debian, so no, it won't work on Debian 11 or any other version. You'll have to wait for Debian to release a fix for it, as Debian and Ubuntu are different operating systems despite the fact that Ubuntu is based on Debian.

[Marietto (marietto2008)]

Unfortunately doing "sudo apt update && sudo apt full-upgrade" didn't solve the problem. So I have installed the package "grub-efi-amd64_2.06-2ubuntu14.1_amd64.deb" individually. Now I'm going to check if the bug has been fixed.

[sh (qbuzpxrhc2sj)]

unfortunately grub-efi-amd64 2.06-2ubuntu14.1 does *not* fix the issue.

as soon if we set more than 32GB of RAM for the IGFX in BIOS-Setup, grub errors out with OOM.

unfortunately at least firefox run into OOM if we only set 32GB with two 4K-monitors attached, so the system is not really usable (it's not a OOM of normal Memory because there's 100GB available while firefox runs for a moment)

please tell us, if there're suggested patches to try. we're able to patch and compile the package by ourselves

[Julian Andres Klode (juliank)]

@sh you want to file a new issue for your issue, this one has been resolved.

You may also be hit by the regression that is fixed in git and the ubuntu-uefi-team PPA (without signed packages obviously), but those regression fixes also cause regressions in other distros, so they were held back too.

[Marietto (marietto2008)]

I want to inform you that your fix for me does not work. The problem disappear when the grub menu is populated with a small number of entries. But when the entries are more,I don't know how many they should be,the error "out of memory" comes back again. I'm sure it is like this for ubuntu 22.10,but probably even for ubuntu 22.04. If you want to help me to troubleshot the error,I'm available to help.

[Julian Andres Klode (juliank)]

Marrietto please do not make fraudulent changes to the bug title. The number of entries is not relevant to the bug. The testing setup did not even have a menu.

If you experience issues, again, file a separate bug. This bug has been resolved.

[jeremyszu (os369510)]

BTW, based on discussion[1].

The grub2@upstream introduce Loadfile2[2] for x86 which may related to this issue, FYI.

[1] https://lists.gnu.org/archive/html/grub-devel/2022-12/msg00135.html
[2] https://lists.gnu.org/archive/html/grub-devel/2023-05/msg00112.html

[TomaszChmielewski (mangoo-wpkg)]

I see the same when booting on Ubuntu 23.04 on LG Gram 17Z90R (32 GB RAM).

Loading initial ramdisk
error: out of memory

Then, it complains something about zstd image being corrupted.

However, after a few seconds, booting continues and everything seems to work correctly.

[Tobin Davis (gruemaster)]

I'm seeing this crash on 22.04 with the latest nVidia driver stack installed from nVidia (535). This system has the updated grub.

Relevant packages:
grub-efi-amd64-bin 2.06-2ubuntu14.1
nvidia-driver-535 535.54.03-0ubuntu1
linux-image-5.19.0-50-generic

My solution was to rebuild initramfs after changing /etc/initramfs-tools/initramfs.conf to have:
MODULES=dep

This reduces the size of the initrd by almost 1/3.

If anyone wants more details on this system, please let me know.

[Julian Andres Klode (juliank)]

@Tobin It's likely fixed now with the latest ubuntu14.2 update.