Can't boot: "error: out of memory." immediately after the grub menu

[Gordon Mckeown]

[Workaround]

Today, 2023-02-14, the following link works to find the workarounds:

https://bugs.launchpad.net/oem-priority/+bug/1842320/comments/125

[Impact]

 * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2.

 * Some real cases from OEM projects:

In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed.

In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file:

```
#0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376
#1 0x000000007dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408
#2 0x000000007dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076)
    at ../../../grub-core/kern/verifiers.c:150
#3 0x000000007dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem",
    type=131076) at ../../../grub-core/kern/file.c:121
#4 0x000000007bcd5a30 in ?? ()
#5 0x000000007fe21247 in ?? ()
#6 0x000000007bc030c8 in ?? ()
#7 0x000000017fe21238 in ?? ()
#8 0x000000007bcd5320 in ?? ()
#9 0x000000007fe21250 in ?? ()
#10 0x0000000000000000 in ?? ()
```

Based on grub_mm_dump, we can see the memory region starvation in <1G addresses:

Type Start End # Pages Attributes
Available 0000000000000000-0000000000086FFF 0000000000000087 000000000000000F
BS_Data 0000000000087000-0000000000087FFF 0000000000000001 000000000000000F
Available 0000000000088000-000000000009EFFF 0000000000000017 000000000000000F
Reserved 000000000009F000-000000000009FFFF 0000000000000001 000000000000000F
Available 0000000000100000-0000000000FFFFFF 0000000000000F00 000000000000000F
LoaderCode 0000000001000000-0000000001021FFF 0000000000000022 000000000000000F
Available 0000000001022000-00000000238A7FFF 0000000000022886 000000000000000F
BS_Data 00000000238A8000-0000000023927FFF 0000000000000080 000000000000000F
Available 0000000023928000-0000000028860FFF 0000000000004F39 000000000000000F
BS_Data 0000000028861000-000000002AB09FFF 00000000000022A9 000000000000000F
LoaderCode 000000002AB0A000-000000002ACF8FFF 00000000000001EF 000000000000000F
BS_Data 000000002ACF9000-000000002B2FAFFF 0000000000000602 000000000000000F
Available 000000002B2FB000-000000002B611FFF 0000000000000317 000000000000000F
BS_Data 000000002B612000-000000002B630FFF 000000000000001F 000000000000000F
Available 000000002B631000-000000002B632FFF 0000000000000002 000000000000000F
BS_Data 000000002B633000-000000002B63CFFF 000000000000000A 000000000000000F
Available 000000002B63D000-000000002B649FFF 000000000000000D 000000000000000F
BS_Data 000000002B64A000-000000002B64EFFF 0000000000000005 000000000000000F
Available 000000002B64F000-000000002B666FFF 0000000000000018 000000000000000F
BS_Data 000000002B667000-000000002D8D5FFF 000000000000226F 000000000000000F
LoaderCode 000000002D8D6000-000000002D8E9FFF 0000000000000014 000000000000000F
BS_Data 000000002D8EA000-000000002D925FFF 000000000000003C 000000000000000F
LoaderCode 000000002D926000-000000002D932FFF 000000000000000D 000000000000000F
BS_Data 000000002D933000-000000002D969FFF 0000000000000037 000000000000000F
BS_Code 000000002D96A000-000000002D973FFF 000000000000000A 000000000000000F
BS_Data 000000002D974000-000000002E377FFF 0000000000000A04 000000000000000F
Available 000000002E378000-000000002E37AFFF 0000000000000003 000000000000000F
...
Reserved 000000003C08B000-000000004192FFFF 00000000000058A5 000000000000000F
ACPI_NVS 0000000041930000-0000000041B2FFFF 0000000000000200 000000000000000F
ACPI_Recl 0000000041B30000-0000000041BFEFFF 00000000000000CF 000000000000000F
BS_Data 0000000041BFF000-0000000041BFFFFF 0000000000000001 000000000000000F
Available 0000000100000000-00000002AB7FFFFF 00000000001AB800 000000000000000F
Reserved 00000000000A0000-00000000000FFFFF 0000000000000060 0000000000000000
Reserved 0000000041C00000-0000000043FFFFFF 0000000000002400 0000000000000000
Reserved 0000000044000000-0000000047FFFFFF 0000000000004000 000000000000000F
Reserved 0000000049400000-00000000495FFFFF 0000000000000200 000000000000000F
Reserved 000000004C000000-000000004FFFFFFF 0000000000004000 0000000000000009
Reserved 0000000050000000-00000000547FFFFF 0000000000004800 0000000000000000
MMIO 00000000C0000000-00000000CFFFFFFF 0000000000010000 8000000000000001
Reserved 00000000FED20000-00000000FED7FFFF 0000000000000060 0000000000000000
MMIO 00000000FF800000-00000000FFFFFFFF 0000000000000800 8000000000001000
...
  LoaderCode: 562 Pages (2,301,952 Bytes)
  LoaderData: 0 Pages (0 Bytes)
...
  Available : 1,917,598 Pages (7,854,481,408 Bytes)

Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages().

As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory.

 * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot).

[Test Plan]
tl;dr: Create an initrd that fails to boot with the old grub and boot it successfully with the new grub.

 * detailed instructions how to reproduce the bug:

1. Any method to grow the initramfs, such as install nvidia-driver.

2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like:

```
$ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file
#!/bin/sh

PREREQ=""

prereqs()
{
        echo "$PREREQ"
}

case $1 in
# get pre-requisites
prereqs)
        prereqs
        exit 0
        ;;
esac

. /usr/share/initramfs-tools/hook-functions
dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500
```

And then update-initramfs

 * After cloning grub2 (2.06-2ubuntu16) from lunar-proposed and built, the results as following:

421M /boot/initrd.img-5.17.0-1021-oem # pass
453M /boot/initrd.img-5.17.0-1024-oem # pass
471M /boot/initrd.img-5.17.0-1024-oem # fail

Only 453M is because verifier will consume the same memory size of initrd in <1G memories.

The loadable initrd will locate at >4G if machine/kernel support it.

[Where problems could occur]

* The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system.
The other change is to modify the “grub-core/kern/efi/mm.c” but it use the original addressing for “arm/arm64/ia64/riscv32/riscv64”.
Thus it should not impact them.

* There are some “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system.

If everything works as expected, then i386 should working good.

If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access.

[Other Info]

 * The package grub2 (2.06-2ubuntu16) is now in lunar-proposed.

---

Upgraded from 19.04 to current 19.10 using "do-release-upgrade -d". Can still boot using the previous 5.0.0-25-generic kernel, but the 5.2.0-15-generic fails to start.

On selecting Ubuntu from Grub, the message "error: out of memory." is immediately shown. Pressing a key attempts to start boot-up but fails to mount root fs.

Machine is HP Spectre X360 with 8GB RAM. Under kernel 5.0.0, free shows the following (run from Gnome terminal):

              total used free shared buff/cache available
Mem: 7906564 1761196 3833240 1020216 2312128 4849224
Swap: 1003516 0 1003516

Kernel packages installed:

linux-generic 5.2.0.15.16 amd64
linux-headers-5.2.0-15 5.2.0-15.16 all
linux-headers-5.2.0-15-generic 5.2.0-15.16 amd64
linux-headers-generic 5.2.0.15.16 amd64
linux-image-5.0.0-25-generic 5.0.0-25.26 amd64
linux-image-5.2.0-15-generic 5.2.0-15.16+signed1 amd64
linux-image-generic 5.2.0.15.16 amd64
linux-modules-5.0.0-25-generic 5.0.0-25.26 amd64
linux-modules-5.2.0-15-generic 5.2.0-15.16 amd64
linux-modules-extra-5.0.0-25-generic 5.0.0-25.26 amd64
linux-modules-extra-5.2.0-15-generic 5.2.0-15.16 amd64

Photo of kernel panic attached.

NVMe drive partition layout (GPT):

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 1050623 1048576 512M EFI System
/dev/nvme0n1p2 1050624 2549759 1499136 732M Linux filesystem
/dev/nvme0n1p3 2549760 1000214527 997664768 475.7G Linux filesystem

$ sudo pvs
  PV VG Fmt Attr PSize PFree
  /dev/mapper/nvme0n1p3_crypt ubuntu-vg lvm2 a-- <475.71g 0

$ sudo lvs
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  root ubuntu-vg -wi-ao---- 474.75g
  swap_1 ubuntu-vg -wi-ao---- 980.00m

Partition 3 is LUKS encrypted. Root LV is ext4.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu7
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: gmckeown 1647 F.... pulseaudio
CurrentDesktop: ubuntu:GNOMEDistroRelease: Ubuntu 19.10
InstallationDate: Installed on 2019-08-15 (18 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 003: ID 8087:0a2b Intel Corp.
 Bus 001 Device 002: ID 04f2:b593 Chicony Electronics Co., Ltd HP Wide Vision FHD Camera
 Bus 001 Device 004: ID 046d:c52b Logitech, Inc. Unifying Receiver
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: HP HP Spectre x360 Convertible 13-ae0xx
Package: linux (not installed)
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.0.0-25-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash
ProcVersionSignature: Ubuntu 5.0.0-25.26-generic 5.0.18
RelatedPackageVersions:
 linux-restricted-modules-5.0.0-25-generic N/A
 linux-backports-modules-5.0.0-25-generic N/A
 linux-firmware 1.181
Tags: eoan
Uname: Linux 5.0.0-25-generic x86_64
UpgradeStatus: Upgraded to eoan on 2019-09-02 (0 days ago)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 05/17/2019
dmi.bios.vendor: AMI
dmi.bios.version: F.25
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: 83B9
dmi.board.vendor: HP
dmi.board.version: 56.43
dmi.chassis.type: 31
dmi.chassis.vendor: HP
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAMI:bvrF.25:bd05/17/2019:svnHP:pnHPSpectrex360Convertible13-ae0xx:pvr:rvnHP:rn83B9:rvr56.43:cvnHP:ct31:cvrChassisVersion:
dmi.product.family: 103C_5335KV HP Spectre
dmi.product.name: HP Spectre x360 Convertible 13-ae0xx
dmi.product.sku: 2QH38EA#ABU
dmi.sys.vendor: HP

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1842320

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

apport information

[Gordon Mckeown (thefluffyone)]

Issue appears to be related to display resolution. Grub was booting into 3840x2160 mode, with the resulting tiny, near-unreadable font. So I set GFXMODE=800x600 and now - unexpectedly - kernel 5.2.0 is able to boot without the memory errors.

Not sure why the high resolution causes an out-of-memory error; a 3840x2160x32 framebuffer should only require ~32MB of RAM.

[MrMEEE (mj-casalogic)]

Same issue here.. workaround works..

[Kai-Heng Feng (kaihengfeng)]

Can you please perform a kernel bisection?

[Steven Mackenzie (z-steven)]

I have a similar issue. I upgraded to 19.10 from 19.04 today.

After grub, I get this message:

"error: out of memory.
Press any key to continue..."

Then kernel panic message (image attached).

I have been unable to successfully boot using the workaround above.

The kernel is shown in grub is 5.3.0-18.
My computer is an Intel NUC (Skull Canyon), 16GB RAM, Intel NVMe, 2 x 2560 x 1440 displays.

[Steven Mackenzie (z-steven)]

Update: My machine boots normally from the USB install media.

[Terry Dawson (tjd-animats)]

ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu8.2
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
DistroRelease: Ubuntu 19.10
HibernationDevice: RESUME=UUID=21a8b99d-8777-4e3e-8fd7-2ad7bb66b24e
InstallationDate: Installed on 2017-01-24 (1068 days ago)
InstallationMedia: Ubuntu 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
IwConfig:
 lo no wireless extensions.

 eno1 no wireless extensions.
Package: linux (not installed)
ProcEnviron:
 LANGUAGE=en_AU:en
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_AU.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-131-generic root=/dev/mapper/ubuntu--vg-root ro plymouth:debug
ProcVersionSignature: Ubuntu 4.4.0-131.157-generic 4.4.134
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-131-generic N/A
 linux-backports-modules-4.4.0-131-generic N/A
 linux-firmware 1.183.3
RfKill:

Tags: eoan
Uname: Linux 4.4.0-131-generic x86_64
UpgradeStatus: Upgraded to eoan on 2019-12-28 (0 days ago)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 08/17/2016
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: KYSKLi70.86A.0041.2016.0817.1130
dmi.board.name: NUC6i7KYB
dmi.board.vendor: Intel Corporation
dmi.board.version: H90766-406
dmi.chassis.type: 3
dmi.chassis.vendor: Intel Corporation
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrKYSKLi70.86A.0041.2016.0817.1130:bd08/17/2016:svn:pn:pvr:rvnIntelCorporation:rnNUC6i7KYB:rvrH90766-406:cvnIntelCorporation:ct3:cvr1.0:

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

apport information

[Terry Dawson (tjd-animats)]

I'd like to echo Steven Mackenzie's report.

The workaround of explicitly configuring GFXMODE does mean that I can hit a key through the "Hit any key to continue" prompt, but the boot still fails. Without the workaround, grub is unresponsive at the "Hit any key.." prompt.

[Terry Dawson (tjd-animats)]

I've opened 1857786 indepdently for my boot issue.

[Ben Hoyt (benhoyt)]

Just noting that I had this same issue on my Dell XPS 9550 laptop, which has a 3840x2160 display. The workaround of setting GFXMODE=800x600 in the grub config worked for me too, thanks! And of course had the nice side effect of actually making the grub menu readable. :-)

[Gordon Mckeown (thefluffyone)]

Just had the problem again after upgrading 20.10 to 21.04.

Kernel 5.8.0-55 boots OK.

Kernel 5.11.0-18 fails with the memory error.

I'm just about to re-add the GFXMODE line to grub, which I hope will (again) work around the problem.

[H (bluesforte)]

I had this issue as well, through ALL of the 5.x kernels. Tried everything mentioned in the thread here, but nothing worked other than going to 4.15 kernel.
This gave me suspicions it was related to something with the UEFI handoff to OS, so I looked for BIOS updates and sure enough there was one from earlier this year. Updated to the latest BIOS from Intel, and everything boots fine and working now.

[Callum L (little-king-john)]

I have this issue on my Dell XPS 9380. I can boot 5.13.0.35, but 0.37 and 0.39 simply show the error: out of memory.

I have tried changing my GFXMODE to 800x600x24, which hwinfo lists as a supported resolution, but grub.... just ignores it? Same with timeout? It just shows in the 4K native res every time, almost as if it's taunting me.

Eventually the 0.35 kernel will be deleted by a system update, and my system will be bricked. I guess I'm going to need to reinstall, aren't I?

[Paulo (ptsneves)]

This is still valid and the issue is more pernicious because with the latest livecd one cannot change the GFXMODE to 800x600. Re-imaging a new ISO with new grub settings is outside most people's abilities nor is it the intention. This means that Ubuntu's livecd lead to a panic in a very popular laptop model.

[rustyx (rustyx2)]

Had the same issue with Ubuntu 22.04 install ISO flashed onto a USB.

Even tried booting "manually" (press "c" to enter GRUB command prompt)

grub> ls
(proc) (hd0) (hd0,gpt1) (hd0,gpt2) (hd0,gpt3) (hd1) . . .
grub> ls (hd0,gpt1)
        Partition hd0,gpt1: Filesystem type ... - Label 'UBUNTU 22_0' ...
grub> set root=(hd0,gpt1)
grub> linux /casper/vmlinuz
grub> initrd /casper/initrd
error: out of memory.
grub>

Couldn't find any way out of this condition (is there?).

Fortunately I already had GRUB installed on the machine so I could proceed from that one.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-signed into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.3~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-unsigned into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Gordon, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.3~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Antoine M (antmfra)]

Hello,

I'm affected by this problem but on my side, it's when I try to install Ubuntu (tested with 22.04 and 23.04dev) with a USB stick. How can I test the resolution with an ISO?

[Matthew L. Dailey (matthew-l-dailey)]

I can confirm that the jammy-proposed package fixes the bug for me on our one affected system at 22.04.1. After installing nvidia-525 drivers, the system would not boot - grub out of memory error and kernel panic.

Updated to grub-efi-amd64_2.06-2ubuntu14.1, grub-efi-amd64-bin_2.06-2ubuntu14.1 and grub-efi-amd64-signed_1.187.3~22.04.1+2.06-2ubuntu14.1 and the system boots normally.

I will change the tag from verification-needed-jammy to verification-done-jammy.

Thanks to everyone for their work on this bug!

[m1st0 (m1st0)]

TESTED
• Reinstalled initramfs-tools-core to undo changes of compression on /sbin/mkinitramfs .
• Confirmed compression set to default values.
• Restored /etc/initramfs-tools/initramfs.conf to default values, with "MODULES=most" .
• Installed grub-efi-amd64/kinetic,now 2.06-2ubuntu14.1 amd64 and related versions of packages.
• Ran the following
  sudo update-initramfs -v -u
  sudo grub-mkconfig -o /boot/grub/grub.cfg
  sudo grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
  sudo update-grub2
• Rebooted

VERIFICATION DONE
• Confirmed fix on MSI GS63VR 6RF laptop. No "out of memory" error on grub boot.
• (For some who mentioned nvidia drivers, I have 525.78.01-0ubuntu0.22.10.1 amd64 installed.)
• My current initrd.img size is larger of course, so if you have multiple versions and your /boot partition is small you may have to remove older versions so you don't run out of room---not in any way meaning the "out of memory" error.

[Kochin (kochinc)]

I also confirm that the proposed update fixed the out of memory error when booting on my computer. The packages I installed and tested:

grub-efi-amd64-bin/jammy-proposed,now 2.06-2ubuntu14.1 amd64 [installed,automatic]
grub-efi-amd64-signed/jammy-proposed,now 1.187.3~22.04.1+2.06-2ubuntu14.1 amd64 [installed]
grub-efi-amd64/jammy-proposed,now 2.06-2ubuntu14.1 amd64 [installed]

The kernel and Nvidia driver installed on the system are:

linux-image-5.15.0-58-generic/jammy-updates,jammy-security,now 5.15.0-58.64 amd64 [installed,automatic]
nvidia-driver-525/jammy-updates,jammy-security,now 525.78.01-0ubuntu0.22.04.1 amd64 [installed]

Since my system hadn't been working because of the error, I had to boot up my computer with a Live CD, mount the filesystem, chroot, add the jammy-proposed repository, and install those proposed packages. My system booted up successfully afterward.

My sincere thanks for the speedy fix.

[Stavros Moiras (wirespecter)]

I get the same error. Is there any ISO image already built that includes the latest fix?

[Julian Andres Klode (juliank)]

Grub update for images is tracked in

[Bug 2004164] 2022v1 signed cd-boot-images

[Julian Andres Klode (juliank)]

The updates for bionic and focal still need verification, though I guess they're the same binary (just different signature due to time of signing) so it's a bit pointless.

I have added blocked tags for those uploads as the changes are a bit larger and we know there are issues with it (three more patches have been posted upstream to avoid OOM and performance regressions), so let's do release jammy and kinetic for now only.

We have decided last week that while we know there are these bugs we tested the existing updates for a while now, and don't want to introduce regression potential by picking up those fixes before the point release.

[Adam (adambzh)]

Is there an expected release date for Kinetic?
And will there be a new version of Kinetic live CD with this patch? since current version just won't boot.

[Julian Andres Klode (juliank)]

No there won't be an updated image for kinetic.

[Olaf Seibert (oseibert-sys11)]

I had a server that suffered from the same problem when netbooting a live system (which has a fairly large initrd).

I tried with the file "grubnetx64.efi.signed" of "grub-efi-amd64-signed 1.187.3~22.04.1+2.06-2ubuntu14.1" from jammy-proposed/main, and the problem was gone.

However I would need this version for focal (our automatic deployment of grub is based on focal at the moment).

[Marietto (marietto2008)]

Hello to everyone. I'm using kinetic and I have this problem. At the very top of this page I read that there is a workaround here :

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125

but that link does not work. So,I would like to ask what should I do to fix this bug. Thanks.

[sudodus (nio-wiklund)]

@ Marietto,

Try this link:

https://bugs.launchpad.net/oem-priority/+bug/1842320/comments/125

[Marietto (marietto2008)]

Sorry,my mistake,I have two ubuntu installations. The first one runs kinetic,the second one jammy. I see the error on both the installations,but kinetic is able to ignore it,jammy can't. I gave a look at the link that you have provided,but the solutions proposed aren't detailed. It's not explained well what should I do technically to fix the error. Sorry If I'm a newbie and that's because I'm looking for the easiest method. For example,is there an alternative to grub to boot jammy ? Maybe if I install rEFIt,will it be able to boot my jammy ? thanks.

[Julian Andres Klode (juliank)]

For focal and bionic I started lxd vms with 4-8GB each (the allocate over 4GB parts I don't think are actually needed, lol), rebuilt initrd with the initrams tools hook, upgraded grub to the ubuntu14.1 and made sure it booted.

Then I downgraded grubs again to ubuntu14 and made sure it failed with out of memory again.

[fantaz (jskific)]

I just updated the the following packages:
firmware-sof-signed:amd64 (2.0-1ubuntu4, 2.0-1ubuntu4.1),
grub-efi-amd64-signed:amd64 (1.182~22.04.1+2.06-2ubuntu10, 1.187.2+2.06-2ubuntu14),
grub-efi-amd64-bin:amd64 (2.06-2ubuntu10, 2.06-2ubuntu14)
and after the restart I get the same message
For what it's worth, I0m on kubuntu 22.04 and have a nvidia-driver-525 installed

[fantaz (jskific)]

I just updated the the following packages:
firmware-sof-signed:amd64 (2.0-1ubuntu4, 2.0-1ubuntu4.1),
grub-efi-amd64-signed:amd64 (1.182~22.04.1+2.06-2ubuntu10, 1.187.2+2.06-2ubuntu14),
grub-efi-amd64-bin:amd64 (2.06-2ubuntu10, 2.06-2ubuntu14)
I'm on kubuntu 22.04, have a 5.15-60-generic kernel, nvidia-driver-525 installed and after the restart I get the same message: "out of memory"

[Julian Andres Klode (juliank)]

@fantaz yes, the version with the fix is the one after that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.1

---------------
grub2-unsigned (2.06-2ubuntu14.1) kinetic; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 11:51:57 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.3

---------------
grub2-signed (1.187.3) kinetic; urgency=medium

  * Build against grub2 2.06-2ubuntu14.1 (LP: #1842320)
  * Add check that we are not signed with a revoked key
  * Incorporate changes from the bionic branch:
    - Bump grub2-common dependency to 2.02~beta2-36ubuntu3.33 in xenial and
      2.02-2ubuntu8.25 in bionic to fix LP #1995751

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 12:09:44 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for grub2-unsigned has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.1

---------------
grub2-unsigned (2.06-2ubuntu14.1) kinetic; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 11:51:57 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.3~22.04.1

---------------
grub2-signed (1.187.3~22.04.1) jammy; urgency=medium

  * Build against grub2 2.06-2ubuntu14.1 (LP: #1842320)
  * Add check that we are not signed with a revoked key
  * Incorporate changes from the bionic branch:
    - Bump grub2-common dependency to 2.02~beta2-36ubuntu3.33 in xenial and
      2.02-2ubuntu8.25 in bionic to fix LP #1995751

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 12:09:44 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.1

---------------
grub2-unsigned (2.06-2ubuntu14.1) kinetic; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 11:51:57 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.3~20.04.1

---------------
grub2-signed (1.187.3~20.04.1) focal; urgency=medium

  * Build against grub2 2.06-2ubuntu14.1 (LP: #1842320)
  * Add check that we are not signed with a revoked key
  * Incorporate changes from the bionic branch:
    - Bump grub2-common dependency to 2.02~beta2-36ubuntu3.33 in xenial and
      2.02-2ubuntu8.25 in bionic to fix LP #1995751

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 12:09:44 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.1

---------------
grub2-unsigned (2.06-2ubuntu14.1) kinetic; urgency=medium

  * Cherry-pick all memory patches from rhboot
    - Allocate initrd > 4 GB (LP: #1842320)
    - Allocate kernels as code, not data (needed for newer firmware)
  * ubuntu: Fix casts on i386-efi target
  * Cherry-pick all the 2.12 memory management changes (LP: #1842320)
  * Allocate executables as CODE, not DATA in chainloader and arm64
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 11:51:57 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.3~18.04.1

---------------
grub2-signed (1.187.3~18.04.1) bionic; urgency=medium

  * Build against grub2 2.06-2ubuntu14.1 (LP: #1842320)
  * Add check that we are not signed with a revoked key
  * Incorporate changes from the bionic branch:
    - Bump grub2-common dependency to 2.02~beta2-36ubuntu3.33 in xenial and
      2.02-2ubuntu8.25 in bionic to fix LP #1995751

 -- Julian Andres Klode <email address hidden> Mon, 30 Jan 2023 12:09:44 +0100

[Marietto (marietto2008)]

Excuse me for my ignorance. I see that the bug has been fixed. Do you confirm that,to fix it,I just need to install the package "grub-efi-amd64_2.06-2ubuntu14.1_amd64.deb" with the command "dpkg -i grub-efi-amd64_2.06-2ubuntu14.1_amd64.deb" or I just need to do something else ? And installing this version of grub is supposed to work with what version of ubuntu ? And it should work even on Debian 11 ? yes,because even on Debian I see the same bug.

[Aaron Rainbolt (arraybolt3)]

@marietto: The bug has a fix released for Ubuntu 18.04, 20.04, 22.04, and 22.10. No user intervention should be needed - the update should be installed in the course of normal system updates. You should be able to make it install with "sudo apt update && sudo apt full-upgrade".

This bug fix does not affect Debian, so no, it won't work on Debian 11 or any other version. You'll have to wait for Debian to release a fix for it, as Debian and Ubuntu are different operating systems despite the fact that Ubuntu is based on Debian.

[Marietto (marietto2008)]

Unfortunately doing "sudo apt update && sudo apt full-upgrade" didn't solve the problem. So I have installed the package "grub-efi-amd64_2.06-2ubuntu14.1_amd64.deb" individually. Now I'm going to check if the bug has been fixed.

[sh (qbuzpxrhc2sj)]

unfortunately grub-efi-amd64 2.06-2ubuntu14.1 does *not* fix the issue.

as soon if we set more than 32GB of RAM for the IGFX in BIOS-Setup, grub errors out with OOM.

unfortunately at least firefox run into OOM if we only set 32GB with two 4K-monitors attached, so the system is not really usable (it's not a OOM of normal Memory because there's 100GB available while firefox runs for a moment)

please tell us, if there're suggested patches to try. we're able to patch and compile the package by ourselves

[Julian Andres Klode (juliank)]

@sh you want to file a new issue for your issue, this one has been resolved.

You may also be hit by the regression that is fixed in git and the ubuntu-uefi-team PPA (without signed packages obviously), but those regression fixes also cause regressions in other distros, so they were held back too.

[Marietto (marietto2008)]

I want to inform you that your fix for me does not work. The problem disappear when the grub menu is populated with a small number of entries. But when the entries are more,I don't know how many they should be,the error "out of memory" comes back again. I'm sure it is like this for ubuntu 22.10,but probably even for ubuntu 22.04. If you want to help me to troubleshot the error,I'm available to help.

[Julian Andres Klode (juliank)]

Marrietto please do not make fraudulent changes to the bug title. The number of entries is not relevant to the bug. The testing setup did not even have a menu.

If you experience issues, again, file a separate bug. This bug has been resolved.

[jeremyszu (os369510)]

BTW, based on discussion[1].

The grub2@upstream introduce Loadfile2[2] for x86 which may related to this issue, FYI.

[1] https://lists.gnu.org/archive/html/grub-devel/2022-12/msg00135.html
[2] https://lists.gnu.org/archive/html/grub-devel/2023-05/msg00112.html

[TomaszChmielewski (mangoo-wpkg)]

I see the same when booting on Ubuntu 23.04 on LG Gram 17Z90R (32 GB RAM).

Loading initial ramdisk
error: out of memory

Then, it complains something about zstd image being corrupted.

However, after a few seconds, booting continues and everything seems to work correctly.

[Tobin Davis (gruemaster)]

I'm seeing this crash on 22.04 with the latest nVidia driver stack installed from nVidia (535). This system has the updated grub.

Relevant packages:
grub-efi-amd64-bin 2.06-2ubuntu14.1
nvidia-driver-535 535.54.03-0ubuntu1
linux-image-5.19.0-50-generic

My solution was to rebuild initramfs after changing /etc/initramfs-tools/initramfs.conf to have:
MODULES=dep

This reduces the size of the initrd by almost 1/3.

If anyone wants more details on this system, please let me know.

[Julian Andres Klode (juliank)]

@Tobin It's likely fixed now with the latest ubuntu14.2 update.