[SRU] unable to boot guest with large memory when SEV is enabled on host
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
grub2-unsigned (Ubuntu) |
Fix Released
|
High
|
gerald.yang | ||
Focal |
Fix Released
|
High
|
gerald.yang | ||
Jammy |
Fix Released
|
High
|
gerald.yang | ||
Kinetic |
Fix Released
|
High
|
gerald.yang |
Bug Description
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https:/
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_
swiotlb_
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x000000000000
[ 0.005881] memory[0x1] [0x000000000010
[ 0.005883] memory[0x2] [0x000000007eb1
[ 0.005885] memory[0x3] [0x000000007fbf
[ 0.005886] memory[0x4] [0x000000010000
The biggest one is:
[ 0.005881] memory[0x1] [0x000000000010
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a
[ 0.005894] reserved[0x3] [0x000000007e13
[ 0.005896] reserved[0x4] [0x000000007e84
[ 0.005897] reserved[0x5] [0x000000007ee9
Now the biggest available range is
[0x000000000010
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x000000000010
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x000000000000
[ 0.005835] memory[0x1] [0x000000000010
[ 0.005837] memory[0x2] [0x000000007eb1
[ 0.005838] memory[0x3] [0x000000007fbf
[ 0.005840] memory[0x4] [0x000000010000
The biggest one is also:
[ 0.005835] memory[0x1] [0x000000000010
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9b
[ 0.005848] reserved[0x3] [0x000000007e13
[ 0.005849] reserved[0x4] [0x000000007e84
[ 0.005851] reserved[0x5] [0x000000007ee9
Now the biggest one is:
[0x000000003e13
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfb
So now, the biggest contiguous memory becomes
[0x000000003c7c
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https:/
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https:/
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/
Make sure the running kernel in VM is 5.15
Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[ Where problems could occur ]
Originally, allocate memory for initrd/
This patch only adjusts the memory allocation that
1. check if it can get memory from less than 0x7fffffff (GRUB_EFI_
2. if step 1 fails, then check if it can get memory from less than 0xffffffffffffffff (GRUB_EFI_
With this patch, initrd/
The only issue is that if initrd is too big that needs to be allocated from higher than 4G (GRUB_EFI_
But this issue is being handled by lp:1842320
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320
Related branches
- Ubuntu Core Development Team: Pending requested
-
Diff: 33477 lines (+26270/-719) (has conflicts)219 files modifiedChangeLog (+5278/-0)
INSTALL (+31/-21)
Makefile.am (+1/-1)
Makefile.in (+270/-54)
Makefile.util.am (+16/-7)
Makefile.util.def (+15/-40)
NEWS (+14/-0)
README (+6/-0)
acinclude.m4 (+36/-2)
aclocal.m4 (+1/-0)
autogen.sh (+1/-1)
conf/Makefile.common (+2/-0)
conf/Makefile.extra-dist (+21/-0)
config-util.h.in (+6/-0)
config.h.in (+0/-2)
configure (+192/-39)
configure.ac (+99/-104)
debian/.git-dpm (+3/-0)
debian/NEWS (+8/-0)
debian/README.source (+3/-0)
debian/apport/source_grub2.py (+14/-5)
debian/build-efi-images (+27/-11)
debian/changelog (+1421/-1)
debian/control (+92/-26)
debian/dirs.in (+1/-0)
debian/grub-check-signatures (+21/-0)
debian/grub-common.service (+13/-0)
debian/grub-efi-amd64-bin.maintscript.in (+1/-0)
debian/grub-efi-arm64-bin.maintscript.in (+1/-0)
debian/grub-extras/915resolution/.gitignore (+3/-0)
debian/grub-extras/915resolution/915resolution.c (+29/-8)
debian/grub-extras/disabled/gpxe/.gitignore (+3/-0)
debian/grub-extras/disabled/zfs/.gitignore (+5/-0)
debian/grub-extras/lua/.gitignore (+3/-0)
debian/grub-extras/ntldr-img/.gitignore (+3/-0)
debian/grub.d/05_debian_theme (+2/-2)
debian/legacy/upgrade-from-grub-legacy (+3/-1)
debian/patches/0076-ubuntu-Make-the-linux-command-in-EFI-grub-always-try.patch (+37/-0)
debian/patches/0077-ubuntu-Update-the-linux-boot-protocol-version-check.patch (+7/-0)
debian/patches/0096-linuxefi-fail-kernel-validation-without-shim-protoco.patch (+52/-0)
debian/patches/0099-chainloader-Avoid-a-double-free-when-validation-fail.patch (+7/-0)
debian/patches/0105-efilinux-Fix-integer-overflows-in-grub_cmd_initrd.patch (+7/-0)
debian/patches/0129-loader-efi-chainloader-grub_load_and_start_image-doe.patch (+68/-0)
debian/patches/0130-loader-efi-chainloader-simplify-the-loader-state.patch (+334/-0)
debian/patches/0131-commands-boot-Add-API-to-pass-context-to-loader.patch (+157/-0)
debian/patches/0132-loader-efi-chainloader-Use-grub_loader_set_ex.patch (+144/-0)
debian/patches/0133-loader-i386-efi-linux-Use-grub_loader_set_ex.patch (+306/-0)
debian/patches/0134-loader-i386-efi-linux-Fix-a-memory-leak-in-the-initr.patch (+72/-0)
debian/patches/0135-kern-efi-sb-Reject-non-kernel-files-in-the-shim_lock.patch (+98/-0)
debian/patches/0136-kern-file-Do-not-leak-device_name-on-error-in-grub_f.patch (+36/-0)
debian/patches/0137-video-readers-png-Abort-sooner-if-a-read-operation-f.patch (+196/-0)
debian/patches/0138-video-readers-png-Refuse-to-handle-multiple-image-he.patch (+26/-0)
debian/patches/0139-video-readers-png-Drop-greyscale-support-to-fix-heap.patch (+167/-0)
debian/patches/0140-video-readers-png-Avoid-heap-OOB-R-W-inserting-huff-.patch (+37/-0)
debian/patches/0141-video-readers-png-Sanity-check-some-huffman-codes.patch (+38/-0)
debian/patches/0142-video-readers-jpeg-Abort-sooner-if-a-read-operation-.patch (+253/-0)
debian/patches/0143-video-readers-jpeg-Do-not-reallocate-a-given-huff-ta.patch (+27/-0)
debian/patches/0144-video-readers-jpeg-Refuse-to-handle-multiple-start-o.patch (+41/-0)
debian/patches/0145-video-readers-jpeg-Block-int-underflow-wild-pointer-.patch (+72/-0)
debian/patches/0146-normal-charset-Fix-array-out-of-bounds-formatting-un.patch (+32/-0)
debian/patches/0147-net-netbuff-Block-overly-large-netbuff-allocs.patch (+44/-0)
debian/patches/0148-net-ip-Do-IP-fragment-maths-safely.patch (+42/-0)
debian/patches/0149-net-dns-Fix-double-free-addresses-on-corrupt-DNS-res.patch (+54/-0)
debian/patches/0150-net-dns-Don-t-read-past-the-end-of-the-string-we-re-.patch (+69/-0)
debian/patches/0151-net-tftp-Prevent-a-UAF-and-double-free-from-a-failed.patch (+110/-0)
debian/patches/0152-net-tftp-Avoid-a-trivial-UAF.patch (+33/-0)
debian/patches/0153-net-http-Do-not-tear-down-socket-if-it-s-already-bee.patch (+39/-0)
debian/patches/0154-net-http-Fix-OOB-write-for-split-http-headers.patch (+44/-0)
debian/patches/0155-net-http-Error-out-on-headers-with-LF-without-CR.patch (+46/-0)
debian/patches/0156-fs-f2fs-Do-not-read-past-the-end-of-nat-journal-entr.patch (+70/-0)
debian/patches/0157-fs-f2fs-Do-not-read-past-the-end-of-nat-bitmap.patch (+130/-0)
debian/patches/0158-fs-f2fs-Do-not-copy-file-names-that-are-too-long.patch (+36/-0)
debian/patches/0159-fs-btrfs-Fix-several-fuzz-issues-with-invalid-dir-it.patch (+74/-0)
debian/patches/0160-fs-btrfs-Fix-more-ASAN-and-SEGV-issues-found-with-fu.patch (+132/-0)
debian/patches/0161-fs-btrfs-Fix-more-fuzz-issues-related-to-chunks.patch (+74/-0)
debian/patches/0241-Call-hwmatch-only-on-the-grub-pc-platform.patch (+47/-0)
debian/patches/RISC-V-Update-image-header.patch (+84/-0)
debian/patches/RISC-V-Use-common-linux-loader.patch (+120/-0)
debian/patches/at_keyboard-module-init.patch (+4/-1)
debian/patches/bash-completion-drop-have-checks.patch (+5/-2)
debian/patches/blacklist-1440x900x32.patch (+4/-1)
debian/patches/bootp-new-net_bootp6-command.patch (+22/-17)
debian/patches/bootp-process-dhcpack-http-boot.patch (+20/-15)
debian/patches/cherrypick-efi-grub_efi_close_protocol.patch (+79/-0)
debian/patches/cherrypick-efinet-correct-closing-snp-protocol.patch (+106/-0)
debian/patches/core-in-fs.patch (+3/-4)
debian/patches/debug_verifiers.patch (+27/-0)
debian/patches/default-grub-d.patch (+34/-17)
debian/patches/dejavu-font-path.patch (+22/-0)
debian/patches/disable-floppies.patch (+1/-2)
debian/patches/dpkg-version-comparison.patch (+3/-4)
debian/patches/efi-EFI-Device-Tree-Fixup-Protocol.patch (+140/-0)
debian/patches/efi-add-definition-of-LoadFile2-protocol.patch (+61/-0)
debian/patches/efi-correct-struct-grub_efi_boot_services.patch (+28/-0)
debian/patches/efi-implement-grub_efi_run_image.patch (+900/-0)
debian/patches/efi-implemented-LoadFile2-initrd-loading-protocol-fo.patch (+183/-0)
debian/patches/efi-variable-storage-minimise-writes.patch (+60/-11)
debian/patches/efinet-set-dns-from-uefi-proto.patch (+13/-8)
debian/patches/efinet-set-network-from-uefi-devpath.patch (+8/-5)
debian/patches/efinet-uefi-ipv6-pxe-support.patch (+8/-5)
debian/patches/efivar-check-that-efivarfs-is-writeable.patch (+74/-0)
debian/patches/fat-fix-listing-the-root-directory.patch (+46/-0)
debian/patches/fdt-add-debug-output-to-devicetree-command.patch (+31/-0)
debian/patches/gettext-quiet.patch (+4/-1)
debian/patches/gfxpayload-dynamic.patch (+23/-7)
debian/patches/gfxpayload-keep-default.patch (+9/-0)
debian/patches/grub-install-pvxen-paths.patch (+14/-3)
debian/patches/grub-legacy-0-based-partitions.patch (+1/-2)
debian/patches/grub.cfg-400.patch (+2/-3)
debian/patches/ieee1275-clear-reset.patch (+4/-1)
debian/patches/ignore-grub_func_test-failures.patch (+4/-1)
debian/patches/insmod-xzio-and-lzopio-on-xen.patch (+7/-0)
debian/patches/install-efi-adjust-distributor.patch (+33/-0)
debian/patches/install-efi-fallback.patch (+5/-2)
debian/patches/install-efi-ubuntu-flavours.patch (+3/-0)
debian/patches/install-locale-langpack.patch (+10/-7)
debian/patches/install-powerpc-machtypes.patch (+18/-11)
debian/patches/install-stage2-confusion.patch (+9/-6)
debian/patches/linux-ignore-FDT-unless-we-need-to-modify-it.patch (+80/-0)
debian/patches/linux_xen-Properly-load-multiple-initrd-files.patch (+123/-0)
debian/patches/linux_xen-Properly-order-multiple-initrd-files.patch (+79/-0)
debian/patches/linuxefi-Invalidate-i-cache-before-starting-the-kern.patch (+111/-0)
debian/patches/linuxefi-do-not-validate-kernels-twice.patch (+227/-0)
debian/patches/loader-Move-arm64-linux-loader-to-common-code.patch (+1091/-0)
debian/patches/loader-drop-argv-argument-in-grub_initrd_load.patch (+178/-0)
debian/patches/maybe-quiet.patch (+30/-21)
debian/patches/minilzo-2.10.patch (+2538/-0)
debian/patches/mkconfig-loopback.patch (+11/-4)
debian/patches/mkconfig-mid-upgrade.patch (+3/-0)
debian/patches/mkconfig-nonexistent-loopback.patch (+11/-8)
debian/patches/mkconfig-other-inits.patch (+14/-3)
debian/patches/mkconfig-recovery-title.patch (+17/-10)
debian/patches/mkconfig-signed-kernel.patch (+9/-0)
debian/patches/mkconfig-ubuntu-distributor.patch (+7/-0)
debian/patches/mkconfig-ubuntu-recovery.patch (+18/-5)
debian/patches/mkimage-fix-section-sizes.patch (+108/-0)
debian/patches/mkrescue-efi-modules.patch (+6/-3)
debian/patches/net-read-bracketed-ipv6-addr.patch (+20/-16)
debian/patches/no-devicetree-if-secure-boot.patch (+8/-5)
debian/patches/no-insmod-on-sb.patch (+8/-58)
debian/patches/olpc-prefix-hack.patch (+1/-2)
debian/patches/pc-verifiers-module.patch (+166/-0)
debian/patches/ppc64el-disable-vsx.patch (+4/-1)
debian/patches/probe-fusionio.patch (+8/-5)
debian/patches/quick-boot-lvm.patch (+6/-3)
debian/patches/quick-boot.patch (+34/-20)
debian/patches/restore-mkdevicemap.patch (+26/-13)
debian/patches/rhboot-f34-dont-use-int-for-efi-status.patch (+7/-0)
debian/patches/rhboot-f34-efinet-also-use-the-firmware-acceleration-for-http.patch (+26/-0)
debian/patches/rhboot-f34-make-exit-take-a-return-code.patch (+68/-0)
debian/patches/rhboot-f34-make-pmtimer-tsc-calibration-fast.patch (+11/-0)
debian/patches/rhboot-try-to-pick-better-locations-for-kernel-and-initrd.patch (+215/-0)
debian/patches/riscv-adjust-march-flags-for-binutils-2.38.patch (+43/-0)
debian/patches/series (+122/-4)
debian/patches/skip-grub_cmd_set_date.patch (+4/-1)
debian/patches/sleep-shift.patch (+3/-0)
debian/patches/suse-AUDIT-0-http-boot-tracker-bug.patch (+68/-0)
debian/patches/suse-add-support-for-UEFI-network-protocols.patch (+4941/-0)
debian/patches/suse-grub.texi-add-net_bootp6-document.patch (+49/-0)
debian/patches/tests-ahci-update-qemu-device-name.patch (+33/-0)
debian/patches/tpm-unknown-error-non-fatal.patch (+30/-0)
debian/patches/ubuntu-add-devicetree-command-support.patch (+7/-0)
debian/patches/ubuntu-add-initrd-less-boot-fallback.patch (+44/-0)
debian/patches/ubuntu-add-initrd-less-boot-messages.patch (+24/-0)
debian/patches/ubuntu-boot-from-multipath-dependent-symlink.patch (+7/-0)
debian/patches/ubuntu-disable-LOAD-FILE2-protocol-for-initrd-on-ARM.patch (+63/-0)
debian/patches/ubuntu-dont-verify-loopback-images.patch (+11/-0)
debian/patches/ubuntu-efi-allow-loopmount-chainload.patch (+27/-0)
debian/patches/ubuntu-fix-lzma-decompressor-objcopy.patch (+10/-0)
debian/patches/ubuntu-fix-reproducible-squashfs-test.patch (+7/-0)
debian/patches/ubuntu-flavour-order.patch (+17/-0)
debian/patches/ubuntu-fuse3.patch (+108/-0)
debian/patches/ubuntu-grub-install-extra-removable.patch (+37/-0)
debian/patches/ubuntu-install-signed.patch (+41/-0)
debian/patches/ubuntu-linuxefi-arm64-set-base-addr.patch (+22/-0)
debian/patches/ubuntu-linuxefi-arm64.patch (+90/-0)
debian/patches/ubuntu-linuxefi.patch (+510/-0)
debian/patches/ubuntu-mkconfig-leave-breadcrumbs.patch (+10/-0)
debian/patches/ubuntu-os-prober-auto.patch (+51/-0)
debian/patches/ubuntu-recovery-dis_ucode_ldr.patch (+15/-0)
debian/patches/ubuntu-resilient-boot-boot-order.patch (+45/-0)
debian/patches/ubuntu-resilient-boot-ignore-alternative-esps.patch (+11/-0)
debian/patches/ubuntu-shorter-version-info.patch (+18/-0)
debian/patches/ubuntu-skip-disk-by-id-lvm-pvm-uuid-entries.patch (+10/-0)
debian/patches/ubuntu-speed-zsys-history.patch (+34/-0)
debian/patches/ubuntu-support-initrd-less-boot.patch (+27/-0)
debian/patches/ubuntu-temp-keep-auto-nvram.patch (+7/-0)
debian/patches/ubuntu-verifiers-last.patch (+59/-0)
debian/patches/ubuntu-zfs-enhance-support.patch (+46/-0)
debian/patches/ubuntu-zfs-gfxpayload-dynamic.patch (+95/-0)
debian/patches/ubuntu-zfs-gfxpayload-keep-default.patch (+38/-0)
debian/patches/ubuntu-zfs-insmod-xzio-and-lzopio-on-xen.patch (+32/-0)
debian/patches/ubuntu-zfs-maybe-quiet.patch (+72/-0)
debian/patches/ubuntu-zfs-mkconfig-recovery-title.patch (+49/-0)
debian/patches/ubuntu-zfs-mkconfig-signed-kernel.patch (+51/-0)
debian/patches/ubuntu-zfs-mkconfig-ubuntu-distributor.patch (+36/-0)
debian/patches/ubuntu-zfs-mkconfig-ubuntu-recovery.patch (+66/-0)
debian/patches/ubuntu-zfs-quick-boot.patch (+50/-0)
debian/patches/ubuntu-zfs-vt-handoff.patch (+77/-0)
debian/patches/uefi-firmware-setup.patch (+3/-0)
debian/patches/uefi-secure-boot-cryptomount.patch (+11/-0)
debian/patches/vsnprintf-upper-case-hex.patch (+3/-0)
debian/patches/vt-handoff.patch (+9/-2)
debian/patches/wubi-no-windows.patch (+6/-3)
debian/patches/xen-no-xsm-policy-in-non-xsm-options.patch (+34/-0)
debian/patches/xfs-fix-v4-superblock.patch (+121/-0)
debian/patches/zpool-full-device-name.patch (+4/-1)
debian/patches/zstd-require-8-byte-buffer.patch (+63/-0)
debian/postinst.in (+91/-7)
debian/postrm.in (+2/-2)
debian/rules (+113/-10)
debian/sbat.debian.csv.in (+3/-0)
debian/sbat.ubuntu.csv.in (+3/-0)
debian/signing-template/control.in (+1/-1)
dev/null (+0/-1)
docs/Makefile.in (+2/-2)
docs/grub-dev.info (+113/-45)
docs/grub-dev.texi (+65/-1)
docs/grub.info (+2/-1)
affects: | linux (Ubuntu) → grub2-unsigned (Ubuntu) |
Changed in grub2-unsigned (Ubuntu): | |
assignee: | nobody → gerald.yang (gerald-yang-tw) |
importance: | Undecided → High |
status: | New → In Progress |
Changed in grub2-unsigned (Ubuntu Jammy): | |
status: | New → In Progress |
Changed in grub2-unsigned (Ubuntu Focal): | |
status: | New → In Progress |
assignee: | nobody → gerald.yang (gerald-yang-tw) |
Changed in grub2-unsigned (Ubuntu Jammy): | |
assignee: | nobody → gerald.yang (gerald-yang-tw) |
importance: | Undecided → High |
Changed in grub2-unsigned (Ubuntu Focal): | |
importance: | Undecided → High |
description: | updated |
description: | updated |
description: | updated |
tags: | added: sts |
description: | updated |
description: | updated |
description: | updated |
tags: | added: foundations-todo |
Changed in grub2-unsigned (Ubuntu Jammy): | |
status: | In Progress → Triaged |
Changed in grub2-unsigned (Ubuntu Focal): | |
status: | In Progress → Triaged |
Changed in grub2-unsigned (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
tags: | removed: foundations-todo |
patch for focal