| 2022-09-13 09:53:26 |
gerald.yang |
bug |
|
|
added bug |
| 2022-09-13 09:54:34 |
gerald.yang |
affects |
linux (Ubuntu) |
grub2-unsigned (Ubuntu) |
|
| 2022-09-13 09:54:34 |
gerald.yang |
grub2-unsigned (Ubuntu): assignee |
|
gerald.yang (gerald-yang-tw) |
|
| 2022-09-13 09:54:42 |
gerald.yang |
grub2-unsigned (Ubuntu): importance |
Undecided |
High |
|
| 2022-09-13 09:54:47 |
gerald.yang |
grub2-unsigned (Ubuntu): status |
New |
In Progress |
|
| 2022-09-13 10:04:25 |
gerald.yang |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-09-13 10:04:25 |
gerald.yang |
bug task added |
|
grub2-unsigned (Ubuntu Jammy) |
|
| 2022-09-13 10:04:25 |
gerald.yang |
nominated for series |
|
Ubuntu Focal |
|
| 2022-09-13 10:04:25 |
gerald.yang |
bug task added |
|
grub2-unsigned (Ubuntu Focal) |
|
| 2022-09-13 10:04:25 |
gerald.yang |
nominated for series |
|
Ubuntu Kinetic |
|
| 2022-09-13 10:04:25 |
gerald.yang |
bug task added |
|
grub2-unsigned (Ubuntu Kinetic) |
|
| 2022-09-13 10:04:32 |
gerald.yang |
grub2-unsigned (Ubuntu Jammy): status |
New |
In Progress |
|
| 2022-09-13 10:04:36 |
gerald.yang |
grub2-unsigned (Ubuntu Focal): status |
New |
In Progress |
|
| 2022-09-13 10:04:38 |
gerald.yang |
grub2-unsigned (Ubuntu Focal): assignee |
|
gerald.yang (gerald-yang-tw) |
|
| 2022-09-13 10:04:40 |
gerald.yang |
grub2-unsigned (Ubuntu Jammy): assignee |
|
gerald.yang (gerald-yang-tw) |
|
| 2022-09-13 10:04:41 |
gerald.yang |
grub2-unsigned (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2022-09-13 10:04:43 |
gerald.yang |
grub2-unsigned (Ubuntu Focal): importance |
Undecided |
High |
|
| 2022-09-13 10:05:42 |
gerald.yang |
description |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Check if it can boot successfully with the above patch
[ Where problems could occur ]
This patch only adjust the memory block layout, it shouldn't affect any other functions
[ Other Info ]
Related bugs:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983625
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320 |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Check if it can boot successfully with the above patch
[ Where problems could occur ]
This patch only adjust the memory block layout, it shouldn't affect any other functions
[ Other Info ]
Related bugs:
#lp:1983624
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983625
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320 |
|
| 2022-09-13 10:06:40 |
gerald.yang |
description |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Check if it can boot successfully with the above patch
[ Where problems could occur ]
This patch only adjust the memory block layout, it shouldn't affect any other functions
[ Other Info ]
Related bugs:
#lp:1983624
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983625
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320 |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Check if it can boot successfully with the above patch
[ Where problems could occur ]
This patch only adjust the memory block layout, it shouldn't affect any other functions
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
|
| 2022-09-15 04:22:39 |
gerald.yang |
description |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Check if it can boot successfully with the above patch
[ Where problems could occur ]
This patch only adjust the memory block layout, it shouldn't affect any other functions
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Make sure the running kernel in VM is 5.15
Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4000-0x0000000054fb4000] (1024MB)
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[ Where problems could occur ]
This patch only adjust the memory block layout, it shouldn't affect any other functions
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
|
| 2022-09-15 07:48:37 |
gerald.yang |
attachment added |
|
focal.patch https://bugs.launchpad.net/ubuntu/+source/grub2-unsigned/+bug/1989446/+attachment/5615992/+files/focal.patch |
|
| 2022-09-15 07:49:10 |
gerald.yang |
attachment added |
|
jammy.patch https://bugs.launchpad.net/ubuntu/+source/grub2-unsigned/+bug/1989446/+attachment/5615993/+files/jammy.patch |
|
| 2022-09-15 07:49:37 |
gerald.yang |
attachment added |
|
kinetic.patch https://bugs.launchpad.net/ubuntu/+source/grub2-unsigned/+bug/1989446/+attachment/5615994/+files/kinetic.patch |
|
| 2022-09-15 08:29:32 |
Ubuntu Foundations Team Bug Bot |
tags |
|
patch |
|
| 2022-09-15 08:29:39 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Review Team |
| 2022-09-15 09:19:11 |
Dariusz Gadomski |
bug |
|
|
added subscriber SE SRU ("STS") Sponsors |
| 2022-09-15 09:42:51 |
gerald.yang |
tags |
patch |
patch sts |
|
| 2022-09-16 07:01:17 |
gerald.yang |
marked as duplicate |
|
1842320 |
|
| 2022-09-20 06:20:54 |
gerald.yang |
description |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Make sure the running kernel in VM is 5.15
Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4000-0x0000000054fb4000] (1024MB)
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[ Where problems could occur ]
This patch only adjust the memory block layout, it shouldn't affect any other functions
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Make sure the running kernel in VM is 5.15
Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4000-0x0000000054fb4000] (1024MB)
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[ Where problems could occur ]
Originally, allocate memory for initrd/params/cmdline/kernel tries to get a memory block from less than 0x3fffffff, if it can not find a contiguous memory block, it will fail
This patch only adjusts the memory allocation that
1. check if it can get memory from less than 0x7fffffff (GRUB_EFI_MAX_ALLOCATION_ADDRESS)
2. if step 1 fails, then check if it can get memory from less than 0xffffffffffffffff (GRUB_EFI_MAX_USABLE_ADDRESS)
With this patch, initrd/params/cmdline/kernel will be located in higher address, e.g. higher than 0x3fffffff, so it gives more room for drivers like SWIOTLB to allocate a larger memory, so it shouldn't affect other functions
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
|
| 2022-09-20 06:25:11 |
gerald.yang |
description |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Make sure the running kernel in VM is 5.15
Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4000-0x0000000054fb4000] (1024MB)
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[ Where problems could occur ]
Originally, allocate memory for initrd/params/cmdline/kernel tries to get a memory block from less than 0x3fffffff, if it can not find a contiguous memory block, it will fail
This patch only adjusts the memory allocation that
1. check if it can get memory from less than 0x7fffffff (GRUB_EFI_MAX_ALLOCATION_ADDRESS)
2. if step 1 fails, then check if it can get memory from less than 0xffffffffffffffff (GRUB_EFI_MAX_USABLE_ADDRESS)
With this patch, initrd/params/cmdline/kernel will be located in higher address, e.g. higher than 0x3fffffff, so it gives more room for drivers like SWIOTLB to allocate a larger memory, so it shouldn't affect other functions
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Make sure the running kernel in VM is 5.15
Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4000-0x0000000054fb4000] (1024MB)
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[ Where problems could occur ]
Originally, allocate memory for initrd/params/cmdline/kernel tries to get a memory block from less than 0x3fffffff, if it can not find a contiguous memory block, it will fail
This patch only adjusts the memory allocation that
1. check if it can get memory from less than 0x7fffffff (GRUB_EFI_MAX_ALLOCATION_ADDRESS)
2. if step 1 fails, then check if it can get memory from less than 0xffffffffffffffff (GRUB_EFI_MAX_USABLE_ADDRESS)
With this patch, initrd/params/cmdline/kernel will be located in higher address, e.g. higher than 0x3fffffff, so it gives more room for drivers like SWIOTLB to allocate a larger memory, so it shouldn't affect other functions
The only issue is that if initrd is too big that needs to be allocated from higher than 4G (GRUB_EFI_MAX_USABLE_ADDRESS), but even without this patch, this issue still exists, because the original policy is to allocate memory from less than 2G (0x3fffffff)
But this issue is being handled by lp:1842320
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
|
| 2022-09-20 06:26:21 |
gerald.yang |
description |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Make sure the running kernel in VM is 5.15
Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4000-0x0000000054fb4000] (1024MB)
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[ Where problems could occur ]
Originally, allocate memory for initrd/params/cmdline/kernel tries to get a memory block from less than 0x3fffffff, if it can not find a contiguous memory block, it will fail
This patch only adjusts the memory allocation that
1. check if it can get memory from less than 0x7fffffff (GRUB_EFI_MAX_ALLOCATION_ADDRESS)
2. if step 1 fails, then check if it can get memory from less than 0xffffffffffffffff (GRUB_EFI_MAX_USABLE_ADDRESS)
With this patch, initrd/params/cmdline/kernel will be located in higher address, e.g. higher than 0x3fffffff, so it gives more room for drivers like SWIOTLB to allocate a larger memory, so it shouldn't affect other functions
The only issue is that if initrd is too big that needs to be allocated from higher than 4G (GRUB_EFI_MAX_USABLE_ADDRESS), but even without this patch, this issue still exists, because the original policy is to allocate memory from less than 2G (0x3fffffff)
But this issue is being handled by lp:1842320
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
[ Impact ]
When booting a large memory guest (both focal and jammy) with 5.15 kernel on a SEV enabled host
it fails to boot and shows the following error in dmesg:
software IO TLB: Cannot allocate buffer
But booting a Fedora36 guest works fine on a SEV enabled host
With this kernel commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12
SWIOTLB could allocate from 64MB to 1G top contiguous memory according to how much memory the system has
in sev_setup_arch:
size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
swiotlb_adjust_size(size);
Look into the memory block layout from Fedora grub, the available memory blocks are:
[ 0.005879] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005883] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005885] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005886] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is:
[ 0.005881] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
The size is close to 2G and sufficient for SWIOTLB to allocate 1G contiguous memory
Then we need to exclude reserved memory blocks overlapped with this region, below is the list
[ 0.005892] reserved[0x2] [0x00000000574a7000-0x0000000059313fff], 0x0000000001e6d000 bytes flags: 0x0
[ 0.005894] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005896] reserved[0x4] [0x000000007e845018-0x000000007e845857], 0x0000000000000840 bytes flags: 0x0
[ 0.005897] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest available range is
[0x0000000000100000-0x00000000574a7000]
Before SWIOTLB allocates memory block, EFI also reserves some memory
the one that overlapped with the above range is
[ 0.005942] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
It’s fine that SWIOTLB can still allocate 1G contiguous memory from [0x0000000000100000-0x00000000574a7000]:
[ 1.089832] software IO TLB: mapped [mem 0x00000000174a7000-0x00000000574a7000] (1024MB)
But if we look into the memory block layout from Ubuntu grub, the available memory blocks are:
[ 0.005833] memory[0x0] [0x0000000000001000-0x000000000009ffff], 0x000000000009f000 bytes flags: 0x0
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
[ 0.005837] memory[0x2] [0x000000007eb1b000-0x000000007fb9afff], 0x0000000001080000 bytes flags: 0x0
[ 0.005838] memory[0x3] [0x000000007fbff000-0x000000007ffdffff], 0x00000000003e1000 bytes flags: 0x0
[ 0.005840] memory[0x4] [0x0000000100000000-0x00000004ffffffff], 0x0000000400000000 bytes flags: 0x0
The biggest one is also:
[ 0.005835] memory[0x1] [0x0000000000100000-0x000000007e9ecfff], 0x000000007e8ed000 bytes flags: 0x0
Then excluding the reserved memory blocks:
[ 0.005846] reserved[0x2] [0x000000003a9ba000-0x000000003c7cdfff], 0x0000000001e14000 bytes flags: 0x0
[ 0.005848] reserved[0x3] [0x000000007e133018-0x000000007e17e057], 0x000000000004b040 bytes flags: 0x0
[ 0.005849] reserved[0x4] [0x000000007e847018-0x000000007e847887], 0x0000000000000870 bytes flags: 0x0
[ 0.005851] reserved[0x5] [0x000000007ee95698-0x000000007ee95af7], 0x0000000000000460 bytes flags: 0x0
Now the biggest one is:
[0x000000003e133018-0x000000007e133018]
Then excluding EFI reserved memory block that overlapped with the above range:
[ 0.005896] memblock_reserve: [0x000000007bfbe000-0x000000007bfddfff] efi_reserve_boot_services+0x8a/0xdb
So now, the biggest contiguous memory becomes
[0x000000003c7ce000-0x000000007bfbe000]
Which is less than 1G, this is why SWIOTLB can not allocate 1G contiguous memory
This commit from rhboot/grub2 fixes this issue:
https://github.com/rhboot/grub2/commit/9e6c1d803ade111b8719502ff25e86d8b4564de8
it adjusts the memory block layout, so SWIOTLB or any other drivers that need more than 1G contiguous memory can be satisfied
[ Test Plan ]
Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1
create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below:
virt-install --name <guest-name> --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/<guest-name.img>,device=disk,bus=scsi --disk /var/lib/libvirt/images/<guest-name>-config.iso,device=cdrom --os-type linux --os-variant <variant> --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole
Make sure the running kernel in VM is 5.15
Then check if it can boot successfully with the above patch
dmesg should show SWIOTLB is correctly mapped to 1G memory
[ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.822210] software IO TLB: mapped [mem 0x0000000014fb4000-0x0000000054fb4000] (1024MB)
[ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[ Where problems could occur ]
Originally, allocate memory for initrd/params/cmdline/kernel tries to get a memory block from less than 0x3fffffff, if it can not find a contiguous memory block, it will fail
This patch only adjusts the memory allocation that
1. check if it can get memory from less than 0x7fffffff (GRUB_EFI_MAX_ALLOCATION_ADDRESS)
2. if step 1 fails, then check if it can get memory from less than 0xffffffffffffffff (GRUB_EFI_MAX_USABLE_ADDRESS)
With this patch, initrd/params/cmdline/kernel will be located in higher address, e.g. between 0x3fffffff - 0x7fffffff, so it gives more room for drivers like SWIOTLB to allocate a larger memory, so it shouldn't affect other functions
The only issue is that if initrd is too big that needs to be allocated from higher than 4G (GRUB_EFI_MAX_USABLE_ADDRESS), but even without this patch, this issue still exists, because the original policy is to allocate memory from less than 2G (0x3fffffff)
But this issue is being handled by lp:1842320
[ Other Info ]
Related bugs:
lp:1983625
lp:1842320 |
|
| 2022-09-22 01:25:47 |
Matthew Ruffell |
bug |
|
|
added subscriber Matthew Ruffell |
| 2022-09-22 01:25:52 |
Jay Vosburgh |
bug |
|
|
added subscriber Jay Vosburgh |
| 2022-09-22 16:33:49 |
Julian Andres Klode |
removed duplicate marker |
1842320 |
|
|
| 2022-10-12 12:37:25 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~juliank/grub/+git/ubuntu/+merge/431421 |
|
| 2022-10-18 09:23:25 |
Julian Andres Klode |
tags |
patch sts |
foundations-todo patch sts |
|
| 2022-10-20 19:52:31 |
Julian Andres Klode |
grub2-unsigned (Ubuntu Jammy): status |
In Progress |
Triaged |
|
| 2022-10-20 19:52:33 |
Julian Andres Klode |
grub2-unsigned (Ubuntu Focal): status |
In Progress |
Triaged |
|
| 2022-10-21 09:08:09 |
Olivier Gayot |
bug |
|
|
added subscriber Canonical Foundations Team |
| 2022-11-16 09:59:50 |
Aaron Whitehouse |
bug |
|
|
added subscriber Aaron Whitehouse |
| 2022-11-18 19:40:48 |
Steve Langasek |
grub2-unsigned (Ubuntu Kinetic): status |
In Progress |
Fix Committed |
|
| 2022-11-18 19:40:50 |
Steve Langasek |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-11-18 19:40:53 |
Steve Langasek |
bug |
|
|
added subscriber SRU Verification |
| 2022-11-18 19:40:58 |
Steve Langasek |
tags |
foundations-todo patch sts |
foundations-todo patch sts verification-needed verification-needed-kinetic |
|
| 2022-11-24 15:23:13 |
Fabio Augusto Miranda Martins |
tags |
foundations-todo patch sts verification-needed verification-needed-kinetic |
foundations-todo patch sts verification-done verification-done-kinetic |
|
| 2022-11-24 17:52:21 |
Launchpad Janitor |
grub2-unsigned (Ubuntu): status |
In Progress |
Fix Released |
|
| 2022-11-29 07:50:22 |
Dariusz Gadomski |
bug |
|
|
added subscriber Dariusz Gadomski |
| 2022-12-01 08:59:04 |
Launchpad Janitor |
grub2-unsigned (Ubuntu Kinetic): status |
Fix Committed |
Fix Released |
|
| 2022-12-01 08:59:09 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2023-01-20 10:02:26 |
Julian Andres Klode |
grub2-unsigned (Ubuntu Jammy): status |
Triaged |
Fix Committed |
|
| 2023-01-20 10:02:28 |
Julian Andres Klode |
grub2-unsigned (Ubuntu Focal): status |
Triaged |
Fix Committed |
|
| 2023-01-20 10:03:18 |
Julian Andres Klode |
grub2-unsigned (Ubuntu Focal): status |
Fix Committed |
Triaged |
|
| 2023-01-20 10:03:21 |
Julian Andres Klode |
grub2-unsigned (Ubuntu Jammy): status |
Fix Committed |
Triaged |
|
| 2023-01-20 10:07:11 |
Julian Andres Klode |
grub2-unsigned (Ubuntu Jammy): status |
Triaged |
Fix Committed |
|
| 2023-01-20 10:07:12 |
Julian Andres Klode |
grub2-unsigned (Ubuntu Focal): status |
Triaged |
Fix Committed |
|
| 2023-01-20 19:24:50 |
Éric St-Jean |
bug |
|
|
added subscriber Éric St-Jean |
| 2023-01-30 15:14:19 |
Launchpad Janitor |
grub2-unsigned (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2023-01-30 15:14:19 |
Launchpad Janitor |
cve linked |
|
2022-2601 |
|
| 2023-01-30 15:14:19 |
Launchpad Janitor |
cve linked |
|
2022-3775 |
|
| 2023-01-30 17:34:55 |
Launchpad Janitor |
grub2-unsigned (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2023-06-02 15:34:18 |
Benjamin Drung |
tags |
foundations-todo patch sts verification-done verification-done-kinetic |
patch sts verification-done verification-done-kinetic |
|
