kernel-5.13.0-23-generic : Unable to boot when Secure Encrypted Virtualization( SEV) is enabled without setting swiotlb boot param

[Louis Bouchard]

While investigating LP: #1955395 by using the -generic kernel image, it appeared that it is impossible to boot the kernel unless the boot parameter swiotlb is set to 512M (swiotlb=262144).

Wnen not set, the kernel tries to adjust the bounce buffer to 1024Mb it fails and later trigger a kernel panic with the following trace :

$ grep TLB /tmp/console.log
[ 0.003665] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.034219] kvm-guest: KVM setup pv remote TLB flush
[ 0.037063] software IO TLB: Cannot allocate buffer
[ 0.223009] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
[ 0.223634] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
[ 0.297424] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.297424] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 1.018860] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 1.019552] software IO TLB: No low mem
[ 1.451497] Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer
[ 1.491589] ---[ end Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer ]---

The SWIOTLB adjustment comes from the following kernel commit : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12

For some reason, the LowMem allocation fails (as seen by the "software IO TLB: No low mem" msg),hence the SWIOTLB adjustment cannot be completed.

When booting with the swiotlb=262144 value, we get the following output :
$ grep TLB /tmp/console.log

[ 0.050908] kvm-guest: KVM setup pv remote TLB flush
[ 0.308896] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
[ 0.309494] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
[ 0.373162] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.373162] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 1.071136] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 1.071837] software IO TLB: mapped [mem 0x000000005bebe000-0x000000007bebe000] (512MB)
[ 1.529804] software IO TLB: Memory encryption is active and system is using DMA bounce buffers

For comparaison, the Fedora 34 kernel (5.15.4-101.fc34.x86_64) with the same adjustment mechanism does correctly adjust the SWIOTLB bounce buffer, without the need to set the swiotlb= value at boot time.

The SWIOTLB buffer adjustment has been introduced in kernel 5.11.

We can make SEV enabled resources available for testing if needed.

...Louis
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Dec 23 13:32 seq
 crw-rw---- 1 root audio 116, 33 Dec 23 13:32 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu74
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:

Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Scaleway SCW-ENT1-S
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.13.0-23-generic root=UUID=1f577236-6bf2-48ef-998a-ba45f71aca7f ro console=tty1 console=ttyS0 swiotlb=262144
ProcVersionSignature: Ubuntu 5.13.0-23.23-generic 5.13.19
RelatedPackageVersions:
 linux-restricted-modules-5.13.0-23-generic N/A
 linux-backports-modules-5.13.0-23-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.13.0-23-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 02/06/2015
dmi.bios.release: 0.0
dmi.bios.vendor: EFI Development Kit II / OVMF
dmi.bios.version: 0.0.0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-3.0
dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnScaleway:pnSCW-ENT1-S:pvrpc-q35-3.0:cvnQEMU:ct1:cvrpc-q35-3.0:sku:
dmi.product.name: SCW-ENT1-S
dmi.product.version: pc-q35-3.0
dmi.sys.vendor: Scaleway

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Louis Bouchard (louis)]

apport information

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Khaled El Mously (kmously)]

I'm not sure I see a panic in the attached kernel log.

[Louis Bouchard (louis)]

Hello,
The kernel panic occurs very early in the boot sequence, even before the root fs is mounted to capture the log. Here is the kenrel backtrace :

[ 1.251773] Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer
[ 1.253440] CPU: 2 PID: 212 Comm: systemd-udevd Not tainted 5.13.0-23-generic #23-Ubuntu
[ 1.255022] Hardware name: Scaleway SCW-ENT1-S, BIOS 0.0.0 02/06/2015
[ 1.256228] Call Trace:
[ 1.256697] show_stack+0x52/0x58
[ 1.257355] dump_stack+0x7d/0x9c
[ 1.257989] panic+0x101/0x2e3
[ 1.258821] swiotlb_tbl_map_single.cold+0x57/0x7c
[ 1.259750] swiotlb_map+0x61/0x240
[ 1.260428] dma_direct_map_page+0x115/0x1a0
[ 1.261255] dma_map_page_attrs+0x35/0x50
[ 1.262016] vring_map_one_sg+0x4c/0x50
[ 1.262787] virtqueue_add_split+0x23c/0x4a0
[ 1.263427] virtqueue_add_inbuf+0x48/0x50
[ 1.263909] virtscsi_kick_event.isra.0+0x8b/0xd0 [virtio_scsi]
[ 1.264616] virtscsi_probe+0x31e/0x33b [virtio_scsi]
[ 1.265216] virtio_dev_probe+0x163/0x220
[ 1.265683] really_probe+0x24b/0x4c0
[ 1.266134] driver_probe_device+0xf0/0x160
[ 1.266647] device_driver_attach+0xab/0xb0
[ 1.267188] __driver_attach+0xb2/0x140
[ 1.267648] ? device_driver_attach+0xb0/0xb0
[ 1.268163] bus_for_each_dev+0x7e/0xc0
[ 1.268620] driver_attach+0x1e/0x20
[ 1.269061] bus_add_driver+0x135/0x1f0
[ 1.269516] driver_register+0x95/0xf0
[ 1.269957] ? 0xffffffffc038e000
[ 1.270344] register_virtio_driver+0x20/0x30
[ 1.270885] init+0x8d/0x1000 [virtio_scsi]
[ 1.271380] ? 0xffffffffc038e000
[ 1.271762] do_one_initcall+0x48/0x1d0
[ 1.272213] ? kmem_cache_alloc_trace+0xfb/0x240
[ 1.272759] do_init_module+0x62/0x290
[ 1.273199] load_module+0xa8f/0xb10
[ 1.273620] __do_sys_finit_module+0xc2/0x120
[ 1.274155] __x64_sys_finit_module+0x18/0x20
[ 1.274680] do_syscall_64+0x61/0xb0
[ 1.275118] ? syscall_exit_to_user_mode+0x27/0x50
[ 1.275667] ? __x64_sys_newfstatat+0x1c/0x20
[ 1.276177] ? do_syscall_64+0x6e/0xb0
[ 1.276614] ? ksys_read+0x67/0xe0
[ 1.277049] ? exit_to_user_mode_prepare+0x37/0xb0
[ 1.277614] ? syscall_exit_to_user_mode+0x27/0x50
[ 1.278180] ? __x64_sys_read+0x19/0x20
[ 1.278653] ? do_syscall_64+0x6e/0xb0
[ 1.279116] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 1.279709] RIP: 0033:0x7fe040ccd94d
[ 1.280140] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b3 64 0f 00 f7 d8 64 89 01 48
[ 1.282337] RSP: 002b:00007ffe2399a6d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 1.283252] RAX: ffffffffffffffda RBX: 0000562e8ab7a030 RCX: 00007fe040ccd94d
[ 1.284077] RDX: 0000000000000000 RSI: 00007fe040e61441 RDI: 0000000000000005
[ 1.284910] RBP: 0000000000020000 R08: 0000000000000000 R09: 00007ffe2399a810
[ 1.285754] R10: 0000000000000005 R11: 0000000000000246 R12: 00007fe040e61441
[ 1.286594] R13: 0000562e8ab79ef0 R14: 0000562e8abd1c50 R15: 0000562e8abd4400
[ 1.288517] Kernel Offset: 0x25a00000 from 0xffffffff810000...

[Louis Bouchard (louis)]

As a complement of information, here are the messages pertaining to the TLB during the log :

$ grep -i tlb swiotlb_crash.out
[ 0.003802] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.037784] kvm-guest: KVM setup pv remote TLB flush
[ 0.040674] software IO TLB: Cannot allocate buffer
[ 0.229918] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
[ 0.230877] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
[ 0.322941] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.324641] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.799064] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.800761] software IO TLB: No low mem
[ 1.251773] Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer
[ 1.258821] swiotlb_tbl_map_single.cold+0x57/0x7c
[ 1.259750] swiotlb_map+0x61/0x240
[ 1.289795] ---[ end Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer ]---

[Dimitri John Ledkov (xnox)]

I believe this is still related to lack of rhboot upstreaming patches to grub to use highmem allocations; such that lowmem remains unused; and thus SEV/SWIOTLB can be used "normally".

It is likely the difference you see is due to fedora's grub patches.

It would be interesting, if you could unpack & use fedora's kernel with ubuntu's grub, and see if fedora's kernel starts failing when booted via ubuntu's grub?

Or vice-versa; unpack and use ubuntu's kernel on fedora with fedora's grub to observe it working fine.

That would help eliminate differences of grub between the distros with the same kernels.

[Louis Bouchard (louis)]

Hi xnox,

Thanks for the reply. I'll test both options and report to you here.

[Louis Bouchard (louis)]

So testing with Fedora's boot/efi/EFI/fedora/grubx64.efi in place of the Ubuntu version solves the problem.

So it is indeed the situation that you describe in comment #15

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Khaled El Mously (kmously)]

Hi @Louis..

Could you please try to reproduce this issue on the latest Jammy images with a 5.15 kernel? (The 5.13 kernel is EOL).

I have tried to reproduce the issue myself on a BM.Standard.E3.128 instance, but I am not sure exactly how to launch the VM correctly with SEV enabled. If you could provide instructions that would be very helpful. Thanks

[Fabio Augusto Miranda Martins (fabio.martins)]

This is a grub bug and it is being tracked here:

[SRU] unable to boot guest with large memory when SEV is enabled on host
https://bugs.launchpad.net/ubuntu/+source/grub2-unsigned/+bug/1989446