kernel-5.13.0-23-generic : Unable to boot when Secure Encrypted Virtualization( SEV) is enabled without setting swiotlb boot param

Bug #1955655 reported by Louis Bouchard
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
grub2 (Ubuntu)
Confirmed
Undecided
Unassigned
Impish
Confirmed
Undecided
Unassigned
Jammy
Confirmed
Undecided
Unassigned
linux (Ubuntu)
Invalid
Undecided
Unassigned
Impish
Invalid
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned

Bug Description

While investigating LP: #1955395 by using the -generic kernel image, it appeared that it is impossible to boot the kernel unless the boot parameter swiotlb is set to 512M (swiotlb=262144).

Wnen not set, the kernel tries to adjust the bounce buffer to 1024Mb it fails and later trigger a kernel panic with the following trace :

$ grep TLB /tmp/console.log
[ 0.003665] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.034219] kvm-guest: KVM setup pv remote TLB flush
[ 0.037063] software IO TLB: Cannot allocate buffer
[ 0.223009] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
[ 0.223634] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
[ 0.297424] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.297424] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 1.018860] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 1.019552] software IO TLB: No low mem
[ 1.451497] Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer
[ 1.491589] ---[ end Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer ]---

The SWIOTLB adjustment comes from the following kernel commit : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12

For some reason, the LowMem allocation fails (as seen by the "software IO TLB: No low mem" msg),hence the SWIOTLB adjustment cannot be completed.

When booting with the swiotlb=262144 value, we get the following output :
$ grep TLB /tmp/console.log

[ 0.050908] kvm-guest: KVM setup pv remote TLB flush
[ 0.308896] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
[ 0.309494] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
[ 0.373162] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.373162] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 1.071136] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 1.071837] software IO TLB: mapped [mem 0x000000005bebe000-0x000000007bebe000] (512MB)
[ 1.529804] software IO TLB: Memory encryption is active and system is using DMA bounce buffers

For comparaison, the Fedora 34 kernel (5.15.4-101.fc34.x86_64) with the same adjustment mechanism does correctly adjust the SWIOTLB bounce buffer, without the need to set the swiotlb= value at boot time.

The SWIOTLB buffer adjustment has been introduced in kernel 5.11.

We can make SEV enabled resources available for testing if needed.

...Louis
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Dec 23 13:32 seq
 crw-rw---- 1 root audio 116, 33 Dec 23 13:32 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu74
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:

Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Scaleway SCW-ENT1-S
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.13.0-23-generic root=UUID=1f577236-6bf2-48ef-998a-ba45f71aca7f ro console=tty1 console=ttyS0 swiotlb=262144
ProcVersionSignature: Ubuntu 5.13.0-23.23-generic 5.13.19
RelatedPackageVersions:
 linux-restricted-modules-5.13.0-23-generic N/A
 linux-backports-modules-5.13.0-23-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.13.0-23-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 02/06/2015
dmi.bios.release: 0.0
dmi.bios.vendor: EFI Development Kit II / OVMF
dmi.bios.version: 0.0.0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-3.0
dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnScaleway:pnSCW-ENT1-S:pvrpc-q35-3.0:cvnQEMU:ct1:cvrpc-q35-3.0:sku:
dmi.product.name: SCW-ENT1-S
dmi.product.version: pc-q35-3.0
dmi.sys.vendor: Scaleway

Revision history for this message
Louis Bouchard (louis) wrote : CurrentDmesg.txt

apport information

tags: added: apport-collected jammy uec-images
description: updated
Revision history for this message
Louis Bouchard (louis) wrote : Lspci.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : Lspci-vt.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : ProcModules.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : UdevDb.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : WifiSyslog.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : acpidump.txt

apport information

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Khaled El Mously (kmously) wrote :

I'm not sure I see a panic in the attached kernel log.

Revision history for this message
Louis Bouchard (louis) wrote :
Download full text (3.3 KiB)

Hello,
The kernel panic occurs very early in the boot sequence, even before the root fs is mounted to capture the log. Here is the kenrel backtrace :

[ 1.251773] Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer
[ 1.253440] CPU: 2 PID: 212 Comm: systemd-udevd Not tainted 5.13.0-23-generic #23-Ubuntu
[ 1.255022] Hardware name: Scaleway SCW-ENT1-S, BIOS 0.0.0 02/06/2015
[ 1.256228] Call Trace:
[ 1.256697] show_stack+0x52/0x58
[ 1.257355] dump_stack+0x7d/0x9c
[ 1.257989] panic+0x101/0x2e3
[ 1.258821] swiotlb_tbl_map_single.cold+0x57/0x7c
[ 1.259750] swiotlb_map+0x61/0x240
[ 1.260428] dma_direct_map_page+0x115/0x1a0
[ 1.261255] dma_map_page_attrs+0x35/0x50
[ 1.262016] vring_map_one_sg+0x4c/0x50
[ 1.262787] virtqueue_add_split+0x23c/0x4a0
[ 1.263427] virtqueue_add_inbuf+0x48/0x50
[ 1.263909] virtscsi_kick_event.isra.0+0x8b/0xd0 [virtio_scsi]
[ 1.264616] virtscsi_probe+0x31e/0x33b [virtio_scsi]
[ 1.265216] virtio_dev_probe+0x163/0x220
[ 1.265683] really_probe+0x24b/0x4c0
[ 1.266134] driver_probe_device+0xf0/0x160
[ 1.266647] device_driver_attach+0xab/0xb0
[ 1.267188] __driver_attach+0xb2/0x140
[ 1.267648] ? device_driver_attach+0xb0/0xb0
[ 1.268163] bus_for_each_dev+0x7e/0xc0
[ 1.268620] driver_attach+0x1e/0x20
[ 1.269061] bus_add_driver+0x135/0x1f0
[ 1.269516] driver_register+0x95/0xf0
[ 1.269957] ? 0xffffffffc038e000
[ 1.270344] register_virtio_driver+0x20/0x30
[ 1.270885] init+0x8d/0x1000 [virtio_scsi]
[ 1.271380] ? 0xffffffffc038e000
[ 1.271762] do_one_initcall+0x48/0x1d0
[ 1.272213] ? kmem_cache_alloc_trace+0xfb/0x240
[ 1.272759] do_init_module+0x62/0x290
[ 1.273199] load_module+0xa8f/0xb10
[ 1.273620] __do_sys_finit_module+0xc2/0x120
[ 1.274155] __x64_sys_finit_module+0x18/0x20
[ 1.274680] do_syscall_64+0x61/0xb0
[ 1.275118] ? syscall_exit_to_user_mode+0x27/0x50
[ 1.275667] ? __x64_sys_newfstatat+0x1c/0x20
[ 1.276177] ? do_syscall_64+0x6e/0xb0
[ 1.276614] ? ksys_read+0x67/0xe0
[ 1.277049] ? exit_to_user_mode_prepare+0x37/0xb0
[ 1.277614] ? syscall_exit_to_user_mode+0x27/0x50
[ 1.278180] ? __x64_sys_read+0x19/0x20
[ 1.278653] ? do_syscall_64+0x6e/0xb0
[ 1.279116] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 1.279709] RIP: 0033:0x7fe040ccd94d
[ 1.280140] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b3 64 0f 00 f7 d8 64 89 01 48
[ 1.282337] RSP: 002b:00007ffe2399a6d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 1.283252] RAX: ffffffffffffffda RBX: 0000562e8ab7a030 RCX: 00007fe040ccd94d
[ 1.284077] RDX: 0000000000000000 RSI: 00007fe040e61441 RDI: 0000000000000005
[ 1.284910] RBP: 0000000000020000 R08: 0000000000000000 R09: 00007ffe2399a810
[ 1.285754] R10: 0000000000000005 R11: 0000000000000246 R12: 00007fe040e61441
[ 1.286594] R13: 0000562e8ab79ef0 R14: 0000562e8abd1c50 R15: 0000562e8abd4400
[ 1.288517] Kernel Offset: 0x25a00000 from 0xffffffff810000...

Read more...

Revision history for this message
Louis Bouchard (louis) wrote :

As a complement of information, here are the messages pertaining to the TLB during the log :

$ grep -i tlb swiotlb_crash.out
[ 0.003802] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB
[ 0.037784] kvm-guest: KVM setup pv remote TLB flush
[ 0.040674] software IO TLB: Cannot allocate buffer
[ 0.229918] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
[ 0.230877] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
[ 0.322941] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.324641] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.799064] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.800761] software IO TLB: No low mem
[ 1.251773] Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer
[ 1.258821] swiotlb_tbl_map_single.cold+0x57/0x7c
[ 1.259750] swiotlb_map+0x61/0x240
[ 1.289795] ---[ end Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer ]---

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

I believe this is still related to lack of rhboot upstreaming patches to grub to use highmem allocations; such that lowmem remains unused; and thus SEV/SWIOTLB can be used "normally".

It is likely the difference you see is due to fedora's grub patches.

It would be interesting, if you could unpack & use fedora's kernel with ubuntu's grub, and see if fedora's kernel starts failing when booted via ubuntu's grub?

Or vice-versa; unpack and use ubuntu's kernel on fedora with fedora's grub to observe it working fine.

That would help eliminate differences of grub between the distros with the same kernels.

Revision history for this message
Louis Bouchard (louis) wrote :

Hi xnox,

Thanks for the reply. I'll test both options and report to you here.

Revision history for this message
Louis Bouchard (louis) wrote :

So testing with Fedora's boot/efi/EFI/fedora/grubx64.efi in place of the Ubuntu version solves the problem.

So it is indeed the situation that you describe in comment #15

Louis Bouchard (louis)
Changed in linux (Ubuntu Impish):
status: New → Invalid
Changed in linux (Ubuntu Jammy):
status: Confirmed → Invalid
tags: added: rls-jj-incoming
tags: removed: rls-jj-incoming
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in grub2 (Ubuntu Impish):
status: New → Confirmed
Changed in grub2 (Ubuntu Jammy):
status: New → Confirmed
Changed in grub2 (Ubuntu):
status: New → Confirmed
Revision history for this message
Khaled El Mously (kmously) wrote :

Hi @Louis..

Could you please try to reproduce this issue on the latest Jammy images with a 5.15 kernel? (The 5.13 kernel is EOL).

I have tried to reproduce the issue myself on a BM.Standard.E3.128 instance, but I am not sure exactly how to launch the VM correctly with SEV enabled. If you could provide instructions that would be very helpful. Thanks

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers