qemu-system-x86_64 crashes inside nested VM

[Dimitri John Ledkov]

Systemd package has autopkgtests
the upstream-2 test cases use upstream systemd testsuite, i.e. make -C str/test/TEST-70-TPM2 setup run
it launches a nested VM to do quick tests inside it.

It appears that qemu-system-x86_64 crashes in such cases:

TEST-70-TPM2 RUN: cryptenroll/cryptsetup with TPM2 devices
+ timeout --foreground 1800 /bin/qemu-system-x86_64 -smp 4 -net none -m 1024M -nographic -vga none -kernel /boot/vmlinuz-6.2.0-1003-lowlatency -drive format=raw,cache=unsafe,file=/var/tmp/systemd-test.G2RH6i/tpm2.img -device virtio-rng-pci,max-bytes=1024,period=1000 -chardev socket,id=chrtpm,path=/tmp/tmp.cRBa43SrLC/sock -tpmdev emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 -initrd /boot/initrd.img-6.2.0-1003-lowlatency -append 'root=LABEL=systemd_boot rw raid=noautodetect rd.luks=0 loglevel=2 init=/lib/systemd/systemd console=ttyS0 SYSTEMD_UNIT_PATH=/usr/lib/systemd/tests/testdata/testsuite-70.units:/usr/lib/systemd/tests/testdata/units: systemd.unit=testsuite.target systemd.wants=testsuite-70.service oops=panic panic=1 softlockup_panic=1 systemd.wants=end.service'
qemu-system-x86_64: ../../util/cacheflush.c:208: init_cache_info: Assertion `(isize & (isize - 1)) == 0' failed.
timeout: the monitored command dumped core
..//test-functions: line 377: 152120 Aborted ( set -x; "${qemu_cmd[@]}" "${qemu_options[@]}" -append "${kernel_params[*]}" )
E: qemu failed with exit code 134

The important bit seems to be:

qemu-system-x86_64: ../../util/cacheflush.c:208: init_cache_info: Assertion `(isize & (isize - 1)) == 0' failed.

Which is an assert inside qemu source code.

Is the systemd test suite VM setup doing something wrong, or is there something wrong in qemu?

[Dimitri John Ledkov (xnox)]

https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/amd64/s/systemd/20230413_175208_af94d@/artifacts.tar.gz

https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/amd64/s/systemd/20230413_175208_af94d@/log.gz

[Christian Ehrhardt  (paelzer)]

No we haven't heard about those crashes.

As we have discussed, this code didn't change recently.

And on the try to recreate this (qemu 7.2 and kernel 6.2 as in lunar) stacked two times in each other it does not fail.

So I'll need a reproducer which you said you'll have a look to try to create - thanks!

[Florian Weimer (fw)]

This commit:

commit 103a469dc7755fd9e8ccf362f3dd4c55dc761908
Author: Sajan Karumanchi <email address hidden>
Date: Wed Jan 18 18:29:04 2023 +0100

    x86: Cache computation for AMD architecture.

    All AMD architectures cache details will be computed based on
    __cpuid__ `0x8000_001D` and the reference to __cpuid__ `0x8000_0006` will be
    zeroed out for future architectures.

    Reviewed-by: Premachandra Mallappa <email address hidden>

changed cache size computation on the AMD architecture.

However, the new way of doing things is not supported by all AMD CPUs. This CPU:

processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 6
model name : AMD Turion(tm) II Neo N40L Dual-Core Processor
stepping : 3
microcode : 0x10000c8
cpu MHz : 800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips : 2995.32
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

reports all zeros for its caches after this change (build from commit cea74a4a24c36202309e8254f1f938e2166488f3, which includes commit mentioned above):

$ ./ld.so --list-diagnostics | grep -E 'level|threshold'
x86.cpu_features.non_temporal_threshold=0x4040
x86.cpu_features.rep_movsb_threshold=0x800
x86.cpu_features.rep_movsb_stop_threshold=0x0
x86.cpu_features.rep_stosb_threshold=0x800
x86.cpu_features.level1_icache_size=0x0
x86.cpu_features.level1_icache_linesize=0x0
x86.cpu_features.level1_dcache_size=0x0
x86.cpu_features.level1_dcache_assoc=0x0
x86.cpu_features.level1_dcache_linesize=0x0
x86.cpu_features.level2_cache_size=0x0
x86.cpu_features.level2_cache_assoc=0x0
x86.cpu_features.level2_cache_linesize=0x0
x86.cpu_features.level3_cache_size=0x0
x86.cpu_features.level3_cache_assoc=0x0
x86.cpu_features.level3_cache_linesize=0x0
x86.cpu_features.level4_cache_size=0xffffffffffffffff

A build from the 2.36 branch (commit b7008a92f505632f32b313d1033d6d15c99a0b31) yields this instead:

$ ./ld.so --list-diagnostics | grep -E 'level|threshold'
x86.cpu_features.non_temporal_threshold=0xc0000
x86.cpu_features.rep_movsb_threshold=0x800
x86.cpu_features.rep_movsb_stop_threshold=0x100000
x86.cpu_features.rep_stosb_threshold=0x800
x86.cpu_features.level1_icache_size=0x10000
x86.cpu_features.level1_icache_linesize=0x40
x86.cpu_features.level1_dcache_size=0x10000
x86.cpu_features.level1_d...

[Florian Weimer (fweimer)]

Just to clarify: this regression affects sysconf (_SC_LEVEL2_CACHE_SIZE) and similar configuration values, so it impacts more than just glibc-internal tuning decisions.

[Launchpad Janitor (janitor)]

[Expired for qemu (Ubuntu) because there has been no activity for 60 days.]

[Florian Weimer (fweimer)]

Initial patch posted (still alters results compared to what we had before):

[PATCH] x86: Fix for cache computation on AMD legacy cpus.
<https://sourceware.org/pipermail/libc-alpha/2023-June/148763.html>

[Nick Rosbrook (enr0n)]

I am seeing the same crash with systemd 253.5-1ubuntu1 upstream-{1,2} tests and qemu 1:8.0.2+dfsg-2ubuntu1[1]. However, the systemd 252.5-2ubuntu3 upstream-{1,2} tests pass against the same version of qemu[2].

[1] https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/amd64/s/systemd/20230720_070515_f604e@/log.gz
[2] https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/amd64/s/systemd/20230721_154608_28f76@/log.gz

[Sergio Durigan Junior (sergiodj)]

The following upstream commit looks interesting:

https://github.com/qemu/qemu/commit/00b5032eaddb7193f03f0a28b10286244d2e2a7b

I'll see if I can reproduce the issue here, and then check if backporting the commit above makes any difference.

[Sergio Durigan Junior (sergiodj)]

Spoke too soon: that commit is already present in qemu 8.0, of course.

The rest of what I wrote still applies, though: I need to see if I can reproduce the failure here.

[Sergio Durigan Junior (sergiodj)]

I was able to reproduce the failure here. It is in fact easy to reproduce:

$ lxc launch ubuntu-daily:mantic qemu-bug --vm
$ lxc shell qemu-bug
# apt update && apt install -y qemu-user qemu-user-static
# qemu-x86_64 /usr/bin/qemu-x86_64 --version
qemu-x86_64: util/cacheflush.c:212: init_cache_info: Assertion `(isize & (isize - 1)) == 0' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted (core dumped)

This looks like it's actually a glibc issue which, unfortunately, hasn't been fixed in the latest release (2.38). I linked the upstream bug and I'm following the discussions in the mailing list.

[Launchpad Janitor (janitor)]

This bug was fixed in the package glibc - 2.38-1ubuntu3

---------------
glibc (2.38-1ubuntu3) mantic; urgency=medium

  * Adjust libc6-dev Breaks to account for binNMU vs no-change uploads

 -- Simon Chopin <email address hidden> Fri, 11 Aug 2023 17:05:47 +0200

[Jamborm (jamborm)]

Has this been addressed(*)? If so, could the respective commit ID's be posted here for reference, please?

(*) https://sourceware.org/pipermail/libc-alpha/2023-June/148812.html would suggest that it was but perhaps not entirely?

[Florian Weimer (fweimer)]

Fixed for 2.39 via:

commit dcad5c8578130dec7f35fd5b0885304b59f9f543
Author: Sajan Karumanchi <email address hidden>
Date: Tue Aug 1 15:20:55 2023 +0000

    x86: Fix for cache computation on AMD legacy cpus.

    Some legacy AMD CPUs and hypervisors have the _cpuid_ '0x8000_001D'
    set to Zero, thus resulting in zeroed-out computed cache values.
    This patch reintroduces the old way of cache computation as a
    fail-safe option to handle these exceptions.
    Fixed 'level4_cache_size' value through handle_amd().

    Reviewed-by: Premachandra Mallappa <email address hidden>
    Tested-by: Florian Weimer <email address hidden>

It seems that we are still missing the backport to 2.37.