apic-split will fail in kvm_units_tests

[Sean Feole]

Reproducible: Yes, every time.
Series: cosmic
Kernel: "linux-aws 4.18.0-1012.14"
Steps:

1.) apt-get install --yes --allow build-essential cpu-checker qemu-kvm git gcc-multilib
2.) git clone --depth=1 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
3.) cd kvm-unit-tests; ./configure; make

4.) TESTNAME=apic-split TIMEOUT=90s ACCEL= ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split

root@ip-172-31-5-254:/home/ubuntu/kvm-unit-tests# TESTNAME=apic-split TIMEOUT=90s ACCEL= ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split # -initrd /tmp/tmp.ZHQwPUOuLD
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
enabling apic
enabling apic
paging enabled
cr0 = 80010011
cr3 = 459000
cr4 = 20
apic version: 1050014
PASS: apic existence
PASS: xapic id matches cpuid
PASS: writeable xapic id
PASS: non-writeable x2apic id
PASS: sane x2apic id
PASS: x2apic id matches cpuid
PASS: correct xapic id after reset
PASS: apic_disable: Local apic disabled
PASS: apic_disable: CPUID.1H:EDX.APIC[bit 9] is clear
PASS: apic_disable: *0xfee00030: ffffffff
PASS: apic_disable: CR8: 0
PASS: apic_disable: CR8: f
PASS: apic_disable: *0xfee00080: ffffffff
PASS: apic_disable: Local apic enabled in xAPIC mode
PASS: apic_disable: CPUID.1H:EDX.APIC[bit 9] is set
PASS: apic_disable: *0xfee00030: 1050014
PASS: apic_disable: *0xfee00080: 0
PASS: apic_disable: *0xfee00080: f0
PASS: apic_disable: Local apic enabled in x2APIC mode
PASS: apic_disable: CPUID.1H:EDX.APIC[bit 9] is set
PASS: apic_disable: *0xfee00030: ffffffff
PASS: apic_disable: CR8: 0
PASS: apic_disable: CR8: f
PASS: apic_disable: *0xfee00080: ffffffff
x2apic enabled
PASS: x2apic enabled to invalid state
PASS: x2apic enabled to apic enabled
PASS: x2apic enabled to disabled state
PASS: disabled to invalid state
PASS: disabled to x2apic enabled
PASS: apic enabled to invalid state
PASS: apicbase: relocate apic
PASS: apicbase: reserved physaddr bits
PASS: apicbase: reserved low bits
PASS: self ipi
starting broadcast (x2apic)
PASS: APIC physical broadcast address
PASS: APIC physical broadcast shorthand
FAIL: PV IPIs testing
PASS: nmi-after-sti
PASS: multiple nmi
PASS: APIC LVT timer one shot
starting apic change mode
PASS: TMICT value reset
PASS: TMCCT should have a non-zero value
PASS: TMCCT should have reached 0
PASS: TMCCT should have a non-zero value
PASS: TMCCT should not be reset to TMICT value
PASS: TMCCT should be reset to the initial-count
PASS: TMCCT should not be reset to init
PASS: TMCCT should have reach zero
PASS: TMCCT should stay at zero
PASS: tsc deadline timer
PASS: tsc deadline timer clearing
SUMMARY: 51 tests, 1 unexpected failures

[Sean Feole (sfeole)]

Also affects Bionic linux-aws 4.15.1035.37

[Sean Feole (sfeole)]

Also affects Xenial linux-aws 4.4.0-1078.88

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1821390

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Sean Feole (sfeole)]

We see a similar failure in some Azure Cloud instances, it appears to be somewhat of the same.

5. 03/27 23:15:29 DEBUG| utils:0153| [stdout] TESTNAME=apic-split TIMEOUT=90s ACCEL= ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split
6. 03/27 23:16:20 DEBUG| utils:0153| [stdout] FAIL apic-split (51 tests, 1 unexpected failures)

[Sean Feole (sfeole)]

affects, 4.18.0-1013.15 linux-aws

[Sean Feole (sfeole)]

also affects aws i3.metal disco , cosmic 4.18.0-1015.17

[Sean Feole (sfeole)]

SRU 2019.12.02 Bionic linux-oracle, Kernel:5.0.0-1009.14~18.04.1

Failure seen in apic-split..

12/06 08:07:06 DEBUG| utils:0153| [stdout] FAIL: TMCCT should not be reset to TMICT value

[Po-Hsu Lin (cypressyew)]

Failed on T-4.4 / X-4.4 AMD64
Passed on X-4.15 / B-4.15 AMD64

[Po-Hsu Lin (cypressyew)]

Passed on E 5.3 AMD64

[Po-Hsu Lin (cypressyew)]

For the timeout issue on Oracle B-5.0, VM.Standard2.16
this test will take about 2 minutes to run:

SUMMARY: 51 tests, 1 unexpected failures
FAIL apic-split (51 tests, 1 unexpected failures)

real 1m53.292s
user 1m22.798s
sys 2m22.239s

SUMMARY: 51 tests, 1 unexpected failures
FAIL apic-split (51 tests, 1 unexpected failures)

real 1m47.981s
user 1m15.764s
sys 2m18.655s

SUMMARY: 51 tests, 1 unexpected failures, 1 skipped
FAIL apic-split (51 tests, 1 unexpected failures, 1 skipped)

real 1m52.938s
user 1m12.339s
sys 2m32.482s

[Guilherme G. Piccoli (gpiccoli)]

Observed in H/KVM, cycle sru-20210531 .

[...]
10:04:03 DEBUG| [stdout] PASS: PV IPIs testing^M
10:04:05 DEBUG| [stdout] PASS: nmi-after-sti^M
10:04:15 DEBUG| [stdout] FAIL: multiple nmi^M
10:04:31 DEBUG| [stdout] PASS: pending nmi^M
10:04:31 DEBUG| [stdout] PASS: APIC LVT timer one shot^M
[...]
10:04:31 DEBUG| [stdout] FAIL apic-split (56 tests, 1 unexpected failures)

[Po-Hsu Lin (cypressyew)]

Still visible on H-kvm 5.11.0-1015.16