vmx_apicv_test in ubuntu_kvm_unit_tests failed on B-KVM (FAIL: x2apic - reading 0x090: x2APIC op triggered GP.)

[Po-Hsu Lin]

vmx_apicv_test from ubuntu_kvm_unit_tests failed on B-KVM:

  FAIL: x2apic - reading 0x090: x2APIC op triggered GP.
  FAIL: x2apic - reading 0x090: x2APIC op triggered GP.
  FAIL: x2apic - reading 0x090: x2APIC op triggered GP.
  FAIL: x2apic - reading 0x090: x2APIC op triggered GP.
  FAIL vmx_apicv_test (9239 tests, 4 unexpected failures)

Complete test log: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1827866/+attachment/5521805/+files/vmx_apicv_test-b-kvm.log

[Po-Hsu Lin (cypressyew)]

vmx_apicv_test timeouted on B-KVM:
FAIL vmx_apicv_test (timeout; duration=10)

[Po-Hsu Lin (cypressyew)]

Can be found on X-Azure 4.15 as well:
 Running '/home/azure/autotest/client/tmp/ubuntu_kvm_unit_tests/src/kvm-unit-tests/tests/vmx_apicv_test'
 BUILD_HEAD=e2c275c4
 timeout -k 1s --foreground 10 /usr/bin/qemu-system-x86_64 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.O2MSEezMNL -smp 1 -cpu host,+vmx -append apic_reg_virt_test virt_x2apic_mode_test # -initrd /tmp/tmp.nFaa5lDEut
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 477000
 cr4 = 20

 Test suite: apic_reg_virt_test
 qemu-system-x86_64: terminating on signal 15 from pid 21120
 FAIL vmx_apicv_test (timeout; duration=10)
 qemu:qemu_cpu_kick_thread: No such process

[Po-Hsu Lin (cypressyew)]

Found on B-aws amd64 4.15

[Po-Hsu Lin (cypressyew)]

Failed on Oracle B-4.15
Passed on Oracle B-5.0

[Po-Hsu Lin (cypressyew)]

Failed on B-AWS-5.3 bare-metal instance i3.metal

 PASS: x2apic - writing 0x78 to 0x080: x2APIC write; got 0x78, want 0x78
 Unhandled exception 13 #GP at ip 00000000004071a3
 error_code=0000 rflags=00010006 cs=00000008
 rax=0000000000000078 rcx=0000000000000809 rdx=0000000000000000 rbx=0000000000000009
 rbp=000000000047efdf rsi=000000000042568d rdi=000000000042568d
 r8=000000000000000a r9=00000000000003f8 r10=000000000000000d r11=0000000000000000
 r12=0000000000000000 r13=0000000000000000 r14=0000000000403dcc r15=0000000000000000
 cr0=0000000080010031 cr2=0000000000000000 cr3=0000000000477000 cr4=0000000000002020
 cr8=0000000000000007
 STACK: @4071a3 40170b 4004dd
 FAIL vmx_apicv_test

[Po-Hsu Lin (cypressyew)]

Passed with B-KVM 4.15.0-1051.51 this cycle.

[Po-Hsu Lin (cypressyew)]

This failure can be found on Eoan bare metal node fili (5.3.0-25.27)

 Unhandled exception 13 #GP at ip 00000000004071bf
 error_code=0000 rflags=00010006 cs=00000008
 rax=0000000000000078 rcx=0000000000000809 rdx=0000000000000000 rbx=0000000000000009
 rbp=000000000047efdf rsi=000000000000000a rdi=00000000000003fd
 r8=000000000042568d r9=00000000000003f8 r10=000000000000000d r11=0000000000000000
 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000403d36
 cr0=0000000080010031 cr2=0000000000000000 cr3=0000000000477000 cr4=0000000000002020
 cr8=0000000000000007
 STACK: @4071bf 40173d 4004dd
 FAIL vmx_apicv_test

[Po-Hsu Lin (cypressyew)]

This issue can be found on AWS Trusty 4.4 as well (4.4.0-1061.65-aws):

 timeout -k 1s --foreground 10 /usr/bin/qemu-system-x86_64 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.QWPJWlQ7uF -smp 1 -cpu host,+vmx -append apic_reg_virt_test virt_x2apic_mode_test # -initrd /tmp/tmp.OCXCEUzFNz
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 477000
 cr4 = 20

 Test suite: apic_reg_virt_test
 --- Virtualize APIC accesses test ---
 FAIL: xapic - reading 0x000: x86/vmx_tests.c:2184: Assertion failed: (expected) == (actual)
 LHS: 0x000000000000002c - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0010'1100 - 44
 RHS: 000000000000000000 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000 - 0
 Expected VMX_APIC_ACCESS, got VMX_EXC_NMI.
 STACK: 405e1c 409a1e 401556 4039f1 400312

 Test suite: virt_x2apic_mode_test
 --- Baseline test ---
 FAIL: xapic - reading 0x000: x2apic - reading 0x000: x86/vmx_tests.c:2184: Assertion failed: (expected) == (actual)
 LHS: 0x0000000000000012 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001'0010 - 18
 RHS: 000000000000000000 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000 - 0
 Expected VMX_VMCALL, got VMX_EXC_NMI.
 STACK: 405e1c 405e46 40c979 401556 4039f1 400312
 SUMMARY: 8 tests, 2 unexpected failures
 FAIL vmx_apicv_test (8 tests, 2 unexpected failures)

[Po-Hsu Lin (cypressyew)]

BTW for AWS T-4.4, this is only failing on:
* r5.metal
* c5.metal

But passed with i3.metal

[Kelsey Steele (kelsey-steele)]

Failed on Bionic/oracle : 5.4.0-1026.26~18.04.1 : amd64

1225. 09/24 08:14:26 DEBUG| utils:0153| [stdout] Unhandled exception 13 #GP at ip 00000000004071a3
1226. 09/24 08:14:26 DEBUG| utils:0153| [stdout] error_code=0000 rflags=00010006 cs=00000008
1227. 09/24 08:14:26 DEBUG| utils:0153| [stdout] rax=0000000000000078 rcx=0000000000000809 rdx=0000000000000000 rbx=0000000000000009
1228. 09/24 08:14:26 DEBUG| utils:0153| [stdout] rbp=000000000047efdf rsi=000000000042568d rdi=000000000042568d
1229. 09/24 08:14:26 DEBUG| utils:0153| [stdout] r8=000000000000000a r9=00000000000003f8 r10=000000000000000d r11=0000000000000000
1230. 09/24 08:14:26 DEBUG| utils:0153| [stdout] r12=0000000000000000 r13=0000000000000000 r14=0000000000403dcc r15=0000000000000000
1231. 09/24 08:14:26 DEBUG| utils:0153| [stdout] cr0=0000000080010031 cr2=0000000000000000 cr3=0000000000477000 cr4=0000000000002020
1232. 09/24 08:14:26 DEBUG| utils:0153| [stdout] cr8=0000000000000007
1233. 09/24 08:14:26 DEBUG| utils:0153| [stdout] STACK: @4071a3 40170b 4004dd
1234. 09/24 08:14:26 DEBUG| utils:0153| [stdout] FAIL vmx_apicv_test

[Po-Hsu Lin (cypressyew)]

Still visible in Xenial 4.4.0-1118.132 AWS c5.metal

[Po-Hsu Lin (cypressyew)]

Found on B-aws-5.4
 Unhandled exception 13 #GP at ip 00000000004071a3
 error_code=0000 rflags=00010006 cs=00000008
 rax=0000000000000078 rcx=0000000000000809 rdx=0000000000000000 rbx=0000000000000009
 rbp=000000000047efdf rsi=000000000042568d rdi=000000000042568d
 r8=000000000000000a r9=00000000000003f8 r10=000000000000000d r11=0000000000000000
 r12=0000000000000000 r13=0000000000000000 r14=0000000000403dcc r15=0000000000000000
 cr0=0000000080010031 cr2=0000000000000000 cr3=0000000000477000 cr4=0000000000002020
 cr8=0000000000000007
 STACK: @4071a3 40170b 4004dd
 FAIL vmx_apicv_test

[Kelsey Steele (kelsey-steele)]

Also found on Bionic/hwe-5.4 5.4.0-67.75~18.04.1 host rumford, passed on Grumman

[Ian May (ian-may)]

bionic/linux-oracle: 4.15.0-1071.79

[Ian May (ian-may)]

bionic/linux-oracle-5.4: 5.4.0-1044.47~18.04.1

[Ian May (ian-may)]

Found on bionic/linux-aws-5.4: 5.4.0-1049.51~18.04.1

[Guilherme G. Piccoli (gpiccoli)]

Observed in B/KVM, cycle sru-20210531.

[Guilherme G. Piccoli (gpiccoli)]

Observed in B-5.4/aws, cycle sru-20210531 .

[Guilherme G. Piccoli (gpiccoli)]

Observed in B-5.4/aws, cycle sru-20210621 .

[Guilherme G. Piccoli (gpiccoli)]

Observed this on B/oracle-5.4, cycle sru-20210621 .

[Po-Hsu Lin (cypressyew)]

Didn't see this anymore on X-azure with updated "hirsute" branch in kvm-unit-tests, remove tags and modify bug title.

[Po-Hsu Lin (cypressyew)]

We don't have X-kvm anymore, modify bug title.

[Po-Hsu Lin (cypressyew)]

This is still visible on B-kvm 4.15.0-1099.101, with different failure. Will update the bug description.

[Po-Hsu Lin (cypressyew)]

With kvm-unit-test repo updated to d8a4f9e5 (ci: Update the list of tests that we run in the Fedora Cirrus-CI), this test is still failing with the same error message.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.