[Potential Regression] hyperv_connections / hyperv_stimer / hyperv_synic failed on B-i386 / B-5.4 i386

[Po-Hsu Lin]

Issue found on 4.15.0-114.115, with i386 generic /lowlatency on node fozzie.

All 3 tests failed with timeout issue:
* hyperv_connections
 timeout -k 1s --foreground 90s /usr/bin/qemu-system-i386 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.UwdQ0PZort -smp 2 -cpu kvm64,hv_vpindex,hv_synic -device hyperv-testdev # -initrd /tmp/tmp.GNmO7BfE7a
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 enabling apic
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 453000
 cr4 = 10
 EAX=00459001 EBX=0040aae0 ECX=40000083 EDX=00000000
 ESI=00000000 EDI=00000000 EBP=0044b158 ESP=0044b130
 EIP=004006b9 EFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
 ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
 SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 GS =0010 0044a1f0 ffffffff 00c09300 DPL=0 DS [-WA]
 LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
 TR =0080 00406280 0000ffff 00008b00 DPL=0 TSS32-busy
 GDT= 00406000 0000027f
 IDT= 00000000 00000fff
 CR0=80010011 CR2=00000000 CR3=00453000 CR4=00000010
 DR0=00000000 DR1=00000000 DR2=664fb17c DR3=00000000
 DR6=ffff0ff0 DR7=00000400
 EFER=0000000000000000
 Code=87 e0 aa 40 00 83 c8 01 ba 00 00 00 00 b9 83 00 00 40 0f 30 <8b> 43 04 83 c8 01 b9 82 00 00 40 0f 30 b8 01 00 00 00 b9 80 00 00 40 0f 30 83 c4 0c 0f b6
 qemu-system-i386: terminating on signal 15 from pid 12753 (timeout)
 FAIL hyperv_connections (timeout; duration=90s)

* hyperv_stimer
 timeout -k 1s --foreground 90s /usr/bin/qemu-system-i386 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.P9Uky1nZUD -smp 2 -cpu kvm64,hv_vpindex,hv_time,hv_synic,hv_stimer -device hyperv-testdev # -initrd /tmp/tmp.uulivUIlYy
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 enabling apic
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 453000
 cr4 = 10
 enabling apic
 cpus = 2
 EAX=00458001 EBX=0040aa9c ECX=40000083 EDX=00000000
 ESI=00000000 EDI=00000004 EBP=0044aab8 ESP=0044aaa0
 EIP=0040070c EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
 ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
 SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 GS =0010 00449b60 ffffffff 00c09300 DPL=0 DS [-WA]
 LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
 TR =0080 00406280 0000ffff 00008b00 DPL=0 TSS32-busy
 GDT= 00406000 0000027f
 IDT= 00000000 00000fff
 CR0=80010011 CR2=00000000 CR3=00453000 CR4=00000010
 DR0=00000000 DR1=00000000 DR2=4036d141 DR3=00000000
 DR6=ffff0ff0 DR7=00000400
 EFER=0000000000000000
 Code=86 64 aa 40 00 83 c8 01 ba 00 00 00 00 b9 83 00 00 40 0f 30 01 00 00 00 b9 80 00 00 40 0f 30 83 ec 04 6a 00 68 f1 00 00 00 6a 02 e8 ac 02 00 00 83
 EAX=00459001 EBX=0040aad8 ECX=40000083 EDX=00000000
 ESI=0000003c EDI=00000004 EBP=0009ffbc ESP=0009ffa4
 EIP=0040070c EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
 ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
 SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 GS =0010 0009f000 ffffffff 00c09300 DPL=0 DS [-WA]
 LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
 TR =0088 004062e8 0000ffff 00008b00 DPL=0 TSS32-busy
 GDT= 00406000 0000027f
 IDT= 00000000 00000fff
 CR0=e0000011 CR2=00000000 CR3=00453000 CR4=00000010
 DR0=00000000 DR1=00000000 DR2=2d1ab8b7 DR3=00000000
 DR6=ffff0ff0 DR7=00000400
 EFER=0000000000000000
 Code=86 64 aa 40 00 83 c8 01 ba 00 00 00 00 b9 83 00 00 40 0f 30 01 00 00 00 b9 80 00 00 40 0f 30 83 ec 04 6a 00 68 f1 00 00 00 6a 02 e8 ac 02 00 00 83
 qemu-system-i386: terminating on signal 15 from pid 12956 (timeout)
 FAIL hyperv_stimer (timeout; duration=90s)

* hyperv_synic
 timeout -k 1s --foreground 90s /usr/bin/qemu-system-i386 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.31xb2AFzst -smp 2 -cpu kvm64,hv_vpindex,hv_synic -device hyperv-testdev # -initrd /tmp/tmp.TrTv7F4PNR
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 enabling apic
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 454000
 cr4 = 10
 enabling apic
 ncpus = 2
 prepare
 EAX=00467001 EBX=00000000 ECX=40000083 EDX=00000000
 ESI=00000001 EDI=0040035d EBP=0044b7a8 ESP=0044b7a0
 EIP=004003bc EFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
 ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
 SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 GS =0010 0044a870 ffffffff 00c09300 DPL=0 DS [-WA]
 LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
 TR =0080 00407280 0000ffff 00008b00 DPL=0 TSS32-busy
 GDT= 00407000 0000027f
 IDT= 00000000 00000fff
 CR0=80010011 CR2=00000000 CR3=00454000 CR4=00000010
 DR0=00000000 DR1=00000000 DR2=21f9ba9d DR3=00000000
 DR6=ffff0ff0 DR7=00000400
 EFER=0000000000000000
 Code=1c 00 00 83 c8 01 bb 00 00 00 00 b9 83 00 00 40 89 da 0f 30 40 1c 00 00 83 c8 01 b9 82 00 00 40 89 da 0f 30 b8 01 00 00 00 b9 80 00 00 40 0f 30 e8
 EAX=00468001 EBX=00000000 ECX=40000083 EDX=00000000
 ESI=0040035d EDI=00454000 EBP=0009ffbc ESP=0009ffb4
 EIP=004003bc EFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
 ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
 SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
 GS =0010 0009f000 ffffffff 00c09300 DPL=0 DS [-WA]
 LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
 TR =0088 004072e8 0000ffff 00008b00 DPL=0 TSS32-busy
 GDT= 00407000 0000027f
 IDT= 00000000 00000fff
 CR0=e0000011 CR2=00000000 CR3=00454000 CR4=00000010
 DR0=00000000 DR1=00000000 DR2=3135efd9 DR3=00000000
 DR6=ffff0ff0 DR7=00000400
 EFER=0000000000000000
 Code=1c 00 00 83 c8 01 bb 00 00 00 00 b9 83 00 00 40 89 da 0f 30 40 1c 00 00 83 c8 01 b9 82 00 00 40 89 da 0f 30 b8 01 00 00 00 b9 80 00 00 40 0f 30 e8
 qemu-system-i386: terminating on signal 15 from pid 13160 (timeout)
 FAIL hyperv_synic (timeout; duration=90s)

These 3 tests were skipped / passed in 4.15.0-112.113:
* hyperv_connections
 timeout -k 1s --foreground 90s /usr/bin/qemu-system-i386 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.kk19hBFFOa -smp 2 -cpu kvm64,hv_vpindex,hv_synic -device hyperv-testdev # -initrd /tmp/tmp.iFsWaupJ8x
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 enabling apic
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 453000
 cr4 = 10
 SKIP: Hyper-V SynIC connections are not supported
 SUMMARY: 1 tests, 1 skipped
 SKIP hyperv_connections (1 tests, 1 skipped)

* hyperv_stimer
 timeout -k 1s --foreground 90s /usr/bin/qemu-system-i386 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.mHE6kfD8aI -smp 2 -cpu kvm64,hv_vpindex,hv_time,hv_synic,hv_stimer -device hyperv-testdev # -initrd /tmp/tmp.px8J6EErWg
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 enabling apic
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 453000
 cr4 = 10
 enabling apic
 cpus = 2
 PASS: Hyper-V SynIC periodic timers test vcpu 0
 PASS: Hyper-V SynIC periodic timers test vcpu 1
 PASS: Hyper-V SynIC one-shot test vcpu 0
 PASS: Hyper-V SynIC one-shot test vcpu 1
 PASS: Hyper-V SynIC auto-enable one-shot timer test vcpu 0
 PASS: Hyper-V SynIC auto-enable one-shot timer test vcpu 1
 PASS: Hyper-V SynIC auto-enable periodic timer test vcpu 0
 PASS: Hyper-V SynIC auto-enable periodic timer test vcpu 1
 PASS: no timer fired while msg slot busy: vcpu 0
 PASS: no timer fired while msg slot busy: vcpu 1
 PASS: timer resumed when msg slot released: vcpu 0
 PASS: timer resumed when msg slot released: vcpu 1
 SUMMARY: 12 tests
 PASS hyperv_stimer (12 tests)

* hyperv_synic
 timeout -k 1s --foreground 90s /usr/bin/qemu-system-i386 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.Uh5zREsVu5 -smp 2 -cpu kvm64,hv_vpindex,hv_synic -device hyperv-testdev # -initrd /tmp/tmp.z95qOX5WXi
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.syscall [bit 11]
 qemu-system-i386: warning: host doesn't support requested feature: CPUID.80000001H:EDX.lm [bit 29]
 enabling apic
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 454000
 cr4 = 10
 enabling apic
 ncpus = 2
 prepare
 test 0 -> 1
 test 1 -> 0
 cleanup
 isr_enter_count[0] = 16
 isr_enter_count[1] = 16
 PASS: Hyper-V SynIC test
 SUMMARY: 1 tests
 PASS hyperv_synic (1 tests)

[Po-Hsu Lin (cypressyew)]

Retested on node pepe with 4.15.0-114.115, this is still failing.

[Marcelo Cerri (mhcerri)]

Sam. Does it happen with generic and amd64?

[Po-Hsu Lin (cypressyew)]

Hi Marcelo,
No, amd64 works fine.

It's just affecting i386 generic and lowlatency.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1892855

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Po-Hsu Lin (cypressyew)]

BTW this test was not tested on X-4.15, the kvm-unit-test cannot be built on X/T
https://bugs.launchpad.net/bugs/1798007

[Po-Hsu Lin (cypressyew)]

A Bisect shows:
40ed11e0049a5856848e59a6275ba08b734accc8 is the first bad commit
commit 40ed11e0049a5856848e59a6275ba08b734accc8
Author: Jon Doron <email address hidden>
Date: Fri Apr 24 14:37:40 2020 +0300

    x86/kvm/hyper-v: Explicitly align hcall param for kvm_hyperv_exit

    BugLink: https://bugs.launchpad.net/bugs/1885176

    [ Upstream commit f7d31e65368aeef973fab788aa22c4f1d5a6af66 ]

    The problem the patch is trying to address is the fact that 'struct
    kvm_hyperv_exit' has different layout on when compiling in 32 and 64 bit
    modes.

    In 64-bit mode the default alignment boundary is 64 bits thus
    forcing extra gaps after 'type' and 'msr' but in 32-bit mode the
    boundary is at 32 bits thus no extra gaps.

    This is an issue as even when the kernel is 64 bit, the userspace using
    the interface can be both 32 and 64 bit but the same 32 bit userspace has
    to work with 32 bit kernel.

    The issue is fixed by forcing the 64 bit layout, this leads to ABI
    change for 32 bit builds and while we are obviously breaking '32 bit
    userspace with 32 bit kernel' case, we're fixing the '32 bit userspace
    with 64 bit kernel' one.

    As the interface has no (known) users and 32 bit KVM is rather baroque
    nowadays, this seems like a reasonable decision.

    Reviewed-by: Vitaly Kuznetsov <email address hidden>
    Signed-off-by: Jon Doron <email address hidden>
    Message-Id: <email address hidden>
    Reviewed-by: Roman Kagan <email address hidden>
    Signed-off-by: Paolo Bonzini <email address hidden>
    Signed-off-by: Sasha Levin <email address hidden>
    Signed-off-by: Kamal Mostafa <email address hidden>
    Signed-off-by: Khalid Elmously <email address hidden>

:040000 040000 e62587476ed3db655527a1e55776ad3dd7756ff6 d085168ac233723af3b4e0e2c079e97d72f6f7e1 M Documentation
:040000 040000 10bf0fe682799f3de914b15a1066636bc25e989a f78b5dad5a85cf65f04c372532d654c0d2c094f8 M include

$ git bisect log
git bisect start
# bad: [0e158155817fc6ac0a55648943e52b1004747d74] UBUNTU: Ubuntu-4.15.0-116.117
git bisect bad 0e158155817fc6ac0a55648943e52b1004747d74
# good: [495149ddc61a5997857fda041ccd4c81cac46e00] UBUNTU: Ubuntu-4.15.0-112.113
git bisect good 495149ddc61a5997857fda041ccd4c81cac46e00
# bad: [672e9dec222925652241f8522bb7e7483d0fa74c] drm/amdgpu: Replace invalid device ID with a valid device ID
git bisect bad 672e9dec222925652241f8522bb7e7483d0fa74c
# bad: [cebf29010efca3b7f7e195f80f5fd9db5f93afea] PCI: Don't disable decoding when mmio_always_on is set
git bisect bad cebf29010efca3b7f7e195f80f5fd9db5f93afea
# good: [8606a3b9726b9efb0a92015372be88a84b291406] s390/mm: fix set_huge_pte_at() for empty ptes
git bisect good 8606a3b9726b9efb0a92015372be88a84b291406
# good: [692fb8958a8fbb0b3b645e73ef89b1e75a543538] ath9k: Fix general protection fault in ath9k_hif_usb_rx_cb
git bisect good 692fb8958a8fbb0b3b645e73ef89b1e75a543538
# bad: [749a10fcdcb6e6e3afa26aa9994325f8fcb19eda] tools api fs: Make xxx__mountpoint() more scalable
git bisect bad 749a10fcdcb6e6e3afa26aa9994325f8fcb19eda
# good: [c0cf7189d1b30c7c5423ce2aa6442f3e2f026568] net: ena: fix error returning...

[Po-Hsu Lin (cypressyew)]

Affecting B-5.4 i386 as well

[Po-Hsu Lin (cypressyew)]

Found on B-4.15 i386 node onza 4.15.0-135.139

[Guilherme G. Piccoli (gpiccoli)]

Issue observed in G/KVM, cycle sru-20210531.

[Po-Hsu Lin (cypressyew)]

With kvm-unit-test repo updated to d8a4f9e5 (ci: Update the list of tests that we run in the Fedora Cirrus-CI), this test is still failing with the same error message.