I have now done additional diagnostics on this bug and it appears to be triggered in nested virtualization case when apic virtualisation is available in Layer 0 HW and then passed forward to Layer 1 VM via Libvirt: <cpu mode='host-model' check='partial'> .
Testing found that in case where Layer 1 FreeBSD host had this feature, see "VID,PostIntr" in "VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr" from CPU Feature below:
3. Check via sysctl that virtual_interupt_delivery is disabled:
# sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 <- should be zero
Questions is:
While FreeBSD triggers this bug, is this a KVM issue or a FreeBSD bhyve one ?
Hi Ubuntu / KVM Maintainers,
I have now done additional diagnostics on this bug and it appears to be triggered in nested virtualization case when apic virtualisation is available in Layer 0 HW and then passed forward to Layer 1 VM via Libvirt: <cpu mode='host-model' check='partial'> .
Testing found that in case where Layer 1 FreeBSD host had this feature, see "VID,PostIntr" in "VT-x: PAT,HLT, MTF,PAUSE, EPT,UG, VPID,VID, PostIntr" from CPU Feature below:
<<START LAYER 1 - FreeBSD CPU Report from dmesg.boot>> "GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2 0xf83fbff< FPU,VME, DE,PSE, TSC,MSR, PAE,MCE, CX8,APIC, SEP,MTRR, PGE,MCA, CMOV,PAT, PSE36,MMX, FXSR,SSE, SSE2,SS> 0xfffa3223< SSE3,PCLMULQDQ, VMX,SSSE3, FMA,CX16, PCID,SSE4. 1,SSE4. 2,x2APIC, MOVBE,POPCNT, TSCDLT, AESNI,XSAVE, OSXSAVE, AVX,F16C, RDRAND, HV> 0x2c100800< SYSCALL, NX,Page1GB, RDTSCP, LM> 0x121<LAHF, ABM,Prefetch> 0x1c0fbb< FSGSBASE, TSCADJ, BMI1,HLE, AVX2,SMEP, BMI2,ERMS, INVPCID, RTM,RDSEED, ADX,SMAP> 0xac000400< MD_CLEAR, IBPB,STIBP, ARCH_CAP, SSBD> 0x1<XSAVEOPT> ARCH_CAPS= 0x8<SKIP_ L1DFL_VME> MTF,PAUSE, EPT,UG, VPID,VID, PostIntr
...
...
CPU: Intel Core Processor (Broadwell, IBRS) (2600.09-MHz K8-class CPU)
Origin=
Features=
Features2=
AMD Features=
AMD Features2=
Structured Extended Features=
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=
XSAVE Features=
IA32_
AMD Extended Feature Extensions ID EBX=0x1001000
VT-x: PAT,HLT,
Hypervisor: Origin = "KVMKVMKVM"
...
...
<END LAYER 1 - dimes.log>>
In my case with Intel Broadwell chipset this is available, in case of desktop "core i5-8250U" chip- this reports as:
VT-x: PAT,HLT, MTF,PAUSE, EPT,UG, VPID
For this case HW case, nested:
Layer 0 - Ubuntu 20.04, Layer 1 - FreeBSD 12.1 with bhyve, Layer 2 - FreeBSD 12.1
Works.
Workaround is to disable APIC virtual interrupt delivery:
1. Add entry into Layer 1 - FreeBSD Guest / Host: /boot/loader.conf: vmx.use_ apic_vid= 0
hw.vmm.
2. Reboot
3. Check via sysctl that virtual_ interupt_ delivery is disabled: vmx.cap. virtual_ interrupt_ delivery vmx.cap. virtual_ interrupt_ delivery: 0 <- should be zero
# sysctl hw.vmm.
hw.vmm.
Questions is:
While FreeBSD triggers this bug, is this a KVM issue or a FreeBSD bhyve one ?
In doing some searching on Web I see that there is already work being done with KVM 5.6 around APIC virtualisation and its handling. So not sure if this a potentially know problem: https:/ /events19. linuxfoundation .org/wp- content/ uploads/ 2017/12/ Improving- KVM-x86- Nested- Virtualization- Liran-Alon- Oracle. pdf
APIC Virtualisation support was introduced back in FreeBSD 11.0 way back in Sept 2016:
https:/ /www.freebsd. org/releases/ 11.0R/relnotes. html#hardware- virtualization
Thanks to Peter Graham on FreeBSD virtualization bug tracker for helping to find source of problem.
Should this BUG go to KVM / QEMU upstream ?
Cheers,
John Hartley.