Recent 5.13 kernel has broken KVM support
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Impish |
Fix Released
|
High
|
Po-Hsu Lin | ||
Jammy |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Impact]
This is caused by commit 08335308 "KVM: x86: check PIR even for vCPUs
with disabled APICv", this patch needs 7e1901f6c "KVM: VMX: prepare
sync_pir_to_irr for running with APICv disabled" otherwise if APICv
is disabled in this vcpu it will trigger warning messages in
vmx_sync_
WARN_
With warnings like:
------------[ cut here ]------------
WARNING: CPU: 13 PID: 6997 at arch/x86/
? xfer_to_
Modules linked in: vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common sb_edac x86_pkg_
vcpu_run+0x4d/0x220 [kvm]
acpi_power_meter sch_fq_codel ipmi_devintf ipmi_msghandler msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul syscopyarea crc32_pclmul sysfillrect sysimgblt ghash_clmulni_intel fb_sys_fops ixgbe cec aesni_intel rc_core crypto_simd xfrm_algo cryptd drm ahci dca i2c_i801 xhci_pci mdio libahci i2c_smbus lpc_ich xhci_pci_renesas wmi
CPU: 13 PID: 6997 Comm: qemu-system-x86 Tainted: G W I 5.13.0-39-generic #44-Ubuntu
Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS SE5C610.
kvm_arch_
RIP: 0010:vmx_
Code: e8 47 f5 18 00 8b 93 00 03 00 00 89 45 ec 83 e2 20 85 d2 74 dc 48 8b 55 f0 65 48 2b 14 25 28 00 00 00 75 1d 48 8b 5d f8 c9 c3 <0f> 0b eb 87 f0 80 4b 39 40 8b 93 00 03 00 00 8b 45 ec 83 e2 20 eb
RSP: 0018:ffffae4d8d
RAX: 0000000000000000 RBX: ffff99c552942640 RCX: ffff99c5043a72f0
RDX: ffff99c552942640 RSI: 0000000000000001 RDI: ffff99c552942640
RBP: ffffae4d8d107cb0 R08: ffff99c86f6a7140 R09: 0000000000027100
R10: 0000000042280000 R11: 000000000000000a R12: ffff99c552942640
R13: 0000000000000000 R14: ffffae4d8d1a63e0 R15: ffff99c552942640
FS: 00007f6ae9be764
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000010b8a6006 CR4: 00000000001726e0
Call Trace:
<TASK>
kvm_vcpu_
vcpu_enter_
? xfer_to_
? kvm_vm_
? __fget_
vcpu_run+0x4d/0x220 [kvm]
__x64_sys_
do_syscall_
? fput+0x13/0x20
? exit_to_
? syscall_
? do_syscall_
? syscall_
? do_syscall_
? do_syscall_
? do_syscall_
entry_SYSCALL_
RIP: 0033:0x7f6aebce1a2b
Code: ff ff ff 85 c0 79 8b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 f3 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007f6ae8
RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f6aebce1a2b
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000000c
RBP: 0000557d3b429b90 R08: 0000557d3a4ebff0 R09: 00000000ffffffff
kvm_arch_
R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000003000 R15: 0000000000000000
</TASK>
---[ end trace 5b722d71a78069b1 ]---
This warning message will be flooding in system log files and
eventually eat up all the disk space then crash the server.
This issue will gone by either reverting it or adding the fixes below.
Reference:
https:/
[Fixes]
* 0b8f11737 KVM: Add infrastructure and macro to mark VM as bugged
* 673692735 KVM: x86: Use KVM_BUG/KVM_BUG_ON to handle bugs that are fatal to the VM
* 7e1901f6c KVM: VMX: prepare sync_pir_to_irr for running with APICv disabled
The fix comes in two fold, the first two patches will fix the warning
message flooding issue, make it only gets printed once. The third
patch will change the prevent this to happen.
The first patch needs to be backported as we're missing:
2fdef3a2ae kvm: add PM-notifier
fcfe1baedd KVM: stats: Support binary stats retrieval for a VM
The second patch needs some context adjustment. And the last one can
be cherry-picked.
[Test]
Test kernels can be found here:
https:/
This issue can be verified with LXD:
1. snap install lxd
2. lxc launch images:ubuntu/20.04 --vm vm1
On affected system, the dmesg output will be flooded with this warning
message. With patched kernel the VM can be started with clean dmesg.
I have this kernel tested on Impish, the F-5.13 has been tested by
Daniël Vos (vosdev) on launchpad. Both are working as expected.
kvm-unit-tests has also been tested on my Impish instance to ensure
there is no other issues.
[Where problems could occur]
This patchset will change how the KVM bug gets reported in the kernel,
if it's incorrect it might affect VMX capability.
[Original Bug Report]
Upgrading to 5.13.0-37 or 5.13.0-39 immediately crashes my production servers as they hit:
https://<email address hidden>
It looks like we need to get e90e51d5f01d included in those kernels.
CVE References
Changed in linux (Ubuntu Impish): | |
assignee: | nobody → Po-Hsu Lin (cypressyew) |
tags: | added: 5.13 impish |
Changed in linux (Ubuntu Impish): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Impish): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Impish): | |
status: | In Progress → Fix Committed |
Mar 25 16:18:30 abydos kernel: [ 1319.549186] ------------[ cut here ]------------ kvm/vmx/ vmx.c:6336 vmx_sync_ pir_to_ irr+0x9f/ 0xc0 [kvm_intel] generic libchacha libblake2s_generic xt_HL xt_MASQUERADE xt_TCPMSS xt_tcpudp binfmt_misc rbd unix_diag nf_conntrack_ netlink veth ceph libceph fscache netfs zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ebtable_filter ebtables ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_raw iptable_mangle iptable_nat iptable_filter bpfilter nf_tables vhost_vsock vmw_vsock_ virtio_ transport_ common vhost vhost_iotlb vsock shiftfs sch_ingress geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 8021q garp mrp nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ipmi_ssif intel_rapl_msr intel_rapl_common sb_edac x86_pkg_ temp_thermal intel_powerclamp coretemp kvm_intel kvm rapl intel_cstate T4T+-ST031/ X10DRU- i+, BIOS 3.2a 11/19/2019 sync_pir_ to_irr+ 0x9f/0xc0 [kvm_intel] 577cd8 EFLAGS: 00010246
Mar 25 16:18:30 abydos kernel: [ 1319.549191] WARNING: CPU: 12 PID: 15052 at arch/x86/
Mar 25 16:18:30 abydos kernel: [ 1319.549213] Modules linked in: wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_64 libcurve25519_
Mar 25 16:18:30 abydos kernel: [ 1319.549305] efi_pstore joydev input_leds cdc_acm mei_me mei ioatdma bridge stp llc bonding acpi_ipmi tls ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad mac_hid sch_fq_codel ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crc32_pclmul ghash_clmulni_intel cec nvme rc_core aesni_intel crypto_simd ixgbe igb i2c_i801 xfrm_algo nvme_core drm i2c_smbus ahci i2c_algo_bit cryptd lpc_ich dca xhci_pci mdio libahci xhci_pci_renesas wmi
Mar 25 16:18:30 abydos kernel: [ 1319.549394] CPU: 12 PID: 15052 Comm: qemu-system-x86 Tainted: P O 5.13.0-39-generic #44~20.04.1-Ubuntu
Mar 25 16:18:30 abydos kernel: [ 1319.549399] Hardware name: Supermicro PIO-618U-
Mar 25 16:18:30 abydos kernel: [ 1319.549402] RIP: 0010:vmx_
Mar 25 16:18:30 abydos kernel: [ 1319.549415] Code: 83 c4 10 5b 5d c3 48 89 df e8 5d dc eb fd 8b 93 00 03 00 00 89 45 ec 83 e2 20 85 d2 75 d2 89 c7 e8 f6 fd ff ff 8b 45 ec eb c6 <0f> 0b eb 86 f0 80 4b 39 40 8b 93 00 03 00 00 8b 45 ec 83 e2 20 eb
Mar 25 16:18:30 abydos kernel: [ 1319.549419] RSP: 0018:ffffaed30c
Mar 25 16:18:30 abydos kernel: [ 1319.549423] RAX: 0000000000000000 RBX: ffff98162a4f8000 RCX: 0000000000000000
Mar 25 16:18:30 abydos kernel: [ 1319.549425] RDX: 0000000000000400 RSI: ffffffffc0c38509 RDI: ffff98162a4f8000
Mar 25 16:18:30 abydos kernel: [ 1319.549428] RBP: ffffaed30c577cf0 R08: 0000000000000400 R09: ffff98161e73fc00
Mar 25 16:18:30 abydos kernel: [ 1319.549430] R10: 0000000000000000 R11: 0000000000000000 R12: ffff98162a4f8000
Mar 25 16:18:30 aby...