Running perf against a QEMU/kvm process results in the guest suffering a page fault in trying to store Precise Event Based Sampling (PEBS) records for the host. This affects both using perf against a single process, in which it crashes the targeted guest, or using perf system wide, in which it crashes all running guests on the system.
This reinstates the logic for setting MSR_CORE_PERF_GLOBAL_CTRL to what it was before "KVM: x86/pmu: Add IA32_PEBS_ENABLE MSR emulation for extended PEBS".
If you install it, then running perf against the PID of qemu processes will no longer crash the guest, and they will be accessible by SSH afterward.
[Where problems could occur]
We are rearranging the logic of setting the PEBS MSRs, which affects processor sampling of events. This will affect any profiling tools running against KVM based virtual machines, namely perf against QEMU.
If a regression were to occur, running perf against a VM could cause it to page fault and subsequently crash, resulting in downtime.
The only workaround will be to disable all profiling tools until a fix is available.
BugLink: https:/ /bugs.launchpad .net/bugs/
[Impact]
Running perf against a QEMU/kvm process results in the guest suffering a page fault in trying to store Precise Event Based Sampling (PEBS) records for the host. This affects both using perf against a single process, in which it crashes the targeted guest, or using perf system wide, in which it crashes all running guests on the system.
The issue was introduced in 6.0 by:
commit c59a1f106f5cd48 43c097069ff1bb2 ad72103a67 /git.kernel. org/pub/ scm/linux/ kernel/ git/torvalds/ linux.git/ commit/ ?id=c59a1f106f5 cd4843c097069ff 1bb2ad72103a67
Author: Like Xu <email address hidden>
Date: Mon Apr 11 18:19:36 2022 +0800
Subject: KVM: x86/pmu: Add IA32_PEBS_ENABLE MSR emulation for extended PEBS
Link: https:/
This affects all 6.2 and 6.5 kernels. There is no known workaround, apart from not using perf on affected systems.
[Fix]
The issue was fixed in 6.7 by:
commit 971079464001c68 56186ca137778e5 34d983174a PERF_GLOBAL_ CTRL /git.kernel. org/pub/ scm/linux/ kernel/ git/torvalds/ linux.git/ commit/ ?id=97107946400 1c6856186ca1377 78e534d983174a
Author: Paolo Bonzini <email address hidden>
Date: Thu Jan 4 16:15:17 2024 +0100
Subject: KVM: x86/pmu: fix masking logic for MSR_CORE_
Link: https:/
This reinstates the logic for setting MSR_CORE_ PERF_GLOBAL_ CTRL to what it was before "KVM: x86/pmu: Add IA32_PEBS_ENABLE MSR emulation for extended PEBS".
- .guest = intel_ctrl & (~cpuc- >intel_ ctrl_host_ mask | ~pebs_mask), intel_ctrl_ host_mask & ~pebs_mask,
+ .guest = intel_ctrl & ~cpuc->
The faulty logic includes any bit that isn't both marked as exclude_guest and using PEBS, while it should really be excluding PEBS from the host.
[Testcase]
Start a bare metal server. Enable KVM, start a few VMs. The VMs can be idle, they don't require any workload.
$ sudo apt-get install qemu-kvm libvirt- daemon- system libvirt-clients bridge-utils uvtool ms-libvirt sync --source http:// cloud-images. ubuntu. com/daily release=jammy arch=amd64 ------- ------- ----
$ sudo reboot
$ ssh-keygen
$ uvt-simplestrea
$ uvt-kvm create --cpu 4 --memory 4096 --disk 10 jammy-a release=jammy arch=amd64
$ uvt-kvm create --cpu 4 --memory 4096 --disk 10 jammy-b release=jammy arch=amd64
$ uvt-kvm create --cpu 4 --memory 4096 --disk 10 jammy-c release=jammy arch=amd64
$ virsh list
Id Name State
-------
2 jammy-a running
3 jammy-b running
4 jammy-c running
$ uvt-kvm ssh jammy-a
Check it works.
$ ps aux | grep qemu
Find the pid of jammy-a
$ perf top -p $PID
$ virsh console jammy-a
Escape character is ^] (Ctrl + ])
[ 357.793039] BUG: unable to handle page fault for address: fffffe49178c6028
$ uvt-kvm ssh jammy-a
(no response)
Test packages are available in the following ppa:
https:/ /launchpad. net/~mruffell/ +archive/ ubuntu/ sf379502- test
If you install it, then running perf against the PID of qemu processes will no longer crash the guest, and they will be accessible by SSH afterward.
[Where problems could occur]
We are rearranging the logic of setting the PEBS MSRs, which affects processor sampling of events. This will affect any profiling tools running against KVM based virtual machines, namely perf against QEMU.
If a regression were to occur, running perf against a VM could cause it to page fault and subsequently crash, resulting in downtime.
The only workaround will be to disable all profiling tools until a fix is available.