| 2021-04-07 05:30:54 |
Christian Ehrhardt |
description |
The kernel keeps track of the time spent in userspace, kernel space and for quite a while (since 2.6.24) also guest_time.
This is accessible to userspace
a) globally via /proc/stat [1]
b) per process via /proc/<PID>/stat
But since ~5.9 this accounting is broken by [3]
The Testcase for this is simple, run any KVM guest content for example even the following without any prep is enough
$ sudo qemu-system-x86_64 -enable-kvm -nographic
That should execute at least a bit of time in the guest for startup code.
Surely, if you want anything for long term debugging you can run a full Guest and do any busy loop in the guest.
This was reported upstream to the kernel in [4] and Debian [5] and qemu [6].
AFAIK there are two suggested [8][9], but no accepted solution yet.
If it would be "only" the global counter (a) this might be prio low-to-mid, because while still inhibiting monitoring you'd at least still have the per guest counters (b). But since both are broken one has no easy means anymore to check if there is massive host-kernel (%sys) or host-userspace (%usr) overhead over the guest (%guest) anymore. Therefore (maybe due to my old perf-engineer heart) IMHO this is rather important to be fixed soon.
Best in the next possible update for Hirsute, but that might be hard. But at lest for an Hirsute based HWE kernel to Focal I'd consider this a hard blocker.
[1]: https://www.kernel.org/doc/Documentation/filesystems/proc.txt see "1.8 Miscellaneous kernel statistics in /proc/stat"
[2]: http://manpages.ubuntu.com/manpages/focal/man5/proc.5.html see "(43) guest_time %lu"
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=87fa7f3e98a1310ef1ac1900e7ee7f9610a038bc
[4]: https://bugzilla.kernel.org/show_bug.cgi?id=209831
[5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986371
[6]: https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg00732.html
[7]: https://lore.kernel.org/kvm/1617011036-11734-1-git-send-email-wanpengli@tencent.com/
[8]: https://lore.kernel.org/kvm/20210206004218.312023-1-seanjc@google.com/
ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: linux-image-5.11.0-13-generic 5.11.0-13.14
ProcVersionSignature: Ubuntu 5.11.0-13.14-generic 5.11.7
Uname: Linux 5.11.0-13-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Apr 6 12:56 seq
crw-rw---- 1 root audio 116, 33 Apr 6 12:56 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu61
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: unknown
Date: Wed Apr 7 05:02:03 2021
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
PciMultimedia:
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-13-generic root=UUID=c941b173-e6b5-485a-a02b-8d966b8d3c73 ro --- console=ttyS1,115200
RelatedPackageVersions:
linux-restricted-modules-5.11.0-13-generic N/A
linux-backports-modules-5.11.0-13-generic N/A
linux-firmware 1.196
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/22/2018
dmi.bios.release: 2.56
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.board.name: ProLiant DL360 Gen9
dmi.board.vendor: HP
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.ec.firmware.release: 2.60
dmi.modalias: dmi:bvnHP:bvrP89:bd01/22/2018:br2.56:efr2.60:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL360 Gen9
dmi.product.sku: 780018-S01
dmi.sys.vendor: HP |
The kernel keeps track of the time spent in userspace, kernel space and for quite a while (since 2.6.24) also guest_time.
This is accessible to userspace
a) globally via /proc/stat [1]
b) per process via /proc/<PID>/stat
But since ~5.9 this accounting is broken by [3]
#An example of a busy guest reporting all zeros in a common userspace tool
ubuntu@node-horsea:~$ pidstat -p $(pidof qemu-system-x86_64) 5
Linux 5.11.0-13-generic (node-horsea) 04/07/21 _x86_64_ (12 CPU)
05:13:13 UID PID %usr %system %guest %wait %CPU CPU Command
05:13:18 64055 8186 10.20 96.80 0.00 0.00 107.00 4 qemu-system-x86
05:13:23 64055 8186 5.00 51.00 0.00 0.00 56.00 5 qemu-system-x86
05:13:28 64055 8186 3.60 109.80 0.00 0.00 113.40 5 qemu-system-x86
05:13:33 64055 8186 1.80 14.20 0.00 0.00 16.00 5 qemu-system-x86
This is due to these stats staying zero:
$ cat /proc/stat | grep "cpu "
cpu 11345 154 10342 71506151 549 0 161 0 0 0
The Testcase for this is simple, run any KVM guest content for example even the following without any prep is enough
$ sudo qemu-system-x86_64 -enable-kvm -nographic
That should execute at least a bit of time in the guest for startup code.
Surely, if you want anything for long term debugging you can run a full Guest and do any busy loop in the guest.
This was reported upstream to the kernel in [4] and Debian [5] and qemu [6].
AFAIK there are two suggested [8][9], but no accepted solution yet.
If it would be "only" the global counter (a) this might be prio low-to-mid, because while still inhibiting monitoring you'd at least still have the per guest counters (b). But since both are broken one has no easy means anymore to check if there is massive host-kernel (%sys) or host-userspace (%usr) overhead over the guest (%guest) anymore. Therefore (maybe due to my old perf-engineer heart) IMHO this is rather important to be fixed soon.
Best in the next possible update for Hirsute, but that might be hard. But at lest for an Hirsute based HWE kernel to Focal I'd consider this a hard blocker.
[1]: https://www.kernel.org/doc/Documentation/filesystems/proc.txt see "1.8 Miscellaneous kernel statistics in /proc/stat"
[2]: http://manpages.ubuntu.com/manpages/focal/man5/proc.5.html see "(43) guest_time %lu"
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=87fa7f3e98a1310ef1ac1900e7ee7f9610a038bc
[4]: https://bugzilla.kernel.org/show_bug.cgi?id=209831
[5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986371
[6]: https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg00732.html
[7]: https://lore.kernel.org/kvm/1617011036-11734-1-git-send-email-wanpengli@tencent.com/
[8]: https://lore.kernel.org/kvm/20210206004218.312023-1-seanjc@google.com/
ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: linux-image-5.11.0-13-generic 5.11.0-13.14
ProcVersionSignature: Ubuntu 5.11.0-13.14-generic 5.11.7
Uname: Linux 5.11.0-13-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Apr 6 12:56 seq
crw-rw---- 1 root audio 116, 33 Apr 6 12:56 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu61
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: unknown
Date: Wed Apr 7 05:02:03 2021
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
PciMultimedia:
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-13-generic root=UUID=c941b173-e6b5-485a-a02b-8d966b8d3c73 ro --- console=ttyS1,115200
RelatedPackageVersions:
linux-restricted-modules-5.11.0-13-generic N/A
linux-backports-modules-5.11.0-13-generic N/A
linux-firmware 1.196
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/22/2018
dmi.bios.release: 2.56
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.board.name: ProLiant DL360 Gen9
dmi.board.vendor: HP
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.ec.firmware.release: 2.60
dmi.modalias: dmi:bvnHP:bvrP89:bd01/22/2018:br2.56:efr2.60:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL360 Gen9
dmi.product.sku: 780018-S01
dmi.sys.vendor: HP |
|
