| 2019-10-16 11:10:14 |
Christian Ehrhardt |
description |
== Comment: #0 - Satheesh Rajendran - 2019-08-20 07:42:03 ==
---Problem Description---
cap-ibs=workaround defaults to "full software flush sequence" though "hardware assisted flush" is available (kvm)
---uname output---
4.15.0-58-generic #64-Ubuntu
---Additional Hardware Info---
# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 160
On-line CPU(s) list: 0-159
Thread(s) per core: 4
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Model: 2.3 (pvr 004e 1203)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2300.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): 0-79
NUMA node8 CPU(s): 80-159
# lsmcode
Version of System Firmware :
Product Name : OpenPOWER Firmware
Product Version : witherspoon-OP9-v2.3-9.52
Product Extra : skiboot-v6.3-231-g0bf01d93ee39
Product Extra : bmc-firmware-version-2.07
Product Extra : occ-e5a2afd
Product Extra : hostboot-2b061c6
Product Extra : buildroot-2019.05-6-g0d430a2
Product Extra : capp-ucode-p9-dd2-v4
Product Extra : machine-xml-a6f4df3
Product Extra : hostboot-binaries-hw062819a.940
Product Extra : sbe-b96edc8
Product Extra : hcode-hw070319a.940
Product Extra : petitboot-v1.10.4
Product Extra : linux-5.1.16-openpower1-pecd2af3
# grep -H . /sys/devices/system/cpu/vulnerabilities/spectre_v2
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Software count cache flush (hardware accelerated)
Machine Type = power9 DD2.3 ppc64le
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1. boot the guest with in above HW DD2.3 with cap-ibs=workaround(same bydefault)
# kvm -M pseries-bionic-sxxm,cap-ibs=workaround -monitor stdio -vga none -nographic -serial /dev/pts/0 -smp 8 -m 8192 /home/sath/ubuntu-18.04-ppc64le.qcow2
QEMU 2.11.1 monitor - type 'help' for more information
(qemu)
2. Check guest dmesg
#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: full software flush sequence enabled.
Expected:
#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: hardware assisted flush sequence enabled
Note: Hardware assisted flush is getting enabled if we set `cap-ccf-assist` explicitly, like below
#kvm -M pseries-bionic-sxxm,cap-ibs=workaround,cap-ccf-assist=on -monitor stdio -vnc none -nographic -serial /dev/pts/0 -smp 8 -m 8192 /home/sath/ubuntu-18.04-ppc64le.qcow2
QEMU 2.11.1 monitor - type 'help' for more information
(qemu)
#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: hardware assisted flush sequence enabled
When hardware assisted flush capability is available it is good to enable bydefault?
Userspace tool common name: Userspace tool common name: ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.18~ppa1 ppc64el QEMU full system emulation binaries (ppc)
Qemu is from Bug 176932 comment 24
External: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1832622/comments/13
== Comment: #3 - Satheesh Rajendran - 2019-08-21 02:29:39 ==
(In reply to comment #2)
> Machine defaults can only be controlled on a per cpu basis, for example
> power7, power8, power9, etc.
> Since the hardware assisted flush is only available on POWER9 DD2.3 it can't
> be enabled by default for the entire power9 processor class. Thus it MUST
> default to the software workaround with the hardware flush enabled
> explicitly on the command line. This is to ensure migration compatibility
> between all power9 cpu models.
then it demands some ways to enable it above layers like libvirt etc.
Today we do not have any provision to do so apart from custom qemu command-line, any thoughts on that?
how we expect the user to exploit this feature in KVM guest?,
It was initially suggested in libvirt to implement and got dropped saying ` drop all features except for HTM, at least for the time being;` , details here, https://www.redhat.com/archives/libvir-list/2018-June/msg01655.html
any updates on reviving it, I know it is a libvirt component question, but if there is any reason we can document it here and move this bug to libvirt to get it fixed.
Regards,
-Satheesh
== Comment: #12 - Daniel Henrique Barboza - 2019-10-10 16:09:28 ==
The patches were accepted upstream:
https://github.com/libvirt/libvirt/commit/86a8e5a84cb6251748925163345c688e0ea794fa
https://github.com/libvirt/libvirt/commit/cab3ea2303617583f31ffb1a9fc73780f796dc06
https://github.com/libvirt/libvirt/commit/8958b47fabe122a8c1dac14198a07906158191c9
These patches will enable the ccf-assist to be used from Libvirt by using this feature:
<features>
<ccf-assist state='on'/>
</features>
Remember: it is a feature Power 9 DD2.3+ only. Enabling it will break guest migration if you migrate a guest with ccf-assist='on' it to a Power 9 DD2.2 or older host.
Canonical: this feature will be rolled out in Libvirt version 5.9.0 at the end of this month/start of November. I'll leave it at your own discretion how to proceed here.
Thanks,
DHB
== Comment: #14 - Murilo Opsfelder Araujo - 2019-10-13 10:44:39 ==
Please mirror to Canonical to get patches from comment 12 back-ported to Bionic, if that's Canonical's will. |
[Impact]
* CCF-assist helps to mitigate some of the spectre related mitigation
costs. The milage varies and depends on workload, but it is generally
recommended to use those features if available.
* Backport the recent upstream change to have libvirt be able to detect
and control that feature
[Test Case]
* On a ppc64 system with the appropriate firmware levels one needs to:
- check virsh capabilities, is ccf-assist detected?
- check virsh domcapabilities, is ccf-assist able to be passed throu?
- configure a guest with ccf-assist and check that the guest
commandline changed to contain e.g. cap-ccf-assist=on
- check if "host-model" (not sure what it is supposed to contain in
this case, but checking it for sanity can't hurt)
[Regression Potential]
* Common issues with changes of this type are related to:
a) why doesn't it work here (FW issues)
b) migrations between different systems might fail (one can't offer
the feature)
But in general the actual "regression" is low as it only changes
behavior when the config is changed.
[Other Info]
* This is the libvirt portion of bug 1832622 - which since it is done
and already had enough content was left alone by the reporter. But
someone being interested can fetch some extra context there.
== Comment: #0 - Satheesh Rajendran - 2019-08-20 07:42:03 ==
---Problem Description---
cap-ibs=workaround defaults to "full software flush sequence" though "hardware assisted flush" is available (kvm)
---uname output---
4.15.0-58-generic #64-Ubuntu
---Additional Hardware Info---
# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 160
On-line CPU(s) list: 0-159
Thread(s) per core: 4
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Model: 2.3 (pvr 004e 1203)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2300.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): 0-79
NUMA node8 CPU(s): 80-159
# lsmcode
Version of System Firmware :
Product Name : OpenPOWER Firmware
Product Version : witherspoon-OP9-v2.3-9.52
Product Extra : skiboot-v6.3-231-g0bf01d93ee39
Product Extra : bmc-firmware-version-2.07
Product Extra : occ-e5a2afd
Product Extra : hostboot-2b061c6
Product Extra : buildroot-2019.05-6-g0d430a2
Product Extra : capp-ucode-p9-dd2-v4
Product Extra : machine-xml-a6f4df3
Product Extra : hostboot-binaries-hw062819a.940
Product Extra : sbe-b96edc8
Product Extra : hcode-hw070319a.940
Product Extra : petitboot-v1.10.4
Product Extra : linux-5.1.16-openpower1-pecd2af3
# grep -H . /sys/devices/system/cpu/vulnerabilities/spectre_v2
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Software count cache flush (hardware accelerated)
Machine Type = power9 DD2.3 ppc64le
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1. boot the guest with in above HW DD2.3 with cap-ibs=workaround(same bydefault)
# kvm -M pseries-bionic-sxxm,cap-ibs=workaround -monitor stdio -vga none -nographic -serial /dev/pts/0 -smp 8 -m 8192 /home/sath/ubuntu-18.04-ppc64le.qcow2
QEMU 2.11.1 monitor - type 'help' for more information
(qemu)
2. Check guest dmesg
#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: full software flush sequence enabled.
Expected:
#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: hardware assisted flush sequence enabled
Note: Hardware assisted flush is getting enabled if we set `cap-ccf-assist` explicitly, like below
#kvm -M pseries-bionic-sxxm,cap-ibs=workaround,cap-ccf-assist=on -monitor stdio -vnc none -nographic -serial /dev/pts/0 -smp 8 -m 8192 /home/sath/ubuntu-18.04-ppc64le.qcow2
QEMU 2.11.1 monitor - type 'help' for more information
(qemu)
#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: hardware assisted flush sequence enabled
When hardware assisted flush capability is available it is good to enable bydefault?
Userspace tool common name: Userspace tool common name: ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.18~ppa1 ppc64el QEMU full system emulation binaries (ppc)
Qemu is from Bug 176932 comment 24
External: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1832622/comments/13
== Comment: #3 - Satheesh Rajendran - 2019-08-21 02:29:39 ==
(In reply to comment #2)
> Machine defaults can only be controlled on a per cpu basis, for example
> power7, power8, power9, etc.
> Since the hardware assisted flush is only available on POWER9 DD2.3 it can't
> be enabled by default for the entire power9 processor class. Thus it MUST
> default to the software workaround with the hardware flush enabled
> explicitly on the command line. This is to ensure migration compatibility
> between all power9 cpu models.
then it demands some ways to enable it above layers like libvirt etc.
Today we do not have any provision to do so apart from custom qemu command-line, any thoughts on that?
how we expect the user to exploit this feature in KVM guest?,
It was initially suggested in libvirt to implement and got dropped saying ` drop all features except for HTM, at least for the time being;` , details here, https://www.redhat.com/archives/libvir-list/2018-June/msg01655.html
any updates on reviving it, I know it is a libvirt component question, but if there is any reason we can document it here and move this bug to libvirt to get it fixed.
Regards,
-Satheesh
== Comment: #12 - Daniel Henrique Barboza - 2019-10-10 16:09:28 ==
The patches were accepted upstream:
https://github.com/libvirt/libvirt/commit/86a8e5a84cb6251748925163345c688e0ea794fa
https://github.com/libvirt/libvirt/commit/cab3ea2303617583f31ffb1a9fc73780f796dc06
https://github.com/libvirt/libvirt/commit/8958b47fabe122a8c1dac14198a07906158191c9
These patches will enable the ccf-assist to be used from Libvirt by using this feature:
<features>
<ccf-assist state='on'/>
</features>
Remember: it is a feature Power 9 DD2.3+ only. Enabling it will break guest migration if you migrate a guest with ccf-assist='on' it to a Power 9 DD2.2 or older host.
Canonical: this feature will be rolled out in Libvirt version 5.9.0 at the end of this month/start of November. I'll leave it at your own discretion how to proceed here.
Thanks,
DHB
== Comment: #14 - Murilo Opsfelder Araujo - 2019-10-13 10:44:39 ==
Please mirror to Canonical to get patches from comment 12 back-ported to Bionic, if that's Canonical's will. |
|
