[Power9][WSP][DD2.3] cap-ibs=workaround defaults to "full software flush sequence" though "hardware assisted flush" is available

[bugproxy]

[Impact]

 * CCF-assist helps to mitigate some of the spectre related mitigation
   costs. The milage varies and depends on workload, but it is generally
   recommended to use those features if available.

 * Backport the recent upstream change to have libvirt be able to detect
   and control that feature

[Test Case]

 * On a ppc64 system with the appropriate firmware levels one needs to:
   - check virsh capabilities, is ccf-assist detected?
   - check virsh domcapabilities, is ccf-assist able to be passed throu?
   - configure a guest with ccf-assist and check that the guest
     commandline changed to contain e.g. cap-ccf-assist=on
   - check if "host-model" (not sure what it is supposed to contain in
     this case, but checking it for sanity can't hurt)

[Regression Potential]

 * Common issues with changes of this type are related to:
   a) why doesn't it work here (FW issues)
   b) migrations between different systems might fail (one can't offer
      the feature)
   But in general the actual "regression" is low as it only changes
   behavior when the config is changed.

[Other Info]

 * This is the libvirt portion of bug 1832622 - which since it is done
   and already had enough content was left alone by the reporter. But
   someone being interested can fetch some extra context there.

== Comment: #0 - Satheesh Rajendran - 2019-08-20 07:42:03 ==
---Problem Description---
cap-ibs=workaround defaults to "full software flush sequence" though "hardware assisted flush" is available (kvm)

---uname output---
4.15.0-58-generic #64-Ubuntu

---Additional Hardware Info---
# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 160
On-line CPU(s) list: 0-159
Thread(s) per core: 4
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Model: 2.3 (pvr 004e 1203)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2300.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): 0-79
NUMA node8 CPU(s): 80-159

# lsmcode
Version of System Firmware :
 Product Name : OpenPOWER Firmware
 Product Version : witherspoon-OP9-v2.3-9.52
 Product Extra : skiboot-v6.3-231-g0bf01d93ee39
 Product Extra : bmc-firmware-version-2.07
 Product Extra : occ-e5a2afd
 Product Extra : hostboot-2b061c6
 Product Extra : buildroot-2019.05-6-g0d430a2
 Product Extra : capp-ucode-p9-dd2-v4
 Product Extra : machine-xml-a6f4df3
 Product Extra : hostboot-binaries-hw062819a.940
 Product Extra : sbe-b96edc8
 Product Extra : hcode-hw070319a.940
 Product Extra : petitboot-v1.10.4
 Product Extra : linux-5.1.16-openpower1-pecd2af3

# grep -H . /sys/devices/system/cpu/vulnerabilities/spectre_v2
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Software count cache flush (hardware accelerated)

Machine Type = power9 DD2.3 ppc64le

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 1. boot the guest with in above HW DD2.3 with cap-ibs=workaround(same bydefault)

# kvm -M pseries-bionic-sxxm,cap-ibs=workaround -monitor stdio -vga none -nographic -serial /dev/pts/0 -smp 8 -m 8192 /home/sath/ubuntu-18.04-ppc64le.qcow2
QEMU 2.11.1 monitor - type 'help' for more information
(qemu)

2. Check guest dmesg
#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: full software flush sequence enabled.

Expected:
#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: hardware assisted flush sequence enabled

Note: Hardware assisted flush is getting enabled if we set `cap-ccf-assist` explicitly, like below

#kvm -M pseries-bionic-sxxm,cap-ibs=workaround,cap-ccf-assist=on -monitor stdio -vnc none -nographic -serial /dev/pts/0 -smp 8 -m 8192 /home/sath/ubuntu-18.04-ppc64le.qcow2
QEMU 2.11.1 monitor - type 'help' for more information
(qemu)

#dmesg | grep cache-flush
[ 0.000000] count-cache-flush: hardware assisted flush sequence enabled

When hardware assisted flush capability is available it is good to enable bydefault?

Userspace tool common name: Userspace tool common name: ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.18~ppa1 ppc64el QEMU full system emulation binaries (ppc)

Qemu is from Bug 176932 comment 24
External: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1832622/comments/13

== Comment: #3 - Satheesh Rajendran - 2019-08-21 02:29:39 ==
(In reply to comment #2)
> Machine defaults can only be controlled on a per cpu basis, for example
> power7, power8, power9, etc.
> Since the hardware assisted flush is only available on POWER9 DD2.3 it can't
> be enabled by default for the entire power9 processor class. Thus it MUST
> default to the software workaround with the hardware flush enabled
> explicitly on the command line. This is to ensure migration compatibility
> between all power9 cpu models.

then it demands some ways to enable it above layers like libvirt etc.

Today we do not have any provision to do so apart from custom qemu command-line, any thoughts on that?
how we expect the user to exploit this feature in KVM guest?,

It was initially suggested in libvirt to implement and got dropped saying ` drop all features except for HTM, at least for the time being;` , details here, https://www.redhat.com/archives/libvir-list/2018-June/msg01655.html

any updates on reviving it, I know it is a libvirt component question, but if there is any reason we can document it here and move this bug to libvirt to get it fixed.

Regards,
-Satheesh

== Comment: #12 - Daniel Henrique Barboza - 2019-10-10 16:09:28 ==
The patches were accepted upstream:

https://github.com/libvirt/libvirt/commit/86a8e5a84cb6251748925163345c688e0ea794fa
https://github.com/libvirt/libvirt/commit/cab3ea2303617583f31ffb1a9fc73780f796dc06
https://github.com/libvirt/libvirt/commit/8958b47fabe122a8c1dac14198a07906158191c9

These patches will enable the ccf-assist to be used from Libvirt by using this feature:

  <features>
    <ccf-assist state='on'/>
  </features>

Remember: it is a feature Power 9 DD2.3+ only. Enabling it will break guest migration if you migrate a guest with ccf-assist='on' it to a Power 9 DD2.2 or older host.

Canonical: this feature will be rolled out in Libvirt version 5.9.0 at the end of this month/start of November. I'll leave it at your own discretion how to proceed here.

Thanks,
DHB

== Comment: #14 - Murilo Opsfelder Araujo - 2019-10-13 10:44:39 ==
Please mirror to Canonical to get patches from comment 12 back-ported to Bionic, if that's Canonical's will.

[Christian Ehrhardt  (paelzer)]

None of the changes is in any released libvirt version yet, so all Ubuntu releases are affected.
Since we only backported the ccf qemu bits up to Bionic that is the limit for these backports as well.

[Christian Ehrhardt  (paelzer)]

Thanks for the report, I already added an SRU Template.
I'll try to prep a PPA for you to test on recent HW/FW combinations to work the expected way even on the backport.

[Christian Ehrhardt  (paelzer)]

The patches won't apply trivially back to libvirt in Bionic as this was before the rework to have these types per machine nor the one to ABI stability checks and validation (maybe more, those were the ones I've seen on a quick check). Eoan with libvirt 5.4 has most of that code, but Bionic being on 4.0 needs quite some fixups.
Obviously I might force things in to maybe-sort-of-work, but if there is an official IBM-backport of these PPC64-only changes that would be helpful.

Therefore before I waste my time to then find there is something let me ask if you happen to have or are going to create backports of those three patches already?

[Andrew Cloke (andrew-cloke)]

Marking as "incomplete" while awaiting the backport discussed in comment #3.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-11-25 10:24 EDT-------
Considering that we have a workaround that works on Bionic, I suggest we just integrate this into future versions of Ubuntu through rebase.

[Christian Ehrhardt  (paelzer)]

Ok, looking forward it now is confirmed that it was accepted in libvirt v5.9.0
Marking this for the merge that I'll do for Ubuntu 20.04

[Frank Heimes (fheimes)]

Just fyi - plan is to go with libvirt 6.0 for 20.04.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.0.0-0ubuntu1

---------------
libvirt (6.0.0-0ubuntu1) focal; urgency=medium

  * Merged with Debian 5.6.0-4 from experimental and v6.0.0 from upstream
    Among many other new features and fixes this includes fixes for:
    - LP: #1859253 - rbd driver fails to create a new volume
    - LP: #1858341 - rbd driver does not list all volumes in pool
    - LP: #1845506 - Libvirt snapshot doesn't update apparmor profile
    - LP: #1854653 - slow libvirt-guests.sh during shutdown if service is off
    - LP: #1848229 - enable ppc64el to use ccf-assist feature
    - LP: #1853315 - Enable CPU Model Comparison and Baselining on s390x
    - LP: #1853317 - CCW IPL support to boot from ECKD DASDs
    - LP: #1859506 - security: AppArmor profile fixes for swtpm
    Remaining changes:
    - Disable libssh2 support (universe dependency)
    - Disable firewalld support (universe dependency)
    - Set qemu-group to kvm (for compat with older ubuntu)
    - Additional apport package-hook
    - Autostart default bridged network (As upstream does, but not Debian).
      In addition to just enabling it our solution provides:
      + do not autostart if subnet is already taken (e.g. in guests).
      + iterate some alternative subnets before giving up
    - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is
      the group based access to libvirt functions as it was used in Ubuntu
      for quite long.
      + d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests
        due to the group access change.
      + d/libvirt-daemon-system.postinst: add users in sudo to the libvirt
        group.
    - ubuntu/parallel-shutdown.patch: set parallel shutdown by default.
    - Update Vcs-Git and Vcs-Browser fields to point to launchpad
    - Update README.Debian with Ubuntu changes
    - Enable some additional features on ppc64el and s390x (for arch parity)
      + systemtap, zfs, numa and numad on s390x.
      + systemtap on ppc64el.
    - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx
    - Further upstreamed apparmor Delta, especially any new one
      Our former delta is split into logical pieces and is either Ubuntu only
      or is part of a continuous upstreaming effort.
      Listing related remaining changes in debian/patches/ubuntu-aa/:
    - fix autopkgtests
      + d/t/control, d/t/smoke-qemu-session: fixup smoke-qemu-session by making
        vmlinuz available and accessible (Debian bug 848314)
      + d/t/control: fix smoke-qemu-session by ensuring the service will run
        installing libvirt-daemon-system
      + d/t/smoke-lxc: fix smoke-lxc by ignoring potential issues on destroy as
        long as the following undefine succeeds
      + d/t/smoke-lxc: use systemd instead of sysV to restart the service
    - dnsmasq related enhancements
      + run dnsmasq as libvirt-dnsmasq (LP: 1743718)
      + d/libvirt-daemon-system.postinst: add libvirt-dnsmasq user and group
      + d/libvirt-daemon-system.postrm: remove libvirt-dnsmasq user and group
        on purge
      + d/p/ubuntu/dnsmasq-as-priv-user: write dnsmasq config with user
        libvirt-dnsmasq and adapt t...