[UBUNTU 20.04] Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu

[bugproxy]

---Problem Description (by <email address hidden>) ---
Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu 20.04/22.04

Contact Information = Eric Farman <email address hidden>

---uname output---
Linux m34mkvmt5 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 21:05:01 UTC 2022 s390x s390x s390x GNU/Linux

---Additional Hardware Info---
ECKD DASD, connected as a mediated device for KVM device passthrough

Machine Type = IBM z14 (3906) LPAR

---Debugger---
A debugger is not configured

---Steps to Reproduce---
Attempting to spawn a guest with a vfio-ccw hostdev device fails with an AppArmor policy restriction on both 20.04 and 22.04, for files that QEMU attempts to open for the device. The failure also occurs when trying to hotplug such a device, which I'll use in these steps to keep the XML simple:

eric@kvmhost:~# chzdev -ea ca8b
eric@kvmhost:~# echo 0.0.ca8b > /sys/bus/ccw/drivers/dasd-eckd/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/io_subchannel/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/vfio_ccw/bind
eric@kvmhost:~# echo 11f2d2bc-4083-431d-a023-eff72715c4f0 > /sys/bus/css/devices/0.0.0b16/mdev_supported_types/vfio_ccw-io/create
eric@kvmhost:~# cat hostdev.xml
    <hostdev mode='subsystem' type='mdev' model='vfio-ccw'>
      <source>
        <address uuid='11f2d2bc-4083-431d-a023-eff72715c4f0'/>
      </source>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0xca8b'/>
    </hostdev>
eric@kvmhost:~# virsh attach-device guest hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add': s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@kvmhost:~# dmesg | grep DENIED
[ 5949.232089] audit: type=1400 audit(1670350246.709:22): apparmor="DENIED" operation="open" profile="libvirt-0e995f6d-f85e-4ffe-b612-e07bfd62116a" name="/sys/devices/css0/0.0.0b16/pimpampom" pid=1497 comm="qemu-system-s39" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0

While the failure occurs with the pimpampom file for the subchannel, there are two others that QEMU would attempt to open after this:

eric:qemu$ git grep -B 2 -pn fopen hw/s390x/
hw/s390x/css.c=2577=static int css_sch_get_chpids(SubchDev *sch, CssDevId *dev_id)
--
hw/s390x/css.c-2585- fid_path = g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/chpids",
hw/s390x/css.c-2586- dev_id->cssid, dev_id->ssid, dev_id->devid);
hw/s390x/css.c:2587: fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2612=static int css_sch_get_path_masks(SubchDev *sch, CssDevId *dev_id)
--
hw/s390x/css.c-2619- fid_path = g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/pimpampom",
hw/s390x/css.c-2620- dev_id->cssid, dev_id->ssid, dev_id->devid);
hw/s390x/css.c:2621: fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2643=static int css_sch_get_chpid_type(uint8_t chpid, uint32_t *type,
--
hw/s390x/css.c-2649- fid_path = g_strdup_printf("/sys/devices/css%x/chp0.%02x/type",
hw/s390x/css.c-2650- dev_id->cssid, chpid);
hw/s390x/css.c:2651: fd = fopen(fid_path, "r");

The first two directories are links to the third, so I made the following entry in /etc/apparmor.d/local/abstractions/libvirt-qemu which Works For Me:

eric@kvmhost:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu
/sys/devices/css0/** r,

This is of course a very broad brush, so perhaps there's a better deterministic way to the files in question for the subchannel(s) that are requested. (I apologize if that deterministic logic is tied up in the "hostdev networks" bug I see here: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398)

For what it's worth, those files are not ones that remain open once the device is connected to the guest:

eric@kvmhost:~# cat /etc/apparmor.d/libvirt/libvirt-43b44ca9-d1c2-46f7-a686-2329a5a1d425.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
  "/var/log/libvirt/**/guest.log" w,
  "/var/lib/libvirt/qemu/domain-guest/monitor.sock" rw,
  "/var/lib/libvirt/qemu/domain-3-guest/*" rw,
  "/run/libvirt/**/guest.pid" rwk,
  "/run/libvirt/**/*.tunnelmigrate.dest.guest" rw,
  "/dev/dasdb" rwk,
  "/dev/pts/2" rw,
  "/dev/vhost-net" rw,
  "/dev/vfio/2" rwk,

(The passed through DASD device is /dev/vfio/2 in the above list, not /dev/dasdb. The latter is the guest rootfs, connected via virtio-blk.)

======================= Comment: <email address hidden> - 2024-03-06 13:30:45 =====================================================================
Verified that this still misbehaves with 20.04.6 and 22.04.4. Both with the manual sysfs changes described in the initial comment, and the more convenient driverctl and mdevctl tooling.

======================= Comment: <email address hidden> - 2024-03-06 13:34:28 =====================================================================
eric@host:~# virsh attach-device guest_3c4c hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add': s390_ccw_realize: Failed to build initial schib: Invalid argument

eric@host:~# dmesg | grep 0165
[ 127.558194] vfio_ccw 0.0.0165: MDEV: Registered
[ 224.657413] audit: type=1400 audit(1709751698.761:42): apparmor="DENIED" operation="open" profile="libvirt-ab758866-8077-4abb-b662-5f8972f93279" name="/sys/devices/css0/0.0.0165/pimpampom" pid=1813 comm="qemu-system-s39" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0
eric@host:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu
eric@host:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
eric@host:~# uname -a
Linux host 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:27:35 UTC 2024 s390x s390x s390x GNU/Linux
eric@host:~# which qemu-system-s390x
/usr/bin/qemu-system-s390x
eric@host:~# qemu-system-s390x --version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.17)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers