Used the Kernel from Proposed: apt-cache policy linux-image-4.15.0-42-generic linux-image-4.15.0-42-generic: Installed: 4.15.0-42.45 Candidate: 4.15.0-42.45 Libvirt/Qemu from PPA [1] Having one device assigned to my LPAR atm: $ ll /sys/bus/ap/devices/ total 0 drwxr-xr-x 2 root root 0 Nov 23 03:29 ./ drwxr-xr-x 4 root root 0 Nov 23 03:29 ../ lrwxrwxrwx 1 root root 0 Nov 23 03:29 00.0016 -> ../../../devices/ap/card00/00.0016/ lrwxrwxrwx 1 root root 0 Nov 23 03:29 card00 -> ../../../devices/ap/card00/ # mask out the adapters/queues of your choice that you want to virtualize # In my case i have card 0 queue 16 (hex 16 dec 22 to match HMC config) $ lszcrypt CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT ------------------------------------------------- 00 CEX5C CCA-Coproc online 5 00.0016 CEX5C CCA-Coproc online 5 # so lets assign that to vfio-ap instead of zcrypt use # Adapter $ cat /sys/bus/ap/apmask 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff $ echo -0x0 | sudo tee /sys/bus/ap/apmask $ cat /sys/bus/ap/apmask 0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff # Domain $ cat /sys/bus/ap/aqmask 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff $ echo -0x16 | sudo tee /sys/bus/ap/aqmask $ cat /sys/bus/ap/aqmask 0xfffffdffffffffffffffffffffffffffffffffffffffffffffffffffffffffff $ sudo modprobe vfio_ap $ dmesg | tail ... [272006.492864] vfio_ap matrix: MDEV: Registered # create a new MDEV $ uuid=$(uuidgen) $ echo ${uuid} | sudo tee /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create in Dmesg: [272197.818811] iommu: Adding device 24f952b3-03d1-4df2-9967-0d5f7d63d5f2 to group 0 [272197.818815] vfio_mdev 24f952b3-03d1-4df2-9967-0d5f7d63d5f2: MDEV: group_id = 0 # Assign adapter 0 to vfio-ap echo +0x0 > /sys/devices/vfio_ap/matrix/${uuid}/assign_adapter # Assign domain 16 (22) to vfio-ap $ echo +0x16 | sudo tee /sys/devices/vfio_ap/matrix/${uuid}/assign_domain $ echo +0x16 | sudo tee /sys/devices/vfio_ap/matrix/${uuid}/assign_control_domain Check the matrix you have set up $ cat /sys/devices/vfio_ap/matrix/${uuid}/matrix 00.0016 Get something to bootable to then start it with the MDEV assigned: $ uvt-kvm create --memory=1024 --password=ubuntu bionic-vfio-ap arch=s390x label=daily release=bionic # wait until initialized and shut it down $ virsh shutdown bionic-vfio-ap # Modify to also use the MDEV $ virsh edit bionic-vfio-ap # add a snippet matching your UUID like:
When restarting the guest this correctly adds the commandline argument: -device vfio-ap,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/24f952b3-03d1-4df2-9967-0d5f7d63d5f2 We also see virt-aa helper generting vfio rules $ grep '/dev/vfio' /etc/apparmor.d/libvirt/$(virsh dominfo bionic-vfio-ap | awk '/^Security label/ {print $3}').files "/dev/vfio/vfio" rw, "/dev/vfio/[0-9]*" rw, And most importantly in the guest the adapter is present: $ lszcrypt CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT ------------------------------------------------- 00 CEX5C CCA-Coproc online 1 00.0016 CEX5C CCA-Coproc online 1 Thanks Halil that I was watching [2] in Edinburgh :-) And thanks cborntra for the WIP script to set those up. [1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3520 [2]: https://events.linuxfoundation.org/wp-content/uploads/2017/12/vfio-ap-The-Perils-of-the-Weird-Halil-Pasic-IBM.pdf