Used the Kernel from Proposed:
apt-cache policy linux-image-4.15.0-42-generic
linux-image-4.15.0-42-generic:
Installed: 4.15.0-42.45
Candidate: 4.15.0-42.45
Libvirt/Qemu from PPA [1]
Having one device assigned to my LPAR atm:
$ ll /sys/bus/ap/devices/
total 0
drwxr-xr-x 2 root root 0 Nov 23 03:29 ./
drwxr-xr-x 4 root root 0 Nov 23 03:29 ../
lrwxrwxrwx 1 root root 0 Nov 23 03:29 00.0016 -> ../../../devices/ap/card00/00.0016/
lrwxrwxrwx 1 root root 0 Nov 23 03:29 card00 -> ../../../devices/ap/card00/
# mask out the adapters/queues of your choice that you want to virtualize
# In my case i have card 0 queue 16 (hex 16 dec 22 to match HMC config)
$ lszcrypt
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT
-------------------------------------------------
00 CEX5C CCA-Coproc online 5
00.0016 CEX5C CCA-Coproc online 5
# so lets assign that to vfio-ap instead of zcrypt use
# create a new MDEV
$ uuid=$(uuidgen)
$ echo ${uuid} | sudo tee /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create
in Dmesg:
[272197.818811] iommu: Adding device 24f952b3-03d1-4df2-9967-0d5f7d63d5f2 to group 0
[272197.818815] vfio_mdev 24f952b3-03d1-4df2-9967-0d5f7d63d5f2: MDEV: group_id = 0
# Assign adapter 0 to vfio-ap
echo +0x0 > /sys/devices/vfio_ap/matrix/${uuid}/assign_adapter
# Assign domain 16 (22) to vfio-ap
$ echo +0x16 | sudo tee /sys/devices/vfio_ap/matrix/${uuid}/assign_domain
$ echo +0x16 | sudo tee /sys/devices/vfio_ap/matrix/${uuid}/assign_control_domain
Check the matrix you have set up
$ cat /sys/devices/vfio_ap/matrix/${uuid}/matrix
00.0016
Get something to bootable to then start it with the MDEV assigned:
$ uvt-kvm create --memory=1024 --password=ubuntu bionic-vfio-ap arch=s390x label=daily release=bionic
# wait until initialized and shut it down
$ virsh shutdown bionic-vfio-ap
# Modify to also use the MDEV
$ virsh edit bionic-vfio-ap
# add a snippet matching your UUID like:
<hostdev mode='subsystem' type='mdev' model='vfio-ap'>
<source>
<address uuid='24f952b3-03d1-4df2-9967-0d5f7d63d5f2'/>
</source>
</hostdev>
When restarting the guest this correctly adds the commandline argument:
-device vfio-ap,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/24f952b3-03d1-4df2-9967-0d5f7d63d5f2
We also see virt-aa helper generting vfio rules
$ grep '/dev/vfio' /etc/apparmor.d/libvirt/$(virsh dominfo bionic-vfio-ap | awk '/^Security label/ {print $3}').files
"/dev/vfio/vfio" rw,
"/dev/vfio/[0-9]*" rw,
And most importantly in the guest the adapter is present:
$ lszcrypt
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT
-------------------------------------------------
00 CEX5C CCA-Coproc online 1
00.0016 CEX5C CCA-Coproc online 1
Thanks Halil that I was watching [2] in Edinburgh :-)
And thanks cborntra for the WIP script to set those up.
Used the Kernel from Proposed: 4.15.0- 42-generic 4.15.0- 42-generic:
apt-cache policy linux-image-
linux-image-
Installed: 4.15.0-42.45
Candidate: 4.15.0-42.45
Libvirt/Qemu from PPA [1]
Having one device assigned to my LPAR atm: ap/devices/ ./devices/ ap/card00/ 00.0016/ ./devices/ ap/card00/
$ ll /sys/bus/
total 0
drwxr-xr-x 2 root root 0 Nov 23 03:29 ./
drwxr-xr-x 4 root root 0 Nov 23 03:29 ../
lrwxrwxrwx 1 root root 0 Nov 23 03:29 00.0016 -> ../../.
lrwxrwxrwx 1 root root 0 Nov 23 03:29 card00 -> ../../.
# mask out the adapters/queues of your choice that you want to virtualize ------- ------- ------- ------- ------- -------
# In my case i have card 0 queue 16 (hex 16 dec 22 to match HMC config)
$ lszcrypt
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT
-------
00 CEX5C CCA-Coproc online 5
00.0016 CEX5C CCA-Coproc online 5
# so lets assign that to vfio-ap instead of zcrypt use
# Adapter fffffffffffffff fffffffffffffff fffffffffffffff ffffff fffffffffffffff fffffffffffffff fffffffffffffff ffffff fffffffffffffff fffffffffffffff fffffffffffffff ffffff fffffffffffffff fffffffffffffff fffffffffffffff ffffff
$ cat /sys/bus/ap/apmask
0xfffffffffffff
$ echo -0x0 | sudo tee /sys/bus/ap/apmask
$ cat /sys/bus/ap/apmask
0x7ffffffffffff
# Domain
$ cat /sys/bus/ap/aqmask
0xfffffffffffff
$ echo -0x16 | sudo tee /sys/bus/ap/aqmask
$ cat /sys/bus/ap/aqmask
0xfffffdfffffff
$ sudo modprobe vfio_ap
$ dmesg | tail
...
[272006.492864] vfio_ap matrix: MDEV: Registered
# create a new MDEV vfio_ap/ matrix/ mdev_supported_ types/vfio_ ap-passthrough/ create
$ uuid=$(uuidgen)
$ echo ${uuid} | sudo tee /sys/devices/
in Dmesg: 03d1-4df2- 9967-0d5f7d63d5 f2 to group 0 03d1-4df2- 9967-0d5f7d63d5 f2: MDEV: group_id = 0
[272197.818811] iommu: Adding device 24f952b3-
[272197.818815] vfio_mdev 24f952b3-
# Assign adapter 0 to vfio-ap vfio_ap/ matrix/ ${uuid} /assign_ adapter vfio_ap/ matrix/ ${uuid} /assign_ domain vfio_ap/ matrix/ ${uuid} /assign_ control_ domain
echo +0x0 > /sys/devices/
# Assign domain 16 (22) to vfio-ap
$ echo +0x16 | sudo tee /sys/devices/
$ echo +0x16 | sudo tee /sys/devices/
Check the matrix you have set up vfio_ap/ matrix/ ${uuid} /matrix
$ cat /sys/devices/
00.0016
Get something to bootable to then start it with the MDEV assigned:
$ uvt-kvm create --memory=1024 --password=ubuntu bionic-vfio-ap arch=s390x label=daily release=bionic
# wait until initialized and shut it down
$ virsh shutdown bionic-vfio-ap
# Modify to also use the MDEV 03d1-4df2- 9967-0d5f7d63d5 f2'/>
$ virsh edit bionic-vfio-ap
# add a snippet matching your UUID like:
<hostdev mode='subsystem' type='mdev' model='vfio-ap'>
<source>
<address uuid='24f952b3-
</source>
</hostdev>
When restarting the guest this correctly adds the commandline argument: id=hostdev0, sysfsdev= /sys/bus/ mdev/devices/ 24f952b3- 03d1-4df2- 9967-0d5f7d63d5 f2 d/libvirt/ $(virsh dominfo bionic-vfio-ap | awk '/^Security label/ {print $3}').files vfio/[0- 9]*" rw,
-device vfio-ap,
We also see virt-aa helper generting vfio rules
$ grep '/dev/vfio' /etc/apparmor.
"/dev/vfio/vfio" rw,
"/dev/
And most importantly in the guest the adapter is present: ------- ------- ------- ------- ------- -------
$ lszcrypt
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT
-------
00 CEX5C CCA-Coproc online 1
00.0016 CEX5C CCA-Coproc online 1
Thanks Halil that I was watching [2] in Edinburgh :-)
And thanks cborntra for the WIP script to set those up.
[1]: https:/ /launchpad. net/~ci- train-ppa- service/ +archive/ ubuntu/ 3520 /events. linuxfoundation .org/wp- content/ uploads/ 2017/12/ vfio-ap- The-Perils- of-the- Weird-Halil- Pasic-IBM. pdf
[2]: https:/