Comment 5 for bug 1678322

# on x86 for vfio set kernel parm have iommu (if not default for you)
$ echo "intel_iommu=on" | sudo tee /etc/default/grub.d/99-force-iommu.cfg
$ sudo update-grub
# reboot

# prep one of my net devices for VF with vfio-pci
$ sudo rmmod ixgbe
$ sudo modprobe ixgbe max_vfs=8
$ lspci -n -s 0000:04:10.0
04:10.0 0200: 8086:1515 (rev 01)
$ sudo modprobe vfio-pci
# assign id to vfio-pci
$ echo 8086 1515 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
# unbind old driver
$ echo "0000:04:10.0" | sudo tee /sys/bus/pci/devices/0000\:04\:10.0/driver/unbind
# usually auto-bound now but be sure
$ echo "0000:04:10.0" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind
0000:04:10.0

#1 get a basic guest
$ sudo apt install uvtool-libvirt
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=zesty
# if you have no keys around also run "ssh-keygen"
$ uvt-kvm create --password=ubuntu zesty-vfio release=zesty arch=amd64 label=daily

# Prep and attach the device to the guest
$ cat vf-04.10.0-pci.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/>
  </source>
</hostdev>
$ virsh attach-device z-testguest vf-04.10.0-pci.xml

With that in place I could confirm your report:
I see
1. the setrlimit deny (all worarounds mentioned before work, but they should not have to)
2. I also see he deny to /dev/vfio/vfio (I'll check the old guides and if we want to make them work more out of the box)
3. I also see an apparmor profile reload which I need to check on its content