apparmor driver blocks access to some hostdev and pcidev devices

[Claas Hilbrecht]

Description: Ubuntu lucid (development branch)
Release: 10.04

If I try to use a "Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)" network device for a VM I can select the device via virt-manager. But if I try to start such a VM the VM won't start because the apparmor profile for the VM is not updated to allow access to the PCI device (the same is true for USB devices). This gives the following messages:

Mar 20 15:33:43 horst kernel: [ 1178.108436] type=1503 audit(1269095623.007:27): operation="open" pid=4902 parent=1 profile="libvirt-fde0ac99-0184-edd3-ab69-d53d6a940c36" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/sys/devices/pci0000:00/0000:00:01.0/0000:0a:10.0/config"
Mar 20 15:33:43 horst libvirtd: 15:33:43.183: error : qemudWaitForMonitor:1536 : internal error unable to start guest: char device redirected to /dev/pts/2#012device: 0a:10.0: driver="pci-assign" host="0a:10.0"#012get_real_device: /sys/bus/pci/devices/0000:0a:10.0/config: Permission denied#012pci-assign: Error: Couldn't get real device (0a:10.0)!#012Error initializing device pci-assign#012

virt-manager should create the neccessary entries in the apparmor profile for Physical Host Devices automaticly.

virt-manager 0.8.2-2ubuntu6
ibvirt-bin 0.7.5-5ubuntu15
apparmor 2.5-0ubuntu1

[Jamie Strandboge (jdstrand)]

Thank you for using Ubuntu and reporting a bug. This is a known issue and a limitation of the AppArmor driver. For now, you need to adjust /etc/apparmor.d/abstractions/libvirt-qemu to allow access to host hardware. For usb devices (hostdev), adjust this:
  # WARNING: uncommenting these gives the guest direct access to host hardware.
  # This is required for USB pass through but is a security risk. You have been
  # warned.
  #/sys/bus/usb/devices/ r,
  #/sys/devices/*/*/usb[0-9]*/** r,
  #/dev/bus/usb/*/[0-9]* rw,

To be:
  # WARNING: uncommenting these gives the guest direct access to host hardware.
  # This is required for USB pass through but is a security risk. You have been
  # warned.
  /sys/bus/usb/devices/ r,
  /sys/devices/*/*/usb[0-9]*/** r,
  /dev/bus/usb/*/[0-9]* rw,

You will have to add similar entries for PCI devices (pcidev).

[Jamie Strandboge (jdstrand)]

I'm going to unmilestone this since it mostly depends on bug #553737. If that bug is fixed, I can add my upstream work to it, otherwise this may have to wait until lucid+1.

[Jamie Strandboge (jdstrand)]

Changes are too big for Lucid. This will be fixed in Maverick and upstream libvirt 0.7.8.

[Jamie Strandboge (jdstrand)]

Actually, I thought about this some more and I can get this to work for Lucid.

[Jamie Strandboge (jdstrand)]

Uploaded 0.7.5-5ubuntu19 which fixes this. Just needs to be approved.

[Steve Langasek (vorlon)]

libvirt 0.7.5-5ubuntu21 is accepted into lucid, but some of the intermediate versions were bounced out of the queue for simplicity's sake - so this didn't get autoclosed. Changelog entry:

libvirt (0.7.5-5ubuntu19) lucid; urgency=low

  * fix for hostdev devices (LP: #545795). This can be dropped in 0.7.8
    - debian/patches/9021-apparmor-fix-hostdev.patch: adjust virt-aa-helper to
      handle pci devices. Update valid_path() to have an override array to
      check against, and add "/sys/devices/pci" to it. Then rename
      file_iterate_cb() to file_iterate_hostdev_cb() and create
      file_iterate_pci_cb() based on it. Update tests suite for this and SDL
    - debian/apparmor/libvirt-qemu: adjust for the above
    - debian/apparmor/usr.lib.libvirt.virt-aa-helper: allow access to
      /sys/devices

 -- Jamie Strandboge <email address hidden> Mon, 05 Apr 2010 19:50:15 -0500

[Francesco Pretto (ceztko)]

I'm using libvirt-bin 0.7.5-5ubuntu25 and the bug is still there. Is the fix lost somewhere?

My /var/log/libvirt/qemu/storage.log have these lines:

usb_create: no bus specified, using "usb.0" for "usb-host"
husb: open device 6.2
/dev/bus/usb/006/002: Permission denied
husb: open device 6.2
/dev/bus/usb/006/002: Permission denied
husb: open device 6.2
/dev/bus/usb/006/002: Permission denied
husb: open device 6.2
/dev/bus/usb/006/002: Permission denied
husb: open device 6.2
/dev/bus/usb/006/002: Permission denied
...

[Francesco Pretto (ceztko)]

Just to confirm something is still not working for me:

$ sudo apt-cache showpkg libvirt-bin
Package: libvirt-bin
Versions:
0.7.5-5ubuntu25

Adding the generic (and unsafe) line:
  /dev/bus/usb/*/[0-9]* rw,

to /etc/apparmor.d/abstractions/libvirt-qemu and reloading profiles works for me so it seems the mechanism to dynamically add host devices to the apparmor profile has been bounced as well or not working anymore.

[Claas Hilbrecht (claas-hilbrecht)]

FYI: I didn't try the latest fix since apparmor causes all kinds of trouble in the last days. Since I need a stable KVM I didn't test the fix so far. Maybe I have some spare time tomorrow.

[Jamie Strandboge (jdstrand)]

Francesco, how did you add this item? Can you attach the XML for the VM in question by using 'virsh dumpxml <vmname>'.

[Francesco Pretto (ceztko)]

I used virt-manager:

<domain type='kvm' id='5'>
  <name>storage</name>
  <uuid>0175b337-5faf-42ba-d6a7-bb60ec8da4ad</uuid>
  <memory>1572864</memory>
  <currentMemory>1572864</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.12'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu'/>
      <source file='/var/vm/storage.img'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
    </disk>
    <interface type='network'>
      <mac address='52:54:00:4a:80:1e'/>
      <source network='server'/>
      <target dev='vnet4'/>
      <model type='virtio'/>
    </interface>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
    </console>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='yes' keymap='it'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x03f0'/>
        <product id='0x0317'/>
      </source>
    </hostdev>
  </devices>
  <seclabel type='dynamic' model='apparmor'>
    <label>libvirt-0175b337-5faf-42ba-d6a7-bb60ec8da4ad</label>
    <imagelabel>libvirt-0175b337-5faf-42ba-d6a7-bb60ec8da4ad</imagelabel>
  </seclabel>
</domain>

[Jamie Strandboge (jdstrand)]

Class,

Can you add the following line to /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper:
  /sys/bus/usb/devices/ r,
  /sys/bus/usb/devices/** r,

Then perform:
$ sudo apparmor_parser -r /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

and try to start the VM again?

[Jamie Strandboge (jdstrand)]

Err... sorry, that last message was for Francesco, not Claas.

[Francesco Pretto (ceztko)]

Yes, that worked! :)

[Jamie Strandboge (jdstrand)]

Francesco. Excellent, thanks for your feedback. This will be added in 0.7.5-5ubuntu26. I am going to mark this back to Fix Released since it works some of the time as is. Please file a new bug if 0.7.5-5ubuntu26 doesn't resolve the issue for you.

[Andreas Ntaflos (daff)]

I'm sorry to post to this bug that has a status of "Fix released" but I am not sure it is really fixed. I have a situation similar too the original poster's concerning a USB card reader that won't make it past AppArmor it seems. Using libvirt-bin 0.7.5-5ubuntu27.

Situation: one of our servers was upgraded from Ubuntu 9.10 to 10.04 today. The server runs a few Ubuntu 9.10 VMs, nothing fancy or out of the ordinary. These VMs were defined and installed a few weeks ago, prior to the release of and update to Ubuntu 10.04 (if that matters at all).

We've had problems with AppArmor and Libvirt/KVM before so we disabled AppArmor and pass-through of the USB card readers worked fine this way. This situation was not ideal from a security point-of-view but since the host and guests are strictly for internal test and development purposes we went with it. Now I see that a lot has happened with regards to AppArmor, USB and PCI pass-through and Libvirt, so tried again enabling AppArmor. Alas, when starting a VM dmesg and /var/log/kern.log show these entries, repeating every second it seems:

May 3 19:44:18 TESTHOST kernel: [ 2407.509182] type=1503 audit(1272908658.618:785): operation="open" pid=1532 parent=1 profile="libvirt-959806d1-327a-cd14-6b3f-ddeee8a19d0e" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/sys/devices/pci0000:00/0000:00:1e.0/0000:01:04.4/usb6/devnum"

The guest of course does not get to see anything of the USB device in question. Please find the XML definition of the guest in question here: https://daff.pseudoterminal.org/files/vm-usb.txt

After disabling AppArmor (/etc/init.d/apparmor stop) the USB device is again visible in the guest.

Why would this happen? The file /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper explicitly states that access to /sys/devices/** should be allowed. Am I missing anything? I can experiment and run tests on this server for the next week or so, so please tell me if I can help debugging anything.

[Andreas Ntaflos (daff)]

Interestingly, or perhaps not, merely running /etc/init.d/apparmor stop isn't enough. I stop AppArmor, restart Libvirt and then start my VMs. However upon starting a VM an AppArmor profile still gets loaded and thus AppArmor denies access to the USB device I want to pass through. I have to run /etc/init.d/apparmor stop again after the VM has been started. Then access to the USB device is allowed.

Looks weird to me but I haven't yet fully understood how and when AppArmor profiles are loaded. But I don't understand why it would deny access to a directory structure that is explicitly permitted in the profile:

May 4 15:56:27 TESTHOST kernel: [75138.174346] type=1503 audit(1272981387.661:879): operation="open" pid=8053 parent=1 profile="libvirt-959806d1-327a-cd14-6b3f-ddeee8a19d0e" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/sys/devices/pci0000:00/0000:00:1e.0/0000:01:04.4/usb6/devnum"

Unfortunately this is quite the blocker for me.

[Andreas Ntaflos (daff)]

Looks like I found it. The VM in my case is trying to access

/sys/devices/pci0000:00/0000:00:1e.0/0000:01:04.4/usb6/devnum

but the abstractions/libvirt-qemu profile only allows

  /sys/bus/usb/devices/ r,
  /sys/devices/*/*/usb[0-9]*/** r,

when it should (also) allow

  /sys/devices/*/*/*/usb[0-9]*/** r,

With this line added the guest boots fine and immediately gets access to the USB device.

I have attached a patch for this one-line fix, hope it helps.

[Jamie Strandboge (jdstrand)]

Andreas, can you adjust this to be:
  /sys/bus/usb/devices/ r,
  /sys/bus/usb/devices/** r,
  /sys/devices/**/usb[0-9]*/** r,

and report back if it fixes it for you?

[Andreas Ntaflos (daff)]

Jamie, yes this fixes it. thank you!

I notice however some redundancies between abstractions/libvirt-qemu and usr.lib.libvirt.virt-aa-helper? At least the line "/sys/bus/usb/devices/ r," appears in both, don't know if that matters any, though. So that's good :)

But now I have discovered something else. When booting a VM that has a USB device included in its XML definition (like here: https://daff.pseudoterminal.org/files/vm-usb.txt) now thanks to this fix works fine. *However* trying to attach a USB device while the VM is running (using virt-manager in my case) results in these messages in /var/log/libvirt/qemu/vm.log:

usb_create: no bus specified, using "usb.0" for "usb-host"
husb: open device 5.2
/dev/bus/usb/005/002: Permission denied
husb: open device 5.2
/dev/bus/usb/005/002: Permission denied
husb: open device 5.2

And in /var/log/kern.log:

May 4 17:01:19 TESTHOST kernel: [79029.932635] type=1503 audit(1272985279.341:1009): operation="open" pid=23782 parent=1 profile="libvirt-959806d1-327a-cd14-6b3f-ddeee8a19d0e" requested_mask="rw::" denied_mask="rw::" fsuid=0 ouid=0 name="/dev/bus/usb/005/002"

So it seems that access to "/dev/bus/usb/**" is needed as well?

[Andreas Ntaflos (daff)]

Oh and it seems that disconnecting/detaching an USB device from the running VM doesn't work at all? virt-manager complains:

Device could not be removed from the running machine.
This change will take effect after the next VM reboot

But this has probably nothing to do with AppArmor and may just be a shortcoming of Libvirt? Just pointing it out here since it seems to fit.

[Jamie Strandboge (jdstrand)]

Andreas, thanks for reporting back. abstractions/libvirt-qemu and usr.lib.libvirt.virt-aa-helper are used by different applications. Eg, virt-aa-helper is confined by the usr.lib.libvirt.virt-aa-helper profile and VMs include the libvirt-qemu abstraction. Please file a different bug regarding hot attach of a USB device.

[James Neave (roboj1m)]

Hi,

I think I may also be having this problem, I'm trying to pass through a PCI WinTV NOVA T 500.
It's actually a pair of USB DVB-T tuners on a PCI card so it needed the 'three stars' in the /sys path in the apparmour profile.

I appended the improved solution:

  /sys/bus/usb/devices/ r,
  /sys/bus/usb/devices/** r,
  /sys/devices/**/usb[0-9]*/** r,

But in the qemu log file I get:

husb: open device 10.2
husb: config #1 need -1
husb: 1 interfaces claimed for configuration 1
husb: grabbed usb device 10.2
usb_linux_update_endp_table: Cannot send after transport endpoint shutdown

Is this the same problem or have I found a new one? Googeling for that last line finds nothing!

I also had to add this to get past the permission denied error:

 /dev/bus/usb/** rw,

Would the output of 'find /sys/devices' be of any use?

Regards,

James.

[James Neave (roboj1m)]

Rebooting the PC fixed that problem so I guess something hadn't been reloaded.
I've also added /dev/** rwk, to the apparmour profile. Taking it away doesn't stop it from working again, although I've not tried rebooting the host yet.

It still doesn't quite work however, as now (on the guest) I see this in lsusb:

james@myth:~$ lsusb
Bus 001 Device 002: ID 2040:9950 Hauppauge
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

but in dmesg I see:

[ 5.806567] dib0700: loaded with support for 13 different device-types
[ 5.816285] dvb-usb: found a 'Hauppauge Nova-T 500 Dual DVB-T' in warm state.
[ 5.816612] dvb-usb: This USB2.0 device cannot be run on a USB1.1 port. (it lacks a hardware PID filter)
[ 5.816682] dvb-usb: Hauppauge Nova-T 500 Dual DVB-T error while loading driver (-19)
[ 5.816787] usbcore: registered new interface driver dvb_usb_dib0700

Different problem?
Why is it on a USB 1 port? Is the virtual USB hub 1.1? Can I change that?

Regards,

James.

[James Neave (roboj1m)]

Hmm, it seems thats actually because usb2 is not supported in KVM?

I tried passing through the PCI card itself, but I'm back to permission denied :(

device: 03:06.0: driver="pci-assign" host="03:06.0"
device: 03:06.1: driver="pci-assign" host="03:06.1"
device: 03:06.2: driver="pci-assign" host="03:06.2"
get_real_device: /sys/bus/pci/devices/0000:03:06.0/config: Permission denied
pci-assign: Error: Couldn't get real device (03:06.0)!
Error initializing device pci-assign

apparmour files contains:

  /sys/bus/usb/devices/ r,
  /sys/bus/usb/devices/** r,
  /sys/devices/**/usb[0-9]*/** r,
  /sys/bus/pci/devices/ r,
  /sys/bus/pci/devices/** r,
  /sys/devices/pci/** r,

  /dev/shm/ r,
  /dev/shm/pulse-shm* r,
  /dev/shm/pulse-shm* rwk,
  /dev/snd/* rw,
  /dev/bus/usb/** rw,
  /dev/** rwk,

Is there any way to get access to an apparmor trace file? See what it's accessing?

Thanks,

James.

[James Neave (roboj1m)]

Found the apparmour errors in syslog, related to virt-aa helper.
I added /sys/devices/** r, to usr.lib.....virt-aa-helper and got a bit further.
Still getting this:

May 10 23:14:25 hal kernel: [ 179.037233] type=1503 audit(1273529665.107:22): operation="open" pid=1601 parent=1 profile="libvirt-28b82cfd-52c0-b743-475e-77dde3933f44" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/sys/devices/pci0000:00/0000:00:14.4/0000:03:06.0/vendor"

in syslog, but now it's some kind of dynamic profile, but I don't know where the template is to edit?

That's it for tonight, I'm going to bed -.- zz

Regards,

James.

[John Ferlito (johnf-inodes)]

util/pci.c in function pciDeviceFileIterate
needs vendor added as a directory match:wq

[John Ferlito (johnf-inodes)]

Looks like it needs device as well. Will attach a patch shortly

[John Ferlito (johnf-inodes)]

Patch to allow PCI pass through to work woth app armor. It's currently missing a couple of files

[Jamie Strandboge (jdstrand)]

John, while this patch seems ok to me, this is really an upstream bug. Can you submit a bug upstream (http://libvirt.org/bugs.html) and link to it from this bug? Once upstream accepts it, we can cherrypick it for Maverick, verify it and backport it to Lucid. Thanks!

[Jamie (jamie-redhat-bugs)]

Someone reported a bug in Ubuntu on pcidev devices not being correctly added with the AppArmor driver and a patch was submitted:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/545795/comments/29

This patch is not tested but suggests that more than just the AppArmor driver is affected, so I am forwarding it here. That bug is kinda all over the place and has various things unrelated to this report.

Here is the comment and submitted patch:

"Patch to allow PCI pass through to work woth app armor. It's currently missing a couple of files"

--- libvirt-0.7.5.orig/src/util/pci.c 2010-06-11 01:43:41 +0000
+++ libvirt-0.7.5/src/util/pci.c 2010-06-11 01:45:06 +0000
@@ -1089,6 +1089,8 @@
          */
         if (STREQ(ent->d_name, "config") ||
             STRPREFIX(ent->d_name, "resource") ||
+ STREQ(ent->d_name, "vendor") ||
+ STREQ(ent->d_name, "device") ||
             STREQ(ent->d_name, "rom")) {
             if (virAsprintf(&file, "%s/%s", pcidir, ent->d_name) < 0) {
                 virReportOOMError(conn);

[Jamie Strandboge (jdstrand)]

I went ahead and filed an upstream bug for the attached patch and linked it to this one.

[Daniel (daniel-redhat-bugs)]

This is rather odd. QEMU appears to use the vendor + device files, but AFAICK, under SElinux QEMU is working fine without this change. The patch certainly looks reasonable though.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 0.8.3-1ubuntu5

---------------
libvirt (0.8.3-1ubuntu5) maverick; urgency=low

  * update to allow pcidev and hostdev to work with AppArmor (LP: #545795)
    - debian/patches/lp-545795.patch: add vendor and device to
      pciDeviceFileIterate(). Patch submitted upstream and they feel it is
      reasonable, but not committed yet. This should fix pcidev.
    - debian/apparmor/usr.lib.libvirt.virt-aa-helper: add read access to
      /sys/bus/usb/devices/**
    - debian/apparmor/libvirt-qemu: adjust read access to be
      /sys/devices/**/usb[0-9]*/** instead of /sys/devices/*/*/usb[0-9]*/**.
      Patched based on work by Andreas Ntaflos.
 -- Jamie Strandboge <email address hidden> Fri, 20 Aug 2010 09:21:15 -0500

[Jamie (jamie-redhat-bugs)]

I'm just following up on this since we are pulling in 0.8.5 into Ubuntu and the above isn't committed yet. I can say that at this point Ubuntu is carrying it in its 10.10 release (libvirt 0.8.3) and there are no reported regressions.

[nutznboltz (nutznboltz-deactivatedaccount)]

For Lucid:
https://launchpad.net/~nutznboltz/+archive/kvm-libvirt-lts

[Sergey Svishchev (svs)]

Will this fix go into Lucid any time soon?

[nutznboltz (nutznboltz-deactivatedaccount)]

@Sergey will you be submitting a debdiff any time soon?

[Sergey Svishchev (svs)]

At the moment, I'm using packages from your PPA, so no.

[Alexander List (alexlist)]

I tried to add a host NIC to one of my VMs using virt-manager. First, I had to enable IOMMU...

https://bugs.launchpad.net/fedora/+source/libvirt/+bug/741706

Now I have a different problem:

Aug 14 11:44:49 thinkpad kernel: [ 63.432692] kvm_iommu_map_guest: No interrupt remapping support, disallowing device assignment. Re-enble with "allow_unsafe_assigned_interrupts=1" module option.

The reason to pass a host NIC to a VM is testing new drivers... guess the use case is not that exotic...

[Alexander List (alexlist)]

Found that this is actually another bug...

https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/639712

[Rolf Leggewie (r0lf)]

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".

[Ján (jn-redhat-bugs)]

Identical patch was already pushed upstream:
commit 28d599c5130ee102d5174c01d59eeb14a75a3747
Author: Cédric Bosdonnat <email address hidden>
AuthorDate: 2015-04-23 09:32:16 +0200
Commit: Cédric Bosdonnat <email address hidden>
CommitDate: 2015-04-24 10:47:41 +0200

    Allow access to vendor and device file for PCI device passthrough

    For some devices, the $PCIDIR/vendor and $PCIDIR/device need to be
    read. Iterate over them to get them as well in the the generated
    apparmor profile.

git describe: v1.2.14-282-g28d599c contains: v1.2.15-rc1~57