libvirt not relabeling devices on USB Passthrough

[Leendert Keus]

After upgrading from Ubuntu 14.04 to Ubuntu 14.10 USB passthrough in QEMU (version is now 2.1.0 - Debian2.1+dfsg-4ubuntu6.1) is not working any more. Already tried to remove and attach the USB devices. I use 1 USB2 HDD + 1 USB3 HDD to a virtual linux machine; 1 USB2 HDD to a virual FNAS machine and a USB 2camera to virtual Win7 machine. All these devices are not working any more.

[Gonglei (arei-gonglei)]

On 2014/11/14 6:12, Leen Keus wrote:

> Public bug reported:
>
> After upgrading from Ubuntu 14.04 to Ubuntu 14.10 USB passthrough in
> QEMU (version is now 2.1.0 - Debian2.1+dfsg-4ubuntu6.1) is not working
> any more. Already tried to remove and attach the USB devices. I use 1
> USB2 HDD + 1 USB3 HDD to a virtual linux machine; 1 USB2 HDD to a
> virual FNAS machine and a USB 2camera to virtual Win7 machine. All these
> devices are not working any more.
>

What's your meaning "are not working any more", any details?

Regards,
-Gonglei

[Leendert Keus (lj-keus)]

Under Ubuntu 14.04 the USB devices where attached to the virtual machines. After upgrading to Ubuntu 14.10 the USB devices are not visible in the virtual machines anymore. Same issue appeard after upgrading from Ubuntu 13.04 to 13.10, but I can not find that bug report anymore.

[Gonglei (arei-gonglei)]

On 2014/11/14 16:47, Leen Keus wrote:

> Under Ubuntu 14.04 the USB devices where attached to the virtual
> machines. After upgrading to Ubuntu 14.10 the USB devices are not
> visible in the virtual machines anymore. Same issue appeard after
> upgrading from Ubuntu 13.04 to 13.10, but I can not find that bug report
> anymore.
>

Please show your command line booting VM.
Or you can get the latest Qemu master code to reproduce your
problem. Maybe it is a resolved issue. :)

Best regards,
-Gonglei

[Leendert Keus (lj-keus)]

Command lines generated by virsh from the libvirt xml files (let me know if you also need the xml configuration files):

qemu-system-x86_64 -enable-kvm -name fsg -S -machine pc-1.2,accel=kvm,usb=off -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 30619811-d285-944a-026a-cdd4a753b77a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fsg.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x5 -drive file=/vm2/fsgorg/disk-sda2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/vm2/fsgorg/disk-sdb.qcow2,if=none,id=drive-virtio-disk1,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 -netdev tap,fd=23,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:76:fd:b6,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:1 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device usb-host,hostbus=1,hostaddr=4,id=hostdev0 -device usb-host,hostbus=4,hostaddr=2,id=hostdev1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on

qemu-system-x86_64 -enable-kvm -name fnas -S -machine pc-i440fx-1.4,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 7c2703e5-b8b8-acd9-df8b-6284382fb58b -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fnas.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=off,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -drive file=/vm2/FreeNAS/fnas.qcow2,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/vm2/FreeNAS/fnas2.qcow2,if=none,id=drive-ide0-0-1,format=qcow2 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:d8:31:b5,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:5 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device usb-host,hostbus=2,hostaddr=3,id=hostdev0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg tim...

[Iggy (iggy-theiggy)]

At first glance I see "-machine pc-1.2,accel=kvm,usb=off" and then "-device nec-usb-xhci,id=usb,bus=pci.0,addr=0x5"... I'm not sure if those should be used together.

Can you look in your libvirt logs from before the upgrade to see if it was using the usb=off option before?

@kraxel-redhat do you know if that's a legitimate set of options?

[Paolo Bonzini (bonzini)]

Yes, it is valid. libvirt is disabling the automagic UHCI controller, which makes sense since he is using XHCI.

[Gonglei (arei-gonglei)]

On 2014/11/14 19:57, Leen Keus wrote:

> Command lines generated by virsh from the libvirt xml files (let me know if you also need the xml configuration files):
>
Does this below patch fix your problem?

http://git.qemu.org/?p=qemu.git;a=commitdiff;h=79ae25af1569a50a0ec799901a1bb280c088f121

Regards,
-Gonglei

[Leendert Keus (lj-keus)]

I greatly appriciate your help to find a solution for this issue, but I have used apt-get to installed QEMU already since version 12.10. I never used the QEMU source code for something. Is it possible that you provide an updated executable or library?
Regards,
Leen

[Serge Hallyn (serge-hallyn)]

@Leen,

I am pushing qemu 2.1+dfsg-4ubuntu6.3~ppa1 (which includes the fix proposed by Gonglei in comment #7) to ppa:serge-hallyn/libvirt-testing. Please let us know whether it does in fact solve your issue.

[Leendert Keus (lj-keus)]

Hi Serge, just tested your solution but unfortunately with no success. I also removed a vm and created it again, also with no luck. Sorry ...

[Leendert Keus (lj-keus)]

Same issue appeard after upgrading from Ubuntu 13.04 to 13.10; bug: 1245251 (Apparmor blocks usb devices in libvirt in Saucy). Could it be related to this issue? Maybe a new or other line in /etc/apparmor.d/abstractions/libvirt-qemu?

[Joe Hickey (jfwd)]

I just wanted to add another data point -- I migrated my old WinXP VM from my 14.04 install to my new 14.10 install and found out that the USB passthrough is not working -- Same libvirt XML definition file. Tried removing and re-adding with no luck.

[Leendert Keus (lj-keus)]

I think it is related to Apparmor, I see the following lines in /etc/syslog:

Nov 22 14:36:23 uvir kernel: [18882.602278] audit: type=1400 audit(1416663383.171:69): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb3/uevent" pid=14279 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 22 14:36:23 uvir kernel: [18882.602295] audit: type=1400 audit(1416663383.171:70): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb3/3-0:1.0/uevent" pid=14279 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 22 14:36:23 uvir kernel: [18882.602306] audit: type=1400 audit(1416663383.171:71): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/uevent" pid=14279 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 22 14:36:23 uvir kernel: [18882.602318] audit: type=1400 audit(1416663383.171:72): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/4-0:1.0/uevent" pid=14279 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 22 14:36:23 uvir kernel: [18882.602330] audit: type=1400 audit(1416663383.171:73): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/4-2/uevent" pid=14279 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 23 00:28:39 uvir kernel: [35109.848778] audit: type=1400 audit(1416698919.255:181): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb3/uevent" pid=25427 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 23 00:28:39 uvir kernel: [35109.848799] audit: type=1400 audit(1416698919.255:182): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb3/3-0:1.0/uevent" pid=25427 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 23 00:28:39 uvir kernel: [35109.848815] audit: type=1400 audit(1416698919.255:183): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/uevent" pid=25427 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 23 00:28:39 uvir kernel: [35109.848831] audit: type=1400 audit(1416698919.255:184): apparmor="DENIED" operation="open" profile="libvirt-30619811-d285-944a-026a-cdd4a753b77a" name="/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/4-0:1.0/uevent" pid=25427 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
Nov 23 00:28:39 uvir kernel: [35109.848847] audit: type=1400 audit(1416698919.255:185): apparmor="DENIE...

[Joe Hickey (jfwd)]

I just noticed that after I applied the latest updates (apt-get dist-ugrade) to my Ubuntu 14.04 system that USB redirection is now failing on THIS system as well - just like the 14.10 system failed.

FWIW, I do not see any apparomor denial messages in /var/log/syslog or anywhere else. So far I've not found anything to indicate why passthrough failed, seems like a silent failure whatever it is.

[Leendert Keus (lj-keus)]

I don't know if it is relevant, but in the error message of apparmor is says: comm="qemu-system-x86"; while libvirt starts qemu-system-x86_64. Anybody searching for a solution? I am not able to make backups since the upgrade to 14.10...

[Joe Hickey (jfwd)]

Please disregard my previous comment about being broken in 14.04 -- In fact I was running 14.10 and didn't realize it (a reboot and 14.10 being the grub default). In fact the passthrough still works in 14.04 just fine, and still fully broken in 14.10.

I did confirm, however, that there are NO apparmor messages regarding qemu on my 14.10 system.

[Leendert Keus (lj-keus)]

Found the following lines in /var/log/libvirt/qemu/<machine>.log
libusbx: error [_get_usbfs_fd] libusbx couldn't open USB device /dev/bus/usb/001/004: Permission denied
libusbx: error [_get_usbfs_fd] libusbx requires write access to USB device nodes.
libusbx: error [_get_usbfs_fd] libusbx couldn't open USB device /dev/bus/usb/001/004: Permission denied
libusbx: error [_get_usbfs_fd] libusbx requires write access to USB device nodes.
libusbx: error [_get_usbfs_fd] libusbx couldn't open USB device /dev/bus/usb/001/004: Permission denied
libusbx: error [_get_usbfs_fd] libusbx requires write access to USB device nodes.

[zebul666 (zebul666)]

I just installed libvirt 1.2.8 on ubuntu utopic and usb passthrough does not work on a linux vm
I got the libusbx error in the livert VM log

[Serge Hallyn (serge-hallyn)]

@lj-keus

if you turn off apparmor

sudo /etc/init.d/apparmor stop
sudo /etc/init.d/apparmor teardown

does that fix the issue for you?

(Please re-enable apparmor immediately after the test using

sudo /etc/init.d/apparmor start
sudo stop libvirt-bin
sudo start libvirt-bin
)

[Leendert Keus (lj-keus)]

Hi Serge,

Following error message appears after I teared down apparmor and when I tried to start the vm:

$sudo virsh start <vm>
error: Failed to start domain <vm>
error: error from service: ListActivatableNames: Failed to query AppArmor policy: No such file or directory

Can anybody provide me a list of settings, for the OS, apparmor, libvirt, qemu, etc., etc. that can I check for this issue?

[Serge Hallyn (serge-hallyn)]

I haven't seen an error like that, and cannot reproduce it here.

Another way to test that apparmor is causing the problem would be to add

 /dev/** rw,

at the bottom of the file /etc/apparmor.d/abstractions/libvirt-qemu. When you next start the vm apparmor should allow it full access to all devices.

[Serge Hallyn (serge-hallyn)]

You also should be able to do

 aa-audit /usr/sbin/libvirtd

which should put informative messages into /var/log/syslog or
/var/log/audit/audit.log as well.

[Kelvin Middleton (kelvin-middleton)]

I don't run apparmor.

Ubuntu 14.10 host

Have this same issue with USB Host Attachment on Windows and Linux guests.
USB Redirection using SPICE is unaffected and works as expected.

Libvirt logs show...
libusbx: error [_get_usbfs_fd] libusbx couldn't open USB device /dev/bus/usb/003/006: Permission denied
libusbx: error [_get_usbfs_fd] libusbx requires write access to USB device nodes.

'chown libvirt-qemu:kvm /dev/bus/usb/003/006' solves the problem and USB Host Attachment now works again.

Host dmesg shows...
usb 3-13: reset high-speed USB device number 9 using xhci_hcd
[ 3346.483029] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff8807ba9beb40
...USB device needs to be physically detached and re-attached for host to see again.

Tested this with a Sandisk Cruzer Blade 16GB and a Logitech HD C525.

[Leendert Keus (lj-keus)]

Hi Kevin, finally somebody that found a solution. Great, thank you very much! It is working now.

[Serge Hallyn (serge-hallyn)]

Has anyone tested whether this is still broken in 15.04?

[Kelvin Middleton (kelvin-middleton)]

I'm running an ACS patched linux mainline 3.18 as well as Ubuntu 14.10 stock. Haven't tested the Ubuntu 15.04 stock but will grab the debs and confirm.

[Kelvin Middleton (kelvin-middleton)]

...actually are you talking kernel or the full 15.04 pre-release?

[Serge Hallyn (serge-hallyn)]

Actually I was talking about just libvirt. cmment #23 (and 24) suggests that the problem is libvirt not re-labeling the devices, so i'm wondering whether 1.2.12 fixes it.

[nickmaelao (nickmaleao)]

I have done the suggested in comment 23 , but the problem persist, after i lunch the guest, i receive the following erros, and lsusb in the guest doesnt show the attached usb device:

Host:

 ll /dev/bus/usb/001/
crw-rw-r-- 1 libvirt-qemu kvm 189, 9 Mar 29 21:21 010

lsusb:
Bus 001 Device 010: ID 0cf3:9271 Atheros Communications, Inc. AR9271 802.11n

syslog

Mar 29 21:21:57 ivan-desktop kernel: [168754.517390] usb 1-1: reset high-speed USB device number 10 using xhci_hcd
Mar 29 21:21:57 ivan-desktop kernel: [168754.666064] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264600
Mar 29 21:21:57 ivan-desktop kernel: [168754.666080] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264648
Mar 29 21:21:57 ivan-desktop kernel: [168754.666086] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264690
Mar 29 21:21:57 ivan-desktop kernel: [168754.666090] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a2646d8
Mar 29 21:21:57 ivan-desktop kernel: [168754.666095] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264720
Mar 29 21:21:57 ivan-desktop kernel: [168754.666099] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264768

libvirt-bin 1.2.8-0ubuntu11.4
qemu 2.1+dfsg-4ubuntu6.4
Ubuntu 14.10 3.19.2-031902-generic

[Kelvin Middleton (kelvin-middleton)]

Hey nickmaelao, your outputs show you've only changed the ownership of the USB bus and not the USB device itself...I'd suspect if you looked at 'ls -la /dev/bus/usb/001/' then the actual USB device will still have root ownership. Ergo if libvirtd is still creating the vm's with qemu and a non-root user you will still have the problem.

chmod libvirt-qemu:kvm /dev/bus/usb/001/xxx (xxx being the USB device numeber) should resolve.

Alternatively you could change /etc/libvirtd/qemu.conf and make libvirtd create guests with qemu as the root user, look for the 'user = ' and 'group = ' lines. I can't comment on the risk associated to this so you'll need to look into that yourself but I've taken this approach and have no problems with USB attachment and I don't need to manually change device ownerships.

[nickmaelao (nickmaleao)]

Sorry, i posted the wrong output, i had modified the ownership of usb the device /dev/bus/usb/001/010 to libvirt-qemu:kvm, but the problem persists.

I configured libvirt to run as root, but the problem is the same, in syslog im still receiving

Mar 29 22:54:42 ivan-desktop kernel: [174324.769452] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264600
Mar 29 22:54:42 ivan-desktop kernel: [174324.769470] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264648
Mar 29 22:54:42 ivan-desktop kernel: [174324.769479] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264690
Mar 29 22:54:42 ivan-desktop kernel: [174324.769489] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a2646d8
Mar 29 22:54:42 ivan-desktop kernel: [174324.769497] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264720
Mar 29 22:54:42 ivan-desktop kernel: [174324.769506] xhci_hcd 0000:00:10.0: xHCI xhci_drop_endpoint called with disabled ep ffff88010a264768
Mar 29 22:54:42 ivan-desktop kernel: [174324.981362] usb 1-1: reset high-speed USB device number 10 using xhci_hcd

Heres the portion of the qemu xml of the guest:

<hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0cf3'/>
        <product id='0x9271'/>
        <address bus='1' device='10'/>
      </source>
      <alias name='hostdev0'/>
 </hostdev>

[Kelvin Middleton (kelvin-middleton)]

The syslog output looks like its from the Host not the guest?

What does the libvirtd log say for the guest...typically at /var/log/libvirt/qemu/XXX (XXX being the name of the guest).

There are 2 separate issues/related in this thread, first being USB attachment to guests not working which is believed ownership related hence my *workaround* in post #23. The second is about how the USB device is re-atrached to the Host once the guest has been destroyed...this is what the syslog/dmesg outputs are about and why Serge has renamed the bug report.

Your posts say you are having problem with both which seems odd and not possible?

[Leendert Keus (lj-keus)]

Hi all, recently upgraded to 15.04 and the issue is solved in this version.

[jim (jm-9)]

Hi, same problem on my machine ... Comment #23 does not work for me. In my VM log I found this line:

libusb_set_configuration: -6 [BUSY]

I'm running 14.04 (fresh install), QEMU 2.0.0. Any suggestions?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

Hi,
seeing this due to the move of upstream qemu->qemu(ubuntu).
This is a libvirt issue, not qemu so marking this as that.

Xenial was already better in this regard which also seems to match the report in comment #33.
Furthermore I fixed quite some usb passthrough issues (4 I think) on the way from xenial to Bionic.
Also I didn't hear about is anymore for years other than in the post-xenial context that I was driving with a few bug reporters since then.

I'll mark the bug invalid, but if anyone really is still affected I would ask you to (also) test with Bionic (18.04) if possible.

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]