USB Passthrough Fails on Start, Needs domain Reset

[Russell Morris]

Hi,

I am seeing (consistently = always), USB Passthrough for my Logitech Keyboard and Mouse ... they don't work / no response on domain (VM) startup. After a reset of the VM they then work - but why are they "dead" on initial startup of the VM? Is this a known issue?

Running,
QEMU emulator version 5.0.0 (Debian 1:5.0-5ubuntu9.1)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

And if it makes a difference, this is a macOS guest (on a Linux host).

Thanks!

[Christian Ehrhardt  (paelzer)]

Hi Russel,
I haven't seen such issues recently, but let us try to sort it out.
For that I beg your pardon but I need to start asking for a few details:

1. what exactly (commands) does "reset of the VM" mean for you?
2. in the guest does the output of lspci -v (or whatever the macos counterpart is) change before/after reset and if so how does it change?
3. Could you track on your host "journalctl -f" output, then start the guest, then reset the guest - and attach that log here. If possible please identify the timestamps when you have reset the guest.

[Russell Morris (6-u3untu-h)]

Sure, NP at all! Let me try to answer, and yell if you need more details - I will get them for you.
1) Reset ... meaning if I start the domain with 'virsh start macOS' => no USB passthrough (Logitech nano receiver in this case). So then, I 'virsh reset macOS' ... USB passthrough works fine! But always needs the reset after start. Make sense?
2) Can't really check that, as I never get in to the guest OS. I can't even get into the UEFI, given that the passthrough device is a receiver for a keyboard and mouse. So it's right at the start where it's not available ... or better said, it's not available / working right from the start.
3) Sure, NP at all! Do you want me to filter journalctl, to only capture a specific target (e.g. libvirtd)?

Thanks!

[Christian Ehrhardt  (paelzer)]

(1) yeah makes sense now, thanks

(2) So we can't get good-vs-bad data, but is the behavior any different at least with non MacOS guests then?

(3) no please no filtering - add all of it to see everything from kernel-apparmor denials to anything odd with udev rules.

[Christian Ehrhardt  (paelzer)]

(2)+(3) bonus, since the device will need to be detached on the host to be passed through and IIRC that depends a bit on USB topology. Can you provide pre/booted/after-reset output of "lsusb -t" of the host as well pelase?

[Russell Morris (6-u3untu-h)]

Sure, will add - NP! One comment on (2) ... I can't say if this happens with other guests or not, as I only bind this USB device to this particular (guest) OS.

[Russell Morris (6-u3untu-h)]

OK, let me try to explain what I did here - and yell if this is not clear!

Step #1, check lsusb -t before any qemu start,
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 3: Dev 26, If 0, Class=Human Interface Device, Driver=usbfs, 1.5M
    |__ Port 4: Dev 3, If 0, Class=Communications, Driver=rndis_host, 480M
    |__ Port 4: Dev 3, If 1, Class=CDC Data, Driver=rndis_host, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
    |__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=uas, 10000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 4, If 2, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 1: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 7: Dev 3, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
    |__ Port 10: Dev 5, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M

Step #2, virsh start macOS, then I have,
a) lsusb -t
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 3: Dev 26, If 0, Class=Human Interface Device, Driver=usbfs, 1.5M
    |__ Port 4: Dev 3, If 0, Class=Communications, Driver=rndis_host, 480M
    |__ Port 4: Dev 3, If 1, Class=CDC Data, Driver=rndis_host, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
    |__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=uas, 10000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 4, If 2, Class=Human Interface Device, Driver=usbfs, 12M
        |__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbfs, 12M
        |__ Port 1: Dev 4, If 1, Class=Human Interface Device, Driver=usbfs, 12M
    |__ Port 7: Dev 3, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
    |__ Port 10: Dev 5, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M

b) Output from, journalctl -f | grep -v x11vnc (avoid x11vnc flooding),
Dec 01 14:35:32 linuxServer audit[845332]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845332 comm="apparmor_parser"
Dec 01 14:35:32 linuxServer kernel: audit: type=1400 audit(1606854932.259:190): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845332 comm="apparmor_parser"
Dec 01 14:35:32 linuxServer audit[845341]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845341 comm="apparmor_parser"
Dec 01 14:35:32 linux...

[Christian Ehrhardt  (paelzer)]

The logs make total sense, thank you!

OK on lsusb:
We see the HID devices flip from "usbhid" driver to "usbfs" with the initial start.
They no more change on the reset.

On the journal entries we see on the initial guest start that gnome has to yield the devices (OK).
USB isn't mentioned at all on this initial start
No errors that would seem obvious there. ...

Then on the reset we see (due to the reset) that the host kernel seems to do a USB reset
Dec 01 14:36:10 linuxServer kernel: usb 1-1.1: reset full-speed USB device number 4 using xhci_hcd
Dec 01 14:36:10 linuxServer upowerd[4208]: treating change event as add on /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb1/1-1/1-1.1
Dec 01 14:36:10 linuxServer upowerd[4208]: treating change event as add on /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb1/1-1/1-1.1

That does not immediately point to the issue unfortunately :-/

I assume the bus reset is done as part of giving the devices back and resetting them.

Maybe we should try to reset this device when the hang happens ourselve to see if it does something:
Be warned, this could be bad as in "need to reboot afterwards"
Device:
  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M

I'm unsure which reset exactly that was, maybe the following (re-)initializes too much but you might try (when the guest hangs):
$ cd /sys/bus/pci/drivers/xhci_hcd/; echo -n "0000:00:01.3" > unbind; echo -n "0000:00:01.3" > bind;

On a similar note, any success if you replug one of those devices?

---

On (2) you said:
"I can't say if this happens with other guests or not, as I only bind this USB device to this particular (guest) OS"
=> But you can for a try shut that guest down and use another e.g. Ubuntu guest with the same forwarding to try right?

[Russell Morris (6-u3untu-h)]

No issue with reboot, worry about that after we debug this :-). I tried the noted command, but I get,
bash: echo: write error: No such device
bash: echo: write error: No such device

FYI,
$ ls -alF /sys/bus/pci/drivers/xhci_hcd/
total 0
drwxr-xr-x 2 root root 0 Dec 2 17:51 ./
drwxr-xr-x 35 root root 0 Dec 2 17:51 ../
lrwxrwxrwx 1 root root 0 Dec 2 17:51 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:02:00.0/
lrwxrwxrwx 1 root root 0 Dec 2 17:51 0000:0b:00.3 -> ../../../../devices/pci0000:00/0000:00:07.1/0000:0b:00.3/
--w------- 1 root root 4096 Dec 2 17:53 bind
lrwxrwxrwx 1 root root 0 Dec 2 17:51 module -> ../../../../module/xhci_pci/
--w------- 1 root root 4096 Dec 2 17:51 new_id
--w------- 1 root root 4096 Dec 2 17:51 remove_id
--w------- 1 root root 4096 Dec 2 17:51 uevent
--w------- 1 root root 4096 Dec 2 17:53 unbind

So I tried echo -n "0000:02:00.0", but no recovery. Also, unplug / plug ... nope. So then I tried my old standby :-).
virsh reset macOS
=> Nope, not working now. Had to destroy the VM, then the usual (start, reset) ... working again.

Thoughts?

Also just stumbled on to this by accident ... on VM start, no USB (as above). But leave it 5 min, and the VM reboots itself (internally, not restart at the virsh "level" ... UEFI watchdog perhaps?). And then USB is OK. So is it some sort of race condition / timing issue at initial VM startup?

Thanks!

[Christian Ehrhardt  (paelzer)]

Hmm,
I was guessing the paths from your former log output.
Thanks for fixing it up as needed on your system.
So we know that just resetting it from the Host POV won't change anything.

And thanks for the info on "auto-recovering" on guest reboot.

The one obvious next test would be to shut down the macOS guest and try an ubuntu guest with the same forwards. That will allow us to see if it is "virtualization-components" or actually "guest OS" that we need to think about.

[Russell Morris (6-u3untu-h)]

> The one obvious next test would be to shut down the macOS guest and try an ubuntu guest with the same forwards. That will allow us to see if it is "virtualization-components" or actually "guest OS" that we need to think about.

Good idea. OK, so I shut down all VM's, added this same USB device to a Windows 10 VM I also had defined (on the same host OS). Started it up (same way, virsh start) => booted up, and ... USB device works!

So it seems to be guest OS related, agreed? And actually, not even that I'd say, as I can't even use it in the bootloader / UEFI, so not really even the OS, but the bootloader / UEFI? And that sort of aligns with the watchdog reset, as then it works, but still hasn't gotten to the OS (boot). Agreed?

Thanks!

[Christian Ehrhardt  (paelzer)]

I think we can agree that it is "guest dependent"
Either Windows has a way to fix up the issue or it doesn't even have it to begin with (compare to MacOS).

Chances are that your (virtual) bootloader/UEFI are what breaks it to begin with (and not the MacOS later). Do you think you could play around with the options you have for that (does it need to be UEFI for example)? Even if you'd need to stick to UEFI you could reset it by e.g. replacing it's VAR file that represents the UEFI-writable space.

[Russell Morris (6-u3untu-h)]

OK, some interesting findings :-). And back to thinking it may be QEMU related? Let me explain, by all means disagree!
1) I found that using OpenCore with macOS, I no longer need a custom UEFI. So, I tried the UEFI that comes with Ubuntu (apt install ovmf, copied the files over). No change. So then, I cloned and built the latest UEFI from Tianocore / EDK II (https://github.com/tianocore/edk2). Copied that over (and VARS, default), no delta. And as I can't get into the UEFI menu on VM startup (i.e. press ESC, F2, etc.) ... seems to be OS independent, just an issue between UEFI and QEMU - agreed?
2) Figured I'd try to clone, build and run the latest version of QEMU, see if that is it - FYI, the current version in Ubuntu that I'm running is,
QEMU emulator version 5.0.0 (Debian 1:5.0-5ubuntu9.2)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

OK, that said ... I can build, install => but can't get apparmor to let me run my custom QEMU executable (arrgh!). I even tried disabling security_driver (set to "none" in /etc/libvirt/qemu.conf, restarted libvirtd.service) ... but no joy. Can't get the latest QEMU to run. Any thoughts here would be appreciated! This is to see if it's an "older" issue, and may be resolved already.

3) An interesting side note, when I did get into the UEFI menu (post VM reset), changed the resolution ... I could no longer boot (fully) my guest OS. Had to reboot my host OS (Ubuntu), then all OK. Very odd, but an observation :-).

Thanks for all the help!

[Russell Morris (6-u3untu-h)]

FYI, was able to get the following version of QEMU running (using the info at https://stackoverflow.com/questions/10817813/apparmor-causes-issues-on-libvirt-with-custom-qemu),
QEMU emulator version 5.1.93 (v5.2.0-rc3)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

No delta - still reacts the same.

Thanks!

[Christian Ehrhardt  (paelzer)]

Thanks for your feedback Russell.
You resolved the apparmor issue faster than I could reply :-)

About the actual issue, it indeed seems like an issue between UEFI and QEMU here.
But I'm still wondering why it then would be any different with Windows and/or Linux guests.
Is the UEFI in those Windows/Linux guests always working fine, or is in those cases maybe the UEFI affected but the later boot of the OS resets/fixes the bad condition (whatever it is).

And if on those non MacOS guests the UEFI is always fine, then we are back wondering why that would be the case if they are using the same UEFI&Qemu combination.

... I feel like I can't really help here, but I'm happy to continue to discuss. Hopefully we can end in a place which either of us can actually action upon to resolve it for you.

[Russell Morris (6-u3untu-h)]

No worries, and thanks for all the help so far - it really is appreciated!

On the apparmor issue, is this the "right" fix? I modified /etc/apparmor.d/abstractions/libvirt-qemu, adding,
/usr/local/bin/qemu-system-x86_64 rmix,
/usr/local/share/qemu/** r,

Or is there a "better" (correct) way?

As for Windows/Linux vs. macOS guests, I don't think they are usign the same UEFI (are they?). I did reboot the Windows guest, and it looks to be a different "BIOS", agreed? Checking the xml configuration files, they are a bit different (macOS copied below, Windows does not include this). So is it (specifically?) UEFI vs. QEMU (i.e. that is the issue)?
<loader readonly="yes" type="pflash">OVMF_CODE.fd</loader>
<nvram>OVMF_VARS.fd</nvram>

I could definitely be wrong here, but thinking that "stock" UEFI should work / be supported?

Thanks again!

[Christian Ehrhardt  (paelzer)]

> On the apparmor issue, is this the "right" fix? I modified /etc/apparmor.d/abstractions/libvirt-qemu, adding,
> /usr/local/bin/qemu-system-x86_64 rmix,
> /usr/local/share/qemu/** r,
>
> Or is there a "better" (correct) way?

Doing the same in /etc/apparmor.d/local/abstractions/libvirt-qemu is
better to now get conflicts on upgrades.

> As for Windows/Linux vs. macOS guests, I don't think they are usign the same UEFI (are they?).
> I did reboot the Windows guest, and it looks to be a different "BIOS", agreed?
> Checking the xml configuration files, they are a bit different (macOS copied below, Windows does not include this). So is it (specifically?) UEFI vs. QEMU (i.e. that is the issue)?
> <loader readonly="yes" type="pflash">OVMF_CODE.fd</loader>
> <nvram>OVMF_VARS.fd</nvram>

The above section belongs to using OVMF and if the windows guest
didn't have that it probably runs on seabios or some other non UEFI
path.

[Russell Morris (6-u3untu-h)]

> Doing the same in /etc/apparmor.d/local/abstractions/libvirt-qemu is
> better to now get conflicts on upgrades.

Makes sense, thanks! Will move the items there. Both entries are needed? Seems like I can just restart apparmor after (or at least that's working for me).

> if the windows guest didn't have that it probably runs on seabios or
> some other non UEFI path.

Agreed! Looks like it runs SeaBIOS (fast flash of the screen :-)). So the issue seems to be UEFI and QEMU?

Thanks!

[Christian Ehrhardt  (paelzer)]

On Mon, Dec 7, 2020 at 4:05 PM Russell Morris
<email address hidden> wrote:
>
> > Doing the same in /etc/apparmor.d/local/abstractions/libvirt-qemu is
> > better to now get conflicts on upgrades.
>
> Makes sense, thanks! Will move the items there. Both entries are needed?
> Seems like I can just restart apparmor after (or at least that's working
> for me).

This particular rule is (re)loaded on guest start, so stop+start a
guest and it is updated.

> > if the windows guest didn't have that it probably runs on seabios or
> > some other non UEFI path.
>
> Agreed! Looks like it runs SeaBIOS (fast flash of the screen :-)). So
> the issue seems to be UEFI and QEMU?

Yeah - kind of, and now the request it to test Ubuntu and Windows
guest with the same OVMF based UEFI.

[Russell Morris (6-u3untu-h)]

> This particular rule is (re)loaded on guest start, so stop+start a
> guest and it is updated.

So no need to restart apparmor? And both entries are required (in the libvirt-qemu file)?

> Yeah - kind of, and now the request it to test Ubuntu and Windows
> guest with the same OVMF based UEFI.

Funny! Only because we're very much on the same page - was already trying to get that going :-). Will let you know!

[Russell Morris (6-u3untu-h)]

Hmmm ... not getting Windows 10 to boot now. Trying to follow https://k3a.me/boot-windows-partition-virtually-kvm-uefi/, set up os in my config (xml),
  <os>
    <type arch="x86_64" machine="pc-q35-5.2">hvm</type>
    <loader readonly="yes" type="pflash">/var/lib/libvirt/qemu/nvram/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd</nvram>
    <boot dev="hd"/>
    <bootmenu enable="yes"/>
  </os>

I get to the UEFI menu, but it won't seem to boot to my drive. Thoughts / suggestions?

Thanks!

[Christian Ehrhardt  (paelzer)]

>
> I get to the UEFI menu, but it won't seem to boot to my drive. Thoughts
> / suggestions?

I guess for that to work you'd have had to install it with UEFI
already present to prep the boot entries accordingly.
I'm not asking you to modify (and break) your existing guest.
Recommendation: just install an Ubuntu in such a Guest - that should
provide the best debugging options and has the most simple install.

[Russell Morris (6-u3untu-h)]

Yep, makes sense ... but I have to admit, I screwed up. Sorry! Complete bonehead on my part.

When I tested Ubuntu (guest), I was connected over VNC, so I didn't correctly / properly test the USB interface. So, I re-checked, making sure to use QEMU + UEFI (Tianocore, stock VARS), and check USB. It does exactly the same thing (now :-))! And this makes more sense, it was very odd that the OS would play into it.

I also here have to do a reset (of the guest), to get the USB to show up. So it really does seem to be an issue between QEMU and UEFI, agreed?

Thanks!

[Christian Ehrhardt  (paelzer)]

> I also here have to do a reset (of the guest), to get the USB to show
> up. So it really does seem to be an issue between QEMU and UEFI, agreed?

As it seems right now - yes
I've not yet seen/reproduced the same on qemu 5.0/edk2 2020.08-1 :-/

I can ping here once I have qemu 5.2 (just released) ready to try, but
quite likely you'll want to ask the ustream people on this.
It might make sense to them and whatever hint they have could bring
this forward.
Would you mind outlining your case on the upstream mailing list?

P.S. Please report the link to the ML archive entry here, that would be nice

[Russell Morris (6-u3untu-h)]

> I've not yet seen/reproduced the same on qemu 5.0/edk2 2020.08-1 :-/

So you don't see this issue, or haven't had a chance to check it? No biggie either way, just trying to understand your comment :-).

> Would you mind outlining your case on the upstream mailing list?

Yes, no issue at all - but should we wait for you to test? If you see the same issue, NP at all flagging this. But if you don't, then somehow configuration related?

Thanks!

[Christian Ehrhardt  (paelzer)]

> > I've not yet seen/reproduced the same on qemu 5.0/edk2 2020.08-1 :-/
>
> So you don't see this issue, or haven't had a chance to check it? No
> biggie either way, just trying to understand your comment :-).

Sorry to be unclear:
- I was not having the issues doing the same in the past.
- Nor did I have "another" report about it yet.
- I was taking this as a chance trying to reproduce it again ...

The device that I pass around is:
Bus 002 Device 004: ID 046d:c069 Logitech, Inc. M-U0007 [Corded Mouse M500]

This is the hierarchy it is attached with:
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/11p, 480M
    |__ Port 2: Dev 2, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
    |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M

Testcase:
- Use UEFI from ovmf and USB passthrough
- boot into a 20.04.1 Desktop installation ISO live environment
- does the forwarded mouse work in the Linux environment

Forwarding:
<hostdev mode="subsystem" type="usb" managed="yes">
  <source>
    <vendor id="0x046d"/>
    <product id="0xc069"/>
  </source>
</hostdev>

Using OVMF
  <os>
    <type arch="x86_64" machine="pc-q35-focal">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
    <boot dev="hd"/>
  </os>

First on 20.04
Qemu: 1:4.2-3ubuntu6.10 ovmf 0~20191122.bd85bf54-2ubuntu3

Note: On guest start I see on the host (dmesg, this is triggered by passing it on):
[ 1471.526982] usb 2-4: reset low-speed USB device number 4 using xhci_hcd
When the guest ends it comes back like:
[ 2417.321453] usb 2-4: reset low-speed USB device number 4 using xhci_hcd
[ 2417.615907] input: Logitech USB Laser Mouse as /devices/pci0000:00/0000:00:14.0/usb2/2-4/2-4:1.0/0003:046D:C069.0005/input/input16
[ 2417.677515] hid-generic 0003:046D:C069.0005: input,hidraw0: USB HID v1.10 Mouse [Logitech USB Laser Mouse] on usb-0000:00:14.0-4/input0

Result:
- the passthrough mouse works fine in the guest
- it is seen in guests lsusb
- There is some struggle with the mouse-pointer as the PassThrough-mouse
  fights with virtual tablet + virtual mouse (visual pointer only follows virtual
  mouse, but real mouse still works completely fine)

If you'd also use e.g. GPU passthrough you'd have less confusion, but since I use spice/qxl there always will still be mouse signals from the host reaching the guest that way.
(TBH for Mouse/keyboard I'm personally favoring just using the virtual devices for the ability to interact with host and guest as needed)
But anyway, if I'd e.g. start an app in the guest I can control it with the passed-through mouse and that is what you look for I guess.

So on 20.04 things worked for me (as in the past).

Upgrading to 20.10
Qemu: 1:5.0-5ubuntu9.2
OVMF: 2020.05-5
=> Still working fine

Upgrading to 21.04
Qemu: 1:5.1+dfsg-4ubuntu3
OVMF: 2020.08-1

P.S. on the fighting input devices.
Qemu/Libvirt doesn't really let you run without these easily (too many guests have fatal fails without input devices). But I have quickly seen that things are fine on e.g. the installer but later on Ubuntu boot change (desktop integration gets enabled, the mouse pointer fight begins). That was due to t...

[Russell Morris (6-u3untu-h)]

Thanks for the details! I thought we may be on to something, but not quite. I saw a couple deltas in your config (vs. mine),
1) No OVMF_VARS - tried removing this, no difference.
2) You are checking the mouse in the OS - checked that as well (not just UEFI / Ubuntu Grub menu), nope, it's still not there. I can't log in, no keyboard or mouse ... but I think yours is up for the login screen, right?

FYI, you note about GPU Passthrough - not sure I'm following you 100%, but I do have a second GPU in the machine, and I enable it for GPU Passthrough. It works fine (actually, very nicely!). Not sure why this matters though?

No issue here sharing config details - is there something in particular that would help?

Thanks!

[Christian Ehrhardt  (paelzer)]

On Thu, Dec 10, 2020 at 1:41 PM Russell Morris
<email address hidden> wrote:
>
> Thanks for the details! I thought we may be on to something, but not quite. I saw a couple deltas in your config (vs. mine),
> 1) No OVMF_VARS - tried removing this, no difference.

If not specified an empty template will be used - so that was not
making a difference

> 2) You are checking the mouse in the OS - checked that as well (not just UEFI / Ubuntu Grub menu), nope, it's still not there. I can't log in, no keyboard or mouse ... but I think yours is up for the login screen, right?

TBH - I never had/tried/used "mouse" in grub or UEFI, I'm just a
keyboard guy I guess.
If you tell me how to initialize a mouse there I can try :-)

> FYI, you note about GPU Passthrough - not sure I'm following you 100%,
> but I do have a second GPU in the machine, and I enable it for GPU
> Passthrough. It works fine (actually, very nicely!). Not sure why this
> matters though?

It only matters if you have your mouse working, but then issues with
the multitude of mouse-inputs.
E.g. one from the Host via Desktop-integration that comes through
spice - and another one via the USB Passthrough.
Due to that if you use GPU passthrough (and no other virtual display)
the channel through spice won't exists and therefore no conflicting
inputs will happen.

[Russell Morris (6-u3untu-h)]

> If you tell me how to initialize a mouse there I can try :-)

Sorry, bad wording on my part. The USB device I am connecting is a Logitech Nano Receiver ... keyboard and mouse both connected to it. So at the UEFI / Grub stage - only the keyboard, as you correctly point out :-).

> It only matters if you have your mouse working, but then issues with
> the multitude of mouse-inputs.

Not 100% sure I follow you :-). But I do have "two" mice connected - one on the guest (USB Passthrough), the other to virt-manager ... which is running over X-Windows from the host to an X-Server. Not sure that makes any difference, but just in case!

> Due to that if you use GPU passthrough (and no other virtual display)
> the channel through spice won't exists and therefore no conflicting
> inputs will happen.

I can disable spice if you want, go only USB Passthrough. Just let me know. I have done that before, can try it (for this particular issue).

BTW, another thought. The VM is defined as Q35 - in fact, a few items below, just in case they could be part of this,
    <type arch="x86_64" machine="pc-q35-5.2">hvm</type>
    <qemu:arg value="-cpu"/>
    <qemu:arg value="Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check"/>
    <qemu:arg value="-smbios"/>
    <qemu:arg value="type=2"/>

Thanks!

[Russell Morris (6-u3untu-h)]

OK, I got to thinking about this other bug I had seen a while back. Initially thought it was not related, but maybe it is after all?
https://bugs.launchpad.net/qemu/+bug/1694808

Perhaps we are using different USB controllers, and that's why we see different results? I have the following ... match to your setup?
    <controller type="usb" index="0" model="ich9-ehci1">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x7"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci1">
      <master startport="0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x0" multifunction="on"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci2">
      <master startport="2"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x1"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci3">
      <master startport="4"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x2"/>
    </controller>

And my USB device,
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc52b"/>
      </source>
      <address type="usb" bus="0" port="3"/>
    </hostdev>

So the USB device is on ich9-uhci2, correct?

Thanks!

[Russell Morris (6-u3untu-h)]

OK, tried other USB controllers (piix4, vt82c686b), same issue - this is really odd!

Thought I'd update to the latest / official v5.2.0 qemu, but now I get this when trying to run it. Huh?!?!
error: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot execute binary /usr/local/bin/qemu-system-x86_64: Permission denied

No changes to my apparmor file - checked :-).

Thanks!

[Russell Morris (6-u3untu-h)]

OK, sort of figured it out :-). Seems I need to also modify /etc/apparmor.d/usr.sbin.libvirtd. Let me dig into that one separately ... LOL.

On the USB front, thinking a basic configuration file should be sufficient for debugging - no drives needed, etc. Do you happen to have one (or a partial) that you can share - so I can try the setup you have?

Thanks!

[Christian Ehrhardt  (paelzer)]

The following should get you the same test environment I used.

$ qemu-img create /var/lib/libvirt/images/ubuntu20.04-UEFI.qcow2 25
# feel free to use a closer mirror
$ wget http://ftp.uni-kl.de/pub/linux/ubuntu.iso/20.04.1/ubuntu-20.04.1-desktop-amd64.iso
$ mv ubuntu-20.04.1-desktop-amd64.iso /var/lib/libvirt/images/ubuntu-20.04.1-desktop-amd64.iso

The guest xml is then configured to boot from the ISO (for testing in the "try Ubuntu" environment.

<domain type='kvm'>
  <name>ubuntu20.04-UEFI</name>
  <uuid>b43803cc-c46f-43ec-8801-cbe805f24770</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/20.04"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/ubuntu20.04-UEFI_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/ubuntu20.04-UEFI.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/ubuntu-20.04.1-desktop-amd64.iso'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
 ...

[Russell Morris (6-u3untu-h)]

OK, this is going to sound crazy, but ...

I did exactly like you note above - and the same thing happens. My USB device isn't seen / avaiable on start, but after I reset the VM .. voila! It's there and works.

OK, wondering what else it could be. I have a USB hub where the device is plugged in - will try removing that (variable) ... nope, that's not it either. Arrgh!?!?

LOL. This really is very odd. I'm open to any thoughts you have. Thanks!

[Russell Morris (6-u3untu-h)]

BTW, one other interesting thing I just noticed, in lsusb. As this Logitech receiver connects to multiple devices (e.g. keyboard, mouse), I see it "3 times",

/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 8, If 2, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 1: Dev 8, If 0, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 1: Dev 8, If 1, Class=Human Interface Device, Driver=usbhid, 12M

Could that be part of the fun here?

[Christian Ehrhardt  (paelzer)]

> LOL. This really is very odd. I'm open to any thoughts you have. Thanks!

Indeed - it might come down to the actual USB devices/Hubs in play and
their exact type and firmware.
OTOH that might also explain why not everyone ever using it complains
about it being dysfunctional but a few do.

> Could that be part of the fun here?

Not uncommon - a single USB device can define multiple "Interfaces" to
support different things at once.
With lsusb -vvv -d <id you got for that dev> you can see what it is -
it can be e.g. a keyboard also registering a mouse for a trackpoint or
such.

P.S. I'm not an USB expert - I know there are even malicious devices
that e.g. have an USB stick also provide Keyboard/Mouse to hijack a
computer. I'd not expect this to be the case here, but you never know
so I wanted to mention.

[Russell Morris (6-u3untu-h)]

Yep, checked the devices (thanks!), and I get,
$ lsusb -v -d 046d:c52b | grep bInterfaceProtocol
      bInterfaceProtocol 1 Keyboard
      bInterfaceProtocol 2 Mouse
      bInterfaceProtocol 0

Pretty much as expected.

Hmm, not sure what else to do - or is it just ... live with it?

Thanks!

[Christian Ehrhardt  (paelzer)]

> Hmm, not sure what else to do - or is it just ... live with it?

You still have the option to report it upstream and hope that someone
in the wider community has seen it before.
Especially now that we already checked plenty of corner cases and
special conditions your report should be more substantial.

[Russell Morris (6-u3untu-h)]

Sure, will do. Just to make sure, this mailing list I assume?
https://lists.nongnu.org/mailman/listinfo/qemu-devel

Thanks!

[Russell Morris (6-u3untu-h)]

FYI, several updates here - for other reasons, but ... BIOS, Linux (Ubuntu), qemu (to master) => now it works on boot. Arrgh! I only say that because I can't duplicate it any more, with all these updates. That's good overall. LOL!

Thanks!

[Christian Ehrhardt  (paelzer)]

Ok, glad to hear that it works now. Were you able to sort out what it was - read this like: "is there anything we need to consider for a fix/backport or are we just happy things work and close it"?

[Russell Morris (6-u3untu-h)]

I really wish I could pinpoint the exact root cause, to be able to feed it back, help others (i.e. fix it once and for all, for everyone :-)). So far though, no luck. I'm still digging, I haven't given up!

Thanks!

[Launchpad Janitor (janitor)]

[Expired for qemu (Ubuntu) because there has been no activity for 60 days.]