could not add physical hardware (ethernet card)

[Michael Zoet]

I am running KVM on a Dell PowerEdge R510 server with 10.04.2. The server has 2 Intel quad port Ethernet cards and 2 Broadcom NICs.

root@kvm01:~# uname -a
Linux kvm01 2.6.32-30-server #59-Ubuntu SMP Tue Mar 1 22:46:09 UTC 2011 x86_64 GNU/Linux
root@kvm01:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04.2 LTS
Release: 10.04
Codename: lucid

I disabled apparmor!

When I try to add a physical network card (eth6 for example) to a VM I get the following error messages:

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 588, in run_domain
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 150, in startup
    self._backend.create()
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 300, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: monitor socket did not show up.: Connection refused

/var/log/libvirt/qemu/VM01.log:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 1 -name VM01 -uuid 6605d7ea-8a19-1cd0-89a1-9ad29d3a6ec8 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/VM01.monitor,server,nowait -monitor chardev:monitor -localtime -boot c -drive file=/storage/vms/VirtualMachines/VM01.img,if=virtio,index=0,boot=on,format=qcow2 -drive if=ide,media=cdrom,index=2,format=raw -net nic,macaddr=52:54:00:06:a3:e4,vlan=0,model=virtio,name=virtio.0 -net tap,fd=57,vlan=0,name=tap.1 -net nic,macaddr=52:54:00:24:99:45,vlan=1,name=nic.0 -net tap,fd=58,vlan=1,name=tap.1 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 127.0.0.1:2 -vga cirrus -pcidevice host=0b:00.1
char device redirected to /dev/pts/3
pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
pci_add_option_rom: failed to find romfile "pxe-rtl8139.bin"
device: 0b:00.1: driver="pci-assign" host="0b:00.1"
No IOMMU found. Unable to assign device "0b:00.1"
Failed to deassign device "0b:00.1" : Invalid argument
Error initializing device pci-assign

Without the physical hardware the VM starts normally. Assigning USB (storage) devices as physical devices also works fine.

I am adding the physical hardware via virt-manager in the corresponding menus.

[Nan (nan-redhat-bugs)]

Description of problem:
NIC passthrough failed during new VM installing in virt-manager.

Version-Release number of selected component (if applicable):
virt-manager-0.8.4-4.el6.noarch

How reproducible:
always

Steps to Reproduce:
1. Install a new VM and passthrough NIC during install wizard.
2.
3.

Actual results:
Unable to complete install '<class 'libvirt.libvirtError'> internal error
Process exited while reading console log output: char device redirected to
/dev/pts/1
No IOMMU found. Unable to assign device "hostdev0"
Failed to deassign device "hostdev0" : Invalid argument
qemu-kvm: -device
pci-assign,host=02:00.0,id=hostdev0,configfd=23,bus=pci.0,addr=0x7: Device
'pci-assign' could not be initialized

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/create.py", line 1559, in
do_install
    dom = guest.start_install(False, meter = meter)
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line 972, in
start_install
    return self._do_install(consolecb, meter, removeOld, wait)
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line 1037, in
_do_install
    "install")
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line 1008, in
_create_guest
    dom = self.conn.createLinux(start_xml, 0)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1270, in
createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed',
conn=self)
libvirtError: internal error Process exited while reading console log output:
char device redirected to /dev/pts/1
No IOMMU found. Unable to assign device "hostdev0"
Failed to deassign device "hostdev0" : Invalid argument
qemu-kvm: -device
pci-assign,host=02:00.0,id=hostdev0,configfd=23,bus=pci.0,addr=0x7: Device
'pci-assign' could not be initialized

'

Expected results:
NIC can be passthrough to guest in new vm wizard.

Additional info:

[RHEL (rhel-redhat-bugs)]

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release. Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release. This request is not yet committed for
inclusion.

[Chris (chris-redhat-bugs)]

This line:

No IOMMU found. Unable to assign device "hostdev0"

seems to suggest that your hardware doesn't have an IOMMU, or it is not configured correctly. What kind of hardware (chipset, in particular), do you have? Also, what is the output from "cat /proc/cmdline" on the host? Finally, can you give us the output from "dmesg"?

Thanks,
Chris Lalancette

[Bill (bill-redhat-bugs)]

Would we want to disable the UI feature if the IOMMU isn't there?

[Nan (nan-redhat-bugs)]

Created attachment 426228
dmesg log

I've set VT-d with ON in bios.

# lspci
00:00.0 Host bridge: Intel Corporation 82Q35 Express DRAM Controller (rev 02)
00:01.0 PCI bridge: Intel Corporation 82Q35 Express PCI Express Root Port (rev 02)
00:03.0 Communication controller: Intel Corporation 82Q35 Express MEI Controller (rev 02)
00:03.2 IDE interface: Intel Corporation 82Q35 Express PT IDER Controller (rev 02)
00:03.3 Serial controller: Intel Corporation 82Q35 Express Serial KT Controller (rev 02)
00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connection (rev 02)
00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IO (ICH9DO) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 VGA compatible controller: ATI Technologies Inc RV610 video device [Radeon HD 2400 PRO]
02:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)

# cat /proc/cmdline
ro root=UUID=d6748640-f848-40fd-a1c3-c57d91fbe84f rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=129M@0M rhgb quiet

[Cole (cole-redhat-bugs)]

(In reply to comment #4)
> Would we want to disable the UI feature if the IOMMU isn't there?

Possibly, we would need that information exported by libvirt though. Definitely would be useful, we could at least show a warning 'Hey, your machine probably doesn't support this'. No idea what detecting it would entail

Anyway, moving to libvirt, which is closer to the source of the issue. Chris, Neil provided the info you asked for in comment #5

[Dave (dave-redhat-bugs)]

This BZ is really a request for more info from libvirt, so moving to 6.1

[Michael Zoet (michazoet-deactivatedaccount)]

Forgot the corresponding dmesg entries:

[ 1921.436178] igb 0000:0b:00.1: PCI INT B disabled
[ 1921.436234] pci-stub 0000:0b:00.1: claimed by stub
[ 1921.437855] device vnet2 entered promiscuous mode
[ 1921.439643] virbr_man: port 2(vnet2) entering forwarding state
[ 1921.441193] device vnet3 entered promiscuous mode
[ 1921.442963] virbr1: topology change detected, propagating
[ 1921.442967] virbr1: port 1(vnet3) entering forwarding state
[ 1921.467693] kvm_vm_ioctl_deassign_device: device hasn't been assigned before, so cannot be deassigned
[ 1921.672335] virbr_man: port 2(vnet2) entering disabled state
[ 1921.750310] device vnet2 left promiscuous mode
[ 1921.750314] virbr_man: port 2(vnet2) entering disabled state
[ 1921.991886] virbr1: port 1(vnet3) entering disabled state
[ 1922.110162] device vnet3 left promiscuous mode
[ 1922.110167] virbr1: port 1(vnet3) entering disabled state
[ 1951.656102] igb 0000:0b:00.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
[ 1951.656135] igb 0000:0b:00.1: setting latency timer to 64
[ 1951.657425] igb 0000:0b:00.1: irq 128 for MSI/MSI-X
[ 1951.657428] igb 0000:0b:00.1: irq 129 for MSI/MSI-X
[ 1951.657432] igb 0000:0b:00.1: irq 130 for MSI/MSI-X
[ 1951.657435] igb 0000:0b:00.1: irq 131 for MSI/MSI-X
[ 1951.657438] igb 0000:0b:00.1: irq 132 for MSI/MSI-X
[ 1951.657441] igb 0000:0b:00.1: irq 133 for MSI/MSI-X
[ 1951.657444] igb 0000:0b:00.1: irq 134 for MSI/MSI-X
[ 1951.657447] igb 0000:0b:00.1: irq 135 for MSI/MSI-X
[ 1951.657450] igb 0000:0b:00.1: irq 136 for MSI/MSI-X
[ 1951.657507] igb 0000:0b:00.1: 0 vfs allocated
[ 1951.888296] igb 0000:0b:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 1951.888301] igb 0000:0b:00.1: eth9: (PCIe:2.5Gb/s:Width x4) 00:1b:21:7b:ff:8d
[ 1951.888385] igb 0000:0b:00.1: eth9: PBA No: e66339-003
[ 1951.888388] igb 0000:0b:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug and helping to make Ubuntu better.

I notice the error messages include:

 No IOMMU found.

Did you boot with 'intel_iommu=on' in the boot arguments? If not, please add that and try again, and let us know whether that worked.

[Michael Zoet (michazoet-deactivatedaccount)]

Hi Serge,

I tried it with 'intel_iommu=on' kernel option but with the same result. It does not work.

Does physical hardware assignment for network cards really work under KVM?

Below my complete dmesg output shortly after I tried to start a VM with a physical hardware assignment.

Bye,

Michael

[Serge Hallyn (serge-hallyn)]

> I tried it with 'intel_iommu=on' kernel option but with the same result.
> It does not work.

Drat. Thanks.

> Does physical hardware assignment for network cards really work under
> KVM?

I've never had hardware to be able to properly test this. As soon as I
did, I was planning to test using:

http://fedoraproject.org/wiki/Features/SR-IOV#How_To_Test

Can you verify that your NIC is in fact supported, and that you followed
the steps there (in particular using 'max_vfs=7' when loading the igb
module)?

[Suzanne (suzanne-redhat-bugs)]

Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as an
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

[Don (don-redhat-bugs)]

First, 'intel_iommu=on' has to be added to the kernel cmdline.
Secondly, libvirt gives a message what the error is: no IOMMU.
Third, the Virtualization Guide states that for Intel-based machines with IOMMU support (a requirement for passthrough/device-assignment), you have to add 'intel_iommu=on' to the kernel cmdline.

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]

[Alexander List (alexlist)]

Confirmed on 12.04. Enabled IOMMU, now virt-manager complains:

Error starting domain: internal error Process exited while reading console log output: char device redirected to /dev/pts/2
Failed to assign device "hostdev0" : Operation not permitted
kvm: -device pci-assign,host=02:00.0,id=hostdev0,configfd=20,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 45, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 66, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1120, in startup
    self._backend.create()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 551, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error Process exited while reading console log output: char device redirected to /dev/pts/2
Failed to assign device "hostdev0" : Operation not permitted
kvm: -device pci-assign,host=02:00.0,id=hostdev0,configfd=20,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized

On a side node, virt-manager should most likely be modified to disable/grey out certain features if IOMMU is disabled.

[Alexander List (alexlist)]

The permission problem is

https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/545795 or
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/639712

Remaining issue is the requirement to enable IOMMU resp. disabling features that lack required kernel support ...