PCI Pass Through via libvirt cannot remap IRQ's

Bug #639712 reported by flashydave on 2010-09-15
46
This bug affects 9 people
Affects Status Importance Assigned to Milestone
Linux
Invalid
Undecided
Unassigned
libvirt (Ubuntu)
Medium
Unassigned
qemu-kvm (Ubuntu)
Medium
Unassigned

Bug Description

Trying to allocate a hardware device (telecoms card) using virt-manager results in errors - most frequently (but not exclusively)
a) Errors claiming IOMMU cannot be found even though the hardware is VT-x and VT-d capable and BIOS has it enabled
b) Failed to assign irq for "hostdev0": Operation not permitted
Perhaps you are assigning a device that shares an IRQ with another device?
and
c) PCI region 0 at address 0xe5200000 has size 0x100, which is not a multiple of 4K

Tried with versions of Lucid and Maverick. Currently using 0.8.3-1ubuntu9

Investigation showed that
i) The kernel does not have DMAR enabled even on Maverick Server.
ii) apparmor profile does not have the right settings
iii) Updates in qemu-kvm git repository which handle small I/O windows havent yet been incorporated in a release

Investigation showed there to be several problems (which perhaps could gave been documented somewhere)
To fix this successfully it was necessary for me to undertake the following:
a) recompiled the kernel with
CONFIG_DMAR=y
CONFIG_DMAR_DEFAULT_ON=y
CONFIG_GART_IOMMU=y

b) recompiled qemu-kvm from git source
c) Added the following to /etc/apparmor.d/abstractions/libvirt-qemu
# required for pci passthru
capability sys_rawio,

# For hostdev access.
  /sys/devices/ r,
  /sys/devices/** r,

You could argue that the small PCI Memory problem was very card specific but given that Ubuntu claims the Server version is ready for virtualisation under KVM and that technology supports VT-d and PCI Pass Through there should be a suitable kernel version and apparmor profile that supports it available.

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: libvirt-bin 0.8.3-1ubuntu9 [modified: usr/bin/virsh usr/sbin/libvirtd usr/share/libvirt/schemas/domain.rng usr/share/libvirt/schemas/storagepool.rng]
ProcVersionSignature: Ubuntu 2.6.35-19.28-dmar 2.6.35.3
Uname: Linux 2.6.35-19-dmar x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Wed Sep 15 16:49:55 2010
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Alpha amd64 (20100803.1)
ProcEnviron:
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
SourcePackage: libvirt

CVE References

Serge Hallyn (serge-hallyn) wrote :

Thanks for reporting this bug and helping to make Ubuntu better.

So the first step, it sounds like, would be to pursue enabling the CONFIGs you need
in Ubuntu's kernel. There may be a specific reason why it isn't yet, but if not we can
see about requesting that it be enabled in for Natty.

Yes. That aspect has been noted already in Bug #552311. Depending on how we deal with the various components of this issue it might be sensible to track the config changes under that reference.

Scott Moser (smoser) wrote :

I'm marking this 'Linux' (upstream) task as based on the bug info. Additionally, Even for Ubuntu project 'linux' changes would be covered under bug 552311

Changed in linux:
status: New → Invalid
Changed in libvirt (Ubuntu):
importance: Undecided → Medium
status: New → Triaged
Changed in qemu-kvm (Ubuntu):
importance: Undecided → Medium
status: New → Triaged
netzwanze (netzwanze) wrote :

I have the same Problem. I will use a 1394 device.

James Neave (roboj1m) wrote :

Hi,

Looks like this is also affecting me. :(

I'm trying to pass a PCI tuner through to a virtual machine, I have an AMD 890FX based board with IOMMU and it's switched on in the BIOS.
Running Ubuntu server 10.10 x64.

Flashydave: It sounds like you've actually got this working, have you written a guide on how to to it by any chance?

/var/log/libvirt/qemu/test.log:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 512 -smp 3,sockets=3,cores=1,threads=1 -name test -uuid 8bda983a-8f07-80ac-da02-2c7298637d96 -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device virtio-net-pci,vlan=0,id=net0,mac=52:54:00:dd:10:7d,bus=pci.0,addr=0x3 -net tap,fd=56,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device pci-assign,host=08:06.0,id=hostdev0,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/1
pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
Failed to assign device "hostdev0" : Device or resource busy
Failed to deassign device "hostdev0" : Invalid argument
Error initializing device pci-assign

James Neave (roboj1m) wrote :

OK, I've tried the following:

1) Recompile kernel with all the same options except enabling DMAR.
2) Edit the apparmour profile to match suggested by flashydave
3) Installed latest qemu-kvm from kvm edgers ppa: https://launchpad.net/~ubuntu-server-edgers/+archive/server-edgers-qemu-kvm

Attached is the dmesg and log output.

James Neave (roboj1m) wrote :

Here is the test.log

James Neave (roboj1m) wrote :

These commands:

echo "1106 3038" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:08:06.0 > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
echo 0000:08:06.0 > /sys/bus/pci/drivers/pci-stub/bind

From here:

http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Did nothing.

However earlier I had the error about 4k pages (although that was before I installed the latest qemu-kvm package)

Going to try after a reboot.

J.

James Neave (roboj1m) wrote :

Reboot did nothing, game over for now.

The driver 'pci-stub' is occupying your device

I thought pci-stub taking the device is what it was supposed to do?

I'm trying with 12.04 LTS KVM host and guest.

RHEL 6 documentation says: {{ Secure device assignment also requires interrupt remapping support. If a platform does not support interrupt remapping, device assignment will fail. To use device assignment without interrupt remapping support in a development environment, set the allow_unsafe_assigned_interrupts KVM module parameter to 1. }}

and I get in dmesg:

[ 5556.500176] kvm_iommu_map_guest: No interrupt remapping support, disallowing device assignment. Re-enble with "allow_unsafe_assigned_interrupts=1" module option.

Easy to create
/etc/modprobe.d/kvm.conf containing

options kvm allow_unsafe_assigned_interrupts=1

However, there is CVE-2011-1898 which this makes your system vulnerable to see:

https://bugzilla.redhat.com/show_bug.cgi?id=715555

allow_unsafe_assigned_interrupts=1 does work:

$ grep QEMU /proc/cpuinfo
model name : QEMU Virtual CPU version 1.0
model name : QEMU Virtual CPU version 1.0

$ lspci | grep Fun
00:05.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

Chris J Arges (arges) wrote :

I was able to assign a PCI device to a VM via PCI pass through/vt-d with Wily/4.2 after adding 'intel_iommu=on' to the kernel cmdline. Can you please re-test with this option and the latest kernel to see if this works now?
Thanks

Changed in libvirt (Ubuntu):
status: Triaged → Incomplete
Changed in qemu-kvm (Ubuntu):
status: Triaged → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.