virt-manager / libvirt fails to create VM due to PCI ordering bug

[Joel Wirāmu Pauling (aenertia)]

Hi there;

libvirt in trusty has a bug where VM's fail to get defined due to PCI bus ordering bugs.

Redhat bug tracker info here: https://bugzilla.redhat.com/show_bug.cgi?id=823535

Appears to be the same bug.

Steps to reproduce:

* Open Virt-manager
* Define a new VM
( I used reactos installer iso)
* Attempt to run.

Get something like this:
"

Unable to complete install: 'internal error: process exited while connecting to monitor: qemu-system-x86_64: -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f4:4b:50,bus=pci.0,addr=0x1: PCI: slot 1 function 0 not available for rtl8139, in use by PIIX3
qemu-system-x86_64: -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f4:4b:50,bus=pci.0,addr=0x1: Device initialization failed.
qemu-system-x86_64: -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f4:4b:50,bus=pci.0,addr=0x1: Device 'rtl8139' could not be initialized
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1983, in do_install
    guest.start_install(False, meter=meter)
  File "/usr/lib/python2.7/dist-packages/virtinst/Guest.py", line 1246, in start_install
    noboot)
  File "/usr/lib/python2.7/dist-packages/virtinst/Guest.py", line 1314, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3199, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f4:4b:50,bus=pci.0,addr=0x1: PCI: slot 1 function 0 not available for rtl8139, in use by PIIX3
qemu-system-x86_64: -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f4:4b:50,bus=pci.0,addr=0x1: Device initialization failed.
qemu-system-x86_64: -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f4:4b:50,bus=pci.0,addr=0x1: Device 'rtl8139' could not be initialized

"

Attempted resolutions:

Removing Devices/reordering them via virt-manager still fails to solve the issue ( you manually need to hack at the .xml definition).

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: libvirt-bin 1.2.2-0ubuntu12
ProcVersionSignature: Ubuntu 3.13.0-23.45-generic 3.13.8
Uname: Linux 3.13.0-23-generic x86_64
ApportVersion: 2.14.1-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Sat Apr 12 17:11:24 2014
InstallationDate: Installed on 2014-03-31 (11 days ago)
InstallationMedia: Ubuntu-GNOME 14.04 LTS "Trusty Tahr" - Daily amd64 (20140330)
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission denied: '/etc/libvirt/qemu.conf']

[Oved (oved-redhat-bugs)]

Description of problem:

Looks like libvirt is sensitive to the order in which the video devices are passed.
Passing them in a wrong order causes the error below.

Version-Release number of selected component (if applicable):

How reproducible:
Always

Steps to Reproduce:
Tried with two VMs, one in which the order was:
<video>
 <address domain="0x0000" function="0x0" slot="0x07" type="pci" bus="0x00"/>
 <model heads="1" type="qxl" vram="65536"/>
</video>
<video>
 <address domain="0x0000" function="0x0" slot="0x02" type="pci" bus="0x00"/>
 <model heads="1" type="qxl" vram="65536"/>
</video>

Failed on (from libvirt.log):
error : virNetClientProgramDispatchError:174 : internal error Primary video card must have PCI address 0:0:2.0

In the other VM the order was:

<video>
 <address domain="0x0000" function="0x0" slot="0x02" type="pci" bus="0x00"/>
 <model heads="1" type="qxl" vram="65536"/>
</video>
<video>
 <address domain="0x0000" function="0x0" slot="0x07" type="pci" bus="0x00"/>
 <model heads="1" type="qxl" vram="65536"/>
</video>

And it succeeded.

Actual results:
The VM in which the order is wrong fails with error:
error : virNetClientProgramDispatchError:174 : internal error Primary video card must have PCI address 0:0:2.0

Expected results:
Both VMs should start correctly.

[Gunannan (gunannan-redhat-bugs)]

New QEMU option -device permits specification of PCI device * address. It doesn't work for VGA devices yet, but QEMU always assigns PCI device address 2 to the VGA.
https://fedoraproject.org/wiki/Features/KVM_Stable_PCI_Addresses

For libvirt, there are hardcoded slot values for the first PCI devices like
 * - Host bridge (slot 0)
 * - PIIX3 ISA bridge, IDE controller, something else unknown, USB controller (slot 1)
 * - Video (slot 2)

So the error message "Primary video card must have PCI address 0:0:2.0" makes sense. Either you assign slot="0x02" to the primary video device or remove <address> line and let libvirt assign it automatically.

[Dave (dave-redhat-bugs)]

Guannan, does qemu now permit the configuration of the video card address, or is there a qemu BZ to make the video card address configurable?

[Gunannan (gunannan-redhat-bugs)]

The i440FX(motherboard), PIIX3, cirrus-vga, rtl8139(or e1000) are qemu built-in pci device, they are allocated pci address from 0x0 to 0x3 in turn by qemu.
libvirt makes sure the assignement of their pci address is valid by users based on this order.
Currently, qemu-kvm doesn't support the address configuration of these built-in PCI devices.

For example, when I try to populate a vga device in 0x0, it will report error as follows:

# /usr/bin/qemu-kvm -m 1024 -smp 1 -name "rhel" -drive file=/var/lib/libvirt/images/rhel6u3qcow2.img \
-device qxl,id=video1,vram_size=67108864,bus=pci.0,addr=0x0

qemu-kvm: -device qxl,id=video1,vram_size=67108864,bus=pci.0,addr=0x0: PCI: slot 0 function 0 not available for qxl, in use by i440FX

The same errors for the rest of other three address(0x1, 0x2, 0x3)
PCI: slot 1 function 0 not available for qxl, in use by PIIX3
PCI: slot 2 function 0 not available for qxl, in use by cirrus-vga
PCI: slot 3 function 0 not available for qxl, in use by rtl8139

About if the qemu will support such configuration of pci address, I have pinged qemu-kvm guys waiting for repsonce.

[Gerd (gerd-redhat-bugs)]

i440FX is fixed at slot 0.
PIIX3 is fixed at slot 1.

That isn't going to change ever, these are parts of the emulated chipset.

vga is fixed at slot 2 in rhel-6. You have the option to run without vga though (-vga none), then assign slot 2 to something else.

Newer qemu versions (qemu 1.2+ IIRC, will work for sure in rhel-7) can create vga devices using -device, which allows to put them into another slot using the addr property, i.e. '-vga none -device qxl-vga,addr=7'.

There is nothing fixed in slot 3. The NIC just ends up there because slot 3 is the first free slot and if not specified otherwise qemu will assign the first free slot to devices you add. You can change the nic slot number in the libvirt xml to something else and it will work just fine.

[Gunannan (gunannan-redhat-bugs)]

Thanks for above infomation, in my box, '-vga none' and '-net none' could explicitly remove built-in VGA and NIC devices. then, we could use '-device qxl-vga' instead for putting VGA device into another available slot.

[Gerd (gerd-redhat-bugs)]

Same goes for all vga devices, not only qxl.

'-vga std' maps to '-device VGA'
'-vga cirrus' maps to '-device cirrus-vga'
'-vga qxl' maps to '-device qxl-vga'
             (there is also '-device qxl' for secondary devices)
'-vga vmware' maps to '-device vmware-svga'

Remember that you have to keep using the -vga switch for older qemu versions.

Also note that '-M isapc' uses different vga devices too (isa-vga + isa-cirrus-vga), it is probably easiest to keep using -vga for isapc.

[Gunannan (gunannan-redhat-bugs)]

patch sent out to upstreamhttps://www.redhat.com/archives/libvir-list/2012-December/msg00406.html

[Gunannan (gunannan-redhat-bugs)]

For this request of non-fixed PCI address for video device, you have to use qemu which version is equal or greater than 1.2, libvirt >= 1.0.2

commit aa51202b723ac6a2b7b99827abf60a7593afcba6
Author: Guannan Ren <email address hidden>
Date: Fri Dec 14 15:08:01 2012 +0800

    qemu: use newer -device video device in qemu commandline

    '-device VGA' maps to '-vga std'
    '-device cirrus-vga' maps to '-vga cirrus'
    '-device qxl-vga' maps to '-vga qxl'
                 (there is also '-device qxl' for secondary devices)
    '-device vmware-svga' maps to '-vga vmware'

    For qemu(>=1.2), we can use -device to replace -vga for video
    device. For the primary video device, the patch tries to use 0x2
    slot for matching old qemu. If the 0x2 slot is allocated already,
    the addr property could help for using any available slot.
    For qemu(< 1.2), we keep using -vga for primary device.

[zhe (zhe-redhat-bugs)]

I can reproduce this with libvirt-1.0.1-1.el7.x86_64

[zhe (zhe-redhat-bugs)]

verify with build:
libvirt-1.1.0-1.el7.x86_64
qemu-kvm-1.5.0-2.el7.x86_64
kernel-3.10.0-0.rc7.64.el7.x86_64

#virsh dumpxml $guest
....
 <video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <alias name='video1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <alias name='video2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

....

# virsh start spice
Domain spice started

#ps -ef | grep qemu-kvm
....
-spice port=5900,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=9437184,bus=pci.0,addr=0x7 -device qxl,id=video1,ram_size=67108864,vram_size=9437184,bus=pci.0,addr=0x8 -device qxl,id=video2,ram_size=67108864,vram_size=9437184,bus=pci.0,addr=0x2
....

verification passed.

[Xuesong (xuesong-redhat-bugs)]

After update the libvirt build to libvirt-1.1.1-1.el7.x86_64, it seems this bug is appeared again in the latest libvirt build. So change the bug status to assigned again.

Version-Release number of selected component (if applicable):
libvirt-1.1.1-1.el7.x86_64
qemu-kvm-1.5.2-1.el7.x86_64
3.10.0-0.rc7.64.el7.x86_64

How reproducible:
100%

Steps:
1. prepare one health guest with the following xml, on one host which the libvirt version is old.

old version build:
libvirt-1.1.0-1.el7.x86_64
qemu-kvm-1.5.0-2.el7.x86_64
kernel-3.10.0-0.rc7.64.el7.x86_64

#virsh dumpxml $guest
....
 <video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <alias name='video1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <alias name='video2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

....

2. make sure the guest can be started normally.
# virsh start a
Domain a started

3. update the libvirt to the latest version, and restart the libvirtd service. Then start the same guest again.
# virsh start a
error: Failed to start domain a
error: internal error: Primary video card must have PCI address 0:0:2.0

Actual results:
The guest failed to be started up in step3.

Expected results:
The guest should be started up in step3.

[Gerd (gerd-redhat-bugs)]

That is intentional. There is a qemu bug (fixed in upcoming 1.6) and disabling this feature (for qxl) is the workaround for it. It should continue working for other vga cards. /me also submitted the qemu fix to stable, so it should show up in 1.5.3 and thereby find it's way into rhel7.

[CongDong (congdong-redhat-bugs)]

Still can reproduce this on rhel7 with:
qemu-kvm-1.5.3-30.el7.x86_64
libvirt-1.1.1-16.el7.x86_64

step:
1. virsh edit $vm
...
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
...
2. Save xml
:wq
error: internal error: Primary video card must have PCI address 0:0:2.0
Failed. Try again? [y,n,f,?]:

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug.

[Serge Hallyn (serge-hallyn)]

Note that the qemu version mentioned in the redhat bug as fixing this is far older than what is in trusty. So if that fix did make it in, then the libvirt fix should be safe to put into trusty. I'd prefer to wait until there is a patch on the libvirt mailing list (or in git) which we can cherrypick for an SRU.

[Luyao (luyao-redhat-bugs)]

Verify this bug with libvirt-1.2.8-7 and qemu-kvm-rhev-2.1.2-6.el7.x86_64:

# virsh edit test5

    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

# virsh start test5
Domain test5 started

# virsh dumpxml test5

    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <alias name='video1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

# ps aux|grep qemu|grep test5|grep qxl
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x7 -device qxl,id=video1,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2

And also test with qemu-kvm still report error, but this is expected.

# virsh edit r6
error: internal error: Primary video card must have PCI address 0:0:2.0
Failed. Try again? [y,n,f,?]:

[errata-xmlrpc (errata-xmlrpc-redhat-bugs)]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html

[Christian Ehrhardt  (paelzer)]

This went upstream, and thereby into Ubuntu a bit later a long time ago. Marking tasks accordingly, no SRUs planned atm.