Libvirt + Xen cannot boot VMs - TAP device issue

[Vangelis Tasoulas]

This bug is being faced on Ubuntu 12.04/13.04 and OpenStack Grizzly.

I cannot start a fully virtualized VM using OpenStack, libvirt and XEN no matter what I have tried so far. The image I am trying is this one: https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img

After a clean hardware (hypervisor) boot, when I try to run the first virtual machine on my libvirt/XEN compute node I get the following error:

      libvirtError: POST operation failed: xend_post: error from xen daemon: (xend.err 'Device 51952 (tap2) could not be connected. Device not found.')

From the second time and after, trying to run any virtual machine returns this error :

      libvirtError: POST operation failed: xend_post: error from xen daemon: (xend.err 'Device /dev/xvdp (51952, tap) is already connected.')

However the second error looks normal because xenstore-ls shows that /dev/xvdp device is still in use from the previous try... xen-hotplug-cleanup script is not running properly?

Trying to debug this further, I created a simple vm and tried to run it using virt-manager.
The configuration is the same one created by nova-compute, except that I removed the network:

<domain type='xen'>
  <name>instance-00000039</name>
  <uuid>2b19dccb-e033-436c-b2bf-64a4eec77b42</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <bootloader></bootloader>
  <os>
    <type>linux</type>
    <cmdline>root=/dev/sda </cmdline>
  </os>
  <clock offset='utc' adjustment='reset'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/lib/xen-4.2/bin/qemu-dm</emulator>
    <disk type='file' device='disk'>
      <driver name='tap' type='qcow2'/>
      <source file='/var/lib/nova/instances/2b19dccb-e033-436c-b2bf-64a4eec77b42/disk'/>
      <target dev='sda' bus='scsi'/>
    </disk>
    <console type='pty'>
      <target type='xen' port='0'/>
    </console>
    <input type='mouse' bus='xen'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
  </devices>
</domain>

Again with this configuration and manually running this VM with Virt-manager, I face the exact same behaviour described above.

Here you can find some pastebin links with the nova configuration files and logs:
http://pastebin.com/E0EAauq4 (nova.conf)
http://pastebin.com/QKMbBDKD (nova-compute.conf)
http://pastebin.com/XHfpS4WS (nova-compute.log)
http://pastebin.com/H3xJdcCD (libvirtd.log)
http://pastebin.com/m1H6fps6 (xend.log)

[Alvaro Lopez (aloga)]

There are two errors here.

First one is related with https://bugs.launchpad.net/nova/+bug/1188517

The second one is that you're using copy on write (i.e. QCOW2) images with Xen+Libvirt, and it does not work since qcow2 is not supported by blktap2.

[Vangelis Tasoulas (cyberang3l)]

Thanks for the response Alvaro,

I have a couple of questions one this one.

1. I have changed nova/virt/libvirt/utils.py to return always "file" according to this: https://bugs.launchpad.net/nova/+bug/1084618/comments/2

After this change, I was able to boot UEC style images but then I encounter the following problem which I am not sure if it is a bug to be reported, or something I am doing wrong at my side. Please re the question here: https://answers.launchpad.net/nova/+question/231408
Have you managed to run this kind of images without issues on your setup?

2. According to this: https://bugs.launchpad.net/nova/+bug/1084618/comments/4 the change introduced by this: https://bugs.launchpad.net/nova/+bug/1084618/comments/2 (return always "file" instead of "tap") was reverted back because "Guest would not boot successfully, as Xen doesn't think of disk format as qcow2. Instead, 'tap' works with both raw and qcow2 disk.".

So qcow2 is supported by tap but not tap2?