xenapi: openstack not working with xenserver core and vhd images

[guillaume thouvenin]

I installed a xenserver and on this server a PV guest. In this guest I deployed devstack. I'm using the following localrc:

MYSQL_PASSWORD=password
SERVICE_TOKEN=password
ADMIN_PASSWORD=password
RABBIT_PASSWORD=password
GUEST_PASSWORD=password
SERVICE_PASSWORD=password

HOST_IP=10.197.217.86

FLOATING_RANGE=192.168.1.224/27
FIXED_RANGE=10.11.12.0/24
FIXED_NETWORK_SIZE=256
FLAT_INTERFACE=eth1
FLAT_NETWORK_BRIDGE=xapi1

XENAPI_PASSWORD=toto
XENAPI_CONNECTION_URL="http://10.197.217.85"

IMAGE_URLS="\
https://github.com/downloads/citrix-openstack/warehouse/cirros-0.3.0-x86_64-disk.vhd.tgz,\
http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz"

VIRT_DRIVER=xenserver

VNCSERVER_PROXYCLIENT_ADDRESS=10.197.217.85

MULTI_HOST=1
ACTIVE_TIMEOUT=45

When I try to boot a VM, I have an error:

  Failure: ['UUID_INVALID', 'VDI', 'a9162793-5e64-41fb-a018-c281a694fcbf']

Complete log is here: http://paste.openstack.org/show/46313/

It seems that the UUID used is not the good one because if I look the list of VDIs I see:

uuid ( RO) : 3aefa6cc-0d75-e356-f934-1c6fe8af1553
          name-label ( RW): a9162793-5e64-41fb-a018-c281a694fcbf.vhd
    name-description ( RW):
             sr-uuid ( RO): 4362c04b-e140-41f8-bcb5-36eed394576e
        virtual-size ( RO): 42025472
            sharable ( RO): false
           read-only ( RO): false

So the UUID that is used is the name-label and not the real UUID.

[guillaume thouvenin (guillaume-thouvenin)]

The image that I'm using has a VHD disk format and the container has the OVF type. I also tried with a bare container :

$ glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| b7b4761a-2b82-4b75-ba42-927da0898c3d | cirros-0.3.0-x86_64-disk | vhd | ovf | 9220018 | active |
| 4e3462a1-72a4-4bad-a57b-8241c997a3da | cirros-0.3.1-x86_64-uec | ami | ami | 25165824 | active |
| 44ff0a3d-1227-45a2-a53c-0239b0cb9274 | cirros-0.3.1-x86_64-uec-kernel | aki | aki | 4955792 | active |
| 2f4bbc75-db94-48f8-8feb-a62adfa58d07 | cirros-0.3.1-x86_64-uec-ramdisk | ari | ari | 3714968 | active |
| 9b13b4a5-c203-4abf-83e0-3de0ee3926a7 | xenimage | vhd | bare | 42025472 | active |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+

[John Garbutt (johngarbutt)]

@guillaume could you please tell us what type of SR your are using EXT or LVM?

[guillaume thouvenin (guillaume-thouvenin)]

I'm using EXT and the version of xenserver is the one installed with xenserver-tech-preview-release-0.3.0-0.x86_64 (Xen version 4.2.2-2021.el6 )

# xe sr-list uuid=4362c04b-e140-41f8-bcb5-36eed394576e
uuid ( RO) : 4362c04b-e140-41f8-bcb5-36eed394576e
          name-label ( RW): /var/run/sr-mount/4362c04b-e140-41f8-bcb5-36eed394576e
    name-description ( RW): Files stored in /var/run/sr-mount/4362c04b-e140-41f8-bcb5-36eed394576e
                host ( RO): r421-e2-5
                type ( RO): ext
        content-type ( RO): default

and the filesystem is :

# df -T
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/sda3 ext4 51606140 25691052 23293648 53% /
tmpfs tmpfs 277404 4 277400 1% /dev/shm
/dev/sda1 ext4 1032088 127320 852340 13% /boot
/dev/sda4 ext4 907762552 2491248 859159560 1% /var/run/sr-mount/4362c04b-e140-41f8-bcb5-36eed394576e
xenstore tmpfs 277404 72 277332 1% /var/lib/xenstored

In /var/run/sr-mount/4362c04b-e140-41f8-bcb5-36eed394576 I can see the file created when I try to start the instance:

# ll /var/run/sr-mount/4362c04b-e140-41f8-bcb5-36eed394576e/
...
-rw-r--r--. 1 1000 1000 42025472 Dec 10 2012 a9162793-5e64-41fb-a018-c281a694fcbf.vhd
...

[John Garbutt (johngarbutt)]

Ah, interesting, so this is the xenserver tech-preview.

Thanks for the extra info.

[guillaume thouvenin (guillaume-thouvenin)]

Yes it is the xenserver tech-preview and I have the same issue with xenserver-core (xenserver-core-0.9.0-9.x86_64)

[guillaume thouvenin (guillaume-thouvenin)]

I dig a little and there is something that I don't understand. The function _fetch_vhd_image() returns a list of dictionaries that describe VDIs. So for example if I have a disk like this one:

# xe vdi-list
uuid ( RO) : 46addb2a-52a8-0010-4751-ad2e5538440e
          name-label ( RW): 7e65ebc1-ac4a-4c35-b66d-73af3d5451c6.vhd
    name-description ( RW):
             sr-uuid ( RO): 1d0018c1-808b-4205-ba69-3ed1806d6839
        virtual-size ( RO): 42025472
            sharable ( RO): false
           read-only ( RO): false

If the disk is the root of the VM, the corresponding VDIs should be something like vdis={'root': {'uuid': '46addb2a-52a8-0010-4751-ad2e5538440e'}}

When I look the following piece of code in _fetch_vhd_image():

1222
1223 vdis = default_handler.download_image(
1224 context, session, instance, image_id)
1225
1226 sr_ref = safe_find_sr(session)
1227 _scan_sr(session, sr_ref)
1228

At 1225, for my case, the file 7e65ebc1-ac4a-4c35-b66d-73af3d5451c6.vhd has been copied in the SR and vdis = {'root': {'uuid': '7e65ebc1-ac4a-4c35-b66d-73af3d5451c6'}}.

When the SR.scan is ran, it creates the vdi shows previously (with the uuid=46a...). So shouldn't we update the vdis after the scan of the SR?

[guillaume thouvenin (guillaume-thouvenin)]

I still have this error with devastack. This error only occurs when I'm using the VHD file and a HVM guest. I can start an PV guest with an image, a kernel and an ramdisk.

[Bob Ball (bob-ball)]

In that version of xenserver-core the ext SR is actually an ffs (flat file system) SR which is an experimental SR type. I guess that there is a strange interaction between what the dom0 plugins expect to be able to do and how the FFS SR is interpreting it.

Delete that SR and create an FFS SR without telling it to pretend to be ext3.

Recent changes accepted into OpenStack should mean this works much better than it did when we added the pretend-ext3 SR.

[guillaume thouvenin (guillaume-thouvenin)]

I modified the pool so the default SR is now the one with ffs-type:

uuid ( RO) : 106ab117-5431-0931-889b-6c534cd061c6
          name-label ( RW): /usr/share/xapi/images
    name-description ( RW): Files stored in /usr/share/xapi/images
                host ( RO): r421-e3-2
                type ( RO): ffs
        content-type ( RO): default

I tried to start an instance but unfortunatly I have the same issue with the uuid that is the name-label without the vhd extension:

nova.compute.manager [...] Failure: ['UUID_INVALID', 'VDI', 'e6b6c517-ad7e-4e42-82c2-1f7e00858075']

and

xe vdi-list name-label=e6b6c517-ad7e-4e42-82c2-1f7e00858075.vhd
uuid ( RO) : 3d719010-a7a7-dea2-19f8-8441bae2a630
          name-label ( RW): e6b6c517-ad7e-4e42-82c2-1f7e00858075.vhd
    name-description ( RW):
             sr-uuid ( RO): 106ab117-5431-0931-889b-6c534cd061c6
        virtual-size ( RO): 2299062272
            sharable ( RO): false
           read-only ( RO): false

[guillaume thouvenin (guillaume-thouvenin)]

Sorry to insist but I don't see how it can work. I probably miss something but in the function _fetch_vhd_image(), first we download the vhd image in the repository that corresponds to the SR. So after

  vdis = handler.download_image(context, session, instance, image_id)

vdis contains the name of the file. Not the UUID of the vdi that will be created when we will invoke the sr-scan. So at this point we send a request to glance to download the image and it is stored in the directory:

# ls /usr/share/xapi/images
951e5aa6-7ef6-43ad-8edd-79edb4ad428e.vhd

the value of vdis is: vdis = 'root': {'uuid': '951e5aa6-7ef...-79edb4ad428e'}}

and the VHD is not seen yet by the SR in Xen.

  The next step is to scan the SR so we will be able to see the VHD file. The result is that the UUID is attributed to the disk and we can see it now:

# xe vdi-list
 uuid ( RO) : acd06ed8-dacf-7b8d-4f95-c5d835691383
          name-label ( RW): 951e5aa6-7ef6-43ad-8edd-79edb4ad428e.vhd
    name-description ( RW):
             sr-uuid ( RO): 106ab117-5431-0931-889b-6c534cd061c6
        virtual-size ( RO): 2299062272
            sharable ( RO): false
           read-only ( RO): false

but the content of vdis is always the same. So I don't understand how we can retrieve information of a disk wih UUID=951e5aa6-7ef6-43ad-8edd-79edb4ad428e from Xen. The UUID seems invalid.

I tried to go deeper but I'm lost when _call_glance_plugin() is called. Don't hesitate to ask for more information. I can get some trace from pdb or things like that.

[Bob Ball (bob-ball)]

GlanceStore.download_image calls through to dom0 with the "glance" plugin, method "download_vhd".

This method will pull down the VHDs and rename them - which _SHOULD_ mean that the SR detects them with the correct UUID.

In this case, it does not, because there is a bug in the FFS SR which means that even when you have a correctly formatted UUID for the VHD name it will pick a new ID when introducing to XAPI.

The only way to do this with the current release is to use an EXT SR, so create a large file, loop-back mount it, and introduce it as an SR to XenServer - or if you have a spare block device, just use that / set up a new LV for it. Some of the instructions you'd need are in http://support.citrix.com/article/ctx116324 - don't delete the original SR, just create a new one and set it to the pool default.

I've also asked for this bug in FFS to be fixed - see https://github.com/xapi-project/xenserver-core/issues/243

OpenStack can also be fixed, but this is a slightly bigger task -

[guillaume thouvenin (guillaume-thouvenin)]

I tried this solution. I mounted an ext4 filesystem on /opt/main-sr:

xe sr-create content-type=user type=ext device-config:path=/opt/main-sr name-label="Main storage"

where

# df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdb2 ext4 628G 2,4G 594G 1% /opt/main-sr

I also set this new SR as the default SR in the pool. Unfortunatly I still have the same error:

Failure: ['UUID_INVALID', 'VDI', '0eda1484-beba-4425-b7d1-1c2b266d3a0b']

and

# ll /opt/main-sr/
total 2245200
-rw-------. 1 1000 users 2299062272 3 oct. 13:52 0eda1484-beba-4425-b7d1-1c2b266d3a0b.vhd

# xe vdi-list name-label=0eda1484-beba-4425-b7d1-1c2b266d3a0b.vhd
uuid ( RO) : 2738a98c-e3fb-9b16-0a3e-3b7c9df4b69e
          name-label ( RW): 0eda1484-beba-4425-b7d1-1c2b266d3a0b.vhd
    name-description ( RW):
             sr-uuid ( RO): 44bf67ce-7fbf-ed56-45e4-b9195c097b30
        virtual-size ( RO): 2299062272
            sharable ( RO): false
           read-only ( RO): false

[Bob Ball (bob-ball)]

FFS SRs are not going to be supported in OpenStack; EXT3 SRs are now supported in buildroot (formally known as xenserver-core).

We've confirmed that EXT3 SRs are indeed supported under buildroot at least from the Juno release of OpenStack.