VMware: Sparse glance vmdk's size property is mistaken for capacity

[Vui Lam]

Scenario:

a sparse vmdk whose file size is 800MB and whose capacity is 4GB is uploaded to glance without specifying the size property.
(glance uses the file's size for the size property in this case)

nova boot with flavor tiny (root disk size of 1GB) said image.

Result:
The vmwareapi driver fails to spawn the VM because the ESX server throws a fault when asked to 'grow' the disk from 4GB to 1GB (driver thinks it is attempt to grow from 800MB to 1GB)

Relevant hostd.log on ESX host:
2013-10-15T17:02:24.509Z [35BDDB90 verbose 'Default'
opID=HB-host-22@3170-d82e35d0-80] ha-license-manager:Validate -> Valid
evaluation detected for "VMware ESX Server 5.0" (lastError=2,
desc.IsValid:Yes)
2013-10-15T17:02:25.129Z [FFBE3D20 info 'Vimsvc.TaskManager'
opID=a3057d82-8e] Task Created :
haTask--vim.VirtualDiskManager.extendVirtualDisk-526626761

2013-10-15T17:02:25.158Z [35740B90 warning 'Vdisksvc' opID=a3057d82-8e]
New capacity (2097152) is not greater than original capacity (8388608).

I am still not exactly sure if this is consider user error on glance import, a glance shortcoming of not introspecting the vmdk, or a bug in the compute driver. Regardless, this bug is to track any potential defensive code we can add to the driver to better handle this scenario.

[Vui Lam (vui)]

For now, sparse image can be used if one of the following is true:

1. the selected flavor to boot with has a configured root disk capacity of 0 or >= capacity of the sparse image.
2. the capacity of the vmdk is what is set as the the size of the image.
(Note: size of sparse vmdk != capacity of vmdk). The capacity of the vmdk should be known a priori, or
inferrable by examining the embedded header of the sparse vmdk.

e.g.

> head -20 precise.vmdk | grep SPARSE
"RW 8388608" SPARSE "precise.vmdk"' in the header of the precise.vmdk,
disk header is reporting 8388608 sectors (of 512 bytes each), for a total of 4GB.

The glance image's size property can therefore be updated like so:
glance image-update --size 4294967296 _image_id_

3. correct the glance image's size to be the capacity of the disk, which is 4294967296.
(glance image-update --size 4294967296 image_id)

[Tracy Jones (tjones-i)]

when i try to convert the size I get the following error

[~/devstack/files/images (master)⚡] ➔ glance image-update --size 2147483648 85d4e127-8dc3-4cc7-8ce0-4c4a7a2bed84
Request returned failure status.
403 Forbidden
Forbidden to modify 'size' of active image.
    (HTTP 403)

[Tracy Jones (tjones-i)]

just asked this question on irc and the answer is

tjones: we cannot resize images
tjones: you have to boot a server and then resize it

[Vui Lam (vui)]

Recent change to make size immutable (except for administrators) is the reason.

commit 9ae9c09e03d68338ac7ec342982489db3f5f54ab
Author: Stuart McLaren <email address hidden>
Date: Wed Aug 7 06:22:12 2013

    Make size/checksum immutable for active images

    Once an image becomes 'active' it should only be possible for an
    administrator (and not a regular user) to change the size and checksum
    values.

    Addresses bug 1209148.

    Change-Id: Ic65afbca4a4161dfa77ce0a296a1bd72d96902ca

[Tracy Jones (tjones-i)]

vui - what is the issue that needs to be solved here?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/81676

[Joe Gordon (jogo)]

this bug cannot be fixed without the glance v2 API.

[Joe Gordon (jogo)]

and adding glance v2 API is a blueprint

[Radoslav Gerganov (rgerganov)]

We can workaround the problem in the Nova driver by updating the image size after fetching the VMDK on the datastore. This problem is affecting also streamOptimized and OVA images and we need to have workaround until this is fixed in Glance.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/168013

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/168013
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2f15d304f01a94a788db4175de3905bdd3a3cc62
Submitter: Jenkins
Branch: master

commit 2f15d304f01a94a788db4175de3905bdd3a3cc62
Author: Radoslav Gerganov <email address hidden>
Date: Thu Mar 26 16:22:30 2015 +0200

    VMware: Use virtual disk size instead of image size

    The image size is different from the virtual disk size when the image is
    sparse. For sparse images we get the size of the virtual disk by quering
    the -flat file after converting the disk to preallocated.

    Co-Authored-By: Qin Zhao <email address hidden>

    Change-Id: I18628fe3a180f722936c680c6be30d5ed4d50bd9
    Closes-Bug: #1240373
    Closes-Bug: #1472955

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/252010

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/258020

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/168024
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bfc5edc405138e3412ae1af01abbf098c319c77c
Submitter: Jenkins
Branch: master

commit bfc5edc405138e3412ae1af01abbf098c319c77c
Author: Radoslav Gerganov <email address hidden>
Date: Thu Mar 26 16:54:05 2015 +0200

    VMware: Handle image size correctly for OVA and streamOptimized images

    The image size is different from the virtual disk size when the image is
    streamOptimized or OVA. In this case we need to use the size of the
    virtual disk which belongs to the temporary VM created by ImportVApp.
    This works for both vSAN and VMFS datastores.

    Related-Bug: #1240373
    Related-Bug: #1472955

    Change-Id: Ie7df9a78632ff01453b07ee9fed3dffdd8b4a0c7

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/277990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277990
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2c6c67422e0e9dac5e4bd0b61085d3ca1b32e789
Submitter: Jenkins
Branch: stable/liberty

commit 2c6c67422e0e9dac5e4bd0b61085d3ca1b32e789
Author: Radoslav Gerganov <email address hidden>
Date: Thu Mar 26 16:54:05 2015 +0200

    VMware: Handle image size correctly for OVA and streamOptimized images

    The image size is different from the virtual disk size when the image is
    streamOptimized or OVA. In this case we need to use the size of the
    virtual disk which belongs to the temporary VM created by ImportVApp.
    This works for both vSAN and VMFS datastores.

    (cherry picked from commit bfc5edc405138e3412ae1af01abbf098c319c77c)

    Related-Bug: #1240373
    Related-Bug: #1472955
    Change-Id: Ie7df9a78632ff01453b07ee9fed3dffdd8b4a0c7

[Chinmaya Bharadwaj (acbharadwaj)]

@Vui Lam : Im getting the same error when booting an "ova" image, in stable-liberty branch. So I have the above mentioned fixes.

39:54.602Z [332C2B70 warning 'Vdisksvc' opID=412bdb89-70 user=vpxuser] New capacity (83886080) is not greater than original capacity (125829120).
-->
2014-08-18T06:39:54.602Z [332C2B70 warning 'Vdisksvc' opID=412bdb89-70 user=vpxuser] Got Methodfault exception: vmodl.fault.InvalidArgument
2014-08-18T06:39:54.602Z [332C2B70 info 'Vimsvc.TaskManager' opID=412bdb89-70 user=vpxuser] Task Completed : haTask--vim.VirtualDiskManager.extendVirtualDisk-351559722 Status error
2014-08-18T06:40:02.012Z [35A80B70 verbose 'SoapAdapter'] Responded to service state request
2014-08-18T06:40:07.619Z [34CC1B70 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {

After the stream optimized image is copied , image_info.file_size is giving the below value.
 > 1286250496 (1.19 gb) which is the size of the ova.

because in my case the image is already existing in _base directory ?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252010
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8a9f711c082e564fb5ac2f983a4616652f5b4d3d
Submitter: Jenkins
Branch: stable/kilo

commit 8a9f711c082e564fb5ac2f983a4616652f5b4d3d
Author: Radoslav Gerganov <email address hidden>
Date: Thu Mar 26 16:22:30 2015 +0200

    VMware: Use virtual disk size instead of image size

    The image size is different from the virtual disk size when the image is
    sparse. For sparse images we get the size of the virtual disk by quering
    the -flat file after converting the disk to preallocated.

    Co-Authored-By: Qin Zhao <email address hidden>

    Change-Id: I18628fe3a180f722936c680c6be30d5ed4d50bd9
    Closes-Bug: #1240373
    Closes-Bug: #1472955
    (cherry picked from commit 2f15d304f01a94a788db4175de3905bdd3a3cc62)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 2015.1.4 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/258020
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 2015.1.4 release.