vmware: vmdk fail to attach to new VMs when using "thin" vmdk

[Shawn Hartsock]

When using "thin" provisioned VMDK with the VMwareAPI VCDriver the VMDK will fail to attach to the VM forcing the VM to be rolled back off of the vCenter inventory. When you keep Nova from rolling back the VM you will be able to boot the VM and attach the VMDK.

The error message received on failure is "a parameter was missing"

[dan wendlandt (danwent)]

I talked to Christian about this on friday. He says that the best near-term workaround is to use a command for creating a VM from OVF in order to transfer to the file from Glance/Nova to the datastore, as there is no other public HTTP API that will correctly transfer thin images. Apparently, the trick will be to create a "fake VM", deploy it as OVF, and then remove the "parent" VM object from the inventory leaving only the VMDK. If what I'm saying doesn't make sense, ping Christian and can give you the real details.

[Shawn Hartsock (hartsock)]

Root cause is likely a Glance bug where thin provisioned VMDK are stored with the wrong image size.

[Tracy Jones (tjones-i)]

Looks like this is the beginning of the problem

2013-06-18 10:39:27.879 DEBUG nova.virt.vmwareapi.vmops [req-73051c93-2870-45d0-aa63-b2d394310570 demo demo] [instance: ea76e43a-18d3-4088-aaea-ba17a6b36485] Copying Virtual Disk of size 0 KB and adapter type lsiLogic on the ESX host local store datastore1 to disk type thin from (pid=6479) _copy_virtual_disk /opt/stack/nova/nova/virt/vmwareapi/vmops.py:324

[dan wendlandt (danwent)]

Hi tracy,

apparently this is a known issue from other VIM integrations that have been done in the past, and using the "create OVF" call to transfer the disk seems to be the best approach. Christian apparently has experience with this, so you could reach out to him.

[Shawn Hartsock (hartsock)]

This may be related:
https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L344

Notice the disk type is hard coded to "eagerZeroedThick" here when we have three modes, thin, eager zeroed, and lazy zeroed. That might be the source of other problems.

[Tracy Jones (tjones-i)]

I will talk with Christian - though i see in the 5.1 documentation that CopyVirtualDisk_Task is publicly available so i can't figure out why this exception is being thrown....

http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.wssdk.apiref.doc/vim.VirtualDiskManager.html?resultof=%2522%2563%256f%2570%2579%2576%2569%2572%2574%2575%2561%256c%2564%2569%2573%256b%2522%2520

[Shawn Hartsock (hartsock)]

Off topic, but you might want to observe this...

Notice here: https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L246

This assumes the object VirtualDiskFlatVer2BackingInfo but you could have any of VirtualDiskFlatVer1BackingInfo, VirtualDiskFlatVer2BackingInfo, VirtualDiskRawDiskMappingVer1BackingInfo, VirtualDiskSeSparseBackingInfo, VirtualDiskSparseVer1BackingInfo, VirtualDiskSparseVer2BackingInfo

I don't think this is practically a problem, but notice that the Sparse disk types are not handled. This complicates supporting formats other than thick + flat. It may be possible to do thin + flat, but we don't have the rest of the parts in place to do ... for example ... thin + sparse. See: http://cormachogan.com/2012/09/05/vsphere-5-1-storage-enhancements-part-2-se-sparse-disks/

tl;dr thin + flat may be almost good to go, but we probably won't get "sparse" on this pass IMHO.

[Tracy Jones (tjones-i)]

Thanks. I'll take a look. I'm also meeting with Christian tomorrow to talk about his experiences with this and vcd

Sent from my iPhone

On Jul 2, 2013, at 8:11 PM, Shawn Hartsock <email address hidden> wrote:

> Off topic, but you might want to observe this...
>
> Notice here:
> https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L246
>
> This assumes the object VirtualDiskFlatVer2BackingInfo but you could
> have any of VirtualDiskFlatVer1BackingInfo,
> VirtualDiskFlatVer2BackingInfo,
> VirtualDiskRawDiskMappingVer1BackingInfo,
> VirtualDiskSeSparseBackingInfo, VirtualDiskSparseVer1BackingInfo,
> VirtualDiskSparseVer2BackingInfo
>
> I don't think this is practically a problem, but notice that the Sparse
> disk types are not handled. This complicates supporting formats other
> than thick + flat. It may be possible to do thin + flat, but we don't
> have the rest of the parts in place to do ... for example ... thin +
> sparse. See: http://cormachogan.com/2012/09/05/vsphere-5-1-storage-
> enhancements-part-2-se-sparse-disks/
>
> tl;dr thin + flat may be almost good to go, but we probably won't get
> "sparse" on this pass IMHO.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1180471
>
> Title:
> vmware: vmdk fail to attach to new VMs when using "thin" vmdk
>
> Status in OpenStack Compute (Nova):
> In Progress
>
> Bug description:
> When using "thin" provisioned VMDK with the VMwareAPI VCDriver the
> VMDK will fail to attach to the VM forcing the VM to be rolled back
> off of the vCenter inventory. When you keep Nova from rolling back the
> VM you will be able to boot the VM and attach the VMDK.
>
> The error message received on failure is "a parameter was missing"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1180471/+subscriptions

[Tracy Jones (tjones-i)]

thin works - i had a user error issue where i used the default adaptor type which is lsilogic. The image i had was ide. once i uploaded it into glance correctly it booted nicely. I will close this and create a new one for sparse disks which do not work.

Cristian's idea of deploying an ovf is a good one but needs a blueprint as it's a larger change. I will work on that at some point

[Tracy Jones (tjones-i)]

closing as user error - one issue i encountered was mis-spelling vmware_adaptor type rather than vmware_adaptertype. May be nice to log the disk type being used to make this easier to debug.