VMwareVCDriver: Sparse disk copy error on spawn

Bug #1171226 reported by dan wendlandt
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Vui Lam
Grizzly
Fix Released
High
Vui Lam
VMwareAPI-Team
Fix Committed
High
Vui Lam

Bug Description

Not sure if this is a real bug, or just a case of inadequate documentation combining with bad error reporting. I get an exception (below) when booting a VM. The exception happens after glance is done streaming the disk image to VC (i.e., I see the image in the vmware_source folder in the DataSource) and it prevents the VM from actually booting.

I tried two different ways of adding the image to glance (both as 'ovf' and as 'bare') neither of which seemed to make a difference:

glance add name="Ubuntu-ovf" disk_format=vmdk container_format=ovf is_public=true vmware_adaptertype="lsiLogic" vmware_ostype="ubuntuGuest" vmware_disktype="sparse" < ~/ubuntu12.04-sparse.vmdk

glance add name="Ubuntu-bare" disk_format=vmdk container_format=bare is_public=true vmware_adaptertype="lsiLogic" vmware_ostype="ubuntuGuest" vmware_disktype="sparse" < ~/ubuntu12.04-sparse.vmdk

In both cases, I see this exception (note: there actually seems to be a second exception to, perhaps due to inproper error handling with the first):

2013-04-21 11:35:07 ERROR [nova.compute.manager] Error: ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/
compute/manager.py", line 905, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/opt/stack/nova/nova/compute/manage
r.py", line 1165, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7
/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1161, in _s
pawn\n block_device_info)\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 176, in spawn\n block_device_inf
o)\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 398, in spawn\n _copy_virtual_disk()\n', ' File "/opt/stac
k/nova/nova/virt/vmwareapi/vmops.py", line 340, in _copy_virtual_disk\n self._session._wait_for_task(instance[\'uuid\'], vmdk_c
opy_task)\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 558, in _wait_for_task\n ret_val = done.wait()\n',
' File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait\n return hubs.get_hub().switch()\n', ' F
ile "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch\n return self.greenlet.switch()\n', 'Nov
aException: The requested operation is not implemented by the server.\n']
2013-04-21 11:35:07 DEBUG [nova.openstack.common.rpc.amqp] Making synchronous call on conductor ...
2013-04-21 11:35:07 DEBUG [nova.openstack.common.rpc.amqp] MSG_ID is 2318255c5a4f4e5783cefb3cfde9e563
2013-04-21 11:35:07 DEBUG [nova.openstack.common.rpc.amqp] UNIQUE_ID is f710f7acfd774af3ba1aa91515b1fd05.
2013-04-21 11:35:10 WARNING [nova.virt.vmwareapi.driver] Task [CopyVirtualDisk_Task] (returnval){
   value = "task-925"
   _type = "Task"
 } status: error The requested operation is not implemented by the server.
2013-04-21 11:35:10 WARNING [nova.virt.vmwareapi.driver] In vmwareapi:_poll_task, Got this error Trying to re-send() an already-triggered event.
2013-04-21 11:35:10 ERROR [nova.utils] in fixed duration looping call
Traceback (most recent call last):
  File "/opt/stack/nova/nova/utils.py", line 595, in _inner
    self.f(*self.args, **self.kw)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 584, in _poll_task
    done.send_exception(excep)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 208, in send_exception
    return self.send(None, args)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 150, in send
    assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
AssertionError: Trying to re-send() an already-triggered event.

Tags: vmware
Revision history for this message
Chuck Short (zulcss) wrote :

Which version is this with?

Changed in nova:
status: New → Incomplete
Revision history for this message
dan wendlandt (danwent) wrote :

working off of master

Changed in nova:
assignee: nobody → Shawn Hartsock (hartsock)
Thierry Carrez (ttx)
Changed in nova:
status: Incomplete → New
Changed in nova:
status: New → Confirmed
importance: Undecided → Low
Tracy Jones (tjones-i)
Changed in nova:
assignee: Shawn Hartsock (hartsock) → Tracy Jones (tjones-i)
Revision history for this message
dan wendlandt (danwent) wrote :

Btw, this bug may be related to a similar issue I am seeing with snapshots: https://bugs.launchpad.net/nova/+bug/1184807

Revision history for this message
Tracy Jones (tjones-i) wrote :

this fix will be similar to https://review.openstack.org/#/c/40298/

Revision history for this message
Vui Lam (vui) wrote :

The root cause is indeed the same as that for 1184807. I am investigating into whether the need to perform disk type conversion (unlike 1184807) will complicate the fix or not.

Changed in nova:
assignee: Tracy Jones (tjones-i) → Vui Lam (vui)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/43974

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/43994

Changed in nova:
assignee: Vui Lam (vui) → Tracy Jones (tjones-i)
Vui Lam (vui)
Changed in nova:
importance: Low → Medium
Vui Lam (vui)
summary: - VMwareVCDriver CopyVirtualDisk returns a "not implemented by server" msg
+ VMwareVCDriver: Sparse disk copy error on spawn
Changed in nova:
assignee: Tracy Jones (tjones-i) → Vui Lam (vui)
Changed in nova:
assignee: Vui Lam (vui) → Gary Kotton (garyk)
Changed in nova:
assignee: Gary Kotton (garyk) → Tracy Jones (tjones-i)
Changed in nova:
assignee: Tracy Jones (tjones-i) → Vui Lam (vui)
Changed in nova:
assignee: Vui Lam (vui) → Gary Kotton (garyk)
Changed in nova:
assignee: Gary Kotton (garyk) → Vui Lam (vui)
Vui Lam (vui)
Changed in nova:
importance: Medium → High
Changed in nova:
milestone: none → havana-rc1
Changed in openstack-vmwareapi-team:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Vui Lam (vui)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/43994
Committed: http://github.com/openstack/nova/commit/42e929e990c7d5c4484c05bc92e57f64ca85629e
Submitter: Jenkins
Branch: master

commit 42e929e990c7d5c4484c05bc92e57f64ca85629e
Author: Vui Lam <email address hidden>
Date: Mon Aug 5 00:37:47 2013 -0700

    VMwareVCDriver Fix sparse disk copy error on spawn

    The same approach of calling VirtualDiskManager.CopyVirtualDisk
    without destination disk spec in the cluster driver for fixing 1184807
    turns out to work for copying a sparse disk as well because vCenter's behavior
    for specless copy is to do the reasonable thing of converting a sparse disk to
    thin-provisioned, and a flat disk to preallocated.

    Tests done:

    Verified using tempest that sparse/thin/preallocated image types deploys
    properly through the cluster driver. Tempest output to be included in
    the review comments.

    Fixes bug 1171226

    Change-Id: I6c48866e06c2aa656c89848564f3262fb7b79039

Changed in nova:
status: In Progress → Fix Committed
Tracy Jones (tjones-i)
Changed in openstack-vmwareapi-team:
status: In Progress → Fix Committed
Gary Kotton (garyk)
tags: added: grizzly-backport-potential
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/49842

Thierry Carrez (ttx)
Changed in nova:
milestone: havana-rc1 → 2013.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/grizzly)

Reviewed: https://review.openstack.org/49842
Committed: http://github.com/openstack/nova/commit/d791181b621302f7c4d0b2e20d4859e64fcd8631
Submitter: Jenkins
Branch: stable/grizzly

commit d791181b621302f7c4d0b2e20d4859e64fcd8631
Author: Vui Lam <email address hidden>
Date: Mon Aug 5 00:37:47 2013 -0700

    VMwareVCDriver Fix sparse disk copy error on spawn

    The same approach of calling VirtualDiskManager.CopyVirtualDisk
    without destination disk spec in the cluster driver for fixing 1184807
    turns out to work for copying a sparse disk as well because vCenter's behavior
    for specless copy is to do the reasonable thing of converting a sparse disk to
    thin-provisioned, and a flat disk to preallocated.

    Tests done:

    Verified using tempest that sparse/thin/preallocated image types deploys
    properly through the cluster driver. Tempest output to be included in
    the review comments.

    Fixes bug 1171226

    (cherry picked from commit 42e929e990c7d5c4484c05bc92e57f64ca85629e)

    Conflicts:

     nova/tests/virt/vmwareapi/test_vmwareapi.py
     nova/virt/vmwareapi/vmops.py

    Change-Id: I51627ddb4b9df8cf80f0b9c6efe68ec9eb68cd99

tags: added: in-stable-grizzly
Alan Pevec (apevec)
tags: removed: grizzly-backport-potential in-stable-grizzly
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.