VMware: Instances stuck in spawning when insufficient disk space

[Ryan Hsu]

When using VC Driver, booting an instance when there is insufficient disk space on the datastore causes the instance to get stuck in SPAWNING state.

The following expected error message is seen in the logs before instance is stuck SPAWNING.

2013-11-26 15:47:58.189 DEBUG nova.virt.vmwareapi.driver [-] Task [CopyVirtualDisk_Task] (returnval){
   value = "task-8768"
   _type = "Task"
 } status: success from (pid=22523) _poll_task /opt/stack/nova/nova/virt/vmwareapi/driver.py:916
2013-11-26 15:47:58.189 DEBUG nova.virt.vmwareapi.vmops [req-bd4ed37e-e5ce-4dd5-a7b7-4a1340c64111 admin demo] [instance: 02b1b31e-ffd3-48fe-b151-68395e80d1ce] Copied Virtual Disk of size 1048576 KB and type preallocated on the ESX host local store iscsi-1 from (pid=22523) _copy_virtual_disk /opt/stack/nova/nova/virt/vmwareapi/vmops.py:458
2013-11-26 15:47:58.190 DEBUG nova.virt.vmwareapi.vmops [req-bd4ed37e-e5ce-4dd5-a7b7-4a1340c64111 admin demo] Extending root virtual disk to 20971520 from (pid=22523) _extend_virtual_disk /opt/stack/nova/nova/virt/vmwareapi/vmops.py:156
2013-11-26 15:48:03.254 WARNING nova.virt.vmwareapi.driver [-] Task [ExtendVirtualDisk_Task] (returnval){
   value = "task-8769"
   _type = "Task"
 } status: error Insufficient disk space on datastore ''.
Instance failed to spawn
Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 1407, in _spawn
    block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 629, in spawn
    admin_password, network_info, block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 544, in spawn
    root_vmdk_path, dc_map.ref)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 166, in _extend_virtual_disk
    vmdk_extend_task)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 898, in _wait_for_task
    ret_val = done.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
NovaException: Insufficient disk space on datastore ''.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/59287

[Gary Kotton (garyk)]

Hi,
When I run this on the latest code I get the following:

nicira@os-devstack:~/devstack$ nova list
+--------------------------------------+------+--------+------------+-------------+----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------------------+
| 9fc6843f-9e41-467a-8d7c-dd974b4f36bc | X | ERROR | None | NOSTATE | private=10.127.1.227 |
| ff3a52f2-545e-48d8-b6ac-72b7e3853487 | Y | ACTIVE | None | Running | private=10.127.1.226 |
+--------------------------------------+------+--------+------------+-------------+----------------------+
nicira@os-devstack:~/devstack$

The instance goes to NOSTATE.

I did see an issue where the temporary files created whilst doing the image extension were left behind and this would have caused problems.

I have addressed these with https://review.openstack.org/59287

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/59287
Committed: http://github.com/openstack/nova/commit/787d36b40b41c1ee1e56ae3f8d17984e70c59a8e
Submitter: Jenkins
Branch: master

commit 787d36b40b41c1ee1e56ae3f8d17984e70c59a8e
Author: Gary Kotton <email address hidden>
Date: Sun Dec 1 05:09:19 2013 -0800

    VMware: fix disk extend bug when no space on datastore

    In the event that thete is no space on the datastore then ensure
    that the disk extension is cleaned up accordingly.

    A number of files would be left behind on the datastore.

    Change-Id: Ica9daf9151e4fb1874fa8b1d7247080860ef4ae2
    Related-Bug: #1255355

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/60652

[Alan Pevec (apevec)]

Is there anything else planned to be fixed for this issue, besides related fix 59287 ?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60652
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=985b5aac409fb7e73c78ef9842da79c9c3950635
Submitter: Jenkins
Branch: stable/havana

commit 985b5aac409fb7e73c78ef9842da79c9c3950635
Author: Gary Kotton <email address hidden>
Date: Sun Dec 1 05:09:19 2013 -0800

    VMware: fix disk extend bug when no space on datastore

    In the event that thete is no space on the datastore then ensure
    that the disk extension is cleaned up accordingly.

    A number of files would be left behind on the datastore.

    Change-Id: Ica9daf9151e4fb1874fa8b1d7247080860ef4ae2
    Related-Bug: #1255355
    (cherry picked from commit 787d36b40b41c1ee1e56ae3f8d17984e70c59a8e)

[Dan Smith (danms)]

I'm marking this as committed since there has been no other activity, Alan's question went unanswered, and it appears to have been fixed and backported.