VMware: Instances stuck in spawning when insufficient disk space

Bug #1255355 reported by Ryan Hsu on 2013-11-27
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Gary Kotton
Havana
High
Gary Kotton
VMwareAPI-Team
High
Unassigned

Bug Description

When using VC Driver, booting an instance when there is insufficient disk space on the datastore causes the instance to get stuck in SPAWNING state.

The following expected error message is seen in the logs before instance is stuck SPAWNING.

2013-11-26 15:47:58.189 DEBUG nova.virt.vmwareapi.driver [-] Task [CopyVirtualDisk_Task] (returnval){
   value = "task-8768"
   _type = "Task"
 } status: success from (pid=22523) _poll_task /opt/stack/nova/nova/virt/vmwareapi/driver.py:916
2013-11-26 15:47:58.189 DEBUG nova.virt.vmwareapi.vmops [req-bd4ed37e-e5ce-4dd5-a7b7-4a1340c64111 admin demo] [instance: 02b1b31e-ffd3-48fe-b151-68395e80d1ce] Copied Virtual Disk of size 1048576 KB and type preallocated on the ESX host local store iscsi-1 from (pid=22523) _copy_virtual_disk /opt/stack/nova/nova/virt/vmwareapi/vmops.py:458
2013-11-26 15:47:58.190 DEBUG nova.virt.vmwareapi.vmops [req-bd4ed37e-e5ce-4dd5-a7b7-4a1340c64111 admin demo] Extending root virtual disk to 20971520 from (pid=22523) _extend_virtual_disk /opt/stack/nova/nova/virt/vmwareapi/vmops.py:156
2013-11-26 15:48:03.254 WARNING nova.virt.vmwareapi.driver [-] Task [ExtendVirtualDisk_Task] (returnval){
   value = "task-8769"
   _type = "Task"
 } status: error Insufficient disk space on datastore ''.
Instance failed to spawn
Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 1407, in _spawn
    block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 629, in spawn
    admin_password, network_info, block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 544, in spawn
    root_vmdk_path, dc_map.ref)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 166, in _extend_virtual_disk
    vmdk_extend_task)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 898, in _wait_for_task
    ret_val = done.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
NovaException: Insufficient disk space on datastore ''.

Ryan Hsu (rhsu) on 2013-11-27
tags: added: drivers
removed: vmdk
Changed in nova:
importance: Undecided → High
Gary Kotton (garyk) wrote :

Hi,
When I run this on the latest code I get the following:

nicira@os-devstack:~/devstack$ nova list
+--------------------------------------+------+--------+------------+-------------+----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------------------+
| 9fc6843f-9e41-467a-8d7c-dd974b4f36bc | X | ERROR | None | NOSTATE | private=10.127.1.227 |
| ff3a52f2-545e-48d8-b6ac-72b7e3853487 | Y | ACTIVE | None | Running | private=10.127.1.226 |
+--------------------------------------+------+--------+------------+-------------+----------------------+
nicira@os-devstack:~/devstack$

The instance goes to NOSTATE.

I did see an issue where the temporary files created whilst doing the image extension were left behind and this would have caused problems.

I have addressed these with https://review.openstack.org/59287

Changed in nova:
assignee: nobody → Gary Kotton (garyk)
milestone: none → icehouse-1
tags: added: havana-backport-potential
Gary Kotton (garyk) on 2013-12-01
Changed in nova:
status: New → Confirmed
status: Confirmed → In Progress
Changed in nova:
milestone: icehouse-1 → icehouse-2

Reviewed: https://review.openstack.org/59287
Committed: http://github.com/openstack/nova/commit/787d36b40b41c1ee1e56ae3f8d17984e70c59a8e
Submitter: Jenkins
Branch: master

commit 787d36b40b41c1ee1e56ae3f8d17984e70c59a8e
Author: Gary Kotton <email address hidden>
Date: Sun Dec 1 05:09:19 2013 -0800

    VMware: fix disk extend bug when no space on datastore

    In the event that thete is no space on the datastore then ensure
    that the disk extension is cleaned up accordingly.

    A number of files would be left behind on the datastore.

    Change-Id: Ica9daf9151e4fb1874fa8b1d7247080860ef4ae2
    Related-Bug: #1255355

Tracy Jones (tjones-i) on 2014-01-10
Changed in openstack-vmwareapi-team:
importance: Undecided → High
status: New → In Progress
Thierry Carrez (ttx) on 2014-01-22
Changed in nova:
milestone: icehouse-2 → icehouse-3
Alan Pevec (apevec) wrote :

Is there anything else planned to be fixed for this issue, besides related fix 59287 ?

tags: removed: havana-backport-potential

Reviewed: https://review.openstack.org/60652
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=985b5aac409fb7e73c78ef9842da79c9c3950635
Submitter: Jenkins
Branch: stable/havana

commit 985b5aac409fb7e73c78ef9842da79c9c3950635
Author: Gary Kotton <email address hidden>
Date: Sun Dec 1 05:09:19 2013 -0800

    VMware: fix disk extend bug when no space on datastore

    In the event that thete is no space on the datastore then ensure
    that the disk extension is cleaned up accordingly.

    A number of files would be left behind on the datastore.

    Change-Id: Ica9daf9151e4fb1874fa8b1d7247080860ef4ae2
    Related-Bug: #1255355
    (cherry picked from commit 787d36b40b41c1ee1e56ae3f8d17984e70c59a8e)

tags: added: in-stable-havana
Thierry Carrez (ttx) on 2014-03-05
Changed in nova:
milestone: icehouse-3 → icehouse-rc1
Dan Smith (danms) wrote :

I'm marking this as committed since there has been no other activity, Alan's question went unanswered, and it appears to have been fixed and backported.

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-03-31
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-04-17
Changed in nova:
milestone: icehouse-rc1 → 2014.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers