OpenStack Compute (nova)

vmware: Instance going to error state while resizing

Bug #1328455 reported by Nileshkumar Patel
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Gary Kotton
OpenStack Compute (nova) 2015.1.0 "kilo"

Bug Description

VM goes to error state in ESX while resizing due to Insufficient disk space on datastore.

Ideal behaviour should bring the Instance back to Active state.

Resize try to clone VM, if datacenter has no free space for that flavor size then it will give error "Insufficient disk space on datastore".

Exception move VM to error state.

Stack Trace:

2014-06-10 09:27:57.013 26472 WARNING nova.virt.vmwareapi.vmops [req-69ff7cf8-52a6-4eb9-a179-82ac29111de7 d90ae18dfd8e43caaa2fbcc00bd5a366 683577f7c17b405686885433f960bbf6] Datacenter does not have a host / host does not have a datastore. Ignoring it.
2014-06-10 09:27:57.647 26472 WARNING nova.virt.vmwareapi.driver [-] Task [CloneVM_Task] (returnval){
   value = "task-28954"
   _type = "Task"
 } status: error Insufficient disk space on datastore 'My ds'.
2014-06-10 09:27:57.648 26472 ERROR nova.compute.manager [req-69ff7cf8-52a6-4eb9-a179-82ac29111de7 d90ae18dfd8e43caaa2fbcc00bd5a366 683577f7c17b405686885433f960bbf6] [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] Setting instance vm_state to ERROR
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] Traceback (most recent call last):
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5012, in _error_out_instance_on_exception
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] yield
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3041, in resize_instance
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] block_device_info)
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 461, in migrate_disk_and_power_off
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] dest, instance_type)
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1301, in migrate_disk_and_power_off
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] self._session._wait_for_task(instance['uuid'], vm_clone_task)
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 932, in _wait_for_task
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] ret_val = done.wait()
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] return hubs.get_hub().switch()
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] return self.greenlet.switch()
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] NovaException: Insufficient disk space on datastore 'My ds'.
2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]
2014-06-10 09:27:58.173 26472 WARNING nova.virt.vmwareapi.driver [-] Task [CloneVM_Task] (returnval){
   value = "task-28954"
   _type = "Task"
 } status: error Insufficient disk space on datastore 'My ds'.
2014-06-10 09:27:58.174 26472 WARNING nova.virt.vmwareapi.driver [-] In vmwareapi:_poll_task, Got this error Trying to re-send() an already-triggered event.
2014-06-10 09:27:58.174 26472 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall self.f(*self.args, **self.kw)
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 989, in _poll_task
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall done.send_exception(excep)
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 208, in send_exception
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall return self.send(None, args)
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 150, in send
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall AssertionError: Trying to re-send() an already-triggered event.
2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall
2014-06-10 09:27:58.623 26472 ERROR nova.openstack.common.rpc.amqp [req-69ff7cf8-52a6-4eb9-a179-82ac29111de7 d90ae18dfd8e43caaa2fbcc00bd5a366 683577f7c17b405686885433f960bbf6] Exception during message handling
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp **args)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 357, in decorated_function
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 93, in wrapped
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp payload)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 76, in wrapped
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 247, in decorated_function
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp pass
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 233, in decorated_function
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 275, in decorated_function
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 262, in decorated_function
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3041, in resize_instance
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp block_device_info)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 461, in migrate_disk_and_power_off
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp dest, instance_type)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1301, in migrate_disk_and_power_off
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp self._session._wait_for_task(instance['uuid'], vm_clone_task)
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 932, in _wait_for_task
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp ret_val = done.wait()
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp return hubs.get_hub().switch()
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp return self.greenlet.switch()
2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp NovaException: Insufficient disk space on datastore 'My ds'.

See original description
Tags: compute vmware
Nileshkumar Patel (nileshkumar-govindbhai-patel)
description: updated
Tracy Jones (tjones-i)
Changed in nova:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Eric Brown (ericwb) wrote :

This is interesting since currently the resize operation in the vmware driver doesn't even resize the disk. There is a patch under review for that:
https://review.openstack.org/#/c/85804/

ravali (ravali-kandula)
Changed in nova:
assignee: nobody → ravali (ravali-kandula)
status: Triaged → Confirmed
ravali (ravali-kandula)
Changed in nova:
status: Confirmed → New
ravali (ravali-kandula)
Changed in nova:
assignee: ravali (ravali-kandula) → nobody
ravali (ravali-kandula)
Changed in nova:
assignee: nobody → ravali (ravali-kandula)
Revision history for this message
ravali (ravali-kandula) wrote :

In juno version of openstack the instance is going to available state after the resize process failed for the instance.Can you please try this in the juno version?

Revision history for this message
melanie witt (melwitt) wrote :

Setting back to Triaged.

@ravali, Are you seeing this behavior using vmware? This bug applies to the vmware driver.

Changed in nova:
status: New → Triaged
Sean Dague (sdague)
Changed in nova:
status: Triaged → Confirmed
Revision history for this message
Gary Kotton (garyk) wrote :

This has been fixed in https://github.com/openstack/nova/commit/f3f1a53fe2bfe23229867c4a90f6456ac204181f
We now use the same instance to do the resize

Changed in nova:
status: Confirmed → Fix Committed
assignee: ravali (ravali-kandula) → Gary Kotton (garyk)
milestone: none → kilo-rc1
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-rc1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.