xenapi VDI_IN_USE error when deleting an instance

[Johannes Erdfelt]

I can't reproduce it by hand, but I can with a testing script I'm working on. I think it may be timing related, but that's not clear yet.

My script has only been able to reproduce when reverting a resize. I don't know how that is related yet.

The symptom is that you will have an instance left still running after a delete instance. The instance will be marked as deleted in the database and won't show up in 'nova list'.

Exceptions like this will end up in the nova-compute logs:

2012-03-24 03:04:53 DEBUG nova.virt.xenapi.vm_utils [req-2d2923e0-efd9-4e50-9cb4-03c816e9f8e9 admin openstack] VDI c50358e4-6171-49dc-acfa-f4114bbae294 is still available from (pid=22488) lookup_vm_vdis /home/johannes/openstack/nova/trunk/nova/virt/xenapi/vm_utils.py:1025
2012-03-24 03:04:53 ERROR nova.virt.xenapi.vm_utils [req-2d2923e0-efd9-4e50-9cb4-03c816e9f8e9 admin openstack] ['VDI_IN_USE', 'OpaqueRef:94e3ae3c-5ed2-3203-7ae9-c7ebdcb26116']
(nova.virt.xenapi.vm_utils): TRACE: Traceback (most recent call last):
(nova.virt.xenapi.vm_utils): TRACE: File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/vm_utils.py", line 318, in destroy_vdi
(nova.virt.xenapi.vm_utils): TRACE: session.call_xenapi('VDI.destroy', vdi_ref)
(nova.virt.xenapi.vm_utils): TRACE: File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi_conn.py", line 574, in call_xenapi
(nova.virt.xenapi.vm_utils): TRACE: return tpool.execute(f, *args)
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/eventlet/tpool.py", line 76, in tworker
(nova.virt.xenapi.vm_utils): TRACE: rv = meth(*args,**kwargs)
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/eventlet/tpool.py", line 76, in tworker
(nova.virt.xenapi.vm_utils): TRACE: rv = meth(*args,**kwargs)
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 229, in __call__
(nova.virt.xenapi.vm_utils): TRACE: return self.__send(self.__name, args)
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 133, in xenapi_request
(nova.virt.xenapi.vm_utils): TRACE: result = _parse_result(getattr(self, methodname)(*full_params))
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 203, in _parse_result
(nova.virt.xenapi.vm_utils): TRACE: raise Failure(result['ErrorDescription'])
(nova.virt.xenapi.vm_utils): TRACE: Failure: ['VDI_IN_USE', 'OpaqueRef:94e3ae3c-5ed2-3203-7ae9-c7ebdcb26116']
(nova.virt.xenapi.vm_utils): TRACE:
2012-03-24 03:04:53 ERROR nova.virt.xenapi.vmops [req-2d2923e0-efd9-4e50-9cb4-03c816e9f8e9 admin openstack] Unable to destroy VDI OpaqueRef:94e3ae3c-5ed2-3203-7ae9-c7ebdcb26116

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/5827

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/5827
Committed: http://github.com/openstack/nova/commit/80a55176e11da1594c9d663df0e4a0a755018c16
Submitter: Jenkins
Branch: master

commit 80a55176e11da1594c9d663df0e4a0a755018c16
Author: Johannes Erdfelt <email address hidden>
Date: Mon Mar 26 17:25:12 2012 +0000

    Don't set instance ACTIVE until it's really active

    Fixes bug 963656

    Reverting a resize would end up setting the instance ACTIVE before the
    driver had finished all of the work starting up the original instance.
    If the instance is deleted quickly after the revert, a race condition
    could occur between restarting the original instance and deleting it.

    Change-Id: Iba26ad7d1bc1049137f94e13898be86698963fb3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/6032

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6032
Committed: http://github.com/openstack/nova/commit/9a21d9f7938a5f1e82f14bdf51502eda78c268c7
Submitter: Jenkins
Branch: milestone-proposed

commit 9a21d9f7938a5f1e82f14bdf51502eda78c268c7
Author: Johannes Erdfelt <email address hidden>
Date: Mon Mar 26 17:25:12 2012 +0000

    Don't set instance ACTIVE until it's really active

    Fixes bug 963656

    Reverting a resize would end up setting the instance ACTIVE before the
    driver had finished all of the work starting up the original instance.
    If the instance is deleted quickly after the revert, a race condition
    could occur between restarting the original instance and deleting it.

    Change-Id: Iba26ad7d1bc1049137f94e13898be86698963fb3