OpenStack Compute (Nova)

xenapi VDI_IN_USE error when deleting an instance

Reported by Johannes Erdfelt on 2012-03-24
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Johannes Erdfelt

Bug Description

I can't reproduce it by hand, but I can with a testing script I'm working on. I think it may be timing related, but that's not clear yet.

My script has only been able to reproduce when reverting a resize. I don't know how that is related yet.

The symptom is that you will have an instance left still running after a delete instance. The instance will be marked as deleted in the database and won't show up in 'nova list'.

Exceptions like this will end up in the nova-compute logs:

2012-03-24 03:04:53 DEBUG nova.virt.xenapi.vm_utils [req-2d2923e0-efd9-4e50-9cb4-03c816e9f8e9 admin openstack] VDI c50358e4-6171-49dc-acfa-f4114bbae294 is still available from (pid=22488) lookup_vm_vdis /home/johannes/openstack/nova/trunk/nova/virt/xenapi/vm_utils.py:1025
2012-03-24 03:04:53 ERROR nova.virt.xenapi.vm_utils [req-2d2923e0-efd9-4e50-9cb4-03c816e9f8e9 admin openstack] ['VDI_IN_USE', 'OpaqueRef:94e3ae3c-5ed2-3203-7ae9-c7ebdcb26116']
(nova.virt.xenapi.vm_utils): TRACE: Traceback (most recent call last):
(nova.virt.xenapi.vm_utils): TRACE: File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/vm_utils.py", line 318, in destroy_vdi
(nova.virt.xenapi.vm_utils): TRACE: session.call_xenapi('VDI.destroy', vdi_ref)
(nova.virt.xenapi.vm_utils): TRACE: File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi_conn.py", line 574, in call_xenapi
(nova.virt.xenapi.vm_utils): TRACE: return tpool.execute(f, *args)
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/eventlet/tpool.py", line 76, in tworker
(nova.virt.xenapi.vm_utils): TRACE: rv = meth(*args,**kwargs)
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/eventlet/tpool.py", line 76, in tworker
(nova.virt.xenapi.vm_utils): TRACE: rv = meth(*args,**kwargs)
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 229, in __call__
(nova.virt.xenapi.vm_utils): TRACE: return self.__send(self.__name, args)
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 133, in xenapi_request
(nova.virt.xenapi.vm_utils): TRACE: result = _parse_result(getattr(self, methodname)(*full_params))
(nova.virt.xenapi.vm_utils): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 203, in _parse_result
(nova.virt.xenapi.vm_utils): TRACE: raise Failure(result['ErrorDescription'])
(nova.virt.xenapi.vm_utils): TRACE: Failure: ['VDI_IN_USE', 'OpaqueRef:94e3ae3c-5ed2-3203-7ae9-c7ebdcb26116']
(nova.virt.xenapi.vm_utils): TRACE:
2012-03-24 03:04:53 ERROR nova.virt.xenapi.vmops [req-2d2923e0-efd9-4e50-9cb4-03c816e9f8e9 admin openstack] Unable to destroy VDI OpaqueRef:94e3ae3c-5ed2-3203-7ae9-c7ebdcb26116

Changed in nova:
assignee: nobody → Johannes Erdfelt (johannes.erdfelt)
status: New → In Progress
description: updated

Reviewed: https://review.openstack.org/5827
Committed: http://github.com/openstack/nova/commit/80a55176e11da1594c9d663df0e4a0a755018c16
Submitter: Jenkins
Branch: master

commit 80a55176e11da1594c9d663df0e4a0a755018c16
Author: Johannes Erdfelt <email address hidden>
Date: Mon Mar 26 17:25:12 2012 +0000

    Don't set instance ACTIVE until it's really active

    Fixes bug 963656

    Reverting a resize would end up setting the instance ACTIVE before the
    driver had finished all of the work starting up the original instance.
    If the instance is deleted quickly after the revert, a race condition
    could occur between restarting the original instance and deleting it.

    Change-Id: Iba26ad7d1bc1049137f94e13898be86698963fb3

Changed in nova:
status: In Progress → Fix Committed
Chuck Short (zulcss) on 2012-03-29
tags: added: essex-rc-potential

Reviewed: https://review.openstack.org/6032
Committed: http://github.com/openstack/nova/commit/9a21d9f7938a5f1e82f14bdf51502eda78c268c7
Submitter: Jenkins
Branch: milestone-proposed

commit 9a21d9f7938a5f1e82f14bdf51502eda78c268c7
Author: Johannes Erdfelt <email address hidden>
Date: Mon Mar 26 17:25:12 2012 +0000

    Don't set instance ACTIVE until it's really active

    Fixes bug 963656

    Reverting a resize would end up setting the instance ACTIVE before the
    driver had finished all of the work starting up the original instance.
    If the instance is deleted quickly after the revert, a race condition
    could occur between restarting the original instance and deleting it.

    Change-Id: Iba26ad7d1bc1049137f94e13898be86698963fb3

Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: none → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers