Instance stuck in ACTIVE/deleting when network deallocation fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
High
|
Russell Bryant |
Bug Description
I set a simple bash script that boots a VM, waits a couple of minutes (doing nova list) then deletes the VM, waits a bit and continues.
I ran this last night, and one of the VMs has not been deleted. Log is showing that the delete was seen and processed without error by the API node but in compute log there is timeout while deallocating network for an instance.
ERROR nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
TRACE nova.openstack.
When timeout exception is raised during network deallocation then instances stuck in Active(deleting) but as nova list displays only vm_state, VM will get displayed as ACTIVE and there is no periodic task which cleans-up instances stuck in Active(deleting) on service restart..
summary: |
- FOLSOM : Silently ignored delete + Instance stuck in ACTIVE/deleting when network deallocation fails |
Changed in nova: | |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | grizzly-rc1 → 2013.1 |
no longer affects: | nova/folsom |
Experienced the same thing and fixed it in the following way:
Use virsh at the compute nodes to see if there are any VMs running which are not registered with OpenStack anymore:
virsh list --all
Destoy them all:
virsh destroy <vm id>
virsh undefine <vm id>
Check in /var/lib/ nova/instances and remove instance's folder
P.S. SRV-ATTR: instance_ name" property in the database or using "nova show <instance id>"
To get "OpenStack instance - to KVM virtual machine" binding, look for "OS-EXT-