Ubuntu
nova package

Undeletable volume-backed instance

Bug #1397893 reported by Francois Deppierraz
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Vladik Romanovsky
nova (Ubuntu)
Fix Released
Medium
Satyanarayana Patibandla

Bug Description

nova-compute package version 1:2014.1.3-0ubuntu1.1 on Ubuntu 14.04.

Trying to delete a volume-backed instance in error state doesn't work. Nova-compute logs the following error.

# nova delete 71a41e09-e8bc-4829-979e-1d175246da00
# tail /var/log/nova/nova-compute.log
[...]
2014-12-01 09:52:25.563 25832 AUDIT nova.compute.manager [req-39acd9da-518c-4804-bf30-1a38eace21bf 4474a81aca524682875658eb8064c33d 7dbed2bcbd7541289c34ae8392acf612] [instance: 71a41e09-e8bc-4829-979e-1d175246da00] Terminating
instance
2014-12-01 09:52:25.569 25832 ERROR nova.virt.libvirt.driver [-] [instance: 71a41e09-e8bc-4829-979e-1d175246da00] During wait destroy, instance disappeared.
2014-12-01 09:52:25.638 25832 ERROR nova.compute.manager [req-39acd9da-518c-4804-bf30-1a38eace21bf 4474a81aca524682875658eb8064c33d 7dbed2bcbd7541289c34ae8392acf612] [instance: 71a41e09-e8bc-4829-979e-1d175246da00] Setting inst
ance vm_state to ERROR
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] Traceback (most recent call last):
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2273, in do_terminate_instance
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] self._delete_instance(context, instance, bdms, quotas)
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 103, in inner
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] rv = f(*args, **kwargs)
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2242, in _delete_instance
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] quotas.rollback()
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] six.reraise(self.type_, self.value, self.tb)
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2214, in _delete_instance
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] self._shutdown_instance(context, db_inst, bdms)
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2156, in _shutdown_instance
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] requested_networks)
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] six.reraise(self.type_, self.value, self.tb)
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2146, in _shutdown_instance
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] block_device_info)
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 963, in destroy
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] destroy_disks)
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1040, in cleanup
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] disk_dev = vol['mount_device'].rpartition("/")[2]
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] AttributeError: 'NoneType' object has no attribute 'rpartition'
2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]
[...]

# nova show 71a41e09-e8bc-4829-979e-1d175246da00
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | openstack110 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstack110.local |
| OS-EXT-SRV-ATTR:instance_name | instance-00000b40 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-10-13T14:14:54Z |
| fault | {"message": "'NoneType' object has no attribute 'rpartition'", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 290, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2285, in terminate_instance |
| | do_terminate_instance(instance, bdms) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py\", line 249, in inner |
| | return f(*args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2283, in do_terminate_instance |
| | self._set_instance_error_state(context, instance['uuid']) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2273, in do_terminate_instance |
| | self._delete_instance(context, instance, bdms, quotas) |
| | File \"/usr/lib/python2.7/dist-packages/nova/hooks.py\", line 103, in inner |
| | rv = f(*args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2242, in _delete_instance |
| | quotas.rollback() |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2214, in _delete_instance |
| | self._shutdown_instance(context, db_inst, bdms) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2156, in _shutdown_instance |
| | requested_networks) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2146, in _shutdown_instance |
| | block_device_info) |
| | File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\", line 963, in destroy |
| | destroy_disks) |
| | File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\", line 1040, in cleanup |
| | disk_dev = vol['mount_device'].rpartition(\"/\")[2] |
| | ", "created": "2014-12-01T08:52:25Z"} |
| flavor | m1.large (4) |
| hostId | 567b019ea0119d6b7c09dcf0d8d859ad64a8cca16254148f439cd8b6 |
| id | 71a41e09-e8bc-4829-979e-1d175246da00 |
| image | openstack100-clone-1 (b472f236-ec54-41a3-9e92-40f091d57d05) |
| key_name | francois |
| metadata | {} |
| name | openstack100-clone |
| os-extended-volumes:volumes_attached | [{"id": "d8d89370-f334-4f78-bb35-e4a74f60d077"}] |
| status | ERROR |
| tenant_id | 7dbed2bcbd7541289c34ae8392acf612 |
| updated | 2014-12-01T08:52:25Z |
| user_id | 4474a81aca524682875658eb8064c33d |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# cinder show d8d89370-f334-4f78-bb35-e4a74f60d077
+--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'id': u'd8d89370-f334-4f78-bb35-e4a74f60d077', u'server_id': u'71a41e09-e8bc-4829-979e-1d175246da00', u'host_name': None, u'volume_id': u'd8d89370-f334-4f78-bb35-e4a74f60d077'}] |
| availability_zone | nova |
| bootable | true |
| created_at | 2014-10-13T14:14:55.000000 |
| display_description | |
| display_name | |
| encrypted | False |
| id | d8d89370-f334-4f78-bb35-e4a74f60d077 |
| metadata | {u'readonly': u'False', u'attached_mode': u'rw'} |
| os-vol-host-attr:host | openstack100 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7dbed2bcbd7541289c34ae8392acf612 |
| size | 100 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'image_name': u'openstack100-clone-1', u'image_id': u'b472f236-ec54-41a3-9e92-40f091d57d05', u'checksum': u'e3898f7109a6b496b39245988bbdd1e0', u'min_disk': u'80', u'size': u'68893671424'} |
| volume_type | None |
+--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

James Page (james-page)
Changed in nova (Ubuntu):
importance: Undecided → Critical
importance: Critical → Medium
Satyanarayana Patibandla (satya-patibandla)
Changed in nova (Ubuntu):
assignee: nobody → Satyanarayana Patibandla (satya-patibandla)
Satyanarayana Patibandla (satya-patibandla)
Changed in nova (Ubuntu):
status: New → In Progress
Revision history for this message
Sean Dague (sdague) wrote :

This is an icehouse bug, do we know if it's still in nova master?

tags: added: volumes
Changed in nova:
status: New → Confirmed
status: Confirmed → Incomplete
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

In response to Sean Dague (comment #1):

I observed the same symptom in nova master (March/April of 2015). Unfortunately I couldn't figure out the exact steps to reproduce this issue.

Revision history for this message
Matt Riedemann (mriedem) wrote :

This was fixed in juno here: https://review.openstack.org/#/c/104048/

Changed in nova:
status: Incomplete → Fix Committed
assignee: nobody → Roman Podoliaka (rpodolyaka)
assignee: Roman Podoliaka (rpodolyaka) → Vladik Romanovsky (vladik-romanovsky)
tags: added: in-stable-juno
Changed in nova:
status: Fix Committed → Fix Released
importance: Undecided → High
Chuck Short (zulcss)
Changed in nova (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.