cinder volume can't be deleted (error_deleting state)

[Max Mazur]

MOS 5.0.1, Cinder + LVM, CentOs

rpm -qa | grep cinder
python-cinderclient-1.0.8-fuel5.0.1.mira2.noarch
python-cinder-2014.1.1-fuel5.0.1.mira3.noarch
openstack-cinder-2014.1.1-fuel5.0.1.mira3.noarch

Steps to reproduce (via horizon):

1. Create 1Gb empty volume (Volume1)
2. Create 1Gb volume, copy of Volume1
3. delete Volume1

Result:
159>Dec 24 14:07:21 node-17 cinder-cinder.openstack.common.processutils DEBUG: Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvs --noheadings --unit=g -o vg_name,name,size --nosuffix cinder
<159>Dec 24 14:07:21 node-17 cinder-cinder.openstack.common.processutils DEBUG: Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvdisplay --noheading -C -o Attr cinder/volume-b1f116e9-05b2-4eb6-9f50-56
656e2361dc
<155>Dec 24 14:07:22 node-17 cinder-cinder.volume.drivers.lvm ERROR: Volume device file path /dev/mapper/cinder-volume--b1f116e9--05b2--4eb6--9f50--56656e2361dc does not exist.
<159>Dec 24 14:07:22 node-17 cinder-cinder.openstack.common.lockutils DEBUG: Released file lock "b1f116e9-05b2-4eb6-9f50-56656e2361dc-delete_volume" at /var/lib/cinder/tmp/cinder-b1f116e9-05b2-4eb6-9f50-56656e2361dc-delete_volume for metho
d "lvo_inner2"...
<155>Dec 24 14:07:22 node-17 cinder-oslo.messaging.rpc.dispatcher ERROR: Exception during message handling: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/mapper/cinder-volume--b1f116e9--05b2--
4eb6--9f50--56656e2361dc does not exist.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 144, in lvo_inner1
    return lvo_inner2(inst, context, volume_id, **kwargs)
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 233, in inner
    retval = f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 143, in lvo_inner2
    return f(*_args, **_kwargs)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 416, in delete_volume
    {'status': 'error_deleting'})
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 405, in delete_volume
    self.driver.delete_volume(volume_ref)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 233, in delete_volume
    self._delete_volume(volume)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 128, in _delete_volume
    self._clear_volume(volume, is_snapshot)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 155, in _clear_volume
    raise exception.VolumeBackendAPIException(data=msg)

Expected result:
deleted volume.

[ruhe (ruhe)]

Max, could you please attach fuel snapshot along with output of http://FUEL_IP:FUEL_PORT/api/version ?

[Max Mazur (mmaxur)]

{u'build_id': u'2014-10-29_12-34-24', u'ostf_sha': u'6664009a4c4b39e4c3a9eb1a52abcd09ea428334', u'build_number': u'22', u'api': u'1.0', u'nailgun_sha': u'13d5da5eba0060ba54c1faf0fdcd8dd232fd5540', u'production': u'docker', u'mirantis': u'no', u'fuelmain_sha': u'bbf2b6fa0acc2976fa8dfb00c97e926fe374e657', u'astute_sha': u'aa9fe5d13741c38e46db0adbba30bab4681b4a8b', u'release': u'5.0.1', u'fuellib_sha': u'166cb6b2eaf7ed844e898e59fbac387116e25b37'}

[Dmitry Mescheryakov (dmitrymex)]

Ivan, is there a workaround for the issue? Also, are 5.1.1 or 5.0.2 or 6.x versions affected?

[Ivan Kolodyazhny (e0ne)]

Dmitry, 6.x it not affected

[Ivan Kolodyazhny (e0ne)]

Root cause if this issue is https://bugs.launchpad.net/mos/+bug/1360173. Fix for #1360173 resolved this bug

[OSCI Robot (oscirobot)]

RPM package cinder has been built for project openstack/cinder
Package version == 2014.1.1, package release == fuel5.0.1.mira4.git.07c4414.4bfc7ce

Changeset: https://review.fuel-infra.org/1669
project: openstack/cinder
branch: openstack-ci/fuel-5.0.1/2014.1.1
author: Ivan Kolodyazhny
committer: Ivan Kolodyazhny
subject: Retry lvremove with ignore_suspended_devices
status: patchset-created

Files placed on repository:
openstack-cinder-2014.1.1-fuel5.0.1.mira4.git.07c4414.4bfc7ce.noarch.rpm
openstack-cinder-doc-2014.1.1-fuel5.0.1.mira4.git.07c4414.4bfc7ce.noarch.rpm
python-cinder-2014.1.1-fuel5.0.1.mira4.git.07c4414.4bfc7ce.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.1-stable-1669/centos

[OSCI Robot (oscirobot)]

DEB package cinder has been built for project openstack/cinder
Package version == 2014.1.1, package release == fuel5.0.1~mira0+git.07c4414.4bfc7ce

Changeset: https://review.fuel-infra.org/1669
project: openstack/cinder
branch: openstack-ci/fuel-5.0.1/2014.1.1
author: Ivan Kolodyazhny
committer: Ivan Kolodyazhny
subject: Retry lvremove with ignore_suspended_devices
status: patchset-created

Files placed on repository:
cinder-api_2014.1.1-fuel5.0.1~mira0+git.07c4414.4bfc7ce_all.deb
cinder-backup_2014.1.1-fuel5.0.1~mira0+git.07c4414.4bfc7ce_all.deb
cinder-common_2014.1.1-fuel5.0.1~mira0+git.07c4414.4bfc7ce_all.deb
cinder-scheduler_2014.1.1-fuel5.0.1~mira0+git.07c4414.4bfc7ce_all.deb
cinder-volume_2014.1.1-fuel5.0.1~mira0+git.07c4414.4bfc7ce_all.deb
python-cinder_2014.1.1-fuel5.0.1~mira0+git.07c4414.4bfc7ce_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.1-stable-1669/ubuntu

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ivan Kolodyazhny <email address hidden> on branch: openstack-ci/fuel-5.0.1/2014.1.1
Review: https://review.fuel-infra.org/1669