ISCSITargetRemoveFailed: Failed to remove iscsi target

[Jorgen Rosink]

Ubuntu 12.10 / Folsom (default Ubuntu packages) / TGT iSCSI

I can successfully create a volume with Horizon or nova CLI, the LVM volume is created and target config is available in /var/lib/cinder/volumes. However, when I try to delete the volume, cinder-volume gives an ISCSITargetRemoveFailed. After the (failed) volume delete command the volume still exists and shows the status "error_deleting".

As far as I can see I only use the following non-default (Folsom Install Guide) configuration options:

volume_name_template = vol-%s [default: volume-%s]
volume_group = openstack-volumes [default: cinder-volumes]

rabbit_virtual_host = /openstack [default: / ]

I've attached my Cinder config for reference, I can reproduce the error with the following commands:

# nova volume-create 1 --display-name TESTVOL
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| created_at | 2013-01-18T04:31:26.621594 |
| display_description | None |
| display_name | TESTVOL |
| id | 04f5af62-a20b-437a-b6fd-c1f6e31499ce |
| metadata | {} |
| size | 1 |
| snapshot_id | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+

# nova volume-list
+--------------------------------------+----------------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+-------------+
| 04f5af62-a20b-437a-b6fd-c1f6e31499ce | available | TESTVOL | 1 | None | |

# nova volume-delete 04f5af62-a20b-437a-b6fd-c1f6e31499ce
!no output!

# nova volume-list
+--------------------------------------+----------------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+-------------+
| 04f5af62-a20b-437a-b6fd-c1f6e31499ce | error_deleting | TESTVOL | 1 | None | |

# cat /var/log/cinder/cinder-volume.log

2013-01-18 05:33:19 4638 DEBUG cinder.openstack.common.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-c854ad68-29e7-4151-b3b6-5641f8888df5', u'_context_quota_class': None, u'args': {u'volume_id': u'04f5af62-a20b-437a-b6fd-c1f6e31499ce'}, u'_context_auth_token': '<SANITIZED>', u'_context_is_admin': True, u'_context_project_id': u'9a48474bc9014d75871decae3d6fd110', u'_context_timestamp': u'2013-01-18T04:33:19.033929', u'_context_read_deleted': u'no', u'_context_user_id': u'7f0a240381004bae898fa7da7460ac73', u'method': u'delete_volume', u'_context_remote_address': u'10.160.50.2'} _safe_log /usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/common.py:195
2013-01-18 05:33:19 4638 DEBUG cinder.openstack.common.rpc.amqp [-] unpacked context: {'user_id': u'7f0a240381004bae898fa7da7460ac73', 'roles': [u'admin'], 'timestamp': u'2013-01-18T04:33:19.033929', 'auth_token': '<SANITIZED>', 'remote_address': u'10.160.50.2', 'quota_class': None, 'is_admin': True, 'request_id': u'req-c854ad68-29e7-4151-b3b6-5641f8888df5', 'project_id': u'9a48474bc9014d75871decae3d6fd110', 'read_deleted': u'no'} _safe_log /usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/common.py:195
2013-01-18 05:33:19 INFO cinder.volume.manager [req-c854ad68-29e7-4151-b3b6-5641f8888df5 7f0a240381004bae898fa7da7460ac73 9a48474bc9014d75871decae3d6fd110] Clear capabilities
2013-01-18 05:33:19 DEBUG cinder.volume.manager [req-c854ad68-29e7-4151-b3b6-5641f8888df5 7f0a240381004bae898fa7da7460ac73 9a48474bc9014d75871decae3d6fd110] volume vol-04f5af62-a20b-437a-b6fd-c1f6e31499ce: removing export delete_volume /usr/lib/python2.7/dist-packages/cinder/volume/manager.py:192
2013-01-18 05:33:19 DEBUG cinder.utils [req-c854ad68-29e7-4151-b3b6-5641f8888df5 7f0a240381004bae898fa7da7460ac73 9a48474bc9014d75871decae3d6fd110] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgt-admin --show execute /usr/lib/python2.7/dist-packages/cinder/utils.py:156
2013-01-18 05:33:19 INFO cinder.volume.iscsi [req-c854ad68-29e7-4151-b3b6-5641f8888df5 7f0a240381004bae898fa7da7460ac73 9a48474bc9014d75871decae3d6fd110] Removing volume: 04f5af62-a20b-437a-b6fd-c1f6e31499ce
2013-01-18 05:33:19 4638 ERROR cinder.openstack.common.rpc.amqp [-] Exception during message handling
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 276, in _process_data
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 206, in delete_volume
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp {'status': 'error_deleting'})
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 193, in delete_volume
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp self.driver.remove_export(context, volume_ref)
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/driver.py", line 417, in remove_export
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp self.tgtadm.remove_iscsi_target(iscsi_target, 0, volume['id'])
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 160, in remove_iscsi_target
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp raise exception.ISCSITargetRemoveFailed(volume_id=vol_id)
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp ISCSITargetRemoveFailed: Failed to remove iscsi target for volume 04f5af62-a20b-437a-b6fd-c1f6e31499ce.
2013-01-18 05:33:19 4638 TRACE cinder.openstack.common.rpc.amqp
2013-01-18 05:33:35 4638 DEBUG cinder.manager [-] Running periodic task VolumeManager._publish_service_capabilities periodic_tasks /usr/lib/python2.7/dist-packages/cinder/manager.py:164
2013-01-18 05:33:35 4638 DEBUG cinder.manager [-] Running periodic task VolumeManager._report_driver_status periodic_tasks /usr/lib/python2.7/dist-packages/cinder/manager.py:164
2013-01-18 05:34:35 4638 DEBUG cinder.manager [-] Running periodic task VolumeManager._publish_service_capabilities periodic_tasks /usr/lib/python2.7/dist-packages/cinder/manager.py:164
2013-01-18 05:34:35 4638 DEBUG cinder.manager [-] Running periodic task VolumeManager._report_driver_status periodic_tasks /usr/lib/python2.7/dist-packages/cinder/manager.py:164

[John Griffith (john-griffith)]

I ran this against a fresh VM and fresh install of devstack/folsom without issue and I also duplicated your settings for the volume paths found in your cinder.conf, but did not have any problems.

I did NOT however run this on Quantal, I'll tear down and try it on Quantal to see if I can reproduce that way.

[John Griffith (john-griffith)]

I believe I know what's wrong here. You've modified the name template to "vol-", however the remove_iscsi_target is hard-coded to look for "volume-".

I'll get a fix out for this, but in the meantime, if you switch to the default volume name template you should be fine.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/20100

[Jorgen Rosink (jrosink)]

Tnx John,

I cleared up my previous volume create testing stuff, started with a fresh cinder DB and changed the "volume_name_template" config option back to default. I can confirm it's working now as expected !

Sorry for my bad habit to change all kinds of default config stuff, well it helped me to learn Cinder a little bit more ;-).

[John Griffith (john-griffith)]

Great!! Actually I'm glad you change everything, otherwise we wouldn't have found this bug :)

[Mark McLoughlin (markmc)]

Proposed for stable/folsom here: https://review.openstack.org/20102

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/20100
Committed: http://github.com/openstack/cinder/commit/f02c575b4b28b69cade6c1c06dd772c62754fc9c
Submitter: Jenkins
Branch: master

commit f02c575b4b28b69cade6c1c06dd772c62754fc9c
Author: John Griffith <email address hidden>
Date: Sun Jan 20 22:52:52 2013 +0000

    Check for non-default volume name template.

    The iscsi delete methods in iscsi.py had hard-coded volume names
    'volume-%s'.

    If a user configured volume_name_template to
    something other than default the result is failure to remove
    the iscsi_target.

    volume create worked appropriately and created the correct name
    in the DB, so the persistence files were the only thing affected
    or named incorrectly.

    Fixes bug: 1101071

    Change-Id: Id5575874b2cebd9b3509d739cb92abf06c503c4e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/20970

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/20970
Committed: http://github.com/openstack/cinder/commit/aeece14ceab7d10acc63742a3dc55eba4f240f63
Submitter: Jenkins
Branch: stable/folsom

commit aeece14ceab7d10acc63742a3dc55eba4f240f63
Author: John Griffith <email address hidden>
Date: Sun Jan 20 22:52:52 2013 +0000

    Check for non-default volume name template.

    The iscsi delete methods in iscsi.py had hard-coded volume names
    'volume-%s'.

    If a user configured volume_name_template to
    something other than default the result is failure to remove
    the iscsi_target.

    volume create worked appropriately and created the correct name
    in the DB, so the persistence files were the only thing affected
    or named incorrectly.

    Fixes bug: 1101071

    Change-Id: Id5575874b2cebd9b3509d739cb92abf06c503c4e
    (cherry picked from commit f02c575b4b28b69cade6c1c06dd772c62754fc9c)