tgtadm: this access control rule does not exist - results in timeout waiting for volume to attach (stuck in detach) or detach

[Matt Riedemann]

http://logs.openstack.org/97/157497/2/check/check-tempest-dsvm-nova-v21-full/3cd8eef/console.html#_2015-02-24_20_12_54_652

Seeing this:

http://logs.openstack.org/97/157497/2/check/check-tempest-dsvm-nova-v21-full/3cd8eef/logs/screen-c-vol.txt.gz#_2015-02-24_19_59_10_603

2015-02-24 19:59:10.603 726 ERROR cinder.volume.targets.tgt [req-316ccffa-b93f-4631-bf9e-9df7429a766d 7b2803e6fe28498aab1356df3e0f294d 8ed6978b84394a24a64359eda55d044c - - -] Failed to remove iscsi target for volume ID: 7bb826e3-c998-4cba-a58e-309c99fd769d: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgt-admin --force --delete iqn.2010-10.org.openstack:volume-7bb826e3-c998-4cba-a58e-309c99fd769d
Exit code: 22
Stdout: u'Command:\n\ttgtadm -C 0 --op unbind --mode target --tid 1 -I ALL\nexited with code: 22.\n'
Stderr: u'tgtadm: this access control rule does not exist\n'
2015-02-24 19:59:10.603 726 ERROR cinder.volume.manager [req-316ccffa-b93f-4631-bf9e-9df7429a766d 7b2803e6fe28498aab1356df3e0f294d 8ed6978b84394a24a64359eda55d044c - - -] Error detaching volume 7bb826e3-c998-4cba-a58e-309c99fd769d, due to remove export failure.
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager Traceback (most recent call last):
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 782, in detach_volume
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager self.driver.remove_export(context.elevated(), volume)
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager return f(*args, **kwargs)
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/lvm.py", line 603, in remove_export
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager self.target_driver.remove_export(context, volume)
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/targets/iscsi.py", line 231, in remove_export
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager volume['name'])
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/targets/tgt.py", line 311, in remove_iscsi_target
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager raise exception.ISCSITargetRemoveFailed(volume_id=vol_id)
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager ISCSITargetRemoveFailed: Failed to remove iscsi target for volume 7bb826e3-c998-4cba-a58e-309c99fd769d.
2015-02-24 19:59:10.603 726 TRACE cinder.volume.manager

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwidGhpcyBhY2Nlc3MgY29udHJvbCBydWxlIGRvZXMgbm90IGV4aXN0XCIgQU5EIHRhZ3M6XCJzY3JlZW4tYy12b2wudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MjQ4MTgyMjI2NzMsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

Looks like this started happening around 2/20. It's only about a 59% failure rate when it shows up so definitely racy.

[Matt Riedemann (mriedem)]

Could be related to bug 1421895.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/162369

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/162369
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9f49be03b2e53b9e5eae36b58ba03ee3ff3e4ff4
Submitter: Jenkins
Branch: master

commit 9f49be03b2e53b9e5eae36b58ba03ee3ff3e4ff4
Author: Mike Perez <email address hidden>
Date: Fri Mar 6 17:24:00 2015 -0800

    Don't fail target_delete if ACL's don't exist

    Seen in the gate a bit where a target delete is called and fails because
    the ACL's don't exist for that target. If we're deleting a target and
    the ACL's don't exist, it's probably safe to move on.

    This patch adds a check for this specific case, logs a warning, and
    continues rather than raising an exception.

    Closes-Bug: #1425310
    Change-Id: Ideada8ddea8624aff7add95e9caad2aba4e31ae3