'iscsiadm ... -o delete' fails occasionally on bulk deployments

[Adam Gandelman]

Attempting to deploy 25 nodes at once, we consistently hit at least one failure caused by iscsiadm on the conductor failing to delete the respective target from the local iscsi db:

2014-05-29 19:42:40.361 5970 ERROR ironic.drivers.modules.pxe [-] PXE deploy failed for instance e7552c1e-cc26-407d-82f6-3293d7731e18. Error: Unexpected error while running command.
Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m node -p 10.22.157.172:3260 -T iqn-fe3d700e-6a02-4c00-ae76-b91a4f8f7d7d -o delete
Exit code: 6
Stdout: ''
Stderr: 'iscsiadm: Could not execute operation on all records: encountered iSCSI database failure\n'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/96611

[OpenStack Infra (hudson-openstack)]

Change abandoned by Adam Gandelman (<email address hidden>) on branch: master
Review: https://review.openstack.org/96611

[Dmitry Tantsur (divius)]

Hi Adam! As your change it abandoned, will you continue working on this bug? If not, could you please unassign yourself and set status back to new?

[aeva black (tenbrae)]

Based on the description (occasional failure, known situation, easy work-around) I'm setting this to Medium sev.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96577
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=af398ee3c47cfd5a0db5f9e760d6b54872c89501
Submitter: Jenkins
Branch: master

commit af398ee3c47cfd5a0db5f9e760d6b54872c89501
Author: Chris Jones <email address hidden>
Date: Thu May 29 21:15:23 2014 +0100

    Work around iscsiadm delete failures

    Attempting to delete an iSCSI target immediately after the initiator has
    logged out of the session can sometimes cause iscsiadm's database update
    to fail. This allows the delete command to be retried up to 5 times on
    failure. We also accept exit code 21 (no records found) as success.

    Closes-bug: #1324670
    Change-Id: I064fcd01e3bd713b0ec19cc1fdb562ca7fd8b666

[Eric Harney (eharney)]

os-brick: https://review.openstack.org/#/c/187811/

[Eric Harney (eharney)]

Cinder stable/kilo: https://review.openstack.org/#/c/232845/