iSCSI login from conductor fails. iscsiadm: No records found

[Adam Gandelman]

First time seeing this... on a deployment of a 30 node overcloud, one node failed deployment. Relevant conductor log pasted, and entire log attached.

2014-06-06 19:14:08.243 5794 DEBUG ironic.common.glance_service.base_image_service [-] Getting image metadata from glance. Image: 3bc59e4b-7b2e-43e3-a2df-16a6600ce92c _show /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/glance_service/base_image_service.py:185
2014-06-06 19:14:08.257 5794 DEBUG ironic.common.utils [-] Execution completed, command line is "iscsiadm -m discovery -t st -p 10.22.167.70:3260" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:65
2014-06-06 19:14:08.257 5794 DEBUG ironic.common.utils [-] Command stdout is: "10.22.167.70:3260,1 iqn-90f14032-1c60-4fe9-9ef0-d64d827e04e8
" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:66
2014-06-06 19:14:08.257 5794 DEBUG ironic.common.utils [-] Command stderr is: "iscsiadm: Could not make /etc/iscsi/nodes: File exists

iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.22.167.70,3260,1 iqn-90f14032-1c60-4fe9-9ef0-d64d827e04e8]
" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:67
2014-06-06 19:14:08.257 5794 DEBUG ironic.openstack.common.processutils [-] Running cmd (subprocess): sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m node -p 10.22.167.70:3260 -T iqn-90f14032-1c60-4fe9-9ef0-d64d827e04e8 --login execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/openstack/common/processutils.py:142
2014-06-06 19:14:08.262 5794 DEBUG ironic.common.utils [-] Execution completed, command line is "iscsiadm -m discovery -t st -p 10.22.167.73:3260" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:65
2014-06-06 19:14:08.262 5794 DEBUG ironic.common.utils [-] Command stdout is: "10.22.167.73:3260,1 iqn-c0c69d97-3490-403d-836a-dd2dd33c8fe6
" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:66
2014-06-06 19:14:08.262 5794 DEBUG ironic.common.utils [-] Command stderr is: "" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:67
2014-06-06 19:14:08.262 5794 DEBUG ironic.openstack.common.processutils [-] Running cmd (subprocess): sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m node -p 10.22.167.73:3260 -T iqn-c0c69d97-3490-403d-836a-dd2dd33c8fe6 --login execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/openstack/common/processutils.py:142
2014-06-06 19:14:08.273 5794 DEBUG ironic.common.utils [-] Execution completed, command line is "iscsiadm -m discovery -t st -p 10.22.167.66:3260" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:65
2014-06-06 19:14:08.274 5794 DEBUG ironic.common.utils [-] Command stdout is: "10.22.167.66:3260,1 iqn-7b390bb2-f040-4382-9b5f-4346a6c14368
" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:66
2014-06-06 19:14:08.274 5794 DEBUG ironic.common.utils [-] Command stderr is: "" execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/utils.py:67
2014-06-06 19:14:08.274 5794 DEBUG ironic.openstack.common.processutils [-] Running cmd (subprocess): sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m node -p 10.22.167.66:3260 -T iqn-7b390bb2-f040-4382-9b5f-4346a6c14368 --login execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/openstack/common/processutils.py:142
2014-06-06 19:14:08.307 5794 DEBUG ironic.common.glance_service.base_image_service [-] Getting image metadata from glance. Image: 12717a13-01e8-45f3-8a96-1328957e5375 _show /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/common/glance_service/base_image_service.py:185
2014-06-06 19:14:08.314 5794 DEBUG ironic.openstack.common.processutils [-] Result was 21 execute /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/openstack/common/processutils.py:167
2014-06-06 19:14:08.314 5794 ERROR ironic.drivers.modules.pxe [-] PXE deploy failed for instance 9f3da368-9666-4b02-a340-413252f3fd20. Error: Unexpected error while running command.
Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m node -p 10.22.167.70:3260 -T iqn-90f14032-1c60-4fe9-9ef0-d64d827e04e8 --login
Exit code: 21
Stdout: ''
Stderr: 'iscsiadm: No records found\n'
2014-06-06 19:14:08.325 5794 DEBUG sqlalchemy.pool.QueuePool [-] Connection <_mysql.connection open to 'localhost' at 264d920> being returned to pool _finalize_fairy /opt/stack/venvs/ironic/local/lib/python2.7/site-packages/sqlalchemy/pool.py:549

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/98510

[aeva black (tenbrae)]

Setting priority to medium as there is a fairly easy work-around -- delete the instance and recreate it.

17:37:28 < adam_g> devananda, sorry, re 1327392.. yes, deleting the instance and re-creating it seemed to work. the issue was random and seemed only to happen during a large number of concurrent deployments

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98510
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=909cff2840c711ac923837ed5bbbad2048b1ea41
Submitter: Jenkins
Branch: master

commit 909cff2840c711ac923837ed5bbbad2048b1ea41
Author: Chris Krelle <email address hidden>
Date: Fri Jun 6 14:26:52 2014 -0600

    Add retry logic to iscsiadm commands

    In order to avoid random iscsiadm errors this patch adds retry logic
    to the iscsiadm commands. All iscsiadm commands will be attempted
    5 times before failing.

    Closes-Bug: #1327392

    Change-Id: I971db541efbaaaf76bb6266570d7606d5b8285e7