Comment 0 for bug 1838820

Cinder 3rd party CI tests fail for multipath iSCSI targets due to an incorrect invocation of iscsiadm:

Called from connect_volume() in a multipath, multiattach configuration, ISCSIConnector._recover_node_startup_values() passes an iqn wrapped in a list to _iscsiadm_update() via the connection properties 'target_iqn' field, but _iscsiadm_update() expects a plain string which will be used as a command line argument to iscsiadm's -T option. This results in an unexpected iscsiadm failure which aborts the connection attempt.

The system is running Ubuntu 16.04 with open-iscsi 2.0.873+git0.3b4b4500-14ubuntu3.7.

Patching _recover_node_startup_values() to pass a scalar instead of a list appears to solve the problem:

diff --git a/os_brick/initiator/connectors/iscsi.py b/os_brick/initiator/connectors/iscsi.py
index a875b76..c428772 100644
--- a/os_brick/initiator/connectors/iscsi.py
+++ b/os_brick/initiator/connectors/iscsi.py
@@ -1178,7 +1178,7 @@ class ISCSIConnector(base.BaseLinuxConnector, base_iscsi.BaseISCSIConnector):
                 # as discovering target.
                 # So target_iqn is updated, and other values aren't updated.
                 recover_connection = copy.deepcopy(connection_properties)
- recover_connection['target_iqn'] = [iqn]
+ recover_connection['target_iqn'] = iqn
                 self._iscsiadm_update(recover_connection,
                                       "node.startup",
                                       old_node_startup)

Finally, here's a backtrace showing the symptom. In this a volume from the target has been attached to one instance and the test case is attempting to attach the same volume to a second instance:

Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi [None req-d3d16630-3ce1-4a4f-a621-7c061eb9e4e6 tempest-TestMultiAttachVolumeSwap-855628845 tempest-TestMultiAttachVolumeSwap-855628845] Exception\
 encountered during portal discovery: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Command: iscsiadm -m node -T ('iqn.2015-11.com.hpe:storage.msa2050.120114328c',) -p 10.135.193.109:3260 --op update -n node.startup -v automatic
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Exit code: 21
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Stdout: ''
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Stderr: 'iscsiadm: No records found\n'
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi Traceback (most recent call last):
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 194, in _get_ips_iqns_luns
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi ips_iqns_luns = method(connection_properties)
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 439, in _discover_iscsi_portals
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi old_node_startups)
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 1182, in _recover_node_startup_values
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi old_node_startup)
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 1003, in _iscsiadm_update
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi **kwargs)
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 990, in _run_iscsiadm
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi delay_on_retry=delay_on_retry)
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/os_brick/executor.py", line 52, in _execute
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi result = self.__execute(*args, **kwargs)
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi return execute_root(*cmd, **kwargs)
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/oslo_privsep/priv_context.py", line 242, in _wrap
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi return self.channel.remote_call(name, args, kwargs)
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi File "/usr/local/lib/python3.7/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi raise exc_type(*result[2])
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi Command: iscsiadm -m node -T ('iqn.2015-11.com.hpe:storage.msa2050.120114328c',) -p 10.135.193.109:3260 --op update -n node.startup -v automatic
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi Exit code: 21
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi Stdout: ''
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi Stderr: 'iscsiadm: No records found\n'
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: ERROR os_brick.initiator.connectors.iscsi
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: DEBUG os_brick.initiator.connectors.iscsi [None req-d3d16630-3ce1-4a4f-a621-7c061eb9e4e6 tempest-TestMultiAttachVolumeSwap-855628845 tempest-TestMultiAttachVolumeSwap-855628845] iscsiadm \
['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS:
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: DiscoveryAddress: 10.135.193.109,3260
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Target: iqn.2015-11.com.hpe:storage.msa2050.120114328c
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Portal: 10.135.193.110:3260,6
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Iface Name: default
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Portal: 10.135.193.109:3260,5
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: Iface Name: default
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: iSNS:
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: No targets found.
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: STATIC:
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: No targets found.
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: FIRMWARE:
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: No targets found.
Aug 01 19:38:03 seagate-ci-656724-13 nova-compute[6560]: stderr= {{(pid=6560) _run_iscsiadm_bare /usr/local/lib/python3.7/dist-packages/os_brick/initiator/connectors/iscsi.py:1121}}