In an iSCSI/multipath environment, a Cinder volume is attached to a tenant server (VM) on a compute host. After the compute host is rebooted, some of the connection(s) to the iSCSI portal(s) are not reinitiated, because of this, missing iSCSI device(s) are observed. This issue is observed for both stable Train and Ussuri releases.
When PS https://review.opendev.org/c/openstack/os-brick/+/796671 is not applied, below stack trace is captured.
Exception in thread Thread-39:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/var/lib/openstack/lib/python3.6/site-packages/os_brick/executor.py", line 84, in run
super(Thread, self).run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/var/lib/openstack/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 626, in _connect_vol
session, manual_scan = self._connect_to_iscsi_portal(props)
File "/var/lib/openstack/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 1042, in _connect_to_iscsi_portal
'--op', 'new'))
File "/var/lib/openstack/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 995, in _run_iscsiadm
delay_on_retry=delay_on_retry)
File "/var/lib/openstack/lib/python3.6/site-packages/os_brick/executor.py", line 52, in _execute
result = self.__execute(*args, **kwargs)
File "/var/lib/openstack/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
return execute_root(*cmd, **kwargs)
File "/var/lib/openstack/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap
return self.channel.remote_call(name, args, kwargs)
File "/var/lib/openstack/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call
raise exc_type(*result[2])
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: iscsiadm -m node -T iqn.xxxxxxxxxxxx -p xxx.xxx.xxx.xxx:3260 --interface default --op new
Exit code: 6
Stdout: ''
Stderr: 'iscsiadm: Could not make /etc/iscsi/nodes/iqn.xxxxxxxxxxxxxxx: File exists\niscsiadm: Error while adding record: encountered iSCSI database failure\n'
After the above PS is applied, the exception is handled, however the underline issue is still exist and missing iSCSI device(s) are observed.
Suggestion:
Is it possible to add some retries to _connect_to_iscsi_portal of class ISCSIConnector in iscsi.py if the second iscsiadm command has a return code of 6?
Please refer to this patchset for a potential fix:
https:/