Infinidat Cinder driver truncates the list of configured iSCSI portals and returns only the first IP for a given network space. And as a result - in case of network path failure we lose access to the data.
Steps to reproduce:
* Configure an OpenStack environment with Infinidat iSCSI backend
* Enable multipath feature on the Infinidat storage side
* Enable multipath feature on the OpenStack environment
* Boot a VM from the Infinidat iSCSI volume
* Simulate failure of one network path
Expected result:
The VM continues to access the disk
Actual result:
The VM loses disk access and no data is available
Infinidat Storage multipath configuration:
admin@localhost> config.net_space.ip.query
NETWORK SPACE IP ADDRESS ENABLED NODE NETWORK INTERFACE TYPE
default_iscsi_space 1.1.1.1 yes 1 PG1 iSCSI
default_iscsi_space 1.1.1.2 yes 1 PG2 iSCSI
default_iscsi_space 1.1.1.3 yes 1 PG3 iSCSI
default_iscsi_space 1.1.1.4 yes 1 PG4 iSCSI
Cinder configuration file:
[iscsi]
san_password = password
san_login = user
san_ip = 2.2.2.2
infinidat_storage_protocol = iSCSI
infinidat_pool_name = iscsi
infinidat_iscsi_netspaces = default_iscsi_space
volume_backend_name = iscsi
volume_driver = cinder.volume.drivers.infinidat.InfiniboxVolumeDriver
Nova configuration file:
nova.conf:
[libvirt]
volume_use_multipath = True
Nova log file on VM boot:
ul 11 18:06:19 devstack-aio-iscsi-master nova-compute[191897]: DEBUG os_brick.initiator.connectors.iscsi [None req-a7002799-7bb1-43bd-9060-2329d0456f54 admin admin] Connected to sdb using {'target_discovered': True, 'target_portal': '1.1.1.1:3260', 'target_iqn': 'iqn.2009-11.com.infinidat:storage:infinibox-sn-12345', 'target_lun': 2, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False} {{_connect_vol /opt/stack/os-brick/os_brick/initiator/connectors/iscsi.py:647}}
Multipath configuration when attaching the volume:
$ sudo multipath -ll
mpathbz (36742b0f0000076de0000000000005ee2) dm-0 NFINIDAT,InfiniBox
size=1.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
`- 3:0:0:2 sdb 8:16 active ready running
Nova iSCSI client:
$ sudo iscsiadm -m node
1.1.1.1:3260,-1 iqn.2009-11.com.infinidat:storage:infinibox-sn-12345
Multipath configuration after network path failure:
$ sudo multipath -ll
mpathbz (36742b0f0000076de0000000000005ee2) dm-0 NFINIDAT,InfiniBox
size=1.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=enabled
`- 3:0:0:2 sdb 8:16 failed faulty running
Nova compute node dmesg after network path failure:
$ sudo dmesg
[40618.846345] connection43:0: detected conn error (1020)
[40717.076034] connection43:0: detected conn error (1020)
[40877.768677] connection43:0: detected conn error (1020)
[40998.808741] session43: session recovery timed out after 120 secs
[40998.824643] sd 3:0:0:2: rejecting I/O to offline device
[40998.826502] blk_update_request: I/O error, dev sdb, sector 885760 op 0x0:(READ) flags 0x4200
phys_seg 256 prio class 0
[40998.828651] sd 3:0:0:2: rejecting I/O to offline device
[40998.830093] sd 3:0:0:2: rejecting I/O to offline device
[40998.832005] device-mapper: multipath: Failing path 8:16.
[40998.834714] blk_update_request: I/O error, dev dm-0, sector 105328 op 0x1:(WRITE) flags 0x880
0 phys_seg 1 prio class 0
[40998.834732] blk_update_request: I/O error, dev sdb, sector 884736 op 0x0:(READ) flags 0x4200
phys_seg 128 prio class 0
[40998.842137] blk_update_request: I/O error, dev dm-0, sector 885760 op 0x0:(READ) flags 0x0 ph
ys_seg 256 prio class 0
[40998.859789] blk_update_request: I/O error, dev dm-0, sector 884736 op 0x0:(READ) flags 0x0 ph
ys_seg 1 prio class 0
[40998.870530] blk_update_request: I/O error, dev dm-0, sector 18536 op 0x1:(WRITE) flags 0x8800
phys_seg 1 prio class 0
[40998.874154] blk_update_request: I/O error, dev dm-0, sector 24960 op 0x1:(WRITE) flags 0x8800
phys_seg 1 prio class 0
To fix this issue, we need to return all configured iSCSI portals for a given network space.