Insufficient error handling when parsing iscsiadm -m node output with blank iscsi target

[Pushkar Acharya]

We encountered the following error when attempting to reboot a VM with multiple attached volumes -

2020-07-02 05:46:05.960 ERROR oslo_messaging.rpc.server [req-0c171deb-bc82-4687-91d6-76e8f95b8e19 **** service] Exception during message handling: IndexError: list index out of range
...
2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 157, in _get_iscsi_nodes
2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server lines.append((info[0].split(',')[0], info[1]))
2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server IndexError: list index out of range
This is observed on os-brick version - 1.15.9

The same code in current master branch -
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L136

iscsiadm -m node output -
172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-f1ff35f1-9716-4929-831f-32e7b207c742
172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-5393e371-337f-4332-b39f-4926e4a1f9f7
172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d
[]:3260,-1
172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb
172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-c1b325b9-7bd2-4d91-a3ef-295736e52eca
172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-6a1a112e-1140-482d-9064-fe1b03391f2b

The blank target causes an unhandled exception. A simple python code snippet to show the same -
>>> out = "172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d\n[]:3260,-1\n172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb"
>>> lines = []
>>> out.splitlines()
['172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d', '[]:3260,-1', '172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb']
>>> for line in out.splitlines():
... if line:
... info = line.split()
... lines.append((info[0].split(',')[0], info[1]))
...
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>

The blank iscsi target was most probably due to corruption of the iscsi data during discovery.

Using strace we could trace that blank target belongs to a OpenStack volume -
open("/var/lib/iscsi/nodes/iqn.2010-10.org.openstack:volume-d88869e6-d27b-4121-9bd1-d8c86ce9d7e1/172.30.0.193,3260", O_RDONLY) = 5

Expected:
Management of that single volume associated with blank iscsi target to be affected.

Observed:
None of the volumes and volume backed VMs can be managed on the affected host.

[Pooja Ghumre (pooja-9)]

Is there a way to handle blank iscsi target files gracefully?

[Luigi Toscano (ltoscano)]

I suspect the gerrit integration didn't work, but this patch should address the issue:
https://review.opendev.org/c/openstack/os-brick/+/763599

[Pooja Ghumre (pooja-9)]

Thanks for the update, Luigi!

Will this fix be backported to the os-brick version used in Openstack Rocky release?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 4.2.0 release.

[Pooja Ghumre (pooja-9)]

Cinder needs to consume the fixed version of os-brick in upper constraints for older releases.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 4.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 3.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 2.10.6 release.