The problem:
If Cinder node reboots or crashes, Cinder doesn't restore LIO target ACL rules for existing targets. That leads to:
1) On Cinder node targetcli reports for example:
o- iqn.2010-10.org.openstack:volume-9b86b9d7-6db6-4a14-afac-a4bd6f1cc6c5 ............................................... [1 TPG]
| o- tpgt1 ........................................................................................................... [enabled]
| o- acls ........................................................................................................... [0 ACLs]
| o- luns ............................................................................................................ [1 LUN]
| | o- lun0 [iblock/iqn.2010-10.org.openstack:volume-9b86b9d7-6db6-4a14-afac-a4bd6f1cc6c5 (/dev/mediumgroup/volume-9b86b9d7-6db6-4a14-afac-a4bd6f1cc6c5)]
| o- portals ...................................................................................................... [1 Portal]
| o- 10.10.21.32:3260 .................................................................................. [OK, iser disabled]
There is expected [1 ACL] instead of [0 ACL] because that volume was in-use at the moment of reboot.
In syslog also there is repeated each second:
iSCSI Initiator Node: iqn.2015-12.ru.ispras.cloud:01:a34486dd48c is not authorized to access iSCSI target portal group
2) On Compute node which hosts this VM, you can see in syslog:
iscsid: conn 0 login rejected: initiator failed authorization with target
3) On VM that uses the volume you can see IO error on every try to use the volume.
The environment:
Openstack multinode setup; each component is setup from git branch stable/liberty (cinder last commit is 6d0981b252835a525149714e39cc1e4ee7568315).
Cinder host: Ubuntu 14.04, python 2.7.6, Cinder from stable/liberty (6d0981b252835a525149714e39cc1e4ee7568315), kernel 3.19.0-47-generic, targetcli 2.1-1 from repo, python-rtslib 2.2-1 from repo.
Config part for these volumes:
[fast-1]
volume_group=fastgroup
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name=LVM_iSCSI_fast
iscsi_protocol = iscsi
iscsi_helper = lioadm
The way to reproduce:
1) Install and setup Cinder from stable/liberty github version with config [fast-1] part and LIO
2) Create Volume and attach it to any VM.
3) Simulate 'crash' - reboot the Cinder node without stopping VM.
4) After reboot start Cinder and look for errors and missing ACL (Volume is not usable anymore).
The clues:
1) After Volume attachment /etc/target/saveconfig.json seems to be correct. After reboot and Cinder restart it becomes incorrect (looses ACLs).
2) May be connected to patches:
the first one - https://github.com/openstack/cinder/commit/6686b896d7ef5e11f1f277ded008d153edc183a8 and the second one - https://github.com/openstack/cinder/commit/706878deaaa5acbb8a577942a5a774c58fe16332
The workaround:
Nethertheless there is a workaround (but it seems painful). If you use "sudo targetctl save backup/before-reboot.json" and "sudo targetctl restore backup/before-reboot.json" (TARGETCTL - just in case - I mean the tool that is in Cinder itself), everything seems to be restored correctly. But it doesn't save us from Cinder node crashes and it should be done with crontab or something (by time not by Cinder actual events)
Targetcli configs (IMPORTANT):
1) Just right after Volume creation and attachment - https://gist.github.com/anonymous/5f1295270c36d19df9d7
2) After Cinder reboot and restart - https://gist.github.com/anonymous/0f110ce214359e41b871
3) Diff between them (that's the actual bug occurance) - https://gist.github.com/anonymous/bc41f4e51b46e8426c79
NOTE: in all the LIO configs there are already broken volumes, so just look at the diff-file first.
Let me try to reproduce and investigate.