Changed SAN iscsi IP addresses in connection_info can prevent VM startup

[Yafei Yu]

The iscsi IP addresses of SAN are stored in nova's block-device-mapping table after connecting volume and are never re-validated down the line. Changing the iscsi IP adresses of SAN will prevent the instance from booting as the stale connection info will enter the instance's XML. We should check weather the iscsi IP addresses of SAN are changed before startup VM, or we need to a way to restore those VMs whose iscsi IP addresses of SAN are changed.

[Matt Riedemann (mriedem)]

There is a similar bug for when the ceph cluster IP changes, and a patch proposed here:

https://review.openstack.org/#/c/289813/

The patch has some issues though and looks more or less abandoned.

We don't really have a good solution for this kind of issue right now for when things change on the backing storage underneath nova.

What version of Nova are you hitting this on?

Do you have a stacktrace of the failure?

[Yafei Yu (yu-yafei)]

The nova version is kilo, and the bug is still existed in liberty and mitaka.

Steps:
1. Create a volume from image and storage is SAN.
2. Boot a instance from volume.
3. Modify the iscsi IP addresses of SAN.
4. Hard reboot the instance.

The stacktrace of the failure:
2016-05-16 17:42:09.328 26645 ERROR oslo_messaging.rpc.dispatcher [req-46ecfc98-97d4-40f9-9e40-e01e14fe4790 5650304261d74dbe8a4f7848661f95a6 55bbe36a87af48c0af04c9204a49a854 - - -] Exception during message handling: iSCSI device not found at [u'/dev/disk/by-path/ip-172.10.150.52:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:51:85-lun-9', u'/dev/disk/by-path/ip-172.10.151.52:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:51:85-lun-9', u'/dev/disk/by-path/ip-172.10.152.52:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:51:85-lun-9', u'/dev/disk/by-path/ip-172.10.153.52:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:51:85-lun-9', u'/dev/disk/by-path/ip-172.10.150.53:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:52:03-lun-9', u'/dev/disk/by-path/ip-172.10.151.53:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:52:03-lun-9', u'/dev/disk/by-path/ip-172.10.152.53:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:52:03-lun-9', u'/dev/disk/by-path/ip-172.10.153.53:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:52:03-lun-9']
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8761, in reboot_instance
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher reboot_type)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher payload)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/441269

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/441269
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Matt Riedemann (mriedem)]

Discussed this at the queens ptg, notes are here: https://etherpad.openstack.org/p/cinder-ptg-queens