Comment 60 for bug 1569925

Anyone facing this..

Could you please make sure to propagate I/O errors properly to filesystem in order for it to call its automatically shutdown procedure. After that, the mount won't reference the super block in kernel and that will allow the device mapper to be destroyed (this can be seen by checking existence of device-mapper node in /sys/fs/{xfs,ext3,ext4}/node, saying that the super block is still being referenced).

I had a recent case where not propagating the error would cause the filesystem to hang forever, causing lockups and not allowing the device-mapper tables to be flushed by multipath (or even the OS to be operational). In this case, if the network is turned off (or interrupted) before the ISCSI is logged out, then the multipath (dm) and iscsi (transport) layers HAVE TO propagate the I/O error to the filesystem for it to shutdown BEFORE the device mapper flush logic is attempted (by issuing ioctls to /dev/dm-X by multipath or lvm, example).

This is can be done by:

node.session.timeo.replacement_timeout = 0 (iscsid.conf)
(This parameter might be persistent on already discovered LUNs)

USAGE:
iscsiadm -m node -T $target_name -p $target_ip:$port -o update -n \
node.session.timeo.replacement_timeout -v $timeout_value
on already logged/discovered LUNs.

---- AND ----

dev_loss_tmo 10 (multipath.conf)

The fact that the network was removed BEFORE the ISCSI logout happened is NOT an issue if ROOT device is NOT on ISCSI: You just have to allow the block device <-> device mapper <-> transport layer SHUTDOWN logic to WORK before flushing the device-mapper. (Hopefully the umount on _netdev devices is done, causes an I/O error, filesystem shuts down, multipath service closes multipath device-mapper devices).

Pay attention because if filesystem is umounted and ISCSI is gone, the timeout will have to be waited before the error is PROPAGATED to other layer. So if filesystem is umounted, ONE last superblock inode UPDATE will be attempted (if no cache in pagecache). This update is to sync superblock state and update metadata for the filesystem itself. This update attempt will create an I/O error for the superblock inode (after time timeout you configured). This will make the filesystem to be shutdown (a half unmounted state). After shutdown, the logic trying to umount (synchronously) the filesystem will continue (possibly in systemd).

If the problem is because of you tried to flush the device mapper and the filesystem was still referencing the device mapper super block, this will fade away if the error is propagated and the filesystem is shutdown, you can attempt doing the flush of device-mapper right after filesystem shutdown happened.

Could you please try setting the timeouts properly for lower timings and let me know if that mitigated the issue ?

Thank you