*sigh* I had this totally wrong earlier, this has nothing to do with trunk ports and is just duplicate iSCSI initiators yet again (I feel like we've fixed this many times in the past) causing the connection to the volume to be cut during post_live_migration when we unmap them from the source.

First of all the instances are clearly being used by test_volume_backed_live_migration tests:

2021-10-01 11:10:18.262 118292 INFO tempest.lib.common.rest_client [req-43af85da-1678-4d4d-950a-69062abb6aae ] Request (LiveMigrationTest:test_volume_backed_live_migration): 202 POST https://198.72.124.126/compute/v2.1/servers/45adbb55-491d-418b-ba68-7db43d1c235b/action 1.164s

2021-10-01 11:10:24.911 118290 INFO tempest.lib.common.rest_client [req-14fb6fc1-9d0a-44f7-9fbc-d24befa6ffeb ] Request (LiveAutoBlockMigrationV225Test:test_volume_backed_live_migration): 202 POST https://198.72.124.126/compute/v2.1/servers/cfdf210a-f2bf-46a4-85f2-56e7d58893c2/action 1.221s

45adbb55-491d-418b-ba68-7db43d1c235b is being live migrated to compute1 from controller with volume 
ba95db3d-c2ff-40c6-ba7c-563c3070c85d attached that is itself served from c-vol on controller.

cfdf210a-f2bf-46a4-85f2-56e7d58893c2 is being live migrated to controller from compute1 with volume 867090f6-2e48-4b0b-97e3-afdbc4cc946b attached that is itself served from c-vol on compute1.

ubuntu-focal-iweb-mtl01-0026751351 == controller
ubuntu-focal-iweb-mtl01-0026751352 == compute1

We see the same initiator used to map both volumes to each compute when these instances are initially spawned:

Oct 01 11:10:11 ubuntu-focal-iweb-mtl01-0026751351 sudo[121117]:   stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool add-initiator iqn.2010-10.org.openstack:volume-ba95db3d-c2ff-40c6-ba7c-563c3070c85d MNgH9PG7eU8gt7QLdi9n Vo78NSpxj9JAFfs4 iqn.1993-08.org.debian:01:ef12d882804f

Oct 01 11:10:18 ubuntu-focal-iweb-mtl01-0026751352 sudo[73346]:    stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool add-initiator iqn.2010-10.org.openstack:volume-867090f6-2e48-4b0b-97e3-afdbc4cc946b h6kS7Yf39w3Xwp4qk63J i9EckeRwVHst5CKD iqn.1993-08.org.debian:01:ef12d882804f

Later during the live migration attempts we again see the same initiators used to map the volumes to the destination hosts:

Oct 01 11:10:21 ubuntu-focal-iweb-mtl01-0026751351 sudo[121346]:    stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool add-initiator iqn.2010-10.org.openstack:volume-ba95db3d-c2ff-40c6-ba7c-563c3070c85d MNgH9PG7eU8gt7QLdi9n Vo78NSpxj9JAFfs4 iqn.1993-08.org.debian:01:ef12d882804f

Oct 01 11:10:28 ubuntu-focal-iweb-mtl01-0026751352 sudo[73576]:    stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool add-initiator iqn.2010-10.org.openstack:volume-867090f6-2e48-4b0b-97e3-afdbc4cc946b h6kS7Yf39w3Xwp4qk63J i9EckeRwVHst5CKD iqn.1993-08.org.debian:01:ef12d882804f

45adbb55-491d-418b-ba68-7db43d1c235b wins the race to migrate first and we then see the ba95db3d-c2ff-40c6-ba7c-563c3070c85d unmapped from the initiator that is being used for both the source and destination hosts:

Oct 01 11:10:29 ubuntu-focal-iweb-mtl01-0026751351 sudo[121509]:    stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool delete-initiator iqn.2010-10.org.openstack:volume-ba95db3d-c2ff-40c6-ba7c-563c3070c85d iqn.1993-08.org.debian:01:ef12d882804f

Straight away the destination host compute1 with the running instance then reports I/O failures:

Oct 01 11:10:29 ubuntu-focal-iweb-mtl01-0026751352 kernel:  connection2:0: detected conn error (1020)
Oct 01 11:10:29 ubuntu-focal-iweb-mtl01-0026751352 kernel: iSCSI/iqn.1993-08.org.debian:01:ef12d882804f: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.

This then leads to the libvirtd lockup previously discussed in bug #1912310 that causes the live migration of cfdf210a-f2bf-46a4-85f2-56e7d58893c2 to fail.

I'll have a quick look at devstack now but this should be trivial to resolve.