*sigh* I had this totally wrong earlier, this has nothing to do with trunk ports and is just duplicate iSCSI initiators yet again (I feel like we've fixed this many times in the past) causing the connection to the volume to be cut during post_live_migration when we unmap them from the source.
First of all the instances are clearly being used by test_volume_backed_live_migration tests:
45adbb55-491d-418b-ba68-7db43d1c235b is being live migrated to compute1 from controller with volume
ba95db3d-c2ff-40c6-ba7c-563c3070c85d attached that is itself served from c-vol on controller.
cfdf210a-f2bf-46a4-85f2-56e7d58893c2 is being live migrated to controller from compute1 with volume 867090f6-2e48-4b0b-97e3-afdbc4cc946b attached that is itself served from c-vol on compute1.
45adbb55-491d-418b-ba68-7db43d1c235b wins the race to migrate first and we then see the ba95db3d-c2ff-40c6-ba7c-563c3070c85d unmapped from the initiator that is being used for both the source and destination hosts:
This then leads to the libvirtd lockup previously discussed in bug #1912310 that causes the live migration of cfdf210a-f2bf-46a4-85f2-56e7d58893c2 to fail.
I'll have a quick look at devstack now but this should be trivial to resolve.
*sigh* I had this totally wrong earlier, this has nothing to do with trunk ports and is just duplicate iSCSI initiators yet again (I feel like we've fixed this many times in the past) causing the connection to the volume to be cut during post_live_migration when we unmap them from the source.
First of all the instances are clearly being used by test_volume_ backed_ live_migration tests:
2021-10-01 11:10:18.262 118292 INFO tempest. lib.common. rest_client [req-43af85da- 1678-4d4d- 950a-69062abb6a ae ] Request (LiveMigrationT est:test_ volume_ backed_ live_migration) : 202 POST https:/ /198.72. 124.126/ compute/ v2.1/servers/ 45adbb55- 491d-418b- ba68-7db43d1c23 5b/action 1.164s
2021-10-01 11:10:24.911 118290 INFO tempest. lib.common. rest_client [req-14fb6fc1- 9d0a-44f7- 9fbc-d24befa6ff eb ] Request (LiveAutoBlockM igrationV225Tes t:test_ volume_ backed_ live_migration) : 202 POST https:/ /198.72. 124.126/ compute/ v2.1/servers/ cfdf210a- f2bf-46a4- 85f2-56e7d58893 c2/action 1.221s
45adbb55- 491d-418b- ba68-7db43d1c23 5b is being live migrated to compute1 from controller with volume c2ff-40c6- ba7c-563c3070c8 5d attached that is itself served from c-vol on controller.
ba95db3d-
cfdf210a- f2bf-46a4- 85f2-56e7d58893 c2 is being live migrated to controller from compute1 with volume 867090f6- 2e48-4b0b- 97e3-afdbc4cc94 6b attached that is itself served from c-vol on compute1.
ubuntu- focal-iweb- mtl01-002675135 1 == controller focal-iweb- mtl01-002675135 2 == compute1
ubuntu-
We see the same initiator used to map both volumes to each compute when these instances are initially spawned:
Oct 01 11:10:11 ubuntu- focal-iweb- mtl01-002675135 1 sudo[121117]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND= /usr/local/ bin/cinder- rootwrap /etc/cinder/ rootwrap. conf cinder-rtstool add-initiator iqn.2010- 10.org. openstack: volume- ba95db3d- c2ff-40c6- ba7c-563c3070c8 5d MNgH9PG7eU8gt7Q Ldi9n Vo78NSpxj9JAFfs4 iqn.1993- 08.org. debian: 01:ef12d882804f
Oct 01 11:10:18 ubuntu- focal-iweb- mtl01-002675135 2 sudo[73346]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND= /usr/local/ bin/cinder- rootwrap /etc/cinder/ rootwrap. conf cinder-rtstool add-initiator iqn.2010- 10.org. openstack: volume- 867090f6- 2e48-4b0b- 97e3-afdbc4cc94 6b h6kS7Yf39w3Xwp4 qk63J i9EckeRwVHst5CKD iqn.1993- 08.org. debian: 01:ef12d882804f
Later during the live migration attempts we again see the same initiators used to map the volumes to the destination hosts:
Oct 01 11:10:21 ubuntu- focal-iweb- mtl01-002675135 1 sudo[121346]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND= /usr/local/ bin/cinder- rootwrap /etc/cinder/ rootwrap. conf cinder-rtstool add-initiator iqn.2010- 10.org. openstack: volume- ba95db3d- c2ff-40c6- ba7c-563c3070c8 5d MNgH9PG7eU8gt7Q Ldi9n Vo78NSpxj9JAFfs4 iqn.1993- 08.org. debian: 01:ef12d882804f
Oct 01 11:10:28 ubuntu- focal-iweb- mtl01-002675135 2 sudo[73576]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND= /usr/local/ bin/cinder- rootwrap /etc/cinder/ rootwrap. conf cinder-rtstool add-initiator iqn.2010- 10.org. openstack: volume- 867090f6- 2e48-4b0b- 97e3-afdbc4cc94 6b h6kS7Yf39w3Xwp4 qk63J i9EckeRwVHst5CKD iqn.1993- 08.org. debian: 01:ef12d882804f
45adbb55- 491d-418b- ba68-7db43d1c23 5b wins the race to migrate first and we then see the ba95db3d- c2ff-40c6- ba7c-563c3070c8 5d unmapped from the initiator that is being used for both the source and destination hosts:
Oct 01 11:10:29 ubuntu- focal-iweb- mtl01-002675135 1 sudo[121509]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND= /usr/local/ bin/cinder- rootwrap /etc/cinder/ rootwrap. conf cinder-rtstool delete-initiator iqn.2010- 10.org. openstack: volume- ba95db3d- c2ff-40c6- ba7c-563c3070c8 5d iqn.1993- 08.org. debian: 01:ef12d882804f
Straight away the destination host compute1 with the running instance then reports I/O failures:
Oct 01 11:10:29 ubuntu- focal-iweb- mtl01-002675135 2 kernel: connection2:0: detected conn error (1020) focal-iweb- mtl01-002675135 2 kernel: iSCSI/iqn. 1993-08. org.debian: 01:ef12d882804f : Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
Oct 01 11:10:29 ubuntu-
This then leads to the libvirtd lockup previously discussed in bug #1912310 that causes the live migration of cfdf210a- f2bf-46a4- 85f2-56e7d58893 c2 to fail.
I'll have a quick look at devstack now but this should be trivial to resolve.