Comment 8 for bug 1807723

Revision history for this message
Matt Riedemann (mriedem) wrote : Re: swap multiattach volume intermittently fails when servers are on different hosts

Details from attaching volume1 to server1:

http://logs.openstack.org/81/606981/4/check/tempest-slow/fafde23/compute1/logs/screen-n-cpu.txt.gz#_Dec_08_01_43_12_388771

Dec 08 01:43:12.388771 ubuntu-xenial-ovh-gra1-0001066278 nova-compute[20206]: INFO nova.compute.manager [None req-15ea8e06-99b2-439b-bad8-d0f0f6bee781 tempest-TestMultiAttachVolumeSwap-1803326020 tempest-TestMultiAttachVolumeSwap-1803326020] [instance: d46fba31-9469-4799-b2bf-1fbad4369a9a] Attaching volume 2a48ae08-2833-488c-8771-eea86f41c7b8 to /dev/vdb

Dec 08 01:43:16.269897 ubuntu-xenial-ovh-gra1-0001066278 nova-compute[20206]: DEBUG os_brick.initiator.connectors.iscsi [None req-15ea8e06-99b2-439b-bad8-d0f0f6bee781 tempest-TestMultiAttachVolumeSwap-1803326020 tempest-TestMultiAttachVolumeSwap-1803326020] ==> connect_volume: call u"{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7fe1aa096d50>, {u'access_mode': u'rw', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-2a48ae08-2833-488c-8771-eea86f41c7b8', u'target_portal': u'149.202.185.218:3260', u'volume_id': u'2a48ae08-2833-488c-8771-eea86f41c7b8', u'target_lun': 1, u'auth_password': u'***', u'auth_username': u'HMHytc239biDCT6NTgMa', u'auth_method': u'CHAP'}), 'kwargs': {}}" {{(pid=20206) trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:146}}

Should target_lun be the same in both cases?

Dec 08 01:43:17.800452 ubuntu-xenial-ovh-gra1-0001066278 nova-compute[20206]: Login to [iface: default, target: iqn.2010-10.org.openstack:volume-2a48ae08-2833-488c-8771-eea86f41c7b8, portal: 149.202.185.218,3260] successful.

Dec 08 01:43:17.825508 ubuntu-xenial-ovh-gra1-0001066278 nova-compute[20206]: DEBUG os_brick.initiator.connectors.iscsi [None req-15ea8e06-99b2-439b-bad8-d0f0f6bee781 tempest-TestMultiAttachVolumeSwap-1803326020 tempest-TestMultiAttachVolumeSwap-1803326020] Connected to sda using {u'auth_password': u'***', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn': u'iqn.2010-10.org.openstack:volume-2a48ae08-2833-488c-8771-eea86f41c7b8', u'target_portal': u'149.202.185.218:3260', u'volume_id': u'2a48ae08-2833-488c-8771-eea86f41c7b8', u'target_lun': 1, u'access_mode': u'rw', u'auth_username': u'HMHytc239biDCT6NTgMa', u'auth_method': u'CHAP'} {{(pid=20206) _connect_vol /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py:663}}

---

We start deleting server1 here:

http://logs.openstack.org/81/606981/4/check/tempest-slow/fafde23/compute1/logs/screen-n-cpu.txt.gz#_Dec_08_01_45_42_745301

Dec 08 01:45:42.745301 ubuntu-xenial-ovh-gra1-0001066278 nova-compute[20206]: INFO nova.compute.manager [None req-69aba1e7-c922-40f2-8136-6e6e0a8c924e tempest-TestMultiAttachVolumeSwap-1803326020 tempest-TestMultiAttachVolumeSwap-1803326020] [instance: d46fba31-9469-4799-b2bf-1fbad4369a9a] Terminating instance

Disconnects from volume2 here:

Dec 08 01:45:43.151301 ubuntu-xenial-ovh-gra1-0001066278 nova-compute[20206]: DEBUG os_brick.initiator.connectors.iscsi [None req-69aba1e7-c922-40f2-8136-6e6e0a8c924e tempest-TestMultiAttachVolumeSwap-1803326020 tempest-TestMultiAttachVolumeSwap-1803326020] Disconnecting from: [(u'213.32.74.238:3260', u'iqn.2010-10.org.openstack:volume-fa56e173-3bd4-4701-bc71-cb4810c53c51')] {{(pid=20206) _disconnect_connection /usr/local/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py:1089}}

And the volume is logically detached in the Cinder API/DB here:

Dec 08 01:45:45.393228 ubuntu-xenial-ovh-gra1-0001066278 nova-compute[20206]: INFO nova.compute.manager [None req-69aba1e7-c922-40f2-8136-6e6e0a8c924e tempest-TestMultiAttachVolumeSwap-1803326020 tempest-TestMultiAttachVolumeSwap-1803326020] [instance: d46fba31-9469-4799-b2bf-1fbad4369a9a] Took 1.28 seconds to detach 1 volumes for instance.