XenAPI: iscsi connection may not show VDI during attach

[Bob Ball]

During an attach to an iSCSI device, the LUN may not appear as soon as the path which contains the LUNs. The XenAPI code copes with multiple (or no) LUNs for an SR, so there is a race condition where the SR is created but no VDIs are registered, leading to the following exception:

2014-01-29 08:19:41.804 ERROR nova.compute.manager [req-144fa10e-7003-45dc-a54c-f45b1b3581fd tempest.scenario.manager-tempest-1638977579-user tempest.scenario.manager-tempest-1638977579-tenant] [instance: cc864540-8650-40ea-9142-ba533c1e39f9] Failed to attach volume 2ac02e92-8880-4bd7-a382-8942ffe14f7c at /dev/xvdb
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] Traceback (most recent call last):
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] File "/opt/stack/nova/nova/compute/manager.py", line 3852, in _attach_volume
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] encryption=encryption)
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] File "/opt/stack/nova/nova/virt/xenapi/driver.py", line 435, in attach_volume
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] mountpoint)
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] File "/opt/stack/nova/nova/virt/xenapi/volumeops.py", line 59, in attach_volume
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] hotplug=hotplug)
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] File "/opt/stack/nova/nova/virt/xenapi/volumeops.py", line 130, in _connect_volume
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] volume_utils.forget_sr(self._session, sr_ref)
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] six.reraise(self.type_, self.value, self.tb)
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] File "/opt/stack/nova/nova/virt/xenapi/volumeops.py", line 108, in _connect_volume
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] target_lun=connection_data['target_lun'])
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] File "/opt/stack/nova/nova/virt/xenapi/volume_utils.py", line 160, in introduce_vdi
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] vdi_rec = session.call_xenapi("VDI.get_record", vdi_ref)
2014-01-29 08:19:41.804 TRACE nova.compute.manager [instance: cc864540-8650-40ea-9142-ba533c1e39f9] UnboundLocalError: local variable 'vdi_ref' referenced before assignment

[Bob Ball (bob-ball)]

Note: This causes tempest test failures:
08:29:59 Traceback (most recent call last):
08:29:59 File "tempest/scenario/test_minimum_basic.py", line 161, in test_minimum_basic_scenario
08:29:59 self.nova_volume_attach()
08:29:59 File "tempest/scenario/test_minimum_basic.py", line 118, in nova_volume_attach
08:29:59 self._wait_for_volume_status('in-use')
08:29:59 File "tempest/scenario/test_minimum_basic.py", line 46, in _wait_for_volume_status
08:29:59 self.volume_client.volumes, volume_id, status)
08:29:59 File "tempest/scenario/manager.py", line 348, in status_timeout
08:29:59 not_found_exception=not_found_exception)
08:29:59 File "tempest/scenario/manager.py", line 409, in _status_timeout
08:29:59 raise exceptions.TimeoutException(message)
08:29:59 TimeoutException: Request timed out
08:29:59 Details: Timed out waiting for thing 2ac02e92-8880-4bd7-a382-8942ffe14f7c to become in-use

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/69879
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=187d1668d43da6977eb9ae4d96cd85243838ed6b
Submitter: Jenkins
Branch: master

commit 187d1668d43da6977eb9ae4d96cd85243838ed6b
Author: Bob Ball <email address hidden>
Date: Wed Jan 29 15:06:54 2014 +0000

    XenAPI: Wait for VDI on introduce

    Some SRs, particularly iSCSI connections, are slow to see the VDIs.

    Introduce a sleep when a VDI that we're definitely expecting is not
    present on the SR.

    Change-Id: I7c71f1c7b811603aeacbb34c913dc236bba2132a
    Closes-bug: 1274088