Attach volume sometimes incorrectly reports success

[Devin Carlen]

Recently while I was refactoring the smoke tests, I found that attach volume is reporting success even when the operation fails. I was able to work around this in the smoke tests by waiting longer before attaching. My initial sense is that the volume is reporting its state as "available" some small amount of time (5-15 seconds?) before it is really available.

Here is a snippet from the volume smoke tests that show how I was forced to work around the issue:

http://paste.openstack.org/show/100/

[Soren Hansen (soren)]

The volume driver's create_export method should block until the device is actually ready. There must be some way to check when vblade (assuming the you're using AoE) has actually gotten its act together and exported the device?

[Vish Ishaya (vishvananda)]

This is apparently happening with iscsi. I'd love to get some tracebacks because I'm not sure what exactly is going to quickly. I guess it is possible that the iscsi target shell is returning before the target is actually created, but this seems unlikely.

Vish

On Nov 22, 2010, at 1:36 AM, Soren Hansen wrote:

> The volume driver's create_export method should block until the device
> is actually ready. There must be some way to check when vblade (assuming
> the you're using AoE) has actually gotten its act together and exported
> the device?
>
> --
> Attach volume sometimes incorrectly reports success
> https://bugs.launchpad.net/bugs/673756
> You received this bug notification because you are a member of Nova
> Bugs, which is subscribed to OpenStack Compute (nova).
>
> Status in OpenStack Compute (Nova): New
>
> Bug description:
> Recently while I was refactoring the smoke tests, I found that attach volume is reporting success even when the operation fails. I was able to work around this in the smoke tests by waiting longer before attaching. My initial sense is that the volume is reporting its state as "available" some small amount of time (5-15 seconds?) before it is really available.
>
> Here is a snippet from the volume smoke tests that show how I was forced to work around the issue:
>
> http://paste.openstack.org/show/100/
>
>

[Mark McLoughlin (markmc)]

Ok, so there's two things allegedly doing on here:

  1) A volume is marked as 'available' before it is actually possible to attach a volume

  2) Attaching a volume sometimes succeeds even when we actually failed to attach because it wasn't ready

Now, in VolumeManager.create_volume(), we mark the volume as available just before returning successfully. That implies to me that the call shouldn't return until the volume is available. And, as Soren points out, that means create_export() should block until it's ready.

That means that this loop in test_002_can_attach_volume() doesn't make sense:

        for x in xrange(10):
            volume.update()
            if volume.status.startswith('available'):
                break
            time.sleep(1)
        else:
            self.fail('cannot attach volume with state %s' % volume.status)

i.e. create_volume() has already allegedly succeeded, so the volume should have 'available' status

To reproduce the bug, I think you just need to remove the two five second sleeps between create_volume() and attach_volume() in the test. I haven't tried reproducing yet, so it'd be nice to know what cases are failing - iSCSI only, or all volume drivers? tgtadm or ietadm? Xen/KVM? With an instance running on the volumes host, or remotely, or both?

In the case of ISCSIDriver, the last thing we do in create_export() is run 'tgtdadm --op new --mode=logicalunit ...'. So, we need to understand the semantics of that. Is it supposed to block until the LUN is available? If not, how do we poll for completion?

And we also need to figure out why attach_volume returns successfully when it has failed.

[yong sheng gong (gongysh)]

I am trying to reproduce it with ISCSIDriver