OSError: [Errno 2] No such file or directory: '/opt/stack/data/cinder/volumes/volume-

[Attila Fazekas]

tempest.api.compute.v3.servers.test_server_rescue.ServerRescueV3Test Failes to delete the volume.

Logstash query:
message: "OSError: [Errno 2] No such file or directory: '/opt/stack/data/cinder" AND filename:"logs/screen-c-vol.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIk9TRXJyb3I6IFtFcnJubyAyXSBObyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5OiAnL29wdC9zdGFjay9kYXRhL2NpbmRlclwiIEFORCBmaWxlbmFtZTpcImxvZ3Mvc2NyZWVuLWMtdm9sLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkxNzU1MDAwODA5LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

http://logs.openstack.org/75/71575/4/gate/gate-tempest-dsvm-postgres-full/7897842/console.html

http://logs.openstack.org/75/71575/4/gate/gate-tempest-dsvm-postgres-full/7897842/logs/screen-c-vol.txt.gz#_2014-02-07_03_58_26_213

[John Griffith (john-griffith)]

This is related to the create_export change. We're doing a double remove somehow it seems in the driver, I'll get it sorted shortly.

[John Griffith (john-griffith)]

Looks like a race between the detach call and the delete call. Result is that when the delete starts the volume_path exists so it tries to go through the target removal process that the detach is already chugging on. By the time delete get's down to remove the volume_path, the detach has finished and already done it for us.

Easy to fix by just adding an if_exists on that removal, but I want to spend a bit of time looking at ways to deal with the race itself.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72265

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72265
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a9112a9cd8d1286e6c62c1a848c10e4d926a45c2
Submitter: Jenkins
Branch: master

commit a9112a9cd8d1286e6c62c1a848c10e4d926a45c2
Author: John Griffith <email address hidden>
Date: Mon Feb 10 05:21:18 2014 +0000

    Fixup persistence file not found on tgt remove

    After commit: beecd769af02ba5915be827d28a7b46d970e41b0
    we're experiencing a race in the tempest gating tests.

    It appears that there are certain cases where a detach
    and a delete call can overlap causing a race in the
    remove_iscsi_target method. The result is that when
    the slower method gets around to the final removal of
    the persistence file it's no longer there, so we
    hit an unhandled exception.

    This patch does two things:
    1. It adds the volume lock to the detach_volume
    method in the manager (already in place for delete).
    This should mitigate the race and allow us to avoid it.

    2. In the iscsi_remove_target method check if the
    persist file has already been removed when we get
    to the last check. The fact is that if it was somehow
    removed before we got here we don't really care, we're
    just going to delete it anyway.

    Change-Id: Ied45e1a66c8fb79c95ada66f8c59bd2839e200ad
    Closes-Bug: 1277362