Instance deletion fails due to race condition on Hyper-V

[Alessandro Pilotti]

A recently merge patch separates the lock required to delete an instance from the one used during instance creation and other operations.

This leads to race conditions in the Hyper-V driver, as the spawn and destroy methods have been written considering that locking already occurs at the compute manager level.

This patch has already been committed, reverted and committed again:

https://review.openstack.org/#/c/55444/
https://review.openstack.org/#/c/69987/
https://review.openstack.org/#/c/70187/

[Matt Riedemann (mriedem)]

Fail:

http://64.119.130.115/63217/7/Hyper-V/hv-compute2/nova-compute.log.gz

2014-02-04 12:04:21.801 444 ERROR nova.virt.hyperv.vmops [req-169d8df4-f190-409e-ae31-3e501e9b6401 0c4d774866944ff78b9222e732172eee f467a9dd25c4477398f3ba92977acc14] [Error 32] The process cannot access the file because it is being used by another process: 'C:\\OpenStack\\Instances\\instance-00000008\\root.vhd'
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops Traceback (most recent call last):
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops File "c:\OpenStack\virtualenv\lib\site-packages\nova\virt\hyperv\vmops.py", line 357, in destroy
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops self._delete_disk_files(instance_name)
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops File "c:\OpenStack\virtualenv\lib\site-packages\nova\virt\hyperv\vmops.py", line 336, in _delete_disk_files
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops remove_dir=True)
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops File "c:\OpenStack\virtualenv\lib\site-packages\nova\virt\hyperv\pathutils.py", line 120, in get_instance_dir
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops create_dir, remove_dir)
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops File "c:\OpenStack\virtualenv\lib\site-packages\nova\virt\hyperv\pathutils.py", line 106, in _get_instances_sub_dir
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops self._check_remove_dir(path)
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops File "c:\OpenStack\virtualenv\lib\site-packages\nova\virt\hyperv\pathutils.py", line 99, in _check_remove_dir
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops self.rmtree(path)
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops File "c:\OpenStack\virtualenv\lib\site-packages\nova\virt\hyperv\pathutils.py", line 75, in rmtree
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops shutil.rmtree(path)
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops File "C:\Python27\Lib\shutil.py", line 252, in rmtree
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops onerror(os.remove, fullname, sys.exc_info())
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops File "C:\Python27\Lib\shutil.py", line 250, in rmtree
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops os.remove(fullname)
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\OpenStack\\Instances\\instance-00000008\\root.vhd'
2014-02-04 12:04:21.801 444 TRACE nova.virt.hyperv.vmops

[Matt Riedemann (mriedem)]

Do we think that this is maybe also the cause of this?

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRXJyb3IgZnJvbSBsaWJ2aXJ0IGR1cmluZyB1bmZpbHRlci4gQ29kZT01NSBFcnJvcj1SZXF1ZXN0ZWQgb3BlcmF0aW9uIGlzIG5vdCB2YWxpZDogbndmaWx0ZXIgaXMgaW4gdXNlXCIgQU5EIGZpbGVuYW1lOlwibG9ncy9zY3JlZW4tbi1jcHUudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImFsbCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTE2MjM5MzE5MDN9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/71363
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7e076df2f5f401eeddce9b1a6697fd19c9857864
Submitter: Jenkins
Branch: master

commit 7e076df2f5f401eeddce9b1a6697fd19c9857864
Author: Alessandro Pilotti <email address hidden>
Date: Wed Feb 5 21:47:27 2014 +0200

    Revert "Allow deleting instances while uuid lock is held"

    This reverts commit 08beabc5e0fc3fffc074a634040f5821d4d1d5f2.

    This patch creates race condition issues in the Hyper-V driver
    due to the fact that spawn, destroy and other methods have been
    written with the instance lock in mind.

    In this context, thread safety needs to be enforced where needed
    in all drivers before this patch can be applied.

    Sample Hyper-V CI log showing the error:
    http://64.119.130.115/63217/7/Hyper-V/hv-compute2/nova-compute.log.gz

    Partial-Bug: #1276772

    Change-Id: I72fb1f9de7767b7206c5fa39b87216fa3f1b5c68

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/88067

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/88067
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.