VMware: failed volume detachment leads to instances remaining on backend and volume still in 'in-use' state

[Gary Kotton]

When the volume detachment fails the termination of the instance will lead to the following:
1. The Nova instance is deleted
2. The Instance on the VC still exists
3. The volume is in 'in-use' state

The nova instance is deleted but the backend is not updated and the volumes are not set as available

One example of this happening is when the spawning of the instance fails with an exception when attaching the volume.
This issue could lead to a DDOS of the backend as the resources on the backend are not cleaned up correctly.

[Gary Kotton (garyk)]

Fix - https://review.openstack.org/220410

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

I've switched this report from private security to public security because it was prematurely disclosed (a proposed fix explicitly mentioning the bug was pushed to public code review rather than uploaded as a bug attachment).

[Tristan Cacqueray (tristan-cacqueray)]

Can someone familiar with the vmware driver please confirm the security issue here ?

[Jeremy Stanley (fungi)]

Does this affect stable/kilo (and perhaps earlier) or only master?

[Gary Kotton (garyk)]

@Fungi - this only affects master

[Jeremy Stanley (fungi)]

Thanks Gary. I've set our security advisory task to won't fix status, since you've confirmed this is a class Y report according to the VMT taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220410
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8d594c264fe5bcfd3ec97698e6c9820e54aa8a0
Submitter: Jenkins
Branch: master

commit d8d594c264fe5bcfd3ec97698e6c9820e54aa8a0
Author: Gary Kotton <email address hidden>
Date: Thu Sep 3 23:25:11 2015 -0700

    VMware: ensure that instance is deleted when volume is missing

    In the event that the volume is missing, the volume detachment fails,
    we need to make sure that the instance deletion is done.

    If this is not done then we will have the following:
    1. Nova instance is deleted
    2. Volume will still be in 'in-use' state
    3. Backend VM will still be on the VC

    In the event that a StorageError error is raised then we will
    proceed to delete the instance.

    All other exceptions will be raised so as to not delete the cinder
    volume.

    Change-Id: I99c59cb94d487322011071d24504dd9401879dba
    Closes-bug: #1492121