Deleting a volume booted server which is still in build state leaves behind volume in 'in-use' state

[Ahmad Faheem]

Deleting a server created using bootable volume which is still in build state leaves behind volume in 'in-use' state and this volume can not be deleted using delete api. Below is the shell script to reproduce it.

curl -g -i -X POST https://10.0.0.5:4523/v2/ee61323896a34bea9c9a5623fbb6f239/os-volumes_boot?format=xml -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: <omitted>" -d '{"server": {"name": "testVolIssue", "imageRef": "", "block_device_mapping_v2": [{"boot_index": "0", "uuid": "5d246189-a666-470c-8cee-36ee489cbd9e", "volume_size": "6", "source_type": "image", "destination_type": "volume", "delete_on_termination": "1"}], "flavorRef": "da9ba7b5-be67-4a62-bb35-a362e05ba2f2", "max_count": 1, "min_count": 1, "networks": [{"uuid": "b5220eb2-e105-4ae0-8fc7-75a7cd468a40"}]}}' > /dev/null ; var=`nova list | grep testVolIssue | awk '{print $2}'`;echo Server Id: $var;nova delete $var; nova show $var
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 839 100 436 100 403 589 544 --:--:-- --:--:-- --:--:-- 589
Server Id: 8f04539c-a0a6-4276-af1f-17c447cc8c53
Request to delete server 8f04539c-a0a6-4276-af1f-17c447cc8c53 has been accepted.
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | deleting |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-05-24T09:24:30Z |
| flavor | small (da9ba7b5-be67-4a62-bb35-a362e05ba2f2) |
| hostId | f96a91b0f7f407d7880f3ee32208c4003f8245431115666dad4bb2e5 |
| id | 8f04539c-a0a6-4276-af1f-17c447cc8c53 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | testVolIssue |
| os-extended-volumes:volumes_attached | [{"id": "97017c0c-b752-4cba-9c42-dcc5600a5f95"}] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | ee61323896a34bea9c9a5623fbb6f239 |
| updated | 2015-05-24T09:24:32Z |
| user_id | 6cb8b12bed5046ff810d71ed73bdcb2e |
+--------------------------------------+----------------------------------------------------------+
nova show 8f04539c-a0a6-4276-af1f-17c447cc8c53
ERROR (CommandError): No server with a name or ID of '8f04539c-a0a6-4276-af1f-17c447cc8c53' exists.

cinder list | grep 8f04539c-a0a6-4276-af1f-17c447cc8c53
| 97017c0c-b752-4cba-9c42-dcc5600a5f95 | in-use | | 6 | None | true | 8f04539c-a0a6-4276-af1f-17c447cc8c53 |
cinder delete 97017c0c-b752-4cba-9c42-dcc5600a5f95
Delete for volume 97017c0c-b752-4cba-9c42-dcc5600a5f95 failed: Bad Request (HTTP 400) (Request-ID: req-30d7843f-6c12-42ad-89e3-4103db630c1c)
ERROR: Unable to delete any of the specified volumes.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/194063
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ecdf331bafddfd2bb8c92d3fd96f301bc7ac644f
Submitter: Jenkins
Branch: master

commit ecdf331bafddfd2bb8c92d3fd96f301bc7ac644f
Author: ankitagrawal <email address hidden>
Date: Wed Sep 23 03:58:19 2015 -0700

    Detach volume after deleting instance with no host

    If an instance is booted from a volume, shelved, and goes into an error
    state due to some reason. Volume from which instance is booted, remains
    in-use state even the instance is deleted because instance has no host
    associated with it.

    Called _local_delete() to detach volume and destroy bdm if instance is
    in shelved_offloaded state or has no host associated with it. This will
    cleanup both volumes and the networks.

    Note:
    I had submitted same patch [1] earlier which was reverted [2] due to a
    race condition on jenkins if an instance is deleted when it is in
    building state. In this patch I have fixed the failure of race condition
    by reverting the ObjectActionError exception handling in _delete.

    [1] Ic630ae7d026a9697afec46ac9ea40aea0f5b5ffb
    [2] Id4e405e7579530ed1c1f22ccc972d45b6d185f41

    Closes-Bug: 1404867
    Closes-Bug: 1408527
    Closes-Bug: 1458308
    Change-Id: Ic107d8edc7ee7a4ebb04eac58ef0cdbf506d6173

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.