cannot deal with soft-deleted VM in Nova

[Li Ma]

When Nova allows soft-delete operation, currently, rally cannot deal with it and throws the exception when clean up all the created VMs.

In benchmark/scenarios/nova/utils.py:

    def _delete_server(self, server):
        ....
        server.delete() <-- this can be soft-delete operation, meaning that VM is recycled temporarily but not completely deleted.

If we use force_delete() function, VM can be safely deleted.

Reproducing: (Nova enables soft-delete)
1. Run task: boot-from-volume-and-delete.json
2. It creates several VMs and volumes.
3. It deletes VMs.
4. VM states change to soft-deleted, but volume state is still in-use because VM is not truly deleted.
5. It throws an exception saying that volume cannot be deleted.
6. The task is failed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135568

[Boris Pavlovic (boris-42)]

Actually not sure that I understand fully:

So we are running this benchmark in gates (on every patch) as a part of functional testing:

http://logs.openstack.org/78/127878/10/check/gate-rally-dsvm-rally/2cf887a/rally-plot/results.html.gz#/NovaServers.boot_server_from_volume_and_delete

It doesn't fail.

As well in LOGs I don't see any messages about errors related to cleanup volume:

http://logs.openstack.org/78/127878/10/check/gate-rally-dsvm-rally/2cf887a/console.html

And after all I run locally this benchmark. After everything finished, this is result:

boris@ubuntu:~$ cinder list --all-tenants
+----+-----------+--------+--------------+------+-------------+----------+-------------+
| ID | Tenant ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+----+-----------+--------+--------------+------+-------------+----------+-------------+
+----+-----------+--------+--------------+------+-------------+----------+-------------+

[Li Ma (nick-ma-z)]

This bug is triggered via a pre-requisite that Nova enables soft-delete operations instead of delete.

Here's the settings:
https://ask.openstack.org/en/question/6548/how-to-use-soft-delete-deferred-delete/

In our environment, soft-delete is enabled and Rally failed.

[Li Ma (nick-ma-z)]

[root@manager1 ~(keystone_admin)]# cinder list --all-tenant | grep rally | grep in-use
| 104de5ab-bdf5-4153-99a4-c9da6159712e | in-use | rally_novaserver_drveooejirkkndys | 25 | None | true | 0478e2b3-0580-4fc2-b810-80729a5b4c86 |
| b15ec005-5960-42cd-8282-36969af82f0c | in-use | rally_novaserver_iwtsmazreyxgouvy | 25 | None | true | f771869c-f115-4d9b-8d7e-07841ba2eff4 |
| ba9d48f9-8845-4225-a72b-5cfcc3ddcf25 | in-use | rally_novaserver_ocbeemcdfqfgrgem | 25 | None | true | c6b023ce-2f47-4ddf-a485-9f28c483b12f |
| f0686468-0603-4786-a5ba-2ca92eee47d1 | in-use | rally_novaserver_llicyxvncjlywwsn | 25 | None | true | bf9d9657-ea64-446b-a797-b800a8ea786e |

Rally cannot delete in-use volumes because the corresponding instance is in soft-deleted status.

[Li Ma (nick-ma-z)]

My Rally version is github master.

[Boris Pavlovic (boris-42)]

What version of OpenStack are you using? Cause it's really strange issue..

[Boris Pavlovic (boris-42)]

Ah I just saw:

https://ask.openstack.org/en/question/6548/how-to-use-soft-delete-deferred-delete/

So, I we should find the way to do handle this case, without affecting everybody else..

[Li Ma (nick-ma-z)]

I'm using Juno stable branch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135568
Committed: https://git.openstack.org/cgit/stackforge/rally/commit/?id=b265aad311a7cfd946e483c40a112ca793b3553a
Submitter: Jenkins
Branch: master

commit b265aad311a7cfd946e483c40a112ca793b3553a
Author: Li Ma <email address hidden>
Date: Wed Nov 19 03:12:24 2014 -0800

    Add force_delete to safely delete created VMs

    When Nova allows soft-delete operation, currently, rally cannot
    deal with it and throws the exception when delete all the created
    VMs.

    Reproducing: (Nova enables soft-delete)
    1. Run task: boot-from-volume-and-delete.json
    2. It creates several VMs and volumes.
    3. It deletes VMs.
    4. VM states change to soft-deleted, but volume state is still in-use
    because VM is not truly deleted.
    5. It throws an exception saying that volume cannot be deleted.
    6. The task is failed.

    This fix adds force_delete when nova calls delete-server.
    You can specify 'force_delete' in task configuration to determine
    whether to use default delete action or force_delete action.

    Change-Id: I56ac5c039b525a8b05960dfbfd3ff0cacec0aef6
    Closes-Bug: #1394119