scenario manager cleans up in reverse order of creation

[John Griffith]

tempst/scenario/manager.py has a teradown class that explicitly deletes resources in reverse order of creation, just popping them off the stack.

The way the tests are written this is great, assuming that that tests are successful. The problem is that if there's an issue as there often is in the boot_from_volume test this might not work.

Common issue being seen in the gates currently is:
1. Boot from volume is called
2. bootable volume is created and succeeds
3. nova boot is called against that volume
        This call fails frequently timing out waiting to become ready

In this situation the test fails and bails out... and then calls clean up. Now, clean up is in a situation where it tries to pop "things" (specifically the volume) and delete it. The problem is that the volume is still in an "in-use" status from the "nova boot" process, so the cleanup fails.

Probably need to implement addCleanup here like we've done in other places that clears up the instance and frees up the volume.

[John Griffith (john-griffith)]

Example occurrence of this can be found here:
http://logs.openstack.org/57/60257/5/check/check-tempest-dsvm-full/d7c1324/console.html

[John Griffith (john-griffith)]

Looking at a number of cases, one thought was to add an admin client for cinder that does a force delete in the teardown routines. This would avoid the confusion of failed instances or other things causing the teardown to fail and confusing folks when they're trying to find and log the recheck bug.

Thoughts?

[Mauro S M Rodrigues (maurorodrigues)]

We hit that 27 times in the last 24 hours
http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIkJhZFJlcXVlc3Q6IEludmFsaWQgdm9sdW1lOiBWb2x1bWUgc3RhdHVzIG11c3QgYmUgYXZhaWxhYmxlIG9yIGVycm9yLCBidXQgY3VycmVudCBzdGF0dXMgaXM6IGluLXVzZSAoSFRUUCA0MDApXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NjkwNzU0OTUxOX0=

[Andrea Frittoli (andrea-frittoli)]

I don't see any hit in Kibana anymore, is this still an issue?

[Luz Cazares (luz-cazares)]

Looks like code has changed a lot since this bug was opened.
Currently, resource clean ups are done via waiters class and called via tearDown on the BaseTestCase instead of tempest/scenario/manager.py

Please add additional information if this is still valid

[Launchpad Janitor (janitor)]

[Expired for tempest because there has been no activity for 60 days.]