Tempest fails spectacularly with "VolumeType c39f18bf-2408-4377-98f6-ef2839d6f70e deletion failed, VolumeType in use."

[Matt Riedemann]

Most if not all volume tests in this tempest run failed. Seeing this error in the c-api logs:

http://logs.openstack.org/26/111526/4/check/check-tempest-dsvm-postgres-full/ab461c3/logs/screen-c-api.txt.gz#_2014-08-04_16_30_14_450

Checking logstash:

message:"VolumeType" AND message:"deletion failed, VolumeType in use" AND tags:"screen-c-api.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVm9sdW1lVHlwZVwiIEFORCBtZXNzYWdlOlwiZGVsZXRpb24gZmFpbGVkLCBWb2x1bWVUeXBlIGluIHVzZVwiIEFORCB0YWdzOlwic2NyZWVuLWMtYXBpLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsiZnJvbSI6IjIwMTQtMDctMjZUMTg6MDM6MTIrMDA6MDAiLCJ0byI6IjIwMTQtMDgtMDRUMTg6MDM6MTIrMDA6MDAiLCJ1c2VyX2ludGVydmFsIjoiMCJ9LCJzdGFtcCI6MTQwNzE3NTUyMzQ3Nn0=

65 hits in 7 days.

The scheduler logs are full of errors, probably due to volumes not getting cleaned up from the host due to cascading failures and then the scheduler can't serve anything because there isn't a valid host (out of space).

http://logs.openstack.org/26/111526/4/check/check-tempest-dsvm-postgres-full/ab461c3/logs/screen-c-sch.txt.gz?level=TRACE

[Matt Riedemann (mriedem)]

Apparently also shows up in stable/icehouse.

[git-harry (git-harry)]

The logstash query picks up on:

http://logs.openstack.org/51/120351/2/check/check-tempest-dsvm-postgres-full/f083f43/logs/tempest.txt.gz?#_2014-09-10_15_33_54_836

This one appears to be a bug in tempest during the tear down after the creation of a volume failed. The volume IDs for the test are only stored once they go available, if the creation fails no attempt is made to delete the volume in error. This prevents the volume type being deleted, generating the error referenced.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120793

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120793
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=9c9f626a3a1031c90e3902aa55303ea76ef99961
Submitter: Jenkins
Branch: master

commit 9c9f626a3a1031c90e3902aa55303ea76ef99961
Author: git-harry <email address hidden>
Date: Thu Sep 11 15:26:23 2014 +0100

    VolumeMultiBackendTest: delete error volumes

    Record the volume ID before waiting for the volume status to go
    available. This means that if the volume goes into error the tearDown
    method can remove the volume.

    Change-Id: I67e74af999862fc37c4f5b2474b50b62cce328e9
    Partial-Bug: #1352498

[jichenjc (jichenjc)]

https://review.openstack.org/#/c/127876/

check-tempest-dsvm-postgres-full: https://bugs.launchpad.net/bugs/1373513 https://bugs.launchpad.net/bugs/1352498

encounter this issue again ...

[John Griffith (john-griffith)]

I think the root or this and https://bugs.launchpad.net/cinder/+bug/1421895 are one in the same. If you take a look at the logstash info it appears these are always encountered in the same run.

[John Griffith (john-griffith)]

This is cleaned up, can't remember if it was something with Cinder AND Tempest, or just the Tempest change.

Also noticed that the ER for the related/original bug is "bunk", so removing the ER as well.