check-heat-dsvm-functional fails with "StackBuildErrorException: Stack VolumeBackupRestoreIntegrationTest-1161201394/7c711e8d-84ba-4363-ab8e-3afa1d6e1dc6 is in CREATE_FAILED status due to 'Resource CREATE failed: WaitConditionFailure: Test Failed'"

[Matt Riedemann]

This is a non-voting job but I saw the failure shows up quite a bit:

http://logs.openstack.org/89/96889/13/check/check-heat-dsvm-functional/9135dca/console.html#_2014-10-22_15_12_51_439

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU3RhY2tCdWlsZEVycm9yRXhjZXB0aW9uOiBTdGFjayBWb2x1bWVCYWNrdXBSZXN0b3JlSW50ZWdyYXRpb25UZXN0XCIgQU5EIG1lc3NhZ2U6XCJpcyBpbiBDUkVBVEVfRkFJTEVEIHN0YXR1cyBkdWUgdG8gJ1Jlc291cmNlIENSRUFURSBmYWlsZWQ6IFdhaXRDb25kaXRpb25GYWlsdXJlOiBUZXN0IEZhaWxlZCdcIiBBTkQgdGFnczpcImNvbnNvbGVcIiBBTkQgYnVpbGRfbmFtZTpcImNoZWNrLWhlYXQtZHN2bS1mdW5jdGlvbmFsXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MTQwMDY4MTA5MDIsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

188 hits in 7 days, all failures. It's all check queue since it's a non-voting job, but it hits on multiple changes so it'd be good to fix this if the job is ever going to be voting.

(stevebaker) The root cause is when attempting to mount a volume which was restored from a backup the volume never appears in cirros /proc/partitions

http://git.openstack.org/cgit/openstack/heat/tree/heat_integrationtests/scenario/test_volumes_create_from_backup.yaml#n74

Assistance from cinder folk will be need to diagnose this.

[Steve Baker (steve-stevebaker)]

Yes, the following change has just landed to mitigate this:
https://review.openstack.org/#/c/129746/

This change just stops the test instead of failing it when this error occurs. We think this test is finding a bug in nova/cinder/swift interaction which is not tested in tempest, namely mounting a volume which has been restored from a backup.

In Paris I'd like to discuss the possibility of using elastic-recheck to check for these non-fail errors so we can monitor progress on fixing it, without having the impact of it failing a gating job.

[Sean Dague (sdague)]

Is there actually a nova component of this problem?

[Steve Baker (steve-stevebaker)]

I'm assuming that change stopped the test failing (even though it is hiding what appears to be a real cinder failure)

[Mike Perez (thingee)]

Testing this out now.

[Steve Baker (steve-stevebaker)]

Mike, take a look at the flow of this test. I think cinder could do with a functional/tempest test which doesn't something similar, but with just nova instead of heat:

http://git.openstack.org/cgit/openstack/heat/tree/heat_integrationtests/scenario/test_volumes.py
http://git.openstack.org/cgit/openstack/heat/tree/heat_integrationtests/scenario/templates/test_volumes_create_from_backup.yaml
http://git.openstack.org/cgit/openstack/heat/tree/heat_integrationtests/scenario/templates/test_volumes_delete_snapshot.yaml

1. Create a volume, attach it to an instance, write some data to it
2. Delete the stack, with 'Snapshot' specified, creates a backup
3. Check the snapshot has created a volume backup
4. Create a new stack, where the volume is created from the backup
5. Verify the test data written in (1) is present in the new volume

[Sean McGinnis (sean-mcginnis)]

Automatically unassigning due to inactivity.