Unable to deactivate LVM snapshot

[Sean Dague]

One of the new error conditions that shows up in the gate now that we are failing on unknown errors is a failure by c-vol to deactivate the LVM snapshot.

The c-vol error condition is the following:

[req-798a0f63-4a42-4f68-9087-54a90ec50376 1b40a47efb8243579e1830d573ac30ed f6fcf023ac3c4fa0a670e28917331b3f] Error reported running lvremove: CMD: sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvremove -f stack-volumes/_snapshot-8a0e5904-d394-4193-97fe-3658e0964c22, RESPONSE: Unable to deactivate open stack--volumes-_snapshot--8a0e5904--d394--4193--97fe--3658e0964c22 (252:4)
  Unable to deactivate logical volume "_snapshot-8a0e5904-d394-4193-97fe-3658e0964c22"

http://logs.openstack.org/14/62514/1/check/check-tempest-dsvm-postgres-full/18e9cf9/logs/screen-c-vol.txt.gz#_2013-12-17_02_10_46_029

It looks like it actually succeeds on attempt 2 after the quiece, so the the fix is probably to reduce the level on these to make them not error/warn unless there is a real failure.

[Sean Dague (sdague)]

Logstash query for this - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIlJFU1BPTlNFOiAgIFVuYWJsZSB0byBkZWFjdGl2YXRlIG9wZW4gc3RhY2stLXZvbHVtZXMtX3NuYXBzaG90XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODcyODI1NjAzNzB9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/62622

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/62622
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=99f3ce38e9ad1e272a4a04858ff1acc709e7c9cb
Submitter: Jenkins
Branch: master

commit 99f3ce38e9ad1e272a4a04858ff1acc709e7c9cb
Author: Sean Dague <email address hidden>
Date: Tue Dec 17 07:29:46 2013 -0500

    make delete recovery messages debug level

    lvm.delete has recovery logic within it to do a second attempt
    if it should fail the first time. It is currently however logging
    those second attempts at error and warn levels. However, as this
    is normal flow these should be debug level instead.

    This is currently causing a few resets in the gate because this
    is a non whitelisted error.

    Change-Id: Ie684331d9d97c089ec23569d06981e90eb7f6a67
    Closes-Bug: #1261731