Checksum error\n Volume group for uuid not found; libdevmapper exiting with 2 device(s) still suspended

[Nathan Kinder]

I'm encountering a number of failed volume related tempest tests in the gate-tempest-dsvm-full job. The errors all look similar to this:

2014-08-20 07:09:32.347 | setUpClass (tempest.api.volume.admin.test_snapshots_actions.SnapshotsActionsTestXML)
2014-08-20 07:09:32.347 | ------------------------------------------------------------------------------------
2014-08-20 07:09:32.347 |
2014-08-20 07:09:32.347 | Captured traceback:
2014-08-20 07:09:32.347 | ~~~~~~~~~~~~~~~~~~~
2014-08-20 07:09:32.347 | Traceback (most recent call last):
2014-08-20 07:09:32.347 | File "tempest/test.py", line 76, in decorator
2014-08-20 07:09:32.347 | f(cls)
2014-08-20 07:09:32.348 | File "tempest/api/volume/admin/test_snapshots_actions.py", line 46, in setUpClass
2014-08-20 07:09:32.348 | 'available')
2014-08-20 07:09:32.348 | File "tempest/services/volume/xml/snapshots_client.py", line 138, in wait_for_snapshot_status
2014-08-20 07:09:32.348 | raise exceptions.TimeoutException(message)
2014-08-20 07:09:32.348 | TimeoutException: Request timed out
2014-08-20 07:09:32.348 | Details: Request timed out
2014-08-20 07:09:32.348 | Details: Time Limit Exceeded! (196s)while waiting for available, but we got creating.

The full log is available here:

  http://logs.openstack.org/08/104408/5/gate/gate-tempest-dsvm-full/f3081d0/console.html

There are some cinder errors seem in the c-vol log:

2014-08-20 06:24:51.343 26531 ERROR cinder.brick.local_dev.lvm [req-374faa66-0448-4db0-912a-da1682aedb18 d0db1e0cb50a44f8b0ce9797b8289862 32bee14e421845b1bf68ce61cf092a4c - - -] Error creating snapshot
2014-08-20 06:24:51.349 26531 ERROR cinder.brick.local_dev.lvm [req-374faa66-0448-4db0-912a-da1682aedb18 d0db1e0cb50a44f8b0ce9797b8289862 32bee14e421845b1bf68ce61cf092a4c - - -] Cmd :sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate --name _snapshot-da519713-3038-4f8a-ac3a-5446d63711fd --snapshot stack-volumes-lvmdriver-1/volume-ebe39b05-7768-4a7f-872f-134e888eabd8 -L 1.00g
2014-08-20 06:24:51.349 26531 ERROR cinder.brick.local_dev.lvm [req-374faa66-0448-4db0-912a-da1682aedb18 d0db1e0cb50a44f8b0ce9797b8289862 32bee14e421845b1bf68ce61cf092a4c - - -] StdOut :
2014-08-20 06:24:51.350 26531 ERROR cinder.brick.local_dev.lvm [req-374faa66-0448-4db0-912a-da1682aedb18 d0db1e0cb50a44f8b0ce9797b8289862 32bee14e421845b1bf68ce61cf092a4c - - -] StdErr : /dev/loop1: Checksum error
  Volume group for uuid not found: eBM7ZaAMcl3tfLoPn79OvlT02cAd30LxD6bozjscwcMF6RX3dGkLiayOO52BkABV
  Problem reactivating origin volume-ebe39b05-7768-4a7f-872f-134e888eabd8
  libdevmapper exiting with 2 device(s) still suspended.

See the full c-vol log for more details:

    http://logs.openstack.org/08/104408/5/gate/gate-tempest-dsvm-full/f3081d0/logs/screen-c-vol.txt.gz?level=ERROR

[John Griffith (john-griffith)]

this is odd, seems to be happening exactly 5x a day every day since the 17'th. Query in Kibana: "libdevmapper exiting with 2 device(s) still suspended"

[John Griffith (john-griffith)]

Not sure, but I think this change in Tempest:
https://github.com/openstack/tempest/commit/d9df38c867d6447dc41ad3fa7bc8b3d732e751b0

May have introduced an issue where we don't reliably get the current status prior to the delete operation.

[Matt Riedemann (mriedem)]

Similar here:

http://logs.openstack.org/87/200787/4/gate/gate-tempest-dsvm-nova-v21-full/62fce1c/logs/screen-c-vol.txt.gz?level=TRACE

Looks like nearly every test related to volumes failed.

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQ2hlY2tzdW0gZXJyb3JcIiBBTkQgbWVzc2FnZTpcIlZvbHVtZSBncm91cCBmb3IgdXVpZCBub3QgZm91bmRcIiBBTkQgdGFnczpcInNjcmVlbi1jLXZvbC50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzNjg4NTE2MzA1OSwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

69 hits in 7 days, check and gate, master and stable/kilo, all failures.

[Jordan Pittier (jordan-pittier)]

...84 failures in the past 7 days....

[Jordan Pittier (jordan-pittier)]

For instance: http://logs.openstack.org/36/248136/1/check/gate-tempest-dsvm-full-juno/e69f2ca/logs/screen-c-vol.txt.gz#_2015-11-20_16_00_59_302

[Ivan Kolodyazhny (e0ne)]

This issue also affects gate-rally-dsvm-cinder job

[Sean McGinnis (sean-mcginnis)]

Appears to have since been fixed indirectly.

[Mike Perez (thingee)]

Appears to still be happening.

[Matt Riedemann (mriedem)]

This is gone again.