Tempest failure because of LVM bug on precise

[David Kranz]

THIS IS NOT THE BUG YOU ARE LOOKING FOR

If you got here because of a recheck issue in gerrit, you've found the wrong bug. Please look deeper into the logs on the VOLUMES exercise to figure out why it failed.

******************************************************************************************

There have been some build failures lately that are not consistent involving test_volumes_get and test_volumes_list. These failures can happen on either the volumes or cinder runs. I could not reproduce this locally. Here are some links and the stacktraces:

http://logs.openstack.org/11936/20/check/gate-tempest-devstack-vm/12844/

12:20:59 ERROR: Create a volume without passing metadata, get details, and delete
12:20:59 ----------------------------------------------------------------------
12:20:59 Traceback (most recent call last):
12:20:59 File "/opt/stack/tempest/tempest/tests/volume/test_volumes_get.py", line 99, in test_volume_get_metadata_none
12:20:59 self.client.wait_for_resource_deletion(volume['id'])
12:20:59 File "/opt/stack/tempest/tempest/common/rest_client.py", line 253, in wait_for_resource_deletion
12:20:59 raise exceptions.TimeoutException
12:20:59 TimeoutException: Request timed out
12:20:59
12:20:59 ======================================================================
12:20:59 ERROR: test suite for <class 'tempest.tests.volume.test_volumes_list.VolumesListTest'>
12:20:59 ----------------------------------------------------------------------
12:20:59 Traceback (most recent call last):
12:20:59 File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 208, in run
12:20:59 self.setUp()
12:20:59 File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 291, in setUp
12:20:59 self.setupContext(ancestor)
12:20:59 File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 314, in setupContext
12:20:59 try_run(context, names)
12:20:59 File "/usr/lib/python2.7/dist-packages/nose/util.py", line 478, in try_run
12:20:59 return func()
12:20:59 File "/opt/stack/tempest/tempest/tests/volume/test_volumes_list.py", line 51, in setUpClass
12:20:59 'available')
12:20:59 File "/opt/stack/tempest/tempest/services/volume/json/volumes_client.py", line 113, in wait_for_volume_status
12:20:59 raise exceptions.TimeoutException(message)
12:20:59 TimeoutException: Request timed out
12:20:59 Details: Volume volume86128337728 failed to reach available status within the required time (300.0 s).
12:20:59

http://logs.openstack.org/13618/2/check/gate-tempest-devstack-vm-cinder/2818/

10:57:21 ERROR: CREATE, GET, DELETE Volume
10:57:21 ----------------------------------------------------------------------
10:57:21 Traceback (most recent call last):
10:57:21 File "/opt/stack/tempest/tempest/tests/compute/test_volumes_get.py", line 73, in test_volume_create_get_delete
10:57:21 self.client.wait_for_resource_deletion(volume['id'])
10:57:21 File "/opt/stack/tempest/tempest/common/rest_client.py", line 253, in wait_for_resource_deletion
10:57:21 raise exceptions.TimeoutException
10:57:21 TimeoutException: Request timed out
10:57:21
10:57:21 ======================================================================
10:57:21 ERROR: Create a volume, Get it's details and Delete the volume
10:57:21 ----------------------------------------------------------------------
10:57:21 Traceback (most recent call last):
10:57:21 File "/opt/stack/tempest/tempest/tests/volume/test_volumes_get.py", line 71, in test_volume_create_get_delete
10:57:21 resp, _ = self.client.delete_volume(volume['id'])
10:57:21 File "/opt/stack/tempest/tempest/services/volume/json/volumes_client.py", line 93, in delete_volume
10:57:21 return self.delete("volumes/%s" % str(volume_id))
10:57:21 File "/opt/stack/tempest/tempest/common/rest_client.py", line 162, in delete
10:57:21 return self.request('DELETE', url, headers)
10:57:21 File "/opt/stack/tempest/tempest/common/rest_client.py", line 201, in request
10:57:21 raise exceptions.BadRequest(resp_body)
10:57:21 BadRequest: Bad request
10:57:21 Details: {u'badRequest': {u'message': u'Invalid volume: Volume status must be available or error', u'code': 400}}
10:57:21
10:57:21 ======================================================================
10:57:21 ERROR: Create a volume without passing metadata, get details, and delete
10:57:21 ----------------------------------------------------------------------
10:57:21 Traceback (most recent call last):
10:57:21 File "/opt/stack/tempest/tempest/tests/volume/test_volumes_get.py", line 98, in test_volume_get_metadata_none
10:57:21 resp, _ = self.client.delete_volume(volume['id'])
10:57:21 File "/opt/stack/tempest/tempest/services/volume/json/volumes_client.py", line 93, in delete_volume
10:57:21 return self.delete("volumes/%s" % str(volume_id))
10:57:21 File "/opt/stack/tempest/tempest/common/rest_client.py", line 162, in delete
10:57:21 return self.request('DELETE', url, headers)
10:57:21 File "/opt/stack/tempest/tempest/common/rest_client.py", line 201, in request
10:57:21 raise exceptions.BadRequest(resp_body)
10:57:21 BadRequest: Bad request
10:57:21 Details: {u'badRequest': {u'message': u'Invalid volume: Volume status must be available or error', u'code': 400}}
10:57:21
10:57:21 ======================================================================
10:57:21 ERROR: test suite for <class 'tempest.tests.volume.test_volumes_list.VolumesListTest'>
10:57:21 ----------------------------------------------------------------------
10:57:21 Traceback (most recent call last):
10:57:21 File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 208, in run
10:57:21 self.setUp()
10:57:21 File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 291, in setUp
10:57:21 self.setupContext(ancestor)
10:57:21 File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 314, in setupContext
10:57:21 try_run(context, names)
10:57:21 File "/usr/lib/python2.7/dist-packages/nose/util.py", line 478, in try_run
10:57:21 return func()
10:57:21 File "/opt/stack/tempest/tempest/tests/volume/test_volumes_list.py", line 51, in setUpClass
10:57:21 'available')
10:57:21 File "/opt/stack/tempest/tempest/services/volume/json/volumes_client.py", line 113, in wait_for_volume_status
10:57:21 raise exceptions.TimeoutException(message)
10:57:21 TimeoutException: Request timed out
10:57:21 Details: Volume volume1064768584 failed to reach available status within the required time (300.0 s).
1

[Thierry Carrez (ttx)]

Could be https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1023755

[John Griffith (john-griffith)]

As Thierry pointed I believe this is in fact a result of bug 1023755.

Basically anytime the delete on a volume is called right now there's a chance that the zeroing operation (dd) will hang things up on the system. As a result we get these cascaded failures after the first test. I'll update with info from Cannonical if/when available.

[Thierry Carrez (ttx)]

Adding openstack-ci as this is #1 problem on the gate right now, training developers to ignore tset failures... and it /might/ not be an openstack or tempest issue, as pointed in comment 1.

[Thierry Carrez (ttx)]

Recent example at http://logs.openstack.org/14629/1/gate/gate-tempest-devstack-vm-cinder/5069/

[Sean Dague (sdague)]

This has been seen as recently as today:

Bug 1056213: Flakey build failures in some volume tests

First seen: 2013-01-29 15:59:12 UTC
Last seen: 2013-02-20 01:06:41 UTC
Rechecks: 8
Affecting projects:
openstack/glance
openstack/nova
openstack/keystone
Affecting changes:
20697
20699
18462
21216
21939
21564

[John Griffith (john-griffith)]

So this bug was actually opened and was due to volume create and delete failures. There were two issue here, both have been addressed:

1. Volume delete on Precise would hang the system when attempting to do a secure delete
        FIX - Introduced "SECURE_DELETE=FALSE" as default in devstack and gate tests

2. Tempest was not waiting for delete when finishing volume tests, as a result there were many cases
     where the delete was slow and we'd attempt to create another volume but we were OUT of space on the Volume Group

        FIX - Added wait_for_resource_deletion to the tempest volume tests
        FIX - Increased the Volume Group size in devstack/tempest gate tests to 10G