tempest test failed test_create_ebs_image_and_check_boot

[Maksym Shalamov]

When running tempest tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_create_ebs_image_and_check_boot fails with the "Invalid volume: Volume still has 1 dependent snapshots." error.

From the console log:
 tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_create_ebs_image_and_check_boot[compute,id-36c34c67-7b54-4b59-b188-02a2f458a63b,image,volume]
------------------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback-2:
~~~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
       File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 791, in wait_for_resource_deletion
         raise exceptions.TimeoutException(message)
     tempest_lib.exceptions.TimeoutException: Request timed out
     Details: (TestVolumeBootPattern:_run_cleanups) Failed to delete volume 973e3dd4-148d-4969-9eee-ffe5daa40d44 within the required time (196 s).

 Captured traceback-1:
 ~~~~~~~~~~~~~~~~~~~~~
     Traceback (most recent call last):
       File "tempest/scenario/manager.py", line 101, in delete_wrapper
         delete_thing(*args, **kwargs)
       File "tempest/services/volume/json/volumes_client.py", line 108, in delete_volume
         resp, body = self.delete("volumes/%s" % str(volume_id))
       File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 290, in delete
        return self.request('DELETE', url, extra_headers, headers, body)
       File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 639, in request
         resp, resp_body)
       File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 697, in _error_checker
         raise exceptions.BadRequest(resp_body, resp=resp)
     tempest_lib.exceptions.BadRequest: Bad request
     Details: {u'code': 400, u'message': u'Invalid volume: Volume still has 1 dependent snapshots.'}

Link on logs:
http://logs.openstack.org/55/245855/1/check/gate-tempest-dsvm-full-ceph/309fe31/

Copy of log file into attachment

[Maksym Shalamov (mshalamov)]

Error found on devstack(branch stable/kilo)

[Matt Riedemann (mriedem)]

The failure on kilo is probably random.

This test has regressed in the ceph jobs in the last 24 hours though on all branches:

http://logs.openstack.org/59/252659/1/check/gate-tempest-dsvm-full-ceph/31dc302/console.html#_2015-12-03_20_09_28_874

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22Details:%20%7Bu'message':%20u'Invalid%20volume:%20Volume%20still%20has%201%20dependent%20snapshots.',%20u'code':%20400%7D%5C%22%20AND%20tags:%5C%22console%5C%22

14 hits in the last 24 hours, check and gate, all failures.

[Matt Riedemann (mriedem)]

Since this started failing on all branches in the last 24 hours, there could have been a regression in tempest.

[Matt Riedemann (mriedem)]

Actually, this looks like the prime suspect, it's the only global ceph change I can find: https://review.openstack.org/#/c/251421/

[Deepak C Shetty (dpkshetty)]

@Matt,
    https://review.openstack.org/#/c/251421/ just adds a new CI job (gate-tempest-dsvm-full-devstack-plugin-ceph) for
ceph which uses the devstack plugin model, doesn't touch/affect the existing CI ceph job (gate-tempest-dsvm-full-ceph, which uses the hooks based approach), so failures in gate-tempest-dsvm-full-ceph job has nothing to do with 251421

FWIW, I saw this error for the new CI job as well

See

http://logs.openstack.org/09/253009/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-nv/6cadba8/console.html.gz#_2015-12-04_11_47_13_962

After a recheck (See patch https://review.openstack.org/#/c/253009/ ) the same CI job passed fine. See

http://logs.openstack.org/09/253009/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-nv/efe84c5/console.html.gz#_2015-12-04_13_54_24_071

So the failure in test_create_ebs_image_and_check_boot seems intermittent (thus racy ? probably)

thanx,
deepak

[Joseph Lanoux (joseph-lanoux)]

Matt,
I don't see why that should be the cause of this failure but this was merged a few hours before the test started to fail and it involves a change in the server creation of the test: https://review.openstack.org/#/c/225575/.
Joseph

[Matt Riedemann (mriedem)]

This is partly a duplicate of bug 1464259 but that bug also includes ec2 volume related test failures (I'm not sure if they are the same root cause), but this one is specifically for the ebs test failures in test_volume_boot_pattern.py.

[Matt Riedemann (mriedem)]

This might eventually help with the races here since snapshot with ceph should be faster:

https://review.openstack.org/#/c/205282/