Several tempest test for nova are failed

[Sofiia Andriichenko]

Detailed bug description:
9.0 mos iso #368
test_list_images_filter_by_changes_since
test_list_images_filter_by_name
test_list_images_filter_by_server_id
test_list_images_filter_by_server_ref
test_list_images_filter_by_status
test_list_images_with_detail_filter_by_changes_since
test_list_images_limit_results
test_list_images_with_detail_filter_by_server_ref
test_list_images_with_detail_filter_by_name
test_list_images_with_detail_filter_by_type
test_list_images_filter_by_type
test_list_images_with_detail_filter_by_status
test_list_images_with_detail_limit_results

Configuration:
Settings:
Storage Backends - Ceph RBD for volumes (Cinder), Ceph RBD for ephemeral volumes (Nova), Ceph RBD for images (Glance), Ceph RadosGW for objects (Swift API)
Additional services - Install Sahara

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR

Nodes: controller, compute, Ceph, Telemetry - MongoDB

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
        #rally --debug verify start --regex tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_filter_by_changes_since

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment
~ 2%

Workaround:
---

Impact:
---

Description of the environment:
See (Detailed bug description)

Additional information:
Error Message

test failed

Stacktrace

Traceback (most recent call last):
  File "tempest/test.py", line 280, in setUpClass
    six.reraise(etype, value, trace)
  File "tempest/test.py", line 273, in setUpClass
    cls.resource_setup()
  File "tempest/api/compute/images/test_list_image_filters.py", line 98, in resource_setup
    cls.server1['id'], wait_until='ACTIVE')
  File "tempest/api/compute/base.py", line 311, in create_image_from_server
    image_id, kwargs['wait_until'])
  File "tempest/common/waiters.py", line 134, in wait_for_image_status
    image = client.show_image(image_id)
  File "tempest/lib/services/compute/images_client.py", line 63, in show_image
    resp, body = self.get("images/%s" % image_id)
  File "tempest/lib/common/rest_client.py", line 274, in get
    return self.request('GET', url, extra_headers, headers)
  File "tempest/lib/services/compute/base_compute_client.py", line 53, in request
    method, url, extra_headers, headers, body)
  File "tempest/lib/common/rest_client.py", line 641, in request
    resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 694, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Image not found.', u'code': 404}

[Sofiia Andriichenko (sandriichenko)]

Snapshot https://drive.google.com/open?id=0B8hkiEm94sEtNUJjNXVQQmxqVjA

[Roman Podoliaka (rpodolyaka)]

Sofiia, please at least provide a diagnostic snapshot.

[Roman Podoliaka (rpodolyaka)]

I checked the diagnostic snapshot Sofiia provided and it looks interesting.

nova-compute log on node-2 contains errors on uploading a snapshot:

http://xsnippet.org/361707/raw/

The traceback looks as following:

http://xsnippet.org/361711/raw/

As you can see, first nova-compute got "Broken pipe" (reading / writing from / to a socket closed by the remote side) on PUT (snapshot upload), then it failed to DELETE the partially uploaded image with 404 Not Found.

haproxy logs claim that PUT must have ended with 404 Not Found too:

http://xsnippet.org/361707/raw/

although there is SD flag, which means that the glance backend aborted the session while sending a response. It's not clear why this happens, glance-api reports a proper 404 error in its logs:

<150>May 18 10:41:54 node-1 glance-api: 2016-05-18 10:41:54.372 22194 INFO eventlet.wsgi.server [req-29c9b0f1-0916-48e3-bf43-21f2543a7806 f260ffd8ac7b48d3b6bed375ac190488 8df70ad8f13140578a278395a57589dd - - -] 10.109.1.9 - - [18/May/2016 10:41:54] "PUT /v1/images/a18296e4-f69f-485d-b756-6f44cbc28efe HTTP/1.1" 404 431 0.150278

This is due to the fact the corresponding image was not found registry:

http://xsnippet.org/361712/raw/

From haproxy logs it's clear that something (or someone) send a request to delete this image via nova-api:

2016-05-18T10:41:49.953628+00:00 info: 10.109.4.2:49619 [18/May/2016:10:41:49.327] nova-api nova-api/node-1 0/0/0/624/626 204 198 - - ---- 0/0/0/0/0 0/0 "DELETE /v2/8df70ad8f13140578a278395a57589dd/images/a18296e4-f69f-485d-b756-6f44cbc28efe HTTP/1.1"

2016-05-18T10:41:49.954023+00:00 info: 10.109.1.2:42576 [18/May/2016:10:41:49.340] glance-api glance-api/node-4 0/0/0/609/612 200 200 - - ---- 159/0/0/0/0 0/0 "DELETE /v1/images/a18296e4-f69f-485d-b756-6f44cbc28efe HTTP/1.1"

which was successfully completed and rightfully deleted the image.

grep'ing for this particular request have not shown anything helpful:

http://xsnippet.org/361710/raw/

If it was tempest, who did this, then it did not log the corresponding HTTP request.

Sofiia, are you sure this environment was not used by someone else, while tests were run? Maybe we run tests in parallel and something tried to clean up all existing images?

I suggest we keep this in Incomplete state until we see another repro.

[Sergii Turivnyi (sturivnyi)]

Can't reproduce