Concurrency in "test_show_share_server" tempest test

Bug #1608600 reported by Valeriy Ponomaryov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Medium
Valeriy Ponomaryov

Bug Description

In test "test_show_share_server" that is located in "manila_tempest_tests/tests/api/admin/test_share_servers.py" module we can (rarely) face concurrency issue, when first share_server from list of share servers gets deleted between "taking list of servers" call and "getting its first element". It can be any share server os any concurrent test. So, we should fix it by usage of share server created specially for that test[suite].

Logs: http://logs.openstack.org/29/348829/1/check/gate-manila-tempest-minimal-dsvm-dummy/94aa130/logs/tempest_1/testr_results.html.gz

paste: http://paste.openstack.org/show/545162/

raw:

ft20.9: manila_tempest_tests.tests.api.admin.test_share_servers.ShareServersAdminTest.test_show_share_server[api_with_backend,positive]_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
2016-07-29 11:39:57,077 15664 INFO [tempest.lib.common.rest_client] Request (ShareServersAdminTest:test_show_share_server): 200 GET http://127.0.0.1:8786/v1/efe50071c67a4023b8e9b07f3e3dea86/share-servers 1.534s
2016-07-29 11:39:57,077 15664 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'content-length': '10829', 'content-location': 'http://127.0.0.1:8786/v1/efe50071c67a4023b8e9b07f3e3dea86/share-servers', 'date': 'Fri, 29 Jul 2016 11:39:57 GMT', 'connection': 'close', 'vary': 'X-OpenStack-Manila-API-Version', 'x-openstack-manila-api-version': '1.0', 'content-type': 'application/json', 'x-compute-request-id': 'req-ef62df21-7884-48cb-bd6d-fc6b16759129', 'status': '200'}
        Body: {"share_servers": [{"status": "active", "updated_at": "2016-07-29T11:39:53.000000", "share_network_name": "9e095535-79d6-4913-9629-d9ee1c49f1ae", "host": "ubuntu-trusty-rax-ord-3015824@beta", "share_network_id": "9e095535-79d6-4913-9629-d9ee1c49f1ae", "project_id": "cf601cbb95fd44dbbd2c4e01dab1f2f0", "id": "0486354b-502b-488e-b71a-3449d02b7a2f"}, {"status": "active", "updated_at": "2016-07-29T11:38:42.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@beta", "share_network_id": "78274dfb-e194-45f0-b188-9a3d3caf4d42", "project_id": "ea93d33873ca41759f657cc7ac6c8aac", "id": "1c711374-d60c-49f8-91a3-37fc13221f7e"}, {"status": "active", "updated_at": "2016-07-29T11:39:53.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@alpha", "share_network_id": "00cdc16d-6ccc-4ef6-96e4-74abcc897ace", "project_id": "27b4c68079d04c15a5f1d855f18b06a2", "id": "201549b2-d1a4-41ea-9216-47d04f37ce5e"}, {"status": "active", "updated_at": "2016-07-29T11:39:49.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@beta", "share_network_id": "f94ab46c-17cc-4730-97ab-9be9e06bc5b9", "project_id": "5ce9b9914bdb42c9b4d4c5617330bd10", "id": "21f75817-8cb8-4080-a790-8f615788d213"}, {"status": "active", "updated_at": "2016-07-29T11:38:28.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@beta", "share_network_id": "337fa66a-7ade-42a2-855c-1c0a6cae58a8", "project_id": "8da07ab9f9a0421f8d8a85cb048c7ed2", "id": "266c6b5b-f207-4235-ade9-13b737d12ab2"}, {"status": "active", "updated_at": "2016-07-29T11:39:49.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@beta", "share_network_id": "00cdc16d-6ccc-4ef6-96e4-74abcc897ace", "project_id": "27b4c68079d04c15a5f1d855f18b06a2", "id": "29ee8908-5adf-4e59-b5fc-e1af82a987e0"}, {"status": "active", "updated_at": "2016-07-29T11:38:44.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@alpha", "share_network_id": "1cf0db49-2c65-4262-b879-6999cec6a82d", "project_id": "b996293ba12a416997118f6114ba5de0", "id": "2fad4cfe-6200-46b4-9f50-2ca496163879"}, {"status": "active", "updated_at": "2016-07-29T11:38:57.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@beta", "share_network_id": "e1ead095-878a-4c74-8b01-6065d1789c6f", "project_id": "043a935d647b4d2d962ecd532f87d58e", "id": "3e45f105-953b-420b-8d6b-9d0d104421aa"}, {"status": "active", "updated_at": "2016-07-29T11:39:27.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@beta", "share_network_id": "8eaeb5c3-98c3-4513-8b7e-a9348b66584d", "project_id": "8ad7dfdb9d47495493aa304dbb933b07", "id": "3eee31a4-4a9e-4b3e-b963-22f8d0bd29bf"}, {"status": "active", "updated_at": "2016-07-29T11:39:20.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@beta", "share_network_id": "3ecea374-f741-44c0-9ffb-df4d3a974f18", "project_id": "a8684df9da0b42279d2b63c01e54966a", "id": "50e60663-0758-43a0-897f-1eee953fc68b"}, {"status": "active", "updated_at": "2016-07-29T11:39:51.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@alpha", "share_network_id": "9f91bf14-09fd-4c35-9ab3-9e2f33fd3552", "project_id": "4ff5896ba128480bb18dc8a0445b6220", "id": "5a3b1dde-1291-40e7-b82e-49b581e9449a"}, {"status": "active", "updated_at": "2016-07-29T11:38:43.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty-rax-ord-3015824@alpha", "share_network_id": "337fa66a-7ade-42a2-855c-1c0a6cae58a8", "project_id": "8da07ab9f9a0421f8d8a85cb048c7ed2", "id": "5a40914e-55cb-45a4-8da4-ac0d806a8249"}, {"status": "active", "updated_at": "2016-07-29T11:38:36.000000", "share_network_name": "autogenerated_by_tempest_reusable", "host": "ubuntu-trusty
2016-07-29 11:39:57,180 15664 INFO [tempest.lib.common.rest_client] Request (ShareServersAdminTest:test_show_share_server): 404 GET http://127.0.0.1:8786/v1/efe50071c67a4023b8e9b07f3e3dea86/share-servers/0486354b-502b-488e-b71a-3449d02b7a2f 0.102s
2016-07-29 11:39:57,181 15664 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'content-length': '115', 'content-location': 'http://127.0.0.1:8786/v1/efe50071c67a4023b8e9b07f3e3dea86/share-servers/0486354b-502b-488e-b71a-3449d02b7a2f', 'date': 'Fri, 29 Jul 2016 11:39:57 GMT', 'connection': 'close', 'vary': 'X-OpenStack-Manila-API-Version', 'x-openstack-manila-api-version': '1.0', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-406e230f-19e3-4023-86e2-1ccc8fe5b2f7', 'status': '404'}
        Body: {"itemNotFound": {"message": "Share server 0486354b-502b-488e-b71a-3449d02b7a2f could not be found.", "code": 404}}
}}}

Traceback (most recent call last):
  File "/opt/stack/new/manila/manila_tempest_tests/tests/api/admin/test_share_servers.py", line 169, in test_show_share_server
    server = self.shares_client.show_share_server(servers[0]["id"])
  File "/opt/stack/new/manila/manila_tempest_tests/services/share/json/shares_client.py", line 735, in show_share_server
    resp, body = self.get(uri)
  File "tempest/lib/common/rest_client.py", line 285, in get
    return self.request('GET', url, extra_headers, headers)
  File "tempest/lib/common/rest_client.py", line 664, in request
    resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 762, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Share server 0486354b-502b-488e-b71a-3449d02b7a2f could not be found.', u'code': 404}

Changed in manila:
importance: Undecided → Medium
assignee: nobody → Valeriy Ponomaryov (vponomaryov)
milestone: none → newton-3
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/349599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/349599
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=0fa609e2f2b31f59589a2e2ce07b47cfd157ab03
Submitter: Jenkins
Branch: master

commit 0fa609e2f2b31f59589a2e2ce07b47cfd157ab03
Author: Valeriy Ponomaryov <email address hidden>
Date: Mon Aug 1 19:59:24 2016 +0300

    [Tempest] Fix concurrency in "test_show_share_server" test

    In test "test_show_share_server" that is located in
    "manila_tempest_tests/tests/api/admin/test_share_servers.py" module
    we can (rarely) face concurrency issue, when first share_server from
    list of share servers gets deleted between "taking list of servers" call and
    "getting its first element". It can be any share server of any concurrent test.
    So, fix it by usage of share server that is being hold by its class and will
    always exist while this test runs.

    Change-Id: I7675a201bcd1eb33f1214ba9837f28a5295f54fc
    Closes-Bug: #1608600

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/352668

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 3.0.0.0b3

This issue was fixed in the openstack/manila 3.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/mitaka)

Reviewed: https://review.openstack.org/352668
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=b8c7fad599063434f1adb0e33ff905b871c660de
Submitter: Jenkins
Branch: stable/mitaka

commit b8c7fad599063434f1adb0e33ff905b871c660de
Author: Valeriy Ponomaryov <email address hidden>
Date: Mon Aug 1 19:59:24 2016 +0300

    [Tempest] Fix concurrency in "test_show_share_server" test

    In test "test_show_share_server" that is located in
    "manila_tempest_tests/tests/api/admin/test_share_servers.py" module
    we can (rarely) face concurrency issue, when first share_server from
    list of share servers gets deleted between "taking list of servers" call and
    "getting its first element". It can be any share server of any concurrent test.
    So, fix it by usage of share server that is being hold by its class and will
    always exist while this test runs.

    Change-Id: I7675a201bcd1eb33f1214ba9837f28a5295f54fc
    Closes-Bug: #1608600
    (cherry picked from commit 0fa609e2f2b31f59589a2e2ce07b47cfd157ab03)

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.