mgmt-datastore tests are failing for concurrency problems

Bug #1492130 reported by Sushil Kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
High
Sushil Kumar

Bug Description

the newly added mgmt-datastore-registration-api tests have been failing few times for the concurrency problems
need to look into the same.

Following are two of such historical reports:

http://logs.openstack.org/55/205355/15/gate/gate-trove-python27/b37398d/console.html#_2015-09-04_02_23_03_652

2015-09-04 02:23:03.651 | ======================================================================
2015-09-04 02:23:03.651 | FAIL: mgmt.test_datastores.TestDatastoreVersion.test_version_index
2015-09-04 02:23:03.651 | tags: worker-1
2015-09-04 02:23:03.651 | ----------------------------------------------------------------------
2015-09-04 02:23:03.651 | Traceback (most recent call last):
2015-09-04 02:23:03.651 | File "/home/jenkins/workspace/gate-trove-python27/trove/tests/unittests/mgmt/test_datastores.py", line 39, in setUp
2015-09-04 02:23:03.651 | self.ds_version1 = models.DatastoreVersion.load(self.ds, 'test_vr1')
2015-09-04 02:23:03.651 | File "trove/datastore/models.py", line 399, in load
2015-09-04 02:23:03.652 | raise exception.DatastoreVersionNotFound(version=id_or_name)
2015-09-04 02:23:03.652 | trove.common.exception.DatastoreVersionNotFound: Datastore version 'test_vr1' cannot be found.
2015-09-04 02:23:03.652 | Ran 1267 tests in 311.575s
2015-09-04 02:23:03.652 | FAILED (id=0, failures=1, skips=1)
2015-09-04 02:23:03.652 | error: testr failed (1)

http://logs.openstack.org/55/205355/15/check/gate-trove-python27/f1d91b0/console.html#_2015-09-04_05_56_39_184

2015-09-04 05:56:39.183 | ======================================================================
2015-09-04 05:56:39.183 | FAIL: mgmt.test_datastores.TestDatastoreVersion.test_version_index
2015-09-04 05:56:39.183 | tags: worker-6
2015-09-04 05:56:39.183 | ----------------------------------------------------------------------
2015-09-04 05:56:39.183 | Traceback (most recent call last):
2015-09-04 05:56:39.183 | File "/home/jenkins/workspace/gate-trove-python27/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
2015-09-04 05:56:39.184 | return func(*args, **keywargs)
2015-09-04 05:56:39.184 | File "/home/jenkins/workspace/gate-trove-python27/trove/tests/unittests/mgmt/test_datastores.py", line 146, in test_version_index
2015-09-04 05:56:39.184 | self.req, self.tenant_id)
2015-09-04 05:56:39.184 | File "trove/common/auth.py", line 91, in wrapper
2015-09-04 05:56:39.184 | return f(*args, **kwargs)
2015-09-04 05:56:39.184 | File "trove/extensions/mgmt/datastores/service.py", line 90, in index
2015-09-04 05:56:39.184 | ds_version.id) for ds_version in db_ds_versions]
2015-09-04 05:56:39.184 | File "trove/datastore/models.py", line 409, in load_by_uuid
2015-09-04 05:56:39.184 | raise exception.DatastoreVersionNotFound(version=uuid)
2015-09-04 05:56:39.184 | trove.common.exception.DatastoreVersionNotFound: Datastore version 'ab2d1574-1512-4296-92f4-ce12f0e7810f' cannot be found.
2015-09-04 05:56:39.184 | Ran 1279 tests in 205.702s
2015-09-04 05:56:39.184 | FAILED (id=0, failures=1, skips=1)
2015-09-04 05:56:39.185 | error: testr failed (1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/220419

Changed in trove:
assignee: nobody → Sushil Kumar (sushil-kumar2)
status: New → In Progress
Changed in trove:
importance: Undecided → High
milestone: none → liberty-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/220419
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=ca576ac6d24aa4bafa43c247e99ccbff482bc098
Submitter: Jenkins
Branch: master

commit ca576ac6d24aa4bafa43c247e99ccbff482bc098
Author: Sushil Kumar <email address hidden>
Date: Fri Sep 4 08:48:38 2015 +0000

    Fixes the mgmt.test_datastores errors

    The test in mgmt.test_datastores.TestDatastoreVersion.test_version_index
    is failing because of concurrency race problems.

    Executing in sequence the delete method would delete one
    of the datastore versions and
    while executing another test for index method,
    the setup method would recreate the deleted object
    so that queries work correctly.

    But because of delete and index methods running concurrently
    on different threads, the delete is deleting the object
    during the execution of setup/index method on another thread
    because of which the object query is failing.

    As a fix, did two things,
    1. Since 'test_vr1' is only used by delete method,
       shifted the query of test_vr1 to delete method.
    2. Updated the mocked method used in test_version_index from
       models.Datastore.load to models.DatastoreVersion.load_by_uuid
       the former is not needed in test_version_index,
       the not-useful mock got left, probably,
       while the code for index method was updated.

    Change-Id: I816b602579892588d3179a0dc08b9fc5668f1e5b
    Closes-Bug: #1492130

Changed in trove:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in trove:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in trove:
milestone: liberty-rc1 → 4.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.