Tempest test "api.test_replication.ReplicationTest.test_promote_in_sync_share_replica" fails sometimes

[Valeriy Ponomaryov]

Tempest test "test_promote_in_sync_share_replica" that is located in "manila_tempest_tests/tests/api/test_replication/ReplicationTest.py" module happens to fail [1].

With following traceback:

Traceback (most recent call last):
  File "/opt/stack/new/manila/manila_tempest_tests/tests/api/test_replication.py", line 223, in test_promote_in_sync_share_replica
    self._verify_in_sync_replica_promotion(share, original_replica)
  File "/opt/stack/new/manila/manila_tempest_tests/tests/api/test_replication.py", line 122, in _verify_in_sync_replica_promotion
    self._verify_active_replica_count(share["id"])
  File "/opt/stack/new/manila/manila_tempest_tests/tests/api/test_replication.py", line 99, in _verify_active_replica_count
    self.assertEqual(1, len(active_replicas))
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 2

as of following [2] manila project state.

[1] http://logs.openstack.org/03/433003/2/check/gate-manila-tempest-minimal-dsvm-dummy-ubuntu-xenial/c34aa9d/logs/tempest_2/testr_results.html.gz

[2] https://github.com/openstack/manila/blob/cffc6e1e/manila_tempest_tests/tests/api/test_replication.py

#########################################################
Update:

We have time window where we update DB for each of replicas [3] and if we get info using API in between DB updates we can get not completely updated info. So, in good case, we should start using "task_state" field of a share to track recent activities of a share in single place as we do it for share migration.

[3] https://github.com/openstack/manila/blob/15b27e8f/manila/share/manager.py#L2022-L2056
#########################################################

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/438892

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438892
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=f7783776b5479761ba9d6a6b63121be1c8957541
Submitter: Jenkins
Branch: master

commit f7783776b5479761ba9d6a6b63121be1c8957541
Author: Valeriy Ponomaryov <email address hidden>
Date: Tue Feb 28 13:05:04 2017 +0300

    Update share replicas after promotion in proper order

    Update replica that becomes active after promotion only as last
    to avoid concurrency in share replica states when we can have 2
    active share replicas in some narrow timeframe.

    Change-Id: Idf01daf31172a21dbb6595044c11980867346d13
    Closes-Bug: #1664201

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 5.0.0.0b1 development milestone.