Tempest test for dr/readable replication fails because share has two active replicas

[Goutham Pacha Ravi]

The tempest test:

manila_tempest_tests.tests.api.test_replication.ReplicationTest#test_promote_in_sync_share_replica

fails sporadically at the gate for both ZFSonLinux and NetApp cDOT Single SVM drivers because of the update sequence in the share manager.

The test executes only when the backend_replication_type is 'dr' or 'readable'. It performs the following actions:

* Creates a share
* Creates a replica
* Waits for replica to become 'in_sync'
* Promotes the replica
* As soon as the promoted replica becomes available, it requests the list of replicas for the share

At this point, if the share manager is still updating the list of replicas, this operation gets the list of replicas in an inconsistent state. Either, we should wait to get the list of replicas or we should change the order of writing data into the database, i.e, update the promoted replica the last.

Automation might always find such bugs because our API service is uncoordinated from updates from the other services in manila.

[Goutham Pacha Ravi (gouthamr)]

Sample failure from NetApp CI attached below.

[Goutham Pacha Ravi (gouthamr)]

Sample failure from NetApp CI attached below.

[Jason Grosso (jgrosso)]

Goutham how goes this defect? Need any assistance?

[Jason Grosso (jgrosso)]

Goutham do we have a milestone set fo this?

[Goutham Pacha Ravi (gouthamr)]

Jason: we'll add a milestone if someone picks this up.

[Goutham Pacha Ravi (gouthamr)]

This bug was discussed during the upstream openstack-manila meeting on Thu 25th June 2020: http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-06-25-15.01.log.html#l-63