Tiramisu: After failover, active_backend_id is not set to secondary backend id

[Vivek Soni]

3PAR Tiramisu: After performing failover, attaching a volume points to primary array and it create a vlun in primary array instead of secondary

Steps:
----------------------
1) group create
    cinder group-create --name GROUP GVGT 3pariscsirep
2) create volumes
3) group update - addition of volumes in group
    cinder --os-volume-api-version=3.38 group-update --add-volumes <vol1,vol2> <group-name>
4) attach volume to a nova instance
4) perform group failover
    cinder --os-volume-api-version=3.38 group-failover-replication <group-id> --allow-attached-volume --secondary-backend-id <target_array>
5) detach and re-attach

re-attach the volume create a vlun in primary array instead of secondary

[Vivek Soni (viveksoni)]

"active_backend_id" is not set to secondary array after fail over, therefore after attachment vluns are created in primary array only instead of secondary array

[TommyLike (hu-husheng)]

I am not sure whether this is designed for purpose, but now we don't have any attribute that stands for the group replication's **active_backend_id** (we have it in service for failover host mechanism.
We can either add this back into group object or ask driver to maintain this relationship. Anyway, this needs Xing Yang or anyone who is more familiar with group replication feature's feedback.

[Xing Yang (xing-yang)]

This works as designed. To change the "active_backend_id", you need to use the promote feature designed by the following spec: https://specs.openstack.org/openstack/cinder-specs/specs/newton/cheesecake-promote-backend.html. I'm not sure if anyone is working on the promote feature though.

[Vivek Soni (viveksoni)]

Hi Xing,

I am facing issue with Tiramisu replication.
'active_backend_id' is not set to failed-over backend i.e. secondary array

For example, arrat A is primary and array B is secondary, so after failover, i am expecting 'active_backend_id' value should be 'array B'
And this is not happening.

[Vivek Soni (viveksoni)]

With Cheesecake, after failing over to secondary array B, 'active_backend_id' is set properly as array B.

[TommyLike (hu-husheng)]

After a conversation with Xing Yang, she mentioned that the active_backend_id will change only when the whole backend fail over, so it seems that the driver need to keep which is primary and which is secondary for one specific replication group in this case:).

[Imran Ansari (imran.ansari)]

Driver can hold the information in-memory but the problem is, how do you persist it across service restarts or reboots? Looks like Cinder framework would need to maintain this information for each GVG.

[Vivek Soni (viveksoni)]

I think this info can be fetched from volume or group object. i am about to identify the code changes required for the same. I will update you on this.

Strange observation seen with cheesecake replication:
----------------------------------------------------------
In Queens setup, attached volume after failover, were able to DETACHED successfully from primary array and reattaching the volumes pointed to secondary backend & this was expected.

whereas in Master(Rocky) setup, attached volume after failover, FAILED to DETACH from primary array itself.

Looking at the HPE driver code from both releases, there seems to be hardly any difference around that flow. I suspect some changes in cinder framework resulting in this different observations amongst both releases.

[Vivek Soni (viveksoni)]

please ignore my comments w.r.t to cheesecake, there was some issue with setup
----------------------------------------------

In Group Replication(Tiramisu), decision on pointing to primary or secondary backend info can be based on the volume attribute (i.e. replication_driver_data).
This will work fine when performing the attachment after failover, but will not work with detachment after failover. please check the below scenario:

1) volume1 is in group1 which is having group_replication enabled
2) volume1 is attached to a nova instances ### here, replication_driver_data is 'None'
3) Performing a group fail-over
4) detaching ### here in terminate_connection, volume object 'replication_driver_data' will have secondary array ID & my configuration points to secondary array and my vluns were present in primary array and this will fail

one way to handle this, we will first try with deleting vlun with secondary array configuration if it fails then try with primary array configuration

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/575702

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/575702
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a433c19ef09c2f3e44dd3dd23fa1283c401aaf30
Submitter: Zuul
Branch: master

commit a433c19ef09c2f3e44dd3dd23fa1283c401aaf30
Author: Vivek Soni <email address hidden>
Date: Fri Jun 15 01:31:17 2018 -0700

    HPE3PAR: Fix pointing to backend in group failover

    Issue: After group failover, subsequent operations
    like attach & detach volume, which are part of that
    failed over group points to primary backend instead
    of secondary.

    This patch fixes the above issue by setting up the
    appropriate backend in case of subsequent operation
    on volume, which are part of group which is failed
    over.

    Change-Id: I679b11317c91ad28cefdf995a8d6849dc71bc1c5
    Closes-Bug: #1773069

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/577044

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577044
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=e6f24570c3b49ffbc7775add8a13223da685baf8
Submitter: Zuul
Branch: stable/queens

commit e6f24570c3b49ffbc7775add8a13223da685baf8
Author: Vivek Soni <email address hidden>
Date: Thu Jun 21 00:52:48 2018 -0400

    HPE3PAR: Fix pointing to backend in group failover

    Issue: After group failover, subsequent operations
    like attach & detach volume, which are part of that
    failed over group points to primary backend instead
    of secondary.

    This patch fixes the above issue by setting up the
    appropriate backend in case of subsequent operation
    on volume, which are part of group which is failed
    over.

    Change-Id: I679b11317c91ad28cefdf995a8d6849dc71bc1c5
    Closes-Bug: #1773069
    (cherry picked from commit a433c19ef09c2f3e44dd3dd23fa1283c401aaf30)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 13.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.4 release.