DS8000 driver terminates volume connection when there still have volume attached to instances

[cao biao]

cinder version: Ussuri 7.0.0
nova version : Ussuri 17.0.0
hypervisor: KVM

the process to reproduce this bug:
1. create a multiattach volume type
2. create 2 instances on same host
3. create a multi-attach volume and attach it to these 2 instances
4. detach the volume from one the instance (the volume still attached to the other instance)

As a result, we can found that the host in storage backend was removed and no more host mapping found.

On KVM compute host, we can see that the path of multipath were all in failed status:

# multipath -l
mpathdd (3600507640083826de000000000008b3e) dm-2 IBM,2145
size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 0:0:2:0 sdb 8:16 failed undef running
| |- 1:0:1:0 sdf 8:80 failed undef running
| |- 1:0:0:0 sdg 8:96 failed undef running
| `- 1:0:6:0 sdh 8:112 failed undef running
`-+- policy='service-time 0' prio=0 status=enabled
  |- 0:0:1:0 sda 8:0 failed undef running
  |- 1:0:2:0 sdc 8:32 failed undef running
  |- 1:0:7:0 sdd 8:48 failed undef running
  `- 1:0:3:0 sde 8:64 failed undef running

The attachment_delete in cinder-volume run as following:

    def attachment_delete(self,
                          context: context.RequestContext,
                          attachment_id: str,
                          vref: objects.Volume) -> None:
        volume_utils.require_driver_initialized(self.driver)
        attachment = objects.VolumeAttachment.get_by_id(context, attachment_id)

        self._notify_about_volume_usage(context, vref, "detach.start")
        has_shared_connection = self._connection_terminate(context,
                                                           vref,
                                                           attachment)
        try:
            LOG.debug('Deleting attachment %(attachment_id)s.',
                      {'attachment_id': attachment.id},
                      resource=vref)
            self.driver.detach_volume(context, vref, attachment)
            if has_shared_connection is not None and not has_shared_connection:
                self.driver.remove_export(context.elevated(), vref)
        except Exception:

which means cinder will always try to terminate connection even if it is a multi-attach volume.

I look up the driver of LVM, its terminate_connection work like this:

        attachments = volume.volume_attachment
        if volume.multiattach:
            if sum(1 for a in attachments if a.connector and
                    a.connector['initiator'] == connector['initiator']) > 1:
                return True

        self.target_driver.terminate_connection(volume, connector, **kwargs)
        return len(attachments) > 1

So, I think the cinder-volume suppose the terminate_connection in driver should do special process for multiattach volumes.
As a contrast , I look up the code of DS8000 driver , but there is no specified operations for multiattach volumes.

So when delete the first attachment of the multiattach volume , the connection was terminated, this is not expected.

[cao biao (flytiger)]

SVC driver has same problem reported in this bug: https://bugs.launchpad.net/cinder/+bug/1941694

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/837699

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/837699
Committed: https://opendev.org/openstack/cinder/commit/56d27404c7f8030b359d63ef5f823da16588d2cc
Submitter: "Zuul (22348)"
Branch: master

commit 56d27404c7f8030b359d63ef5f823da16588d2cc
Author: haailani <email address hidden>
Date: Wed Apr 13 08:02:38 2022 +0000

    [IBM DS8000] Fixed Detach for multi-attach volumes

    During detach operation for Multi attach type volumes
    the first detach from the host is successful.
    In the second detach operation the volume remains in
    detaching state.

    Fixed the above issue by adding a check for multi-attach
    volumes and get the host mapping info of a volume.
    If the volume is attached to multiple hosts, then skip
    the delete host operation and return.

    Closes-bug: #1951046

    Change-Id: I8f4586292d1310d6f7bd9fd22b98715f3414d23c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 21.0.0.0rc1 release candidate.