Dell SC terminates volume connection wrongly when instances with the multi-attach volume are on same host.

Bug #1822229 reported by Sam Wan on 2019-03-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Rajini Karthik

Bug Description

If a volume is multi-attached to multiple instances on the same host and when the volume is removed from one instance, Dell SC removes the volume connection from the host which causes rest instances lose connection to the volume.

steps to reproduce:
1. create a volume type with multiattach enabled.
2. create a volume of multi-attach type
3. create 2 server instances and make sure they're on the same host.
(can use '--availability-zone ZONE:HOST:NODE')
4. attach volume to instances
5. remove volume from 1 instance.
6. check on Dell SC DSM that the volume has no 'Mappings' for the host
7. remove volume from rest instance.
errors can be observed from nova log for FC connection
======
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 314, in driver_detach
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server encryption=encryption)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1624, in detach_volume
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server encryption=encryption)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1267, in _disconnect_volume
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server vol_driver.disconnect_volume(connection_info, instance)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/fibrechannel.py", line 72, in disconnect_volume
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server connection_info['data'])
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 328, in disconnect_volume
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server wwn = self._linuxscsi.get_scsi_wwn(path)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py", line 150, in get_scsi_wwn
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server root_helper=self._root_helper)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server raise exc_type(*result[2])
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:06:00.0-fc-0x5000d3100101d531-lun-3
======

Eric Harney (eharney) on 2019-04-08
Changed in cinder:
assignee: nobody → Rajini Karthik (rajini-karthik)
tags: added: multiattach

Fix proposed to branch: master
Review: https://review.opendev.org/656835

Changed in cinder:
status: New → In Progress
Changed in cinder:
assignee: Rajini Karthik (rajini-karthik) → Eric Harney (eharney)
Eric Harney (eharney) on 2019-08-29
Changed in cinder:
assignee: Eric Harney (eharney) → Rajini Karthik (rajini-karthik)

Reviewed: https://review.opendev.org/656835
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=08d35b099113679eb7d343f7463d63cb6f1adbbc
Submitter: Zuul
Branch: master

commit 08d35b099113679eb7d343f7463d63cb6f1adbbc
Author: rajinir <email address hidden>
Date: Thu May 2 12:43:40 2019 -0500

    Dell EMC SC: Handle the mappings of multiattached volume

    For volumes attached to multiple instances, this handles
    the deletion of mappings in the backend when the volume
    is attached to multiple instances on the same host.

    Change-Id: I33d691d1dfa835b70726bcfacf868faab52de16a
    Closes-Bug: #1822229
    Co-Authored-By: Sam Wan <email address hidden>

Changed in cinder:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/679857
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9b5b380a6cb1a7e99d277ffe177ad02405f9fd00
Submitter: Zuul
Branch: stable/stein

commit 9b5b380a6cb1a7e99d277ffe177ad02405f9fd00
Author: rajinir <email address hidden>
Date: Thu May 2 12:43:40 2019 -0500

    Dell EMC SC: Handle the mappings of multiattached volume

    For volumes attached to multiple instances, this handles
    the deletion of mappings in the backend when the volume
    is attached to multiple instances on the same host.

    Change-Id: I33d691d1dfa835b70726bcfacf868faab52de16a
    Closes-Bug: #1822229
    Co-Authored-By: Sam Wan <email address hidden>
    (cherry picked from commit 08d35b099113679eb7d343f7463d63cb6f1adbbc)

tags: added: in-stable-stein

Reviewed: https://review.opendev.org/682776
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=767bdf1a5e28f18c12dba03adc429766f9be79d6
Submitter: Zuul
Branch: stable/rocky

commit 767bdf1a5e28f18c12dba03adc429766f9be79d6
Author: rajinir <email address hidden>
Date: Thu May 2 12:43:40 2019 -0500

    Dell EMC SC: Handle the mappings of multiattached volume

    For volumes attached to multiple instances, this handles
    the deletion of mappings in the backend when the volume
    is attached to multiple instances on the same host.

    Change-Id: I33d691d1dfa835b70726bcfacf868faab52de16a
    Closes-Bug: #1822229
    Co-Authored-By: Sam Wan <email address hidden>
    (cherry picked from commit 08d35b099113679eb7d343f7463d63cb6f1adbbc)

tags: added: in-stable-rocky

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

This issue was fixed in the openstack/cinder 14.0.2 release.

This issue was fixed in the openstack/cinder 13.0.7 release.

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/679643
Reason: "For this driver, multiattach is not supported in stable/queens (upstream), so I would abandon this patch."

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers