Cinder

Dell SC terminates volume connection wrongly when instances with the multi-attach volume are on same host.

Bug #1822229 reported by Sam Wan on 2019-03-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Fix Released	Undecided	Rajini Karthik

Bug Description

If a volume is multi-attached to multiple instances on the same host and when the volume is removed from one instance, Dell SC removes the volume connection from the host which causes rest instances lose connection to the volume.

steps to reproduce:
1. create a volume type with multiattach enabled.
2. create a volume of multi-attach type
3. create 2 server instances and make sure they're on the same host.
(can use '--availability-zone ZONE:HOST:NODE')
4. attach volume to instances
5. remove volume from 1 instance.
6. check on Dell SC DSM that the volume has no 'Mappings' for the host
7. remove volume from rest instance.
errors can be observed from nova log for FC connection
======
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 314, in driver_detach
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server encryption=encryption)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1624, in detach_volume
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server encryption=encryption)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1267, in _disconnect_volume
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server vol_driver.disconnect_volume(connection_info, instance)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/fibrechannel.py", line 72, in disconnect_volume
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server connection_info['data'])
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 328, in disconnect_volume
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server wwn = self._linuxscsi.get_scsi_wwn(path)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py", line 150, in get_scsi_wwn
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server root_helper=self._root_helper)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs)
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server raise exc_type(*result[2])
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2019-03-29 02:06:44.070 1 ERROR oslo_messaging.rpc.server Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:06:00.0-fc-0x5000d3100101d531-lun-3
======

Tags:

Eric Harney (eharney) on 2019-04-08

Changed in cinder:
assignee:	nobody → Rajini Karthik (rajini-karthik)
tags:	added: multiattach

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-02: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/656835

Changed in cinder:
status:	New → In Progress

OpenStack Infra (hudson-openstack) on 2019-08-29

Changed in cinder:
assignee:	Rajini Karthik (rajini-karthik) → Eric Harney (eharney)

Eric Harney (eharney) on 2019-08-29

Changed in cinder:
assignee:	Eric Harney (eharney) → Rajini Karthik (rajini-karthik)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-30: Fix merged to cinder (master)

Reviewed: https://review.opendev.org/656835
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=08d35b099113679eb7d343f7463d63cb6f1adbbc
Submitter: Zuul
Branch: master

commit 08d35b099113679eb7d343f7463d63cb6f1adbbc
Author: rajinir <email address hidden>
Date: Thu May 2 12:43:40 2019 -0500

Dell EMC SC: Handle the mappings of multiattached volume

    For volumes attached to multiple instances, this handles
    the deletion of mappings in the backend when the volume
    is attached to multiple instances on the same host.

    Change-Id: I33d691d1dfa835b70726bcfacf868faab52de16a
    Closes-Bug: #1822229
    Co-Authored-By: Sam Wan <email address hidden>

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-02: Fix proposed to cinder (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/679643

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-03: Fix proposed to cinder (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/679857

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-18: Fix proposed to cinder (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/682776

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-18: Fix merged to cinder (stable/stein)

Reviewed: https://review.opendev.org/679857
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9b5b380a6cb1a7e99d277ffe177ad02405f9fd00
Submitter: Zuul
Branch: stable/stein

commit 9b5b380a6cb1a7e99d277ffe177ad02405f9fd00
Author: rajinir <email address hidden>
Date: Thu May 2 12:43:40 2019 -0500

Dell EMC SC: Handle the mappings of multiattached volume

    For volumes attached to multiple instances, this handles
    the deletion of mappings in the backend when the volume
    is attached to multiple instances on the same host.

    Change-Id: I33d691d1dfa835b70726bcfacf868faab52de16a
    Closes-Bug: #1822229
    Co-Authored-By: Sam Wan <email address hidden>
    (cherry picked from commit 08d35b099113679eb7d343f7463d63cb6f1adbbc)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-25: Fix merged to cinder (stable/rocky)

Reviewed: https://review.opendev.org/682776
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=767bdf1a5e28f18c12dba03adc429766f9be79d6
Submitter: Zuul
Branch: stable/rocky

commit 767bdf1a5e28f18c12dba03adc429766f9be79d6
Author: rajinir <email address hidden>
Date: Thu May 2 12:43:40 2019 -0500

Dell EMC SC: Handle the mappings of multiattached volume

    For volumes attached to multiple instances, this handles
    the deletion of mappings in the backend when the volume
    is attached to multiple instances on the same host.

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-27: Fix included in openstack/cinder 15.0.0.0rc1

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-04: Fix included in openstack/cinder 14.0.2

This issue was fixed in the openstack/cinder 14.0.2 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-04: Fix included in openstack/cinder 13.0.7

#10

This issue was fixed in the openstack/cinder 13.0.7 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-09: Change abandoned on cinder (stable/queens)

#11

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/679643
Reason: "For this driver, multiattach is not supported in stable/queens (upstream), so I would abandon this patch."

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-21: Fix proposed to cinder (master)

#12

Fix proposed to branch: master
Review: https://review.opendev.org/742162

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.