tempest.api.volume.admin.test_volumes_actions.VolumesActionsTest.test_force_detach_volume failed

[Dell EMC PS Series CI]

test case 'tempest.api.volume.admin.test_volumes_actions.VolumesActionsTest.test_force_detach_volume' introduced by https://review.openstack.org/#/c/436400/ failed without connector information.
======
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager [req-ba63e390-9653-465c-afd8-810eed42c74d tempest-VolumesActionsTest-975408085 -] Terminate volume connection failed: argument of type 'NoneType' is not iterable
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager Traceback (most recent call last):
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 1555, in terminate_connection
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager force=force)
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/zonemanager/utils.py", line 104, in decorator
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager conn_info = terminate_connection(self, *args, **kwargs)
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/dell_emc/vnx/driver.py", line 207, in terminate_connection
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager conn_info = self.adapter.terminate_connection(volume, connector)
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/dell_emc/vnx/adapter.py", line 989, in terminate_connection
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager return self._terminate_connection(volume, connector)
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/dell_emc/vnx/adapter.py", line 999, in _terminate_connection
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager host = self.build_host(connector)
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/dell_emc/vnx/adapter.py", line 1484, in build_host
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager if 'wwnns' not in connector or 'wwpns' not in connector:
2017-04-25 22:09:41.157 9375 ERROR cinder.volume.manager TypeError: argument of type 'NoneType' is not iterable
====

seems the force_detach_volume does not pass any 'connector' information to driver.

detailed logs:

http://publiclogs.emc.com/vnx_ostack/EMC_VNX_FC/317/logs/screen-c-vol.txt.gz
http://openstack.falconstor.com/40/445440/7/check/dsvm-cinder-fss-iscsi/ec83331/logs/screen-c-vol.txt.gz

[Jeremy Zhang (jeremy.zhang)]

    Codes below is in cinder/api/contrib/admin_actions.py. That we can find that the connector can be set to None. Here I think /cinder/cinder/volume/drivers/dell_emc/vnx/driver.py should handle the scenario when connector is None.

    @wsgi.action('os-force_detach')
    def _force_detach(self, req, id, body):
        """Roll back a bad detach after the volume been disconnected."""
        context = req.environ['cinder.context']
        self.authorize(context, 'force_detach')
        # Not found exception will be handled at the wsgi level
        volume = self._get(context, id)
        try:
            connector = body['os-force_detach'].get('connector', None)
        except KeyError:
            raise webob.exc.HTTPBadRequest(
                explanation=_("Must specify 'connector'."))
        try:
            self.volume_api.terminate_connection(context, volume, connector)
        except exception.VolumeBackendAPIException as error:
            msg = _("Unable to terminate volume connection from backend.")
            raise webob.exc.HTTPInternalServerError(explanation=msg)
     ......

[Matthew Treinish (treinish)]

This isn't a bug in tempest, tempest is just exposing an issue in emc vnx cinder driver. (it's what tests are supposed to do)

[nikeshkm (nike-niec)]

"tempest.api.volume.admin.test_volumes_actions.VolumesActionsTest.test_force_detach_volume" tempest test is failing for Kaminario CI for both FC and iSCSI Kaminario Cinder Drivers

Following is Traceback for Kaminario FC Cinder Driver:
tempest-VolumesActionsTest-922037769 -] Terminate volume connection failed: 'NoneType' object has no attribute 'get'
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager Traceback (most recent call last):
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 1555, in terminate_connection
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager force=force)
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/zonemanager/utils.py", line 104, in decorator
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager conn_info = terminate_connection(self, *args, **kwargs)
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/kaminario/kaminario_common.py", line 106, in func_wrapper
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager ret = func(*args, **kwargs)
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "<decorator-gen-251>", line 2, in terminate_connection
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/coordination.py", line 301, in _synchronized
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager return f(*a, **k)
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/kaminario/kaminario_fc.py", line 97, in terminate_connection
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager super(KaminarioFCDriver, self).terminate_connection(volume, connector)
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/kaminario/kaminario_common.py", line 106, in func_wrapper
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager ret = func(*args, **kwargs)
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/kaminario/kaminario_common.py", line 931, in terminate_connection
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager host_name = self.get_initiator_host_name(connector)
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/drivers/kaminario/kaminario_common.py", line 844, in get_initiator_host_name
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager return re.sub('[^0-9a-zA-Z-_]', '_', connector.get('host', ''))[:32]
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager AttributeError: 'NoneType' object has no attribute 'get'
2017-04-26 08:52:25.372 7484 ERROR cinder.volume.manager
2017-04-26 08:52:25.374 7484 ERROR oslo_messaging.rpc.server [req-ae17f87f-9616-49be-9e30-1c401ca8bac4 tempest-VolumesActionsTest-922037769 -] Exception during message handling
2017-04-26 08:52:25.374 7484 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-26 08:52:25.374 7484 ERROR oslo_messaging.rpc.server File "/usr/local/lib...

[Goutham Pacha Ravi (gouthamr)]

@Matthew Treinish,

The API reference states that "connector" is a required parameter and the example also shows how the connector object looks like, depending on the protocol being used, information passed into the connector object is different.

A sample connector should look like the example provided here:

https://developer.openstack.org/api-ref/block-storage/v2/?expanded=force-detach-volume-detail#force-detach-volume

In the most general case, when attaching a cinder volume, the connector information is always validated to be containing "initiator" or ["wwpns", "wwnns"] information:

See: https://github.com/openstack/cinder/blob/f6fa604/cinder/volume/manager.py#L1501
See the base iSCSI driver: https://github.com/openstack/cinder/blob/f6fa604/cinder/volume/driver.py#L2815
See the base FC driver:
https://github.com/openstack/cinder/blob/f6fa604/cinder/volume/driver.py#L2961

As far as i can see, NFS drivers don't need any connector information. So, this test case isn't generic enough in my opinion.

[Goutham Pacha Ravi (gouthamr)]

The reason that the reference driver (LVM with tgtadm) didn't fail this test is because the terminate_connection implementation is a no-op. There's nothing for this driver to do:

LOG:
http://logs.openstack.org/00/436400/14/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/6ca812b/logs/

Cinder.conf:
http://logs.openstack.org/00/436400/14/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/6ca812b/logs/etc/cinder/cinder.conf.txt.gz

LVM driver:
https://github.com/openstack/cinder/blob/a55a6b5c/cinder/volume/drivers/lvm.py#L846

Target:
https://github.com/openstack/cinder/blob/a55a6b5c/cinder/volume/targets/iscsi.py#L374

[Shay Halsband (shay-halsband)]

this issue also affects XtremIO driver, it doesn't make sense to pass a None connector to the driver, it should first validate it.

[Xing Yang (xing-yang)]

Hi Shay,

Please take a look of this patch: https://review.openstack.org/#/c/471861/. You may need to submit a similar fix for the XtremIO driver. Thanks.