While testing the VMAX FC driver using CiscoFCZoneDriver, attach volume failed when trying to create and activate zone set.
Errors in cinder-volume.log
-----------------------------------
2014-09-19 23:24:16.670 DEBUG cinder.zonemanager.drivers.cisco.cisco_fc_zone_client_cli [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Exit Status from ssh:16 from (pid=7959) _ssh_execute /opt/stack/cinder/cinder/zonemanager/drivers/cisco/cisco_fc_zone_client_cli.py:436
2014-09-19 23:24:16.670 DEBUG cinder.zonemanager.drivers.cisco.cisco_fc_zone_client_cli [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Result was 16 from (pid=7959) _ssh_execute /opt/stack/cinder/cinder/zonemanager/drivers/cisco/cisco_fc_zone_client_cli.py:439
Exception AttributeError: "'ChannelFile' object has no attribute '_wbuffer'" in ignored
Exception AttributeError: "'ChannelStderrFile' object has no attribute '_wbuffer'" in ignored
2014-09-19 23:24:16.674 ERROR cinder.zonemanager.drivers.cisco.cisco_fc_zone_client_cli [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Exception: Unexpected error while running command.
Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
Exit code: 16
Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
2014-09-19 23:24:21.155 DEBUG cinder.zonemanager.drivers.cisco.cisco_fc_zone_client_cli [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Handling error case after SSH:Unexpected error while running command.
Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
Exit code: 16
Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>' from (pid=7959) _ssh_execute /opt/stack/cinder/cinder/zonemanager/drivers/cisco/cisco_fc_zone_client_cli.py:455
Exception AttributeError: "'ChannelFile' object has no attribute '_wbuffer'" in ignored
Exception AttributeError: "'ChannelStderrFile' object has no attribute '_wbuffer'" in ignored
2014-09-19 23:24:21.159 ERROR cinder.zonemanager.drivers.cisco.cisco_fc_zone_client_cli [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Error executing command via ssh: Unexpected error while running command.
Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
Exit code: 16
Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
2014-09-19 23:24:21.159 ERROR cinder.zonemanager.drivers.cisco.cisco_fc_zone_client_cli [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Creating and activating zone set failed: (Zone set=xbulaasio2p error=Unexpected error while running command.
Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
Exit code: 16
Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>').
2014-09-19 23:24:21.160 DEBUG cinder.openstack.common.lockutils [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Released file lock "cisco" at /opt/stack/data/cinder/fcfabric-cisco for method "add_connection"... from (pid=7959) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:239
2014-09-19 23:24:21.161 ERROR cinder.zonemanager.fc_zone_manager [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Failed adding connection for fabric=faba: Error:Exception: Fibre Channel Zoning CLI error: Creating and activating zone set failed: (Zone set=xbulaasio2p error=Unexpected error while running command.
Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
Exit code: 16
Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>').
2014-09-19 23:24:21.168 DEBUG paramiko.transport [-] EOF in transport thread from (pid=7959) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1269
2014-09-19 23:24:21.171 ERROR cinder.volume.manager [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Unable to fetch connection information from backend: Fibre Channel connection control failure: Failed adding connection for fabric=faba: Error:Exception: Fibre Channel Zoning CLI error: Creating and activating zone set failed: (Zone set=xbulaasio2p error=Unexpected error while running command.
Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
Exit code: 16
Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>').
2014-09-19 23:24:21.174 ERROR oslo.messaging.rpc.dispatcher [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Exception during message handling: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Fibre Channel connection control failure: Failed adding connection for fabric=faba: Error:Exception: Fibre Channel Zoning CLI error: Creating and activating zone set failed: (Zone set=xbulaasio2p error=Unexpected error while running command.
Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
Exit code: 16
Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>').
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 901, in initialize_connection
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher raise exception.VolumeBackendAPIException(data=err_msg)
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Fibre Channel connection control failure: Failed adding connection for fabric=faba: Error:Exception: Fibre Channel Zoning CLI error: Creating and activating zone set failed: (Zone set=xbulaasio2p error=Unexpected error while running command.
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher Exit code: 16
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>').
2014-09-19 23:24:21.174 TRACE oslo.messaging.rpc.dispatcher
2014-09-19 23:24:21.176 ERROR oslo.messaging._drivers.common [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] Returning exception Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Fibre Channel connection control failure: Failed adding connection for fabric=faba: Error:Exception: Fibre Channel Zoning CLI error: Creating and activating zone set failed: (Zone set=xbulaasio2p error=Unexpected error while running command.
Command: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end
Exit code: 16
Stdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'
Stderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'). to caller
2014-09-19 23:24:21.177 ERROR oslo.messaging._drivers.common [req-00e7fce6-47a1-4383-8b47-3bff49eb5a45 b58f3b9456e3464e8376eb25d5c40edb ecc06b248db44562af5599dab17bea56] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 901, in initialize_connection\n raise exception.VolumeBackendAPIException(data=err_msg)\n', "VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Fibre Channel connection control failure: Failed adding connection for fabric=faba: Error:Exception: Fibre Channel Zoning CLI error: Creating and activating zone set failed: (Zone set=xbulaasio2p error=Unexpected error while running command.\nCommand: conf ; zoneset name xbulaasio2p vsan 2007 ; zone name openstack20000025b582ad1e50000973f0025188 ; member pwwn 20:00:00:25:b5:82:ad:1e ; member pwwn 50:00:09:73:f0:02:51:88 ; end\nExit code: 16\nStdout: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>'\nStderr: u'<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x3b77910L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>').\n"]
2014-09-19 23:24:44.292 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._publish_service_capabilities from (pid=7959) run_periodic_tasks /opt/stack/cinder/cinder/openstack/common/periodic_task.py:193
cinder.conf
---------------
zone_driver=cinder.zonemanager.drivers.cisco.cisco_fc_zone_driver.CiscoFCZoneDriver
zoning_mode=fabric
[fc-zone-manager]
fc_fabric_names=faba,fabb
zoning_policy=initiator-target
cisco_sb_connector=cinder.zonemanager.drivers.cisco.cisco_fc_zone_client_cli.CiscoFCZoneClientCLI
[faba]
cisco_fc_fabric_address=1.1.1.1
cisco_fc_fabric_user=user1
cisco_fc_fabric_password=password1
cisco_fc_fabric_port=22
cisco_zoning_policy=initiator-target
cisco_zone_activate=true
cisco_zone_name_prefix=OpenStack-autozone
cisco_zoning_vsan=2007
[fabb]
cisco_fc_fabric_address=1.1.1.2
cisco_fc_fabric_user=user1
cisco_fc_fabric_password=password1
cisco_fc_fabric_port=22
cisco_zoning_policy=initiator-target
cisco_zone_activate=true
cisco_zone_name_prefix=OpenStack-autozone
cisco_zoning_vsan=2006
This is no longer a problem as it was fixed as part of some other bug fixes. Mark this as Invalid.