Cinder issues with HPE 3par FC and boot from volume or creation of volume from a glance image. New test 3par volume creation is successful if we create manually but when we try to launch instance, it was failing. When looking at the cinder volume logs, we first see: ~~~ 2017-11-20 14:32:07.061 560051 DEBUG cinder.volume.drivers.hpe.hpe_3par_fc [req-0968d084-b50c-4832-b92d-5c637ea0688d 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - default default] <== decorator: exception (1841ms) KeyError('remoteName',) trace_logging_wrapper /usr/lib/python2.7/site-packages/cinder/utils.py:901 ~~~ Then: ~~~ 2017-11-20 18:45:07.200 560051 ERROR cinder.volume.driver [req-7b9019fd-02f6-4f10-822a-24e4a4530d2e 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - default default] Unable to fetch connection information from backend: 'remoteName' 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume [req-7b9019fd-02f6-4f10-822a-24e4a4530d2e 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - default default] Failed to copy image ce69e81e-cfc7-47e1-a857-5c7710ba0501 to volume: 87e53204-8ff1-4bb5-b6a5-6b0852dda2ea 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last): 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 536, in _copy_image_to_volume 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id) 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 788, in copy_image_to_volume 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id, False) 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 805, in _copy_image_data_to_volume 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume attach_info, volume = self._attach_volume(context, volume, properties) 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 975, in _attach_volume 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume raise exception.VolumeBackendAPIException(data=err_msg) 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: 'remoteName' 2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager Traceback (most recent call last): 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager result = task.execute(**arguments) 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 873, in execute 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager **volume_spec) 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 831, in _create_from_image 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service) 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 285, in wrapped 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager return f(*a, **k) 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 758, in _create_from_image_cache_or_download 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 670, in _create_from_image_download 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service) 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 551, in _copy_image_to_volume 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager raise exception.ImageCopyFailure(reason=ex) 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: 'remoteName' 2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager ~~~ And ~~~ 2017-11-20 18:45:07.266 560051 ERROR cinder.volume.flows.manager.create_volume [req-7b9019fd-02f6-4f10-822a-24e4a4530d2e 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - default default] Volume 87e53204-8ff1-4bb5-b6a5-6b0852dda2ea: create failed 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server [req-7b9019fd-02f6-4f10-822a-24e4a4530d2e 07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 - default default] Exception during message handling 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4370, in create_volume 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server allow_reschedule=allow_reschedule, volume=volume) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 635, in create_volume 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server _run_flow() 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 627, in _run_flow 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server flow_engine.run() 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server failures[0].reraise() 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server result = task.execute(**arguments) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 873, in execute 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server **volume_spec) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 831, in _create_from_image 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server image_service) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 285, in wrapped 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server return f(*a, **k) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 758, in _create_from_image_cache_or_download 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server image_service 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 670, in _create_from_image_download 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server image_service) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 551, in _copy_image_to_volume 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server raise exception.ImageCopyFailure(reason=ex) 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: 'remoteName' 2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server ~~~ Looks as if 3par is returning this: ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: 'remoteName' We have to figure out why this is happening. Either OSP provides bad data or the 3par's control plane has issues. From an internal update on this case which was made yesterday, it seems that this issue can be reproduced when a volume is created from a glance image: ~~~ [stack@director ~]$ cinder create --image-id ce69e81e-cfc7-47e1-a857-5c7710ba0501 --name test-2 50 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2017-11-20T11:24:16.000000 | | description | None | | encrypted | False | | id | 41892b85-652c-43a9-ac40-4f8ee2fb6043 | | metadata | {} | | migration_status | None | | multiattach | False | | name | test-2 | | os-vol-host-attr:host | hostgroup@3par01#CPG_SSD_R6 | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 1b6c62dd0a6a4f7c8917304292298cb0 | | replication_status | disabled | | size | 50 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | 2017-11-20T11:24:16.000000 | | user_id | 07330a460401434bb02fed5acb0e3d3c | | volume_type | None | +--------------------------------+-------------------------------------- [stack@director ~]$ cinder show 41892b85-652c-43a9-ac40-4f8ee2fb6043 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2017-11-20T11:24:16.000000 | | description | None | | encrypted | False | | id | 41892b85-652c-43a9-ac40-4f8ee2fb6043 | | metadata | {} | | migration_status | None | | multiattach | False | | name | test-2 | | os-vol-host-attr:host | hostgroup@3par01#CPG_SSD_R6 | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 1b6c62dd0a6a4f7c8917304292298cb0 | | replication_status | disabled | | size | 50 | | snapshot_id | None | | source_volid | None | | status | error | | updated_at | 2017-11-20T11:24:21.000000 | | user_id | 07330a460401434bb02fed5acb0e3d3c | | volume_type | None | +--------------------------------+--------------------------------------+ ~~~ /etc/cinder/cinder.conf ~~~ #enabled_backends = enabled_backends = 3par01 (...) [3par01] hpe3par_api_url=https://10.3.160.105:8080/api/v1 hpe3par_username=redhat hpe3par_password= hpe3par_cpg=CPG_SSD_R6 san_ip=10.3.160.105 san_login=redhat san_password=********* volume_driver=cinder.volume.drivers.hpe.hpe_3par_fc.HPE3PARFCDriver hpe3par_debug=false reserved_percentage=15 volume_backend_name=CPG_SSD_R6 ~~~ Installed cinder versions: ~~~ $ grep cinder ./controller0-0.stack.enoc-airtel.com/installed-rpms openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16 00:22:31 2017 puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16 00:29:02 2017 python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16 00:21:02 2017 python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15 23:56:15 2017 $ grep cinder ./controller0-1.stack.enoc-airtel.com/installed-rpms openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16 00:22:31 2017 puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16 00:29:02 2017 python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16 00:21:02 2017 python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15 23:56:15 2017 $ grep cinder ./controller0-1.stack.enoc-airtel.com/installed-rpms openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16 00:22:31 2017 puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16 00:29:02 2017 python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16 00:21:02 2017 python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15 23:56:15 2017 ~~~