2017-11-28 16:18:04 |
Andreas Karis |
description |
Cinder issues with HPE 3par FC and boot from volume or creation of volume from a glance image.
New test 3par volume creation is successful
if we create manually but when we try to launch instance, it was
failing.
When looking at the cinder volume logs, we first see:
~~~
2017-11-20 14:32:07.061 560051 DEBUG
cinder.volume.drivers.hpe.hpe_3par_fc
[req-0968d084-b50c-4832-b92d-5c637ea0688d
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] <== decorator: exception (1841ms)
KeyError('remoteName',) trace_logging_wrapper
/usr/lib/python2.7/site-packages/cinder/utils.py:901
~~~
Then:
~~~
2017-11-20 18:45:07.200 560051 ERROR cinder.volume.driver
[req-7b9019fd-02f6-4f10-822a-24e4a4530d2e
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] Unable to fetch connection information from backend:
'remoteName'
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume
[req-7b9019fd-02f6-4f10-822a-24e4a4530d2e
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] Failed to copy image
ce69e81e-cfc7-47e1-a857-5c7710ba0501 to volume:
87e53204-8ff1-4bb5-b6a5-6b0852dda2ea
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume Traceback (most recent call
last):
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 536, in _copy_image_to_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume context, volume,
image_service, image_id)
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume File
"/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 788,
in copy_image_to_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume context, volume,
image_service, image_id, False)
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume File
"/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 805,
in _copy_image_data_to_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume attach_info, volume =
self._attach_volume(context, volume, properties)
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume File
"/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 975,
in _attach_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume raise
exception.VolumeBackendAPIException(data=err_msg)
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume VolumeBackendAPIException:
Bad or unexpected response from the storage volume backend API: Unable
to fetch connection information from backend: 'remoteName'
2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager Traceback
(most recent call last):
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
line 53, in _execute_task
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager result
= task.execute(**arguments)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 873, in execute
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager **volume_spec)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 831, in _create_from_image
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/coordination.py", line 285,
in wrapped
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager return f(*a, **k)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 758, in _create_from_image_cache_or_download
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 670, in _create_from_image_download
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 551, in _copy_image_to_volume
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager raise
exception.ImageCopyFailure(reason=ex)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager
ImageCopyFailure: Failed to copy image to volume: Bad or unexpected
response from the storage volume backend API: Unable to fetch
connection information from backend: 'remoteName'
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager
~~~
And
~~~
2017-11-20 18:45:07.266 560051 ERROR
cinder.volume.flows.manager.create_volume
[req-7b9019fd-02f6-4f10-822a-24e4a4530d2e
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] Volume 87e53204-8ff1-4bb5-b6a5-6b0852dda2ea: create
failed
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
[req-7b9019fd-02f6-4f10-822a-24e4a4530d2e
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] Exception during message handling
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
Traceback (most recent call last):
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
133, in _process_incoming
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server res
= self.dispatcher.dispatch(message)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 150, in dispatch
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
return self._do_dispatch(endpoint, method, ctxt, args)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 121, in _do_dispatch
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
result = func(ctxt, **new_args)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line
4370, in create_volume
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
allow_reschedule=allow_reschedule, volume=volume)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 635,
in create_volume
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server _run_flow()
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 627,
in _run_flow
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
flow_engine.run()
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py",
line 247, in run
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server for
_state in self.run_iter(timeout=timeout):
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py",
line 340, in run_iter
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
failure.Failure.reraise_if_any(er_failures)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line
336, in reraise_if_any
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
failures[0].reraise()
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line
343, in reraise
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
six.reraise(*self._exc_info)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
line 53, in _execute_task
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
result = task.execute(**arguments)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 873, in execute
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
**volume_spec)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 831, in _create_from_image
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
image_service)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/coordination.py", line 285,
in wrapped
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
return f(*a, **k)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 758, in _create_from_image_cache_or_download
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server image_service
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 670, in _create_from_image_download
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
image_service)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 551, in _copy_image_to_volume
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
raise exception.ImageCopyFailure(reason=ex)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
ImageCopyFailure: Failed to copy image to volume: Bad or unexpected
response from the storage volume backend API: Unable to fetch
connection information from backend: 'remoteName'
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
~~~
Looks as if 3par is returning this: ImageCopyFailure: Failed to copy
image to volume: Bad or unexpected response from the storage volume
backend API: Unable to fetch connection information from backend:
'remoteName'
We have to figure out why this is happening. Either OSP provides bad
data or the 3par's control plane has issues.
From an internal update on this case which was made yesterday, it
seems that this issue can be reproduced when a volume is created from
a glance image:
~~~
[stack@director ~]$ cinder create --image-id
ce69e81e-cfc7-47e1-a857-5c7710ba0501 --name test-2 50
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-11-20T11:24:16.000000 |
| description | None |
| encrypted | False |
| id | 41892b85-652c-43a9-ac40-4f8ee2fb6043 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | test-2 |
| os-vol-host-attr:host | hostgroup@3par01#CPG_SSD_R6 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 1b6c62dd0a6a4f7c8917304292298cb0 |
| replication_status | disabled |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2017-11-20T11:24:16.000000 |
| user_id | 07330a460401434bb02fed5acb0e3d3c |
| volume_type | None |
+--------------------------------+--------------------------------------
[stack@director ~]$ cinder show 41892b85-652c-43a9-ac40-4f8ee2fb6043
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-11-20T11:24:16.000000 |
| description | None |
| encrypted | False |
| id | 41892b85-652c-43a9-ac40-4f8ee2fb6043 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | test-2 |
| os-vol-host-attr:host | hostgroup@3par01#CPG_SSD_R6 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 1b6c62dd0a6a4f7c8917304292298cb0 |
| replication_status | disabled |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | error |
| updated_at | 2017-11-20T11:24:21.000000 |
| user_id | 07330a460401434bb02fed5acb0e3d3c |
| volume_type | None |
+--------------------------------+--------------------------------------+
~~~
/etc/cinder/cinder.conf
~~~
#enabled_backends = <None>
enabled_backends = 3par01
(...)
[3par01]
hpe3par_api_url=https://10.3.160.105:8080/api/v1
hpe3par_username=redhat
hpe3par_password=<password>
hpe3par_cpg=CPG_SSD_R6
san_ip=10.3.160.105
san_login=redhat
san_password=*********
volume_driver=cinder.volume.drivers.hpe.hpe_3par_fc.HPE3PARFCDriver
hpe3par_debug=false
reserved_percentage=15
volume_backend_name=CPG_SSD_R6
~~~
Installed cinder versions:
~~~
$ grep cinder
./controller0-0.stack.enoc-airtel.com/installed-rpms
openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:22:31 2017
puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16
00:29:02 2017
python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:21:02 2017
python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15
23:56:15 2017
$ grep cinder
./controller0-1.stack.enoc-airtel.com/installed-rpms
openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:22:31 2017
puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16
00:29:02 2017
python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:21:02 2017
python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15
23:56:15 2017
$ grep cinder
./controller0-1.stack.enoc-airtel.com/installed-rpms
openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:22:31 2017
puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16
00:29:02 2017
python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:21:02 2017
python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15
23:56:15 2017
~~~ |
Cinder issues with HPE 3par FC and boot from volume or creation of volume from a glance image.
New test 3par volume creation is successful
if we create manually but when we try to launch instance, it was
failing.
When looking at the cinder volume logs, we first see:
~~~
2017-11-20 14:32:07.061 560051 DEBUG
cinder.volume.drivers.hpe.hpe_3par_fc
[req-0968d084-b50c-4832-b92d-5c637ea0688d
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] <== decorator: exception (1841ms)
KeyError('remoteName',) trace_logging_wrapper
/usr/lib/python2.7/site-packages/cinder/utils.py:901
~~~
Then:
~~~
2017-11-20 18:45:07.200 560051 ERROR cinder.volume.driver
[req-7b9019fd-02f6-4f10-822a-24e4a4530d2e
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] Unable to fetch connection information from backend:
'remoteName'
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume
[req-7b9019fd-02f6-4f10-822a-24e4a4530d2e
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] Failed to copy image
ce69e81e-cfc7-47e1-a857-5c7710ba0501 to volume:
87e53204-8ff1-4bb5-b6a5-6b0852dda2ea
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume Traceback (most recent call
last):
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 536, in _copy_image_to_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume context, volume,
image_service, image_id)
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume File
"/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 788,
in copy_image_to_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume context, volume,
image_service, image_id, False)
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume File
"/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 805,
in _copy_image_data_to_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume attach_info, volume =
self._attach_volume(context, volume, properties)
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume File
"/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 975,
in _attach_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume raise
exception.VolumeBackendAPIException(data=err_msg)
2017-11-20 18:45:07.201 560051 ERROR
cinder.volume.flows.manager.create_volume VolumeBackendAPIException:
Bad or unexpected response from the storage volume backend API: Unable
to fetch connection information from backend: 'remoteName'
2017-11-20 18:45:07.201 560051 ERROR cinder.volume.flows.manager.create_volume
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager Traceback
(most recent call last):
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
line 53, in _execute_task
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager result
= task.execute(**arguments)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 873, in execute
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager **volume_spec)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 831, in _create_from_image
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/coordination.py", line 285,
in wrapped
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager return f(*a, **k)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 758, in _create_from_image_cache_or_download
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 670, in _create_from_image_download
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager image_service)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 551, in _copy_image_to_volume
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager raise
exception.ImageCopyFailure(reason=ex)
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager
ImageCopyFailure: Failed to copy image to volume: Bad or unexpected
response from the storage volume backend API: Unable to fetch
connection information from backend: 'remoteName'
2017-11-20 18:45:07.210 560051 ERROR cinder.volume.manager
~~~
And
~~~
2017-11-20 18:45:07.266 560051 ERROR
cinder.volume.flows.manager.create_volume
[req-7b9019fd-02f6-4f10-822a-24e4a4530d2e
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] Volume 87e53204-8ff1-4bb5-b6a5-6b0852dda2ea: create
failed
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
[req-7b9019fd-02f6-4f10-822a-24e4a4530d2e
07330a460401434bb02fed5acb0e3d3c 1b6c62dd0a6a4f7c8917304292298cb0 -
default default] Exception during message handling
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
Traceback (most recent call last):
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
133, in _process_incoming
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server res
= self.dispatcher.dispatch(message)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 150, in dispatch
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
return self._do_dispatch(endpoint, method, ctxt, args)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 121, in _do_dispatch
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
result = func(ctxt, **new_args)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line
4370, in create_volume
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
allow_reschedule=allow_reschedule, volume=volume)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 635,
in create_volume
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server _run_flow()
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 627,
in _run_flow
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
flow_engine.run()
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py",
line 247, in run
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server for
_state in self.run_iter(timeout=timeout):
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py",
line 340, in run_iter
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
failure.Failure.reraise_if_any(er_failures)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line
336, in reraise_if_any
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
failures[0].reraise()
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line
343, in reraise
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
six.reraise(*self._exc_info)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py",
line 53, in _execute_task
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
result = task.execute(**arguments)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 873, in execute
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
**volume_spec)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 831, in _create_from_image
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
image_service)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/coordination.py", line 285,
in wrapped
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
return f(*a, **k)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 758, in _create_from_image_cache_or_download
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server image_service
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 670, in _create_from_image_download
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
image_service)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 551, in _copy_image_to_volume
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
raise exception.ImageCopyFailure(reason=ex)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
ImageCopyFailure: Failed to copy image to volume: Bad or unexpected
response from the storage volume backend API: Unable to fetch
connection information from backend: 'remoteName'
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.rpc.server
~~~
Looks as if 3par is returning this: ImageCopyFailure: Failed to copy
image to volume: Bad or unexpected response from the storage volume
backend API: Unable to fetch connection information from backend:
'remoteName'
We have to figure out why this is happening. Either OSP provides bad
data or the 3par's control plane has issues.
From an internal update on this case which was made yesterday, it
seems that this issue can be reproduced when a volume is created from
a glance image:
~~~
[stack@director ~]$ cinder create --image-id
ce69e81e-cfc7-47e1-a857-5c7710ba0501 --name test-2 50
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-11-20T11:24:16.000000 |
| description | None |
| encrypted | False |
| id | 41892b85-652c-43a9-ac40-4f8ee2fb6043 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | test-2 |
| os-vol-host-attr:host | hostgroup@3par01#CPG_SSD_R6 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 1b6c62dd0a6a4f7c8917304292298cb0 |
| replication_status | disabled |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2017-11-20T11:24:16.000000 |
| user_id | 07330a460401434bb02fed5acb0e3d3c |
| volume_type | None |
+--------------------------------+--------------------------------------
[stack@director ~]$ cinder show 41892b85-652c-43a9-ac40-4f8ee2fb6043
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-11-20T11:24:16.000000 |
| description | None |
| encrypted | False |
| id | 41892b85-652c-43a9-ac40-4f8ee2fb6043 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | test-2 |
| os-vol-host-attr:host | hostgroup@3par01#CPG_SSD_R6 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 1b6c62dd0a6a4f7c8917304292298cb0 |
| replication_status | disabled |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | error |
| updated_at | 2017-11-20T11:24:21.000000 |
| user_id | 07330a460401434bb02fed5acb0e3d3c |
| volume_type | None |
+--------------------------------+--------------------------------------+
~~~
/etc/cinder/cinder.conf
~~~
#enabled_backends = <None>
enabled_backends = 3par01
(...)
[3par01]
hpe3par_api_url=https://10.3.160.105:8080/api/v1
hpe3par_username=redhat
hpe3par_password=<password>
hpe3par_cpg=CPG_SSD_R6
san_ip=10.3.160.105
san_login=redhat
san_password=*********
volume_driver=cinder.volume.drivers.hpe.hpe_3par_fc.HPE3PARFCDriver
hpe3par_debug=false
reserved_percentage=15
volume_backend_name=CPG_SSD_R6
~~~
Installed cinder versions:
~~~
$ grep cinder
./controller0-0/installed-rpms
openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:22:31 2017
puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16
00:29:02 2017
python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:21:02 2017
python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15
23:56:15 2017
$ grep cinder
./controller0-1/installed-rpms
openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:22:31 2017
puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16
00:29:02 2017
python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:21:02 2017
python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15
23:56:15 2017
$ grep cinder
./controller0-1/installed-rpms
openstack-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:22:31 2017
puppet-cinder-9.5.0-1.el7ost.noarch Fri Jun 16
00:29:02 2017
python-cinder-9.1.4-3.el7ost.noarch Fri Jun 16
00:21:02 2017
python-cinderclient-1.9.0-6.el7ost.noarch Thu Jun 15
23:56:15 2017
~~~ |
|