Cinder issues with HPE 3par FC and boot from volume or creation of volume from a glance image
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
Medium
|
Raghavendra Tilay |
Bug Description
Cinder issues with HPE 3par FC and boot from volume or creation of volume from a glance image.
New test 3par volume creation is successful
if we create manually but when we try to launch instance, it was
failing.
When looking at the cinder volume logs, we first see:
~~~
2017-11-20 14:32:07.061 560051 DEBUG
cinder.
[req-0968d084-
07330a460401434
default default] <== decorator: exception (1841ms)
KeyError(
/usr/lib/
~~~
Then:
~~~
2017-11-20 18:45:07.200 560051 ERROR cinder.
[req-7b9019fd-
07330a460401434
default default] Unable to fetch connection information from backend:
'remoteName'
2017-11-20 18:45:07.201 560051 ERROR
cinder.
[req-7b9019fd-
07330a460401434
default default] Failed to copy image
ce69e81e-
87e53204-
2017-11-20 18:45:07.201 560051 ERROR
cinder.
last):
2017-11-20 18:45:07.201 560051 ERROR
cinder.
"/usr/lib/
line 536, in _copy_image_
2017-11-20 18:45:07.201 560051 ERROR
cinder.
image_service, image_id)
2017-11-20 18:45:07.201 560051 ERROR
cinder.
"/usr/lib/
in copy_image_
2017-11-20 18:45:07.201 560051 ERROR
cinder.
image_service, image_id, False)
2017-11-20 18:45:07.201 560051 ERROR
cinder.
"/usr/lib/
in _copy_image_
2017-11-20 18:45:07.201 560051 ERROR
cinder.
self._attach_
2017-11-20 18:45:07.201 560051 ERROR
cinder.
"/usr/lib/
in _attach_volume
2017-11-20 18:45:07.201 560051 ERROR
cinder.
exception.
2017-11-20 18:45:07.201 560051 ERROR
cinder.
Bad or unexpected response from the storage volume backend API: Unable
to fetch connection information from backend: 'remoteName'
2017-11-20 18:45:07.201 560051 ERROR cinder.
2017-11-20 18:45:07.210 560051 ERROR cinder.
(most recent call last):
2017-11-20 18:45:07.210 560051 ERROR cinder.
"/usr/lib/
line 53, in _execute_task
2017-11-20 18:45:07.210 560051 ERROR cinder.
= task.execute(
2017-11-20 18:45:07.210 560051 ERROR cinder.
"/usr/lib/
line 873, in execute
2017-11-20 18:45:07.210 560051 ERROR cinder.
2017-11-20 18:45:07.210 560051 ERROR cinder.
"/usr/lib/
line 831, in _create_from_image
2017-11-20 18:45:07.210 560051 ERROR cinder.
2017-11-20 18:45:07.210 560051 ERROR cinder.
"/usr/lib/
in wrapped
2017-11-20 18:45:07.210 560051 ERROR cinder.
2017-11-20 18:45:07.210 560051 ERROR cinder.
"/usr/lib/
line 758, in _create_
2017-11-20 18:45:07.210 560051 ERROR cinder.
2017-11-20 18:45:07.210 560051 ERROR cinder.
"/usr/lib/
line 670, in _create_
2017-11-20 18:45:07.210 560051 ERROR cinder.
2017-11-20 18:45:07.210 560051 ERROR cinder.
"/usr/lib/
line 551, in _copy_image_
2017-11-20 18:45:07.210 560051 ERROR cinder.
exception.
2017-11-20 18:45:07.210 560051 ERROR cinder.
ImageCopyFailure: Failed to copy image to volume: Bad or unexpected
response from the storage volume backend API: Unable to fetch
connection information from backend: 'remoteName'
2017-11-20 18:45:07.210 560051 ERROR cinder.
~~~
And
~~~
2017-11-20 18:45:07.266 560051 ERROR
cinder.
[req-7b9019fd-
07330a460401434
default default] Volume 87e53204-
failed
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
[req-7b9019fd-
07330a460401434
default default] Exception during message handling
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
Traceback (most recent call last):
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
133, in _process_incoming
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
= self.dispatcher
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 150, in dispatch
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
return self._do_
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 121, in _do_dispatch
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
result = func(ctxt, **new_args)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
4370, in create_volume
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
allow_reschedul
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
in create_volume
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
in _run_flow
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
flow_engine.run()
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 247, in run
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
_state in self.run_
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 340, in run_iter
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
failure.
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
336, in reraise_if_any
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
failures[
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
343, in reraise
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
six.reraise(
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 53, in _execute_task
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
result = task.execute(
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 873, in execute
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
**volume_spec)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 831, in _create_from_image
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
image_service)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
in wrapped
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
return f(*a, **k)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 758, in _create_
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 670, in _create_
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
image_service)
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
"/usr/lib/
line 551, in _copy_image_
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
raise exception.
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
ImageCopyFailure: Failed to copy image to volume: Bad or unexpected
response from the storage volume backend API: Unable to fetch
connection information from backend: 'remoteName'
2017-11-20 18:45:07.273 560051 ERROR oslo_messaging.
~~~
Looks as if 3par is returning this: ImageCopyFailure: Failed to copy
image to volume: Bad or unexpected response from the storage volume
backend API: Unable to fetch connection information from backend:
'remoteName'
We have to figure out why this is happening. Either OSP provides bad
data or the 3par's control plane has issues.
From an internal update on this case which was made yesterday, it
seems that this issue can be reproduced when a volume is created from
a glance image:
~~~
[stack@director ~]$ cinder create --image-id
ce69e81e-
+------
| Property | Value |
+------
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-11-
| description | None |
| encrypted | False |
| id | 41892b85-
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | test-2 |
| os-vol-
| os-vol-
| os-vol-
| os-vol-
| replication_status | disabled |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2017-11-
| user_id | 07330a460401434
| volume_type | None |
+------
[stack@director ~]$ cinder show 41892b85-
+------
| Property | Value |
+------
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-11-
| description | None |
| encrypted | False |
| id | 41892b85-
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | test-2 |
| os-vol-
| os-vol-
| os-vol-
| os-vol-
| replication_status | disabled |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | error |
| updated_at | 2017-11-
| user_id | 07330a460401434
| volume_type | None |
+------
~~~
/etc/cinder/
~~~
#enabled_backends = <None>
enabled_backends = 3par01
(...)
[3par01]
hpe3par_api_url=https:/
hpe3par_
hpe3par_
hpe3par_
san_ip=10.3.160.105
san_login=redhat
san_password=
volume_
hpe3par_debug=false
reserved_
volume_
~~~
Installed cinder versions:
~~~
$ grep cinder
./controller0-
openstack-
00:22:31 2017
puppet-
00:29:02 2017
python-
00:21:02 2017
python-
23:56:15 2017
$ grep cinder
./controller0-
openstack-
00:22:31 2017
puppet-
00:29:02 2017
python-
00:21:02 2017
python-
23:56:15 2017
$ grep cinder
./controller0-
openstack-
00:22:31 2017
puppet-
00:29:02 2017
python-
00:21:02 2017
python-
23:56:15 2017
~~~
description: | updated |
tags: | added: 3par drivers |
Changed in cinder: | |
importance: | Undecided → Medium |
Changed in cinder: | |
assignee: | Sean McGinnis (sean-mcginnis) → Raghavendra Tilay (raghavendrat) |
@raghavendra., can you look into this case ? we have 2 cases related to the same issue i guess..