Cinder

Error while creating a nova instance with volume backed image by block device method as hpe 3par backend

Bug #1546392 reported by Bharath kumar on 2016-02-17

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Fix Released	Medium	Kurt Martin	Cinder mitaka-3 "m3"

Bug Description

i was running this command :
nova boot --flavor 1 --block-device source=image,id=c2d917fd-5b16-446f-8258-f2e839e2d834,dest=volume,size=1,shutdown=preserve,bootindex=0 with_cinder

For the first time it created a instance, and a volume and a cache of image [i enabled image volume cache] but for the second time it is giving the below error :

2016-02-17 05:59:23.693 DEBUG hpe3parclient.http [req-71a5fe47-d67f-453f-a171-0c2621d76fce 1b4e0ac99bba48b2883eaa6153f94e9d 5080728719bc4ce68bc03a96b98ecc01] RESP BODY:
from (pid=13332) _http_log_resp /usr/local/lib/python2.7/dist-packages/hpe3parclient/http.py:174
2016-02-17 05:59:23.706 DEBUG oslo_concurrency.lockutils [req-71a5fe47-d67f-453f-a171-0c2621d76fce 1b4e0ac99bba48b2883eaa6153f94e9d 5080728719bc4ce68bc03a96b98ecc01] Lock "88112269-8056-435c-87b6-d9538940f9bd-detach_volume" released by "cinder.volume.manager.ldo_inner2" :: held 0.668s from (pid=13332) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282
2016-02-17 05:59:23.706 ERROR oslo_messaging.rpc.dispatcher [req-71a5fe47-d67f-453f-a171-0c2621d76fce 1b4e0ac99bba48b2883eaa6153f94e9d 5080728719bc4ce68bc03a96b98ecc01] Exception during message handling: Bad or unexpected response from the storage volume backend API: Failure in clear_volume_key_value_pair: Not found (HTTP 404) 180 - Object key does not exist
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 178, in ldo_inner1
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher return ldo_inner2(inst, context, volume_id, attachment_id, **kwargs)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 177, in ldo_inner2
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher return f(*_args, **_kwargs)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1024, in detach_volume
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher {'attach_status': 'error_detaching'})
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher self.force_reraise()
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1019, in detach_volume
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher self.driver.detach_volume(context, volume, attachment)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_iscsi.py", line 876, in detach_volume
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher common.detach_volume(volume, attachment)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_common.py", line 2295, in detach_volume
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher LOG.error(_LE("Error detaching volume %s"), volume)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher self.force_reraise()
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_common.py", line 2292, in detach_volume
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher self.clear_volume_key_value_pair(volume, 'HPQ-CS-instance_uuid')
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_common.py", line 2267, in clear_volume_key_value_pair
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher raise exception.VolumeBackendAPIException(data=msg)
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Failure in clear_volume_key_value_pair: Not found (HTTP 404) 180 - Object key does not exist
2016-02-17 05:59:23.706 TRACE oslo_messaging.rpc.dispatcher
2016-02-17 05:59:23.710 ERROR oslo_messaging._drivers.common [req-71a5fe47-d67f-453f-a171-0c2621d76fce 1b4e0ac99bba48b2883eaa6153f94e9d 5080728719bc4ce68bc03a96b98ecc01] Returning exception Bad or unexpected response from the storage volume backend API: Failure in clear_volume_key_value_pair: Not found (HTTP 404) 180 - Object key does not exist to caller
2016-02-17 05:59:23.710 ERROR oslo_messaging._drivers.common [req-71a5fe47-d67f-453f-a171-0c2621d76fce 1b4e0ac99bba48b2883eaa6153f94e9d 5080728719bc4ce68bc03a96b98ecc01] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch\n executor_callback)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 178, in ldo_inner1\n return ldo_inner2(inst, context, volume_id, attachment_id, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner\n return f(*args, **kwargs)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 177, in ldo_inner2\n return f(*_args, **_kwargs)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 1024, in detach_volume\n {\'attach_status\': \'error_detaching\'})\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 1019, in detach_volume\n self.driver.detach_volume(context, volume, attachment)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_iscsi.py", line 876, in detach_volume\n common.detach_volume(volume, attachment)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_common.py", line 2295, in detach_volume\n LOG.error(_LE("Error detaching volume %s"), volume)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_common.py", line 2292, in detach_volume\n self.clear_volume_key_value_pair(volume, \'HPQ-CS-instance_uuid\')\n', ' File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_common.py", line 2267, in clear_volume_key_value_pair\n raise exception.VolumeBackendAPIException(data=msg)\n', 'VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Failure in clear_volume_key_value_pair: Not found (HTTP 404) 180 - Object key does not exist\n']

Tags:

Bharath kumar (bharath-kumar3) on 2016-02-17

tags:	added: 3par cinder
tags:	added: nova

Bharath kumar (bharath-kumar3) on 2016-02-18

summary:	- Error while creating a nova instance with image by block device + Error while creating a nova instance with image by block device method
summary:	Error while creating a nova instance with image by block device method + as hpe 3par backend

Bharath kumar (bharath-kumar3) on 2016-02-23

summary:

- Error while creating a nova instance with image by block device method
- as hpe 3par backend
+ Error while creating a nova instance with volume backed image by block
+ device method as hpe 3par backend

Revision history for this message

Kurt Martin (kurt-f-martin) wrote on 2016-03-02:

What version of OpenStack is this against? master? Liberty?

Changed in cinder:
assignee:	nobody → Kurt Martin (kurt-f-martin)

Revision history for this message

Bharath kumar (bharath-kumar3) wrote on 2016-03-02:

master branch

Walt Boring (walter-boring) on 2016-03-02

Changed in cinder:
importance:	Undecided → Medium
status:	New → Confirmed
milestone:	none → mitaka-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-02: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/287502

Changed in cinder:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-05: Fix merged to cinder (master)

Reviewed: https://review.openstack.org/287502
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c86b150d5fb4bc5a238da1d84b94408c68e74a04
Submitter: Jenkins
Branch: master

commit c86b150d5fb4bc5a238da1d84b94408c68e74a04
Author: Kurt Martin <email address hidden>
Date: Wed Mar 2 14:57:06 2016 -0800

Don't fail on clearing 3PAR object volume key

    The 3PARs drivers write a key value pair on the 3PAR backend volumes
    to track the instance that the volume is exported to. However, in
    certain cases the key is not present and we would throw an
    exception and not allow the detach to continue. If the key is not
    present then we do not need to clear it. This patch will just
    log a warning that it wasn't present and continue with the detaching
    the volume.

Change-Id: Ie19c9050d514d68ba162bcbfa95dfc6e96d5d7bb
Closes-Bug: #1546392

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

Thierry Carrez (ttx) wrote on 2016-03-17: Fix included in openstack/cinder 8.0.0.0rc1

This issue was fixed in the openstack/cinder 8.0.0.0rc1 release candidate.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.