Live-migration of an instance with Lefthand volumes triggers Cinder errors

Bug #1311350 reported by Sean Severson
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Anthony Lee

Bug Description

If a Lefthand volume is attached to an instance, and a live migration is attempted on that instance, Cinder will throw errors.

2014-04-22 14:23:32.876 DEBUG cinder.openstack.common.lockutils [req-ec047dcb-f3da-42ab-9ae4-3739155cf445 0ae03dba92094cfaabe9c0d6e5f359d9 4650989f8e1841b28f6123b66660e848] Got file lock "lefthand" at /opt/stack/data/cinder/cinder-lefthand for method "initialize_connection"... from (pid=57883) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:232
2014-04-22 14:23:32.912 DEBUG cinder.openstack.common.lockutils [req-ec047dcb-f3da-42ab-9ae4-3739155cf445 0ae03dba92094cfaabe9c0d6e5f359d9 4650989f8e1841b28f6123b66660e848] Released file lock "lefthand" at /opt/stack/data/cinder/cinder-lefthand for method "initialize_connection"... from (pid=57883) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:239
2014-04-22 14:23:32.913 ERROR cinder.volume.manager [req-ec047dcb-f3da-42ab-9ae4-3739155cf445 0ae03dba92094cfaabe9c0d6e5f359d9 4650989f8e1841b28f6123b66660e848] Unable to fetch connection information from backend: Error (HTTP 500) OPERATION_FAILED - The operation failed: The association for the volume list group cannot be added because the volume list group 'cld5b15VolumeList1398189674' already exists on volume 'volume-cadbc12b-8f77-4496-afa6-94630f223557'.
2014-04-22 14:23:32.913 DEBUG cinder.openstack.common.lockutils [req-ec047dcb-f3da-42ab-9ae4-3739155cf445 0ae03dba92094cfaabe9c0d6e5f359d9 4650989f8e1841b28f6123b66660e848] Got semaphore "lefthand" for method "remove_export"... from (pid=57883) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:191
2014-04-22 14:23:32.914 DEBUG cinder.openstack.common.lockutils [req-ec047dcb-f3da-42ab-9ae4-3739155cf445 0ae03dba92094cfaabe9c0d6e5f359d9 4650989f8e1841b28f6123b66660e848] Attempting to grab file lock "lefthand" for method "remove_export"... from (pid=57883) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:202
2014-04-22 14:23:32.914 DEBUG cinder.openstack.common.lockutils [req-ec047dcb-f3da-42ab-9ae4-3739155cf445 0ae03dba92094cfaabe9c0d6e5f359d9 4650989f8e1841b28f6123b66660e848] Got file lock "lefthand" at /opt/stack/data/cinder/cinder-lefthand for method "remove_export"... from (pid=57883) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:232
2014-04-22 14:23:32.915 DEBUG cinder.openstack.common.lockutils [req-ec047dcb-f3da-42ab-9ae4-3739155cf445 0ae03dba92094cfaabe9c0d6e5f359d9 4650989f8e1841b28f6123b66660e848] Released file lock "lefthand" at /opt/stack/data/cinder/cinder-lefthand for method "remove_export"... from (pid=57883) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:239
2014-04-22 14:23:32.915 ERROR oslo.messaging.rpc.dispatcher [req-ec047dcb-f3da-42ab-9ae4-3739155cf445 0ae03dba92094cfaabe9c0d6e5f359d9 4650989f8e1841b28f6123b66660e848] Exception during message handling: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Error (HTTP 500) OPERATION_FAILED - The operation failed: The association for the volume list group cannot be added because the volume list group 'cld5b15VolumeList1398189674' already exists on volume 'volume-cadbc12b-8f77-4496-afa6-94630f223557'.
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 804, in initialize_connection
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher raise exception.VolumeBackendAPIException(data=err_msg)
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Error (HTTP 500) OPERATION_FAILED - The operation failed: The association for the volume list group cannot be added because the volume list group 'cld5b15VolumeList1398189674' already exists on volume 'volume-cadbc12b-8f77-4496-afa6-94630f223557'.
2014-04-22 14:23:32.915 TRACE oslo.messaging.rpc.dispatcher

Changed in cinder:
assignee: nobody → Ramy Asselin (ramy-asselin)
status: New → Confirmed
importance: Undecided → High
milestone: none → juno-1
tags: added: drivers
Changed in cinder:
milestone: juno-1 → juno-2
Changed in cinder:
assignee: Ramy Asselin (ramy-asselin) → nobody
assignee: nobody → Ramy Asselin (ramy-asselin)
Changed in cinder:
assignee: Ramy Asselin (ramy-asselin) → Anthony Lee (anthony-mic-lee)
Changed in cinder:
milestone: juno-2 → juno-3
Changed in cinder:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/112692

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/112692
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b5791432e5126f47b55e958f1686d3dd474b97f6
Submitter: Jenkins
Branch: master

commit b5791432e5126f47b55e958f1686d3dd474b97f6
Author: Anthony Lee <email address hidden>
Date: Thu Aug 7 13:58:34 2014 -0700

    Fixing LeftHand live migration error

    Fixed a bug in the LeftHand driver that was causing an error to occur during
    live migration of an instance that is attached to a volume. The solution was
    to make sure server access is only requested for a volume if the iSCSI
    session doesn't already have access enabled on the LeftHand backend.

    Change-Id: I41046beb66f2a6138659ec3e4d1d3f37b37859b8
    Closes-Bug: 1311350

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.