Cinder

Failed to migrate volume using LIO scsi target

Bug #1433360 reported by Mitsuhiro Tanino
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Mitsuhiro Tanino
Cinder 2015.1.0 "kilo"

Bug Description

When I tried to migrate volume between two back-ends using LIO scsi target, it was failed due to the error of "Unable to terminate volume connection: 'NoneType' object has no attribute 'split'.
Using tgt scsi target, this problem doesn't happen because tgt does not have terminate_connection.

In my understanding, volume migration using LIO target is used to work fine, but recent commit (I'm not sure which one) breaks this unfortunately.

Test summary.
---------------------------------------------------------------------------------------------------------------------------
$ cinder service-list
+------------------+-------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | host1 | nova | enabled | up | 2015-03-17T22:02:04.000000 | None |
| cinder-volume | host2@lvmdriver-2 | nova | enabled | up | 2015-03-17T22:02:10.000000 | None |
| cinder-volume | host1@lvmdriver-1 | nova | enabled | up | 2015-03-17T22:02:03.000000 | None |
+------------------+-------------------+------+---------+-------+----------------------------+-----------------+

$ cinder extra-specs-list
+--------------------------------------+-------------+------------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+-------------+------------------------------------------+
| 1b787af4-31fd-4bd1-8765-a30faaca66f8 | lvmdriver-1 | {u'volume_backend_name': u'lvmdriver-1'} |
+--------------------------------------+-------------+------------------------------------------+

c-vol on host1
---------------------
[lvmdriver-1]
iscsi_helper = lioadm
volume_group = abcde
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name = lvmdriver-1

c-vol on host2
---------------------
[lvmdriver-2]
iscsi_helper = lioadm
volume_group = fghij
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name = lvmdriver-1

migration command
$ cinder migrate vol2 host2@lvmdriver-2#lvmdriver-1
---------------------------------------------------------------------------------------------------------------------------

c-vol error log
----------------------
2015-03-17 16:11:00.210 ERROR cinder.volume.driver [-] Unable to terminate volume connection: 'NoneType' object has no attribute 'split'
2015-03-17 16:11:00.210 ERROR cinder.volume.manager [-] Failed to copy volume db5a6c34-e271-4d1b-a768-07995ca86f82 to 89a74435-e7ac-4d19-85b5-1298512bc1ae
2015-03-17 16:11:00.230 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is 3ad6ddb8d55643fe8c3042e83f33fa29. from (pid=4863) _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:224
2015-03-17 16:11:00.256 INFO cinder.volume.targets.lio [-] Creating iscsi_target for volume: volume-db5a6c34-e271-4d1b-a768-07995ca86f82
2015-03-17 16:11:00.257 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool create /dev/stack-vg1/volume-db5a6c34-e271-4d1b-a768-07995ca86f82 iqn.2010-10.org.openstack:volume-db5a6c34-e271-4d1b-a768-07995ca86f82 Tg88KXQTjHoG3hJc5vJe 4ihLxkC3AZqZnvQ7 False from (pid=4863) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:191
2015-03-17 16:11:00.461 DEBUG oslo_concurrency.processutils [-] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool create /dev/stack-vg1/volume-db5a6c34-e271-4d1b-a768-07995ca86f82 iqn.2010-10.org.openstack:volume-db5a6c34-e271-4d1b-a768-07995ca86f82 Tg88KXQTjHoG3hJc5vJe 4ihLxkC3AZqZnvQ7 False" returned: 0 in 0.204s from (pid=4863) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:216
2015-03-17 16:11:00.461 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool get-targets from (pid=4863) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:191
2015-03-17 16:11:00.691 DEBUG oslo_concurrency.processutils [-] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool get-targets" returned: 0 in 0.230s from (pid=4863) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:216
2015-03-17 16:11:00.692 DEBUG cinder.volume.targets.iscsi [-] Set provider_location to: 192.168.100.50:3260,iqn.2010-10.org.openstack:volume-db5a6c34-e271-4d1b-a768-07995ca86f82 iqn.2010-10.org.openstack:volume-db5a6c34-e271-4d1b-a768-07995ca86f82 0 from (pid=4863) create_export /opt/stack/cinder/cinder/volume/targets/iscsi.py:205
2015-03-17 16:11:00.728 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: 'NoneType' object has no attribute 'split'
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1398, in migrate_volume
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher self.db.volume_update(ctxt, volume_ref['id'], updates)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1383, in migrate_volume
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher new_type_id)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1248, in _migrate_volume_generic
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher {'vol': new_volume['id']})
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1205, in _migrate_volume_generic
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher remote='dest')
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/driver.py", line 540, in copy_volume_data
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher remote=src_remote)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/driver.py", line 354, in _detach_volume
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher raise exception.VolumeBackendAPIException(data=err_msg)
2015-03-17 16:11:00.728 TRACE oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: 'NoneType' object has no attribute 'split'

full c-vol log
http://paste.openstack.org/show/192986/

See original description
Tags: lvm migration
Mitsuhiro Tanino (mitsuhiro-tanino)
description: updated
Mitsuhiro Tanino (mitsuhiro-tanino)
Changed in cinder:
assignee: nobody → Mitsuhiro Tanino (mitsuhiro-tanino)
Revision history for this message
Jay Bryant (jsbryant) wrote :

Adding Vincent Hou as an interested party.

Changed in cinder:
importance: Undecided → High
Revision history for this message
Vincent Hou (houshengbo) wrote :

Thanks for logging this issue and adding me. This is an very interesting issue. I will help to double check as well.

Revision history for this message
Mitsuhiro Tanino (mitsuhiro-tanino) wrote :

Hi Vincent,

Yes, please double check and let me know your test result.
It's helpful.

Thanks,

Revision history for this message
Vincent Hou (houshengbo) wrote :

This is a real existing issue. I have got exactly the same when I test cinder migrate with lioadm.

Changed in cinder:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/165554

Changed in cinder:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/165554
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=bbf1b4950fed90338aa38cd93f41fc86d5a66f99
Submitter: Jenkins
Branch: master

commit bbf1b4950fed90338aa38cd93f41fc86d5a66f99
Author: Mitsuhiro Tanino <email address hidden>
Date: Wed Mar 18 11:48:25 2015 -0400

    Return updated volume object to the caller of _attach_volume()

    Currently, migrate volume between two back-ends using LIO scsi
    target fails due to the error of "Unable to terminate volume
    connection: 'NoneType' object has no attribute 'split'".

    The root cause of this error is _detach_volume() during volume
    migration does not use updated volume object which is updated
    during _attach_volume() by create_export().
    As a result terminate_connection() which is called from
    _detach_volume() tries to split volume['provider_location']
    even if the entry is None and causes above error.

    Also same error would be happened following methods.

    - copy_volume_data(-> volume migration)
    - backup_volume
    - restore_backup
    - copy_image_to_volume
    - copy_volume_to_image

    This change adds updated volume as an additional return value
    for _attach_volume().

    Change-Id: I7b76941d8b2e761c2c0147b13a201583d7e0a1c4
    Closes-Bug: #1433360

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → kilo-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.