unable to re-attach a volume to instance in VMWare deployment

Bug #1379830 reported by satyadev svn
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Critical
Vipin Balachandran

Bug Description

Attach a volume to instance fails with following exception :

014-10-10 19:29:54.112 DEBUG cinder.volume.drivers.vmware.vmdk [req-90f52202-4720-41ca-aa01-cc1bd2e1a4be 304c157ff8fe4136a279d5f157e63cfc eb53ada20d274ed8832cf46afb21636b] The instance: (Property){
   value = "vm-2360"
   _type = "VirtualMachine"
 } for which initialize connection is called, exists. from (pid=10484) _initialize_connection /opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py:656
2014-10-10 19:29:55.114 DEBUG cinder.volume.drivers.vmware.vmdk [req-90f52202-4720-41ca-aa01-cc1bd2e1a4be 304c157ff8fe4136a279d5f157e63cfc eb53ada20d274ed8832cf46afb21636b] Backing exists from (pid=10484) _initialize_connection /opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py:666
2014-10-10 19:29:57.068 DEBUG cinder.volume.drivers.vmware.vmdk [req-90f52202-4720-41ca-aa01-cc1bd2e1a4be 304c157ff8fe4136a279d5f157e63cfc eb53ada20d274ed8832cf46afb21636b] Datastore: (ManagedObjectReference){
   value = "datastore-34"
   _type = "Datastore"
 }, profile: None from (pid=10484) _relocate_backing /opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py:1934
2014-10-10 19:29:59.015 ERROR cinder.volume.manager [req-90f52202-4720-41ca-aa01-cc1bd2e1a4be 304c157ff8fe4136a279d5f157e63cfc eb53ada20d274ed8832cf46afb21636b] Unable to fetch connection information from backend: 'Text' object has no attribute 'value'
2014-10-10 19:29:59.017 ERROR oslo.messaging.rpc.dispatcher [req-90f52202-4720-41ca-aa01-cc1bd2e1a4be 304c157ff8fe4136a279d5f157e63cfc eb53ada20d274ed8832cf46afb21636b] Exception during message handling: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: 'Text' object has no attribute 'value'
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  incoming.message))
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  return self._do_dispatch(endpoint, method, ctxt, args)
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  result = getattr(endpoint, method)(ctxt, **new_args)
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  return f(*args, **kwargs)
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  File "/opt/stack/cinder/cinder/volume/manager.py", line 901, in initialize_connection
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher  raise exception.VolumeBackendAPIException(data=err_msg)
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: 'Text' object has no attribute 'value'
2014-10-10 19:29:59.017 TRACE oslo.messaging.rpc.dispatcher 
2014-10-10 19:29:59.021 ERROR oslo.messaging._drivers.common [req-90f52202-4720-41ca-aa01-cc1bd2e1a4be 304c157ff8fe4136a279d5f157e63cfc eb53ada20d274ed8832cf46afb21636b] Returning exception Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: 'Text' object has no attribute 'value' to caller
2014-10-10 19:29:59.022 ERROR oslo.messaging._drivers.common [req-90f52202-4720-41ca-aa01-cc1bd2e1a4be 304c157ff8fe4136a279d5f157e63cfc eb53ada20d274ed8832cf46afb21636b] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 901, in initialize_connection\n raise exception.VolumeBackendAPIException(data=err_msg)\n', "VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: 'Text' object has no attribute 'value'\n"]
2014-10-10 19:30:02.547 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._publish_service_capabilities from (pid=10484) run_periodic_tasks /opt/stack/cinder/cinder/openstack/common/periodic_task.py:193
2014-10-10 19:30:02.547 DEBUG cinder.manager [-] Notifying Schedulers of capabilities ... from (pid=10484) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:128
2014-10-10 19:30:02.552 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._report_driver_status from (pid=10484) run_periodic_tasks /opt/stack/cinder/cinder/openstack/common/periodic_task.py:193
2014-10-10 19:30:02.553 INFO cinder.volume.manager [-] Updating volume status

Revision history for this message
satyadev svn (svnsatya) wrote :
Revision history for this message
satyadev svn (svnsatya) wrote :

Issue is happens only on reattachment same volume.

steps :
1) create nova instance
2) create cinder volume
3) attach volume to instance
4) detach volume from instance
5) re-attach volume to instance

exception observed

satyadev svn (svnsatya)
summary: - unable to attach a volume to instance
+ unable to re-attach a volume to instance
Revision history for this message
satyadev svn (svnsatya) wrote : Re: unable to re-attach a volume to instance

other test scenario :

1) create volume from image. [ cinder create 1 --image-id 452fd43a-1503-41cd-bd67-7331e3b05679 --name imgvol1]
2) create instance [ nova boot --image 452fd43a-1503-41cd-bd67-7331e3b05679 --flavor 1 vm1 ]
3) now attach volume to instance

observe same exception.

summary: - unable to re-attach a volume to instance
+ unable to re-attach a volume to instance in VMWare deployment
Revision history for this message
Vipin Balachandran (vbala) wrote :

This is a regression caused by commit 4be8913520f5e9fe4109ade101da9509e4a83360. Currently testing the commit to revert this change.

Changed in cinder:
assignee: nobody → Vipin Balachandran (vbala)
status: New → Confirmed
importance: Undecided → Critical
Revision history for this message
satyadev svn (svnsatya) wrote :

Verified patch: https://review.openstack.org/#/c/127604/

After applying above patch, above mentioned exception not observed during testing following test scenarios.
- Re attachment of volume with backing volume in vcenter.
- Re attachment of volume without backing volume in vcenter.
- Migration of volume across data stores.
- Volumes present to different types of data stores and reattached to instance

Changed in cinder:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/127604
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=48cb82971e0418f9a629e2b39d0433dc2c0e6919
Submitter: Jenkins
Branch: master

commit 48cb82971e0418f9a629e2b39d0433dc2c0e6919
Author: Vipin Balachandran <email address hidden>
Date: Fri Oct 10 23:06:27 2014 +0530

    Revert "Relocate volume to compliant datastore"

    Commit 4be8913520f5e9fe4109ade101da9509e4a83360 introduced a regression
    which causes failures during cinder volume re-attach. This patch reverts
    commit 4be8913520f5e9fe4109ade101da9509e4a83360 as an immediate fix.

    Closes-Bug: #1379830
    Change-Id: I5dfbd45533489c3c81db8d256bbfd2f85614a357

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
tags: added: juno-rc-potential
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → juno-rc3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (proposed/juno)

Fix proposed to branch: proposed/juno
Review: https://review.openstack.org/128431

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (proposed/juno)

Reviewed: https://review.openstack.org/128431
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=01e7c516852e53df661b2eedc970c327c1ff10ce
Submitter: Jenkins
Branch: proposed/juno

commit 01e7c516852e53df661b2eedc970c327c1ff10ce
Author: Vipin Balachandran <email address hidden>
Date: Fri Oct 10 23:06:27 2014 +0530

    Revert "Relocate volume to compliant datastore"

    Commit 4be8913520f5e9fe4109ade101da9509e4a83360 introduced a regression
    which causes failures during cinder volume re-attach. This patch reverts
    commit 4be8913520f5e9fe4109ade101da9509e4a83360 as an immediate fix.

    Closes-Bug: #1379830
    Change-Id: I5dfbd45533489c3c81db8d256bbfd2f85614a357
    (cherry picked from commit 48cb82971e0418f9a629e2b39d0433dc2c0e6919)

Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: juno-rc3 → 2014.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/128920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by Mike Perez (<email address hidden>) on branch: master
Review: https://review.openstack.org/128920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)
Download full text (11.8 KiB)

Reviewed: https://review.openstack.org/128920
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=66494f54112fdfa135b3974c75aa388c8d1fb49e
Submitter: Jenkins
Branch: master

commit be3d4604dc0566e0838959d998ff1d37755de6d3
Author: Tomoki Sekiyama <email address hidden>
Date: Tue Oct 14 19:09:44 2014 -0400

    Fix LVM iSCSI driver tgtadm CHAP authentication

    Currently CHAP Authentication in LVM iSCSI driver with tgtadm does not work.
    This is because the tgtadm helper creates the target configuration file
    with an 'IncomingUser' entry, which is ignored by tgtd.
    This patch fixes it to 'incominguser'.

    Change-Id: I14871985a2a916834122f849238f05b75726bc1a
    Closes-Bug: #1329214
    (cherry picked from commit e3563891545c801726d227f752cf99488ed5c7dd)

commit f7ee62cc58d8b642af67510a310f6259492a4508
Author: Mitsuhiro Tanino <email address hidden>
Date: Tue Oct 14 12:41:41 2014 -0400

    Export cinder volumes only if the status is 'in-use'

    Currently, cinder volumes are exported both 'in-use' and 'available'
    after restarting cinder-volume service.
    This behavior was introduced following commit.

      commit ffefe18334a9456250e1b6ff88b7b47fb366f374
      Author: Zhiteng Huang <email address hidden>
      Date: Sat Aug 23 18:32:57 2014 +0000

    If the volumes are attached to nova instances, they should be exported
    via tgtd after restarting cinder-volume.
    But the volumes which are not attached to instances must not be exported
    because everyone can connect these volumes.

    This patch changes volume export behavior that exports a volume only if
    the volume status is 'in-use'.

    Change-Id: I4c598c240b9290c81bd8001e5a0720c8c329aeb9
    Signed-off-by: Mitsuhiro Tanino <email address hidden>
    Closes-bug: #1381106
    (cherry picked from commit e2f28b967910625432be0eab6a851adf53ac58ea)

commit 01e7c516852e53df661b2eedc970c327c1ff10ce
Author: Vipin Balachandran <email address hidden>
Date: Fri Oct 10 23:06:27 2014 +0530

    Revert "Relocate volume to compliant datastore"

    Commit 4be8913520f5e9fe4109ade101da9509e4a83360 introduced a regression
    which causes failures during cinder volume re-attach. This patch reverts
    commit 4be8913520f5e9fe4109ade101da9509e4a83360 as an immediate fix.

    Closes-Bug: #1379830
    Change-Id: I5dfbd45533489c3c81db8d256bbfd2f85614a357
    (cherry picked from commit 48cb82971e0418f9a629e2b39d0433dc2c0e6919)

commit 900d49723f65e87658381ff955559f54ac98c487
Author: Andreas Jaeger <email address hidden>
Date: Thu Oct 9 12:25:28 2014 +0200

    Updated translations

    Commands run:-
    $ python setup.py extract_messages
    $ python setup.py update_catalog --no-fuzzy-matching \
      --ignore-obsolete=true
    $ source \
      ../openstack-infra/project-config/jenkins/scripts/common_translation_update.sh
    $ setup_loglevel_vars
    $ cleanup_po_files cinder

    Change-Id: I73f3bdccb4be98df95fa853864e465f4d83a8884

commit 8e94aaa2b28b491314fe8642061ac73e3fe8e966
Author: Navneet Singh <email address hidden>
Date: Thu Aug 28 16:03:41 2014 +0530

    NetApp fix eseries unit test mock clean

 ...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.