Unexpected cinder volumes export occurs after restarting cinder-volume

Bug #1381106 reported by Mitsuhiro Tanino
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Critical
Mitsuhiro Tanino

Bug Description

When I restarted cinder-volume service, I noticed that all cinder volumes which use LVMISCSIDriver are exported via tgtd unexpectedly.
If the volumes are attached to nova instances, they should be exported via tgtd after restarting cinder-volume.
But the volumes which are not attached to instances must not be exported;
(a) Everyone can connect these volumes because the volumes are exported via tgtd
    without user and password authentication.
(b) Exported volume(LVMiSCSI) via tgtd can't delete from 'cinder delete'
    because the LV is locked by tgtd, therefore lvremove fails to
    delete target LV. This makes volume status as a deleting error.

Here is my cinder.conf and some logs.
  http://paste.openstack.org/show/121028/

I think this is a patch of the root cause for this problem.
Originally, ensure_export() was only called if the volume status was 'in-use'.
This means that if a volume was attached to an instance, the volume was exported after restarting cinder-volume service.

However, after applying this patch, ensure_export() is called if the volume is 'available' status.
Despite the volume is not attached to specific instance but the volume is exported to outside.
This is wrong behavior.

In my understanding. ensure_export() must be called only the volume status is 'in-use'.

Commit
=========================================
commit ffefe18334a9456250e1b6ff88b7b47fb366f374
Author: Zhiteng Huang <email address hidden>
Date: Sat Aug 23 18:32:57 2014 +0000

-------
             for volume in volumes:
- if volume['status'] in ['in-use']:
+ # available volume should also be counted into allocated
+ if volume['status'] in ['in-use', 'available']:
                     # calculate allocated capacity for driver
- sum += volume['size']
- self.stats['allocated_capacity_gb'] = sum
+ self._count_allocated_capacity(ctxt, volume)
+
                     try:
                         self.driver.ensure_export(ctxt, volume)
                     except Exception as export_ex:
-------

Changed in cinder:
assignee: nobody → Mitsuhiro Tanino (mitsuhiro-tanino)
summary: - Unexpected cinder volumes export occur after restarting cinder-volume
+ Unexpected cinder volumes export occurs after restarting cinder-volume
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/128437

Changed in cinder:
status: New → In Progress
Revision history for this message
Eric Harney (eharney) wrote :

Possible security implications due to tgtd ACL misconfiguration...

Changed in cinder:
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/128437
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=e2f28b967910625432be0eab6a851adf53ac58ea
Submitter: Jenkins
Branch: master

commit e2f28b967910625432be0eab6a851adf53ac58ea
Author: Mitsuhiro Tanino <email address hidden>
Date: Tue Oct 14 12:41:41 2014 -0400

    Export cinder volumes only if the status is 'in-use'

    Currently, cinder volumes are exported both 'in-use' and 'available'
    after restarting cinder-volume service.
    This behavior was introduced following commit.

      commit ffefe18334a9456250e1b6ff88b7b47fb366f374
      Author: Zhiteng Huang <email address hidden>
      Date: Sat Aug 23 18:32:57 2014 +0000

    If the volumes are attached to nova instances, they should be exported
    via tgtd after restarting cinder-volume.
    But the volumes which are not attached to instances must not be exported
    because everyone can connect these volumes.

    This patch changes volume export behavior that exports a volume only if
    the volume status is 'in-use'.

    Change-Id: I4c598c240b9290c81bd8001e5a0720c8c329aeb9
    Signed-off-by: Mitsuhiro Tanino <email address hidden>
    Closes-bug: #1381106

Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (proposed/juno)

Fix proposed to branch: proposed/juno
Review: https://review.openstack.org/128483

Changed in cinder:
milestone: none → juno-rc3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (proposed/juno)

Reviewed: https://review.openstack.org/128483
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f7ee62cc58d8b642af67510a310f6259492a4508
Submitter: Jenkins
Branch: proposed/juno

commit f7ee62cc58d8b642af67510a310f6259492a4508
Author: Mitsuhiro Tanino <email address hidden>
Date: Tue Oct 14 12:41:41 2014 -0400

    Export cinder volumes only if the status is 'in-use'

    Currently, cinder volumes are exported both 'in-use' and 'available'
    after restarting cinder-volume service.
    This behavior was introduced following commit.

      commit ffefe18334a9456250e1b6ff88b7b47fb366f374
      Author: Zhiteng Huang <email address hidden>
      Date: Sat Aug 23 18:32:57 2014 +0000

    If the volumes are attached to nova instances, they should be exported
    via tgtd after restarting cinder-volume.
    But the volumes which are not attached to instances must not be exported
    because everyone can connect these volumes.

    This patch changes volume export behavior that exports a volume only if
    the volume status is 'in-use'.

    Change-Id: I4c598c240b9290c81bd8001e5a0720c8c329aeb9
    Signed-off-by: Mitsuhiro Tanino <email address hidden>
    Closes-bug: #1381106
    (cherry picked from commit e2f28b967910625432be0eab6a851adf53ac58ea)

Changed in cinder:
status: Fix Committed → Fix Released
description: updated
description: updated
description: updated
Thierry Carrez (ttx)
Changed in cinder:
milestone: juno-rc3 → 2014.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/128920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by Mike Perez (<email address hidden>) on branch: master
Review: https://review.openstack.org/128920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)
Download full text (11.8 KiB)

Reviewed: https://review.openstack.org/128920
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=66494f54112fdfa135b3974c75aa388c8d1fb49e
Submitter: Jenkins
Branch: master

commit be3d4604dc0566e0838959d998ff1d37755de6d3
Author: Tomoki Sekiyama <email address hidden>
Date: Tue Oct 14 19:09:44 2014 -0400

    Fix LVM iSCSI driver tgtadm CHAP authentication

    Currently CHAP Authentication in LVM iSCSI driver with tgtadm does not work.
    This is because the tgtadm helper creates the target configuration file
    with an 'IncomingUser' entry, which is ignored by tgtd.
    This patch fixes it to 'incominguser'.

    Change-Id: I14871985a2a916834122f849238f05b75726bc1a
    Closes-Bug: #1329214
    (cherry picked from commit e3563891545c801726d227f752cf99488ed5c7dd)

commit f7ee62cc58d8b642af67510a310f6259492a4508
Author: Mitsuhiro Tanino <email address hidden>
Date: Tue Oct 14 12:41:41 2014 -0400

    Export cinder volumes only if the status is 'in-use'

    Currently, cinder volumes are exported both 'in-use' and 'available'
    after restarting cinder-volume service.
    This behavior was introduced following commit.

      commit ffefe18334a9456250e1b6ff88b7b47fb366f374
      Author: Zhiteng Huang <email address hidden>
      Date: Sat Aug 23 18:32:57 2014 +0000

    If the volumes are attached to nova instances, they should be exported
    via tgtd after restarting cinder-volume.
    But the volumes which are not attached to instances must not be exported
    because everyone can connect these volumes.

    This patch changes volume export behavior that exports a volume only if
    the volume status is 'in-use'.

    Change-Id: I4c598c240b9290c81bd8001e5a0720c8c329aeb9
    Signed-off-by: Mitsuhiro Tanino <email address hidden>
    Closes-bug: #1381106
    (cherry picked from commit e2f28b967910625432be0eab6a851adf53ac58ea)

commit 01e7c516852e53df661b2eedc970c327c1ff10ce
Author: Vipin Balachandran <email address hidden>
Date: Fri Oct 10 23:06:27 2014 +0530

    Revert "Relocate volume to compliant datastore"

    Commit 4be8913520f5e9fe4109ade101da9509e4a83360 introduced a regression
    which causes failures during cinder volume re-attach. This patch reverts
    commit 4be8913520f5e9fe4109ade101da9509e4a83360 as an immediate fix.

    Closes-Bug: #1379830
    Change-Id: I5dfbd45533489c3c81db8d256bbfd2f85614a357
    (cherry picked from commit 48cb82971e0418f9a629e2b39d0433dc2c0e6919)

commit 900d49723f65e87658381ff955559f54ac98c487
Author: Andreas Jaeger <email address hidden>
Date: Thu Oct 9 12:25:28 2014 +0200

    Updated translations

    Commands run:-
    $ python setup.py extract_messages
    $ python setup.py update_catalog --no-fuzzy-matching \
      --ignore-obsolete=true
    $ source \
      ../openstack-infra/project-config/jenkins/scripts/common_translation_update.sh
    $ setup_loglevel_vars
    $ cleanup_po_files cinder

    Change-Id: I73f3bdccb4be98df95fa853864e465f4d83a8884

commit 8e94aaa2b28b491314fe8642061ac73e3fe8e966
Author: Navneet Singh <email address hidden>
Date: Thu Aug 28 16:03:41 2014 +0530

    NetApp fix eseries unit test mock clean

 ...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.