Cinder

Premature volume connection termination with LVM/iSCSI multiattached volumes when using different mountpoints

Bug #1825957 reported by Lee Yarwood
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Lee Yarwood

Bug Description

This was uncovered downstream and is a regression introduced by the fix to bug #1786327 where we started to compare the entire provided connector dict to that stored against each attachment. This fails when the mountpoints stored within each connector dict differ:

https://bugzilla.redhat.com/show_bug.cgi?id=1701172

Easily reproducible with the following steps:

- Create two instances on the same compute.
- Attach a multi-attach volume to both using different mountpoints within the instance.
- Detach the volume from one of the instances.
- Confirm that the underlying volume connection has been removed from the host.

OpenStack Infra (hudson-openstack)
Changed in cinder:
assignee: nobody → Lee Yarwood (lyarwood)
status: New → In Progress
Eric Harney (eharney)
Changed in cinder:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/653837
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c1388ce1c05f3e3e53210d52bc2efafbe191135d
Submitter: Zuul
Branch: master

commit c1388ce1c05f3e3e53210d52bc2efafbe191135d
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 18 20:17:54 2019 +0100

    lvm: Only use initiators when comparing connector dicts

    Ib5aa1b7578f7d3200185566ff5f8634dd519d020 previously fixed the premature
    termination of multi attached iSCSI/LVM volumes by comparing the
    provided connector dict against that stored with any remaining active
    attachments. Only allowing the termination to proceed if a single active
    attachment remained using the provided connector.

    However this check fails if two instances on the same host are connected
    to the same multi attach volume using different mountpoints as these are
    now stored within the connector. This behaviour was introduced during
    Queens by If3afe8d8bd6b8c327ccc7d1140053bccaf7e1ad7 and
    I11ba269c3f7a2e7707b2b7e27d26eb7a2c948a82 to workaround differences
    between the cinderv2 and cinderv3 APIs.

    This change corrects this by only comparing the initiator key of the
    connectors ensuring terminate connection is only called when a single
    attachment remains on a given host using a specific initiator.

    Closes-Bug: #1825957
    Change-Id: Icabc4a67d3f5462fe24e4027e84e56a001e1b2b8

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/660458

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/660586

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/660587

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/stein)

Reviewed: https://review.opendev.org/660458
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=4910ca4d7337298553f3d0b58ceccf925ae39f1b
Submitter: Zuul
Branch: stable/stein

commit 4910ca4d7337298553f3d0b58ceccf925ae39f1b
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 18 20:17:54 2019 +0100

    lvm: Only use initiators when comparing connector dicts

    Ib5aa1b7578f7d3200185566ff5f8634dd519d020 previously fixed the premature
    termination of multi attached iSCSI/LVM volumes by comparing the
    provided connector dict against that stored with any remaining active
    attachments. Only allowing the termination to proceed if a single active
    attachment remained using the provided connector.

    However this check fails if two instances on the same host are connected
    to the same multi attach volume using different mountpoints as these are
    now stored within the connector. This behaviour was introduced during
    Queens by If3afe8d8bd6b8c327ccc7d1140053bccaf7e1ad7 and
    I11ba269c3f7a2e7707b2b7e27d26eb7a2c948a82 to workaround differences
    between the cinderv2 and cinderv3 APIs.

    This change corrects this by only comparing the initiator key of the
    connectors ensuring terminate connection is only called when a single
    attachment remains on a given host using a specific initiator.

    Closes-Bug: #1825957
    Change-Id: Icabc4a67d3f5462fe24e4027e84e56a001e1b2b8
    (cherry picked from commit c1388ce1c05f3e3e53210d52bc2efafbe191135d)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/rocky)

Reviewed: https://review.opendev.org/660586
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d5291871bb0d21dc74e1003480b3cde8fdbef1a3
Submitter: Zuul
Branch: stable/rocky

commit d5291871bb0d21dc74e1003480b3cde8fdbef1a3
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 18 20:17:54 2019 +0100

    lvm: Only use initiators when comparing connector dicts

    Ib5aa1b7578f7d3200185566ff5f8634dd519d020 previously fixed the premature
    termination of multi attached iSCSI/LVM volumes by comparing the
    provided connector dict against that stored with any remaining active
    attachments. Only allowing the termination to proceed if a single active
    attachment remained using the provided connector.

    However this check fails if two instances on the same host are connected
    to the same multi attach volume using different mountpoints as these are
    now stored within the connector. This behaviour was introduced during
    Queens by If3afe8d8bd6b8c327ccc7d1140053bccaf7e1ad7 and
    I11ba269c3f7a2e7707b2b7e27d26eb7a2c948a82 to workaround differences
    between the cinderv2 and cinderv3 APIs.

    This change corrects this by only comparing the initiator key of the
    connectors ensuring terminate connection is only called when a single
    attachment remains on a given host using a specific initiator.

    NOTE(lyarwood): Conflict as I253123d5451b32f0e3143916e41aaa1af75561c2
    is not present in stable/rocky and as such fake_volume_attachment_obj
    is now used to create the required test attachment objects.

    Conflicts:
        cinder/tests/unit/volume/drivers/test_lvm_driver.py

    Closes-Bug: #1825957
    Change-Id: Icabc4a67d3f5462fe24e4027e84e56a001e1b2b8
    (cherry picked from commit c1388ce1c05f3e3e53210d52bc2efafbe191135d)
    (cherry picked from commit 4910ca4d7337298553f3d0b58ceccf925ae39f1b)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/queens)

Reviewed: https://review.opendev.org/660587
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=e8d68422ed082718bf9de3c58fb4b1cbfabd9ccd
Submitter: Zuul
Branch: stable/queens

commit e8d68422ed082718bf9de3c58fb4b1cbfabd9ccd
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 18 20:17:54 2019 +0100

    lvm: Only use initiators when comparing connector dicts

    Ib5aa1b7578f7d3200185566ff5f8634dd519d020 previously fixed the premature
    termination of multi attached iSCSI/LVM volumes by comparing the
    provided connector dict against that stored with any remaining active
    attachments. Only allowing the termination to proceed if a single active
    attachment remained using the provided connector.

    However this check fails if two instances on the same host are connected
    to the same multi attach volume using different mountpoints as these are
    now stored within the connector. This behaviour was introduced during
    Queens by If3afe8d8bd6b8c327ccc7d1140053bccaf7e1ad7 and
    I11ba269c3f7a2e7707b2b7e27d26eb7a2c948a82 to workaround differences
    between the cinderv2 and cinderv3 APIs.

    This change corrects this by only comparing the initiator key of the
    connectors ensuring terminate connection is only called when a single
    attachment remains on a given host using a specific initiator.

    NOTE(lyarwood): Conflict as I253123d5451b32f0e3143916e41aaa1af75561c2
    is not present in stable/rocky and as such fake_volume_attachment_obj
    is now used to create the required test attachment objects.

    Conflicts:
        cinder/tests/unit/volume/drivers/test_lvm_driver.py

    Closes-Bug: #1825957
    Change-Id: Icabc4a67d3f5462fe24e4027e84e56a001e1b2b8
    (cherry picked from commit c1388ce1c05f3e3e53210d52bc2efafbe191135d)
    (cherry picked from commit 4910ca4d7337298553f3d0b58ceccf925ae39f1b)
    (cherry picked from commit d5291871bb0d21dc74e1003480b3cde8fdbef1a3)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 14.0.1

This issue was fixed in the openstack/cinder 14.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 13.0.6

This issue was fixed in the openstack/cinder 13.0.6 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 12.0.8

This issue was fixed in the openstack/cinder 12.0.8 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 15.0.0.0rc1

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.