race in backup manager remove_export call

Bug #1920237 reported by Eric Harney
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Unassigned

Bug Description

In _detach_device in the backup manager, rpcapi.remove_export() is called to detach a volume. This then quickly allows subsequent operations to occur, but since remove_export() is called with a "cast" and may take a moment to complete, it can interfere with operations happening shortly after it.

This manifests in the lio-barbican CI job, where a call to "cinder-rtstool add-initiator" is made and the target is not found. It presumably affects other volume drivers in similar ways, maybe with varying frequency.

Revision history for this message
Eric Harney (eharney) wrote :
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 18.0.0.0rc1

This issue was fixed in the openstack/cinder 18.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/785970
Committed: https://opendev.org/openstack/cinder/commit/34120d49808537d0523375b173db9c657114a4cb
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 34120d49808537d0523375b173db9c657114a4cb
Author: Eric Harney <email address hidden>
Date: Fri Mar 19 16:01:33 2021 +0000

    Backup manager: Synchronously call remove_export

    Call the volume's rpcapi remove_export method with
    a "call" instead of a "cast" when detaching volumes
    in the backup manager.

    This prevents _detach_volume() from returning before
    the remove_export() has completed, which helps prevent
    problems where remove_export() races against a
    subsequent operation.

    Closes-Bug: #1920237

    Change-Id: I482c3d5520a7bbb9971942cdb4c4208052106d70
    (cherry picked from commit d6895f9d2aab84c116efc2f853bb5f9e57fac539)
    (cherry picked from commit 5402ac8e2d30872b526c3ac7d9f10645452fbf66)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/cinder/+/789859

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/train)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/789859
Committed: https://opendev.org/openstack/cinder/commit/3bb28c637e98078a4da2b6930e56bfce0cce9f73
Submitter: "Zuul (22348)"
Branch: stable/train

commit 3bb28c637e98078a4da2b6930e56bfce0cce9f73
Author: Eric Harney <email address hidden>
Date: Fri Mar 19 16:01:33 2021 +0000

    Backup manager: Synchronously call remove_export

    Call the volume's rpcapi remove_export method with
    a "call" instead of a "cast" when detaching volumes
    in the backup manager.

    This prevents _detach_volume() from returning before
    the remove_export() has completed, which helps prevent
    problems where remove_export() races against a
    subsequent operation.

    Closes-Bug: #1920237

    Change-Id: I482c3d5520a7bbb9971942cdb4c4208052106d70
    (cherry picked from commit d6895f9d2aab84c116efc2f853bb5f9e57fac539)
    (cherry picked from commit 5402ac8e2d30872b526c3ac7d9f10645452fbf66)
    (cherry picked from commit 34120d49808537d0523375b173db9c657114a4cb)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 15.6.0

This issue was fixed in the openstack/cinder 15.6.0 release.

Gorka Eguileor (gorka)
Changed in cinder:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 16.4.0

This issue was fixed in the openstack/cinder 16.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 17.2.0

This issue was fixed in the openstack/cinder 17.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.