os-brick

NFS mount command error detection is faulty

Bug #1797233 reported by Eric Harney
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
os-brick
Fix Released
Undecided
Unassigned

Bug Description

NFS mount calls are considered successful in os-brick if "already mounted" appears in the stderr output of the mount command.

However, "busy or already mounted" can appear when the mount was not actually successful.

This causes Cinder to write data to a local disk instead of to an NFS target when it occurs.

Revision history for this message
Eric Harney (eharney) wrote :

The test case for this behavior is to mount an NFS export with one SELinux context, and then have Cinder try to mount it with a different SELinux context. This hits a limitation in the Linux NFS client that produces this error.

See details in https://bugzilla.redhat.com/show_bug.cgi?id=253530 .

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-brick (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/623035

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-brick (master)

Reviewed: https://review.openstack.org/609543
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=360a20c7c7b4c385255200a574edc0ca1ac8d87c
Submitter: Zuul
Branch: master

commit 360a20c7c7b4c385255200a574edc0ca1ac8d87c
Author: Eric Harney <email address hidden>
Date: Wed Oct 10 16:46:01 2018 -0400

    Fix NFS "already mounted" detection

    Previous fix

    2f32c98a RemoteFS: don't fail in do_mount if already mounted

    was made with the assumption that "already mounted" appearing
    in stderr from an NFS mount means that the NFS share was already
    mounted.

    However, the NFS client can fail with "busy or already mounted"
    in cases where it fails to mount the share as well.

    This results in the Cinder NFS backup driver believing that the
    NFS mount has succeeded when it didn't, and as a result, data
    can be written to the local disk instead of the NFS target.

    Fail if the share isn't actually mounted rather than succeeding.

    Related-Bug: #1780813
    Closes-Bug: #1797233
    Change-Id: Iebd1afb3340fcaeb1969784966c4f9be35a28417

Changed in os-brick:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-brick (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/623238

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-brick (stable/rocky)

Reviewed: https://review.openstack.org/623035
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=4c94477cb525b25f178d52ff1b01bfe1fc3d3dd3
Submitter: Zuul
Branch: stable/rocky

commit 4c94477cb525b25f178d52ff1b01bfe1fc3d3dd3
Author: Eric Harney <email address hidden>
Date: Wed Oct 10 16:46:01 2018 -0400

    Fix NFS "already mounted" detection

    Previous fix

    2f32c98a RemoteFS: don't fail in do_mount if already mounted

    was made with the assumption that "already mounted" appearing
    in stderr from an NFS mount means that the NFS share was already
    mounted.

    However, the NFS client can fail with "busy or already mounted"
    in cases where it fails to mount the share as well.

    This results in the Cinder NFS backup driver believing that the
    NFS mount has succeeded when it didn't, and as a result, data
    can be written to the local disk instead of the NFS target.

    Fail if the share isn't actually mounted rather than succeeding.

    Related-Bug: #1780813
    Closes-Bug: #1797233
    Change-Id: Iebd1afb3340fcaeb1969784966c4f9be35a28417
    (cherry picked from commit 360a20c7c7b4c385255200a574edc0ca1ac8d87c)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-brick (stable/queens)

Reviewed: https://review.openstack.org/623238
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=30d7bc124e564d4a2c288f3666d8d14dc96fb9f5
Submitter: Zuul
Branch: stable/queens

commit 30d7bc124e564d4a2c288f3666d8d14dc96fb9f5
Author: Eric Harney <email address hidden>
Date: Wed Oct 10 16:46:01 2018 -0400

    Fix NFS "already mounted" detection

    Previous fix

    2f32c98a RemoteFS: don't fail in do_mount if already mounted

    was made with the assumption that "already mounted" appearing
    in stderr from an NFS mount means that the NFS share was already
    mounted.

    However, the NFS client can fail with "busy or already mounted"
    in cases where it fails to mount the share as well.

    This results in the Cinder NFS backup driver believing that the
    NFS mount has succeeded when it didn't, and as a result, data
    can be written to the local disk instead of the NFS target.

    Fail if the share isn't actually mounted rather than succeeding.

    Related-Bug: #1780813
    Closes-Bug: #1797233
    Change-Id: Iebd1afb3340fcaeb1969784966c4f9be35a28417
    (cherry picked from commit 360a20c7c7b4c385255200a574edc0ca1ac8d87c)
    (cherry picked from commit 4c94477cb525b25f178d52ff1b01bfe1fc3d3dd3)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 2.7.0

This issue was fixed in the openstack/os-brick 2.7.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 2.5.5

This issue was fixed in the openstack/os-brick 2.5.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 2.3.5

This issue was fixed in the openstack/os-brick 2.3.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-brick (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/647766

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-brick (stable/pike)

Reviewed: https://review.openstack.org/647766
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=f64ad529bfd836a6e18199c3b7d22ff4e07be2c5
Submitter: Zuul
Branch: stable/pike

commit f64ad529bfd836a6e18199c3b7d22ff4e07be2c5
Author: Eric Harney <email address hidden>
Date: Wed Oct 10 16:46:01 2018 -0400

    Fix NFS "already mounted" detection

    Previous fix

    2f32c98a RemoteFS: don't fail in do_mount if already mounted

    was made with the assumption that "already mounted" appearing
    in stderr from an NFS mount means that the NFS share was already
    mounted.

    However, the NFS client can fail with "busy or already mounted"
    in cases where it fails to mount the share as well.

    This results in the Cinder NFS backup driver believing that the
    NFS mount has succeeded when it didn't, and as a result, data
    can be written to the local disk instead of the NFS target.

    Fail if the share isn't actually mounted rather than succeeding.

    Related-Bug: #1780813
    Closes-Bug: #1797233
    Change-Id: Iebd1afb3340fcaeb1969784966c4f9be35a28417
    (cherry picked from commit 360a20c7c7b4c385255200a574edc0ca1ac8d87c)
    (cherry picked from commit 4c94477cb525b25f178d52ff1b01bfe1fc3d3dd3)
    (cherry picked from commit 30d7bc124e564d4a2c288f3666d8d14dc96fb9f5)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 1.15.9

This issue was fixed in the openstack/os-brick 1.15.9 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.