Manila Ceph driver resets root folder mode on recent Ceph clusters

[Luis Fernández Álvarez]

Description
===========

On a Manila deployment using the Ceph driver against a recent Ceph cluster, when the Manila share service starts the logic of "ensure_share" will have the side effect of resetting the mode of the share root folder to the default specified via "cephfs_volume_mode", even if the user changed the mode to a custom value.

After some investigation, this looks connected to a change of behaviour introduced in "fs subvolume create" to fix this issue: https://tracker.ceph.com/issues/54375. That change has been ported to Pacific and Quincy, so users are affected depending on the cluster version running.

Steps to reproduce
==================

Requirements: Ceph cluster running > Pacific (or any other version patched with https://tracker.ceph.com/issues/54375).

* Create a Manila share.
* Changed the mode of the root share folder to a mode different of the default. (e.g. 777).
* Restart the Manila share service.

Expected result
===============

The mode set by the user is kept after the restart of the share service. In this example, 777.

Actual result
=============

The mode was reset to the default specified in our deployment by "cephfs_volume_mode" (755).

Environment
===========

* Manila: Wallaby.
* Storage backend: Ceph.
* Ceph cluster version: Pacific.

[Vida Haririan (vhariria)]

See additional related discussions at https://meetings.opendev.org/meetings/manila/2023/manila.2023-01-12-15.00.log.html

[Victoria Martinez de la Cruz (vkmc)]

In order to fix this we should get the mode information from the share instead of using the value from the configuration file. Precisely, we would need to change this line https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/driver.py#L468.

[Carlos Eduardo (silvacarlose)]

Hello, Luis! After a round of discussions, we were thinking about how could we tackle this issue, there are few approaches and one of them is through the use of share type extra specs. Admins would be able to enforce the value of an extra spec similar to cephfs_volume_mode on share type, and we would read from the extra specs in the ensure_shares. If no share type extra spec is defined, we would fall back to the config option. What do you think about this approach?

[Luis Fernández Álvarez (luis-fernandez-alvarez)]

Hello, Carlos. First of all, many thanks for looking into this bug.

The solution you proposed based on extra specs wouldn't fix the issue in our deployment. We don't enforce a cephfs volume module for all the shares of our users. Depending on their use case, they might need to define different modes. These different use cases is not something we isolate per share types.

Our ideal scenario would be:
* Manila creates a share with a default mode (a global default or default per share type, it doesn't matter).
* Once the share is created, Manila shouldn't touch the mode again.

This should avoid situations such as:
* user creates the share, modifies the mode (maybe he's not even aware of an enforced default for the mode) to his needs and the application is running perfectly. Then, we restart the services few months later for a configuration change and modes are reset, applications stop working... etc.

This is a similar situation with the share gid/uid. Manila doesn't currently worry about those settings and it doesn't set them during the life-cycle of the share. I think it should be the same for the mode.

What do you think?

Finally, if for whatever reason you decide to go ahead with the per-type mode enforcement, it would be good if this feature is optional and operators we can chose if Manila enforces the share mode or not.

Cheers,
Luis

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/manila/+/871611

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Carlos Eduardo <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/manila/+/871611

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/871643

[Carlos Eduardo (silvacarlose)]

Hello, Luis! Thank you for the feedback. We are attempting a simple fix, that will avoid the mode being overwritten in the ensure_shares. That will be an easy one to backport as soon as we have it merged. The solution with extra-specs can come in the future as an RFE. But yes, it would be easily turned on/off.

[Luis Fernández Álvarez (luis-fernandez-alvarez)]

Thanks, Carlos. I'm taking a look to the proposed patch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/871643
Committed: https://opendev.org/openstack/manila/commit/c61c1977c25e151fd2839febd08a7fdac6b1f7b8
Submitter: "Zuul (22348)"
Branch: master

commit c61c1977c25e151fd2839febd08a7fdac6b1f7b8
Author: silvacarloss <email address hidden>
Date: Tue Jan 24 11:49:14 2023 -0300

    [Cephfs] Fix erroneous share mode override on ensure_shares

    In order to ensure that a share existed, we used to send the same
    request to Ceph as we would do with the share creation. In that
    request, we will set the share mode, which is a value set by
    the deployer through the `cephfs_volume_mode` config option.

    If the share was already created, Ceph would not override the value
    we were forwarding for the permission mode. After a bugfix [1], the
    value of mode started being reinforced when sent, even if the
    subvolume was already created. If the value for
    `cephfs_volume_mode` changed after the share was created, the mode
    is now being overwritten after manila startup on ensure_shares.

    Fixed this issue by modifying the ensure_shares to only confirm if
    the export exists.

    [1] https://tracker.ceph.com/issues/54375

    Closes-Bug: #2002394
    Change-Id: Ic5b5b42b882ce1346b4b46d1b29aa31740933e0e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/871611
Committed: https://opendev.org/openstack/manila/commit/f8d0e0cad271baaac0f86aa3fe156272deabb48e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit f8d0e0cad271baaac0f86aa3fe156272deabb48e
Author: silvacarloss <email address hidden>
Date: Tue Jan 24 11:49:14 2023 -0300

    [Cephfs] Fix erroneous share mode override on ensure_shares

    In order to ensure that a share existed, we used to send the same
    request to Ceph as we would do with the share creation. In that
    request, we will set the share mode, which is a value set by
    the deployer through the `cephfs_volume_mode` config option.

    If the share was already created, Ceph would not override the value
    we were forwarding for the permission mode. After a bugfix [1], the
    value of mode started being reinforced when sent, even if the
    subvolume was already created. If the value for
    `cephfs_volume_mode` changed after the share was created, the mode
    is now being overwritten after manila startup on ensure_shares.

    Fixed this issue by modifying the ensure_shares to only confirm if
    the export exists.

    [1] https://tracker.ceph.com/issues/54375

    Closes-Bug: #2002394
    Change-Id: Ic5b5b42b882ce1346b4b46d1b29aa31740933e0e
    (cherry picked from commit c61c1977c25e151fd2839febd08a7fdac6b1f7b8)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/manila/+/874500

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/874500
Committed: https://opendev.org/openstack/manila/commit/3d6fcc99458b07fa6f8e2983038a8c035b6dc98a
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 3d6fcc99458b07fa6f8e2983038a8c035b6dc98a
Author: silvacarloss <email address hidden>
Date: Tue Jan 24 11:49:14 2023 -0300

    [Cephfs] Fix erroneous share mode override on ensure_shares

    In order to ensure that a share existed, we used to send the same
    request to Ceph as we would do with the share creation. In that
    request, we will set the share mode, which is a value set by
    the deployer through the `cephfs_volume_mode` config option.

    If the share was already created, Ceph would not override the value
    we were forwarding for the permission mode. After a bugfix [1], the
    value of mode started being reinforced when sent, even if the
    subvolume was already created. If the value for
    `cephfs_volume_mode` changed after the share was created, the mode
    is now being overwritten after manila startup on ensure_shares.

    Fixed this issue by modifying the ensure_shares to only confirm if
    the export exists.

    [1] https://tracker.ceph.com/issues/54375

    Closes-Bug: #2002394
    Change-Id: Ic5b5b42b882ce1346b4b46d1b29aa31740933e0e
    (cherry picked from commit c61c1977c25e151fd2839febd08a7fdac6b1f7b8)
    (cherry picked from commit f8d0e0cad271baaac0f86aa3fe156272deabb48e)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/manila/+/874743

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/874746

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/874743
Committed: https://opendev.org/openstack/manila/commit/f320288f47b3f986a6df834922503527f446a728
Submitter: "Zuul (22348)"
Branch: stable/xena

commit f320288f47b3f986a6df834922503527f446a728
Author: silvacarloss <email address hidden>
Date: Tue Jan 24 11:49:14 2023 -0300

    [Cephfs] Fix erroneous share mode override on ensure_shares

    In order to ensure that a share existed, we used to send the same
    request to Ceph as we would do with the share creation. In that
    request, we will set the share mode, which is a value set by
    the deployer through the `cephfs_volume_mode` config option.

    If the share was already created, Ceph would not override the value
    we were forwarding for the permission mode. After a bugfix [1], the
    value of mode started being reinforced when sent, even if the
    subvolume was already created. If the value for
    `cephfs_volume_mode` changed after the share was created, the mode
    is now being overwritten after manila startup on ensure_shares.

    Fixed this issue by modifying the ensure_shares to only confirm if
    the export exists.

    [1] https://tracker.ceph.com/issues/54375

    Closes-Bug: #2002394
    Change-Id: Ic5b5b42b882ce1346b4b46d1b29aa31740933e0e
    (cherry picked from commit c61c1977c25e151fd2839febd08a7fdac6b1f7b8)
    (cherry picked from commit f8d0e0cad271baaac0f86aa3fe156272deabb48e)
    (cherry picked from commit 3d6fcc99458b07fa6f8e2983038a8c035b6dc98a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 16.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 15.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 13.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 14.1.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/874746
Reason: stable/wallaby branch of openstack/manila is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/911620