[DOC]CephFS driver in manila

[Buddhika Sanjeewa]

This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes:

- [X] This doc is inaccurate in this way: ______
- [ ] This is a doc addition request.
- [ ] I have a fix to the document that I can paste below including example: input and output.

If you have a troubleshooting or support issue, use the following resources:

 - The mailing list: https://lists.openstack.org
 - IRC: 'openstack' channel on OFTC

-----------------------------------
Release: 12.1.0.dev37 on 2021-03-24 23:00:34
SHA: 341ea59e4061884836054b595da34a38f3ffa35f
Source: https://opendev.org/openstack/manila/src/doc/source/admin/cephfs_driver.rst
URL: https://docs.openstack.org/manila/latest/admin/cephfs_driver.html

The section "Authorizing the driver to communicate with Ceph" in latest branch says the ceph capabilities are different in Wallaby release. But the Wallaby branch doc does not specify that detail.

===Latest=====

 Important

The driver in the Wallaby (or later) release requires a Ceph identity with a different set of Ceph capabilities when compared to the driver in a pre-Wallaby release.

When upgrading to Wallaby you’ll also have to update the capabilities of the Ceph identity used by the driver (refer to Ceph user capabilities docs) E.g. a native driver that already uses client.manila Ceph identity, issue command ceph auth caps client.manila mon ‘allow r’ mgr ‘allow rw’

===End Latest===

===Wallaby====
Nothing like that. The old capabilities command (also in Victoria) is present

[Buddhika Sanjeewa (bsanjeewa)]

Anyway I ran into an error when using ceph auth caps client.manila mon ‘allow r’ mgr ‘allow rw’ in pacific, with Wallaby.

The way described in wallaby doc seems working.

[Vida Haririan (vhariria)]

Additional comments https://meetings.opendev.org/meetings/manila/2021/manila.2021-07-08-15.00.log.html

[Goutham Pacha Ravi (gouthamr)]

Hi bsanjeewa

Thank you for raising this bug. We were missing a backport of the documentation changes that were made after the Wallaby release was shipped. We're going to make that now, as a bug fix [1]

> Anyway I ran into an error when using ceph auth caps client.manila mon ‘allow r’ mgr ‘allow rw’ in pacific, with Wallaby.

Any chance you can share a log of this error?
Were you upgrading an existing manila installation, or were you deploying the wallaby release freshly?

[1] https://review.opendev.org/c/openstack/manila/+/799984

[Buddhika Sanjeewa (bsanjeewa)]

Hi Goutham,
It was an access denied error, sorry cannot remember the details. I was
installing from the scratch using kayobe. Even ceph was deployed after
formatting all. Kind of PoC. Unfortunately I don't have the logs with me,
because I changed some other settings with other services and redeployed
the whole.

If you can give me a couple of days, I might be able to recreate it, if
it's OK with you.

Thanks

On Thu, 8 Jul 2021, 23:30 Goutham Pacha Ravi, <email address hidden>
wrote:

> Hi bsanjeewa
>
> Thank you for raising this bug. We were missing a backport of the
> documentation changes that were made after the Wallaby release was
> shipped. We're going to make that now, as a bug fix [1]
>
> > Anyway I ran into an error when using ceph auth caps client.manila mon
> ‘allow r’ mgr ‘allow rw’ in pacific, with Wallaby.
>
> Any chance you can share a log of this error?
> Were you upgrading an existing manila installation, or were you deploying
> the wallaby release freshly?
>
>
> [1] https://review.opendev.org/c/openstack/manila/+/799984
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1933696
>
> Title:
> [DOC]CephFS driver in manila
>
> Status in OpenStack Shared File Systems Service (Manila):
> Confirmed
>
> Bug description:
>
> This bug tracker is for errors with the documentation, use the
> following as a template and remove or add fields as you see fit.
> Convert [ ] into [x] to check boxes:
>
> - [X] This doc is inaccurate in this way: ______
> - [ ] This is a doc addition request.
> - [ ] I have a fix to the document that I can paste below including
> example: input and output.
>
> If you have a troubleshooting or support issue, use the following
> resources:
>
> - The mailing list: https://lists.openstack.org
> - IRC: 'openstack' channel on OFTC
>
> -----------------------------------
> Release: 12.1.0.dev37 on 2021-03-24 23:00:34
> SHA: 341ea59e4061884836054b595da34a38f3ffa35f
> Source:
> https://opendev.org/openstack/manila/src/doc/source/admin/cephfs_driver.rst
> URL: https://docs.openstack.org/manila/latest/admin/cephfs_driver.html
>
> The section "Authorizing the driver to communicate with Ceph" in
> latest branch says the ceph capabilities are different in Wallaby
> release. But the Wallaby branch doc does not specify that detail.
>
>
> ===Latest=====
>
> Important
>
> The driver in the Wallaby (or later) release requires a Ceph identity
> with a different set of Ceph capabilities when compared to the driver
> in a pre-Wallaby release.
>
> When upgrading to Wallaby you’ll also have to update the capabilities
> of the Ceph identity used by the driver (refer to Ceph user
> capabilities docs) E.g. a native driver that already uses
> client.manila Ceph identity, issue command ceph auth caps
> client.manila mon ‘allow r’ mgr ‘allow rw’
>
> ===End Latest===
>
> ===Wallaby====
> Nothing like that. The old capabilities command (also in Victoria) is
> present
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/manila/+bug/1933696/+subscriptions
>

--
+++++++++...

[Buddhika Sanjeewa (bsanjeewa)]

Hi Goutham,
I destroyed the overcloud, thrn ran
ceph auth caps client.manila mon 'allow r' mgr 'allow rw'
on my existing ceph cluster and then tried to deploy with kayobe.
This error occurred.

TASK [manila : include_tasks]
***************************************************************************************************************************************************************************************************************
included:
/root/kayobe/live/venvs/kolla-ansible/share/kolla-ansible/ansible/roles/manila/tasks/fix_cephfs_owner.yml
for ctrl-s1-001, ctrl-s1-002, ctrl-s2-001, ctrl-s2-002

TASK [manila : Check /tmp/cephfs path]
******************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Create /tmp/cephfs path]
*****************************************************************************************************************************************************************************************************
changed: [ctrl-s1-001]

TASK [manila : Get monitor dump]
************************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Get cephfs addr]
*************************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Get cephfs secret]
***********************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Umount cephfs]
***************************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Mount cephfs]
****************************************************************************************************************************************************************************************************************
fatal: [ctrl-s1-001]: FAILED! => {"changed": false, "cmd": ["docker",
"exec", "-u", "0", "manila_share", "mount", "-t", "ceph", "10.244.0.5:6789,
10.244.0.6:6789,10.244.0.7:6789:/", "/tmp/cephfs", "-o",
"name=manila,secret=AQA/JONgCFtfLhAAeS8IY8k1CP0j9cslUUxCZQ=="], "delta":
"0:00:00.162060", "end": "2021-07-08 19:55:06.926873", "msg": "non-zero
return code", "rc": 32, "start": "2021-07-08 19:55:06.764813", "stderr":
"mount error 13 = Permission denied", "stderr_lines": ["mount error 13 =
Permission denied"], "stdout": "", "stdout_lines": []}

I have attached the manila-share.log on ctrl-s1-001 too.

Then I ran

read -d '' MON_CAPS << EOF
allow r,
allow command "auth del",
allow command "auth caps",
allow command "auth get",
allow command "auth get-or-create"
EOF

AND

ceph auth caps client.manila...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/799984
Committed: https://opendev.org/openstack/manila/commit/2433bca5641035d2e13b7d2fb8ef855133e9db2e
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 2433bca5641035d2e13b7d2fb8ef855133e9db2e
Author: Victoria Martinez de la Cruz <email address hidden>
Date: Wed Mar 24 23:00:34 2021 +0000

    Updates CephFS drivers docs

    As part of our work on updating CephFS drivers
    to use the ceph-mgr interface, the related docs
    on the admin guide need to be updated

    - Updates ceph caps for manila to include mgr caps,
    remove mds caps, and constrain mon caps
    - Add the new cephfs_filesystem_name param to the
    config
    - Add upgrade to Wallaby considerations (minimum Ceph
    version required)

    Closes-Bug: #1933696
    Partially-Implement: bp update-cephfs-drivers
    Change-Id: I113639cc9989bbaf224d19969993870d30b81c29
    (cherry picked from commit 992ec82575655e9153af4c75fab911dcf598d423)

[Goutham Pacha Ravi (gouthamr)]

Hi,

Thank you so much for sharing your log and reproduction steps. It looks like the error is coming from this portion of kolla-ansible:

https://opendev.org/openstack/kolla-ansible/src/commit/2ecf0a8783a71777ab1082329bb3ae778f41e949/ansible/roles/manila/tasks/fix_cephfs_owner.yml#L48-L84

The bootstrap/setup steps in that code look unnecessary in my view; The code's creating a "/volumes" directory under the "cephfs" filesystem and changing the ownership of that directory.

In releases prior to wallaby, manila shares would be created as subdirectories under the "/volumes" directory inside the cephfs filesystem. In Wallaby and beyond, the shares are created using the "subvolumes" interface of CephFS [1], and this "/volumes" directory is not longer created/needed. (Shares created with older releases will be left under the /volumes directory, new shares won't be created there)

Even when the we were creating manila shares as subdirectories under /volumes, the "/volumes" directory is created automatically - there is no need for a bootstrap script to explicitly create this directory. [2]

To be able to mount the cephfs filesystem, you'll need "osd" capabilities/permissions - the manila driver never mounts any cephfs directories from the wallaby release, so "osd" permissions are no longer required.

So this should be reported as a bug on kolla-ansible: https://bugs.launchpad.net/kolla-ansible/ - it's possible someone who works on that community can provide further information on this explicit directory creation.

[1] https://docs.ceph.com/en/latest/cephfs/fs-volumes/#fs-subvolumes
[2] https://docs.openstack.org/manila/wallaby/admin/cephfs_driver.html

[Buddhika Sanjeewa (bsanjeewa)]

Reported @ kolla-ansible
https://bugs.launchpad.net/kolla-ansible/+bug/1935784

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 12.1.0 release.