kayobe overcloud service deploy fails at TASK [manila : Mount cephfs]

[Buddhika Sanjeewa]

When I run kayobe overcloud service deploy, TASK [manila : Mount cephfs] fails.
OS: CentOS Stream
Release: Wallaby
Images built from source

TASK [manila : include_tasks]
***************************************************************************************************************************************************************************************************************
included:
/root/kayobe/live/venvs/kolla-ansible/share/kolla-ansible/ansible/roles/manila/tasks/fix_cephfs_owner.yml
for ctrl-s1-001, ctrl-s1-002, ctrl-s2-001, ctrl-s2-002

TASK [manila : Check /tmp/cephfs path]
******************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Create /tmp/cephfs path]
*****************************************************************************************************************************************************************************************************
changed: [ctrl-s1-001]

TASK [manila : Get monitor dump]
************************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Get cephfs addr]
*************************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Get cephfs secret]
***********************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Umount cephfs]
***************************************************************************************************************************************************************************************************************
ok: [ctrl-s1-001]

TASK [manila : Mount cephfs]
****************************************************************************************************************************************************************************************************************
fatal: [ctrl-s1-001]: FAILED! => {"changed": false, "cmd": ["docker",
"exec", "-u", "0", "manila_share", "mount", "-t", "ceph", "10.244.0.5:6789,
10.244.0.6:6789,10.244.0.7:6789:/", "/tmp/cephfs", "-o",
"name=manila,secret=AQA/JONgCFtfLhAAeS8IY8k1CP0j9cslUUxCZQ=="], "delta":
"0:00:00.162060", "end": "2021-07-08 19:55:06.926873", "msg": "non-zero
return code", "rc": 32, "start": "2021-07-08 19:55:06.764813", "stderr":
"mount error 13 = Permission denied", "stderr_lines": ["mount error 13 =
Permission denied"], "stdout": "", "stdout_lines": []}

I have attached the manila-share.log on ctrl-s1-001 too.

Then I ran

read -d '' MON_CAPS << EOF
allow r,
allow command "auth del",
allow command "auth caps",
allow command "auth get",
allow command "auth get-or-create"
EOF

AND

ceph auth caps client.manila -o manila.keyring mds 'allow *' osd 'allow rw'
mgr 'allow r' mon "$MON_CAPS"

on a ceph node, and redeployed using kayobe. It completed as expected. The issue is the manila capabilities working here is for Victoria.
Wallaby has a different set of capabilities.

When communicated with manila, they suggested an update in kolla-ansible.

Please see bug report: https://bugs.launchpad.net/manila/+bug/1933696
Kolla-Ansible patch suggestion: https://bugs.launchpad.net/manila/+bug/1933696/comments/7

[Buddhika Sanjeewa (bsanjeewa)]

ceph capabilities used originally was

ceph auth caps client.manila mon 'allow r' mgr 'allow rw'

as suggested in https://docs.openstack.org/manila/wallaby/admin/cephfs_driver.html

[Mark Goddard (mgoddard)]

Thanks Buddhika - I suggest we follow manila's direction here. Are you able to propose a patch?

[Buddhika Sanjeewa (bsanjeewa)]

Dear Mark,
I am not very familiar with the code, and the idea of ceph and manila
drivers. Also I do not have a place to test at the moment.
So I am afraid I won't be much help at the moment.

On Tue, 20 Jul 2021, 14:15 Mark Goddard, <email address hidden> wrote:

> Thanks Buddhika - I suggest we follow manila's direction here. Are you
> able to propose a patch?
>
> ** Changed in: kolla-ansible
> Status: New => Triaged
>
> ** Changed in: kolla-ansible
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1935784
>
> Title:
> kayobe overcloud service deploy fails at TASK [manila : Mount cephfs]
>
> Status in kolla-ansible:
> Triaged
>
> Bug description:
> When I run kayobe overcloud service deploy, TASK [manila : Mount cephfs]
> fails.
> OS: CentOS Stream
> Release: Wallaby
> Images built from source
>
> TASK [manila : include_tasks]
>
> ***************************************************************************************************************************************************************************************************************
> included:
>
> /root/kayobe/live/venvs/kolla-ansible/share/kolla-ansible/ansible/roles/manila/tasks/fix_cephfs_owner.yml
> for ctrl-s1-001, ctrl-s1-002, ctrl-s2-001, ctrl-s2-002
>
> TASK [manila : Check /tmp/cephfs path]
>
> ******************************************************************************************************************************************************************************************************
> ok: [ctrl-s1-001]
>
> TASK [manila : Create /tmp/cephfs path]
>
> *****************************************************************************************************************************************************************************************************
> changed: [ctrl-s1-001]
>
> TASK [manila : Get monitor dump]
>
> ************************************************************************************************************************************************************************************************************
> ok: [ctrl-s1-001]
>
> TASK [manila : Get cephfs addr]
>
> *************************************************************************************************************************************************************************************************************
> ok: [ctrl-s1-001]
>
> TASK [manila : Get cephfs secret]
>
> ***********************************************************************************************************************************************************************************************************
> ok: [ctrl-s1-001]
>
> TASK [manila : Umount cephfs]
>
> ***************************************************************************************************************************************************************************************************************
> ok: [ctrl-s1-001]
>
> TASK [manila : Mount cephfs]
>
> ***************************************************************************************************************************************************************...

[Skylar Tristan Kelty (skylarkelty)]

Looks like we need to tidy up fix_cephfs_owner.yml for wallaby+ and update the documentation. I'm happy to work on this.
This work would fix https://bugs.launchpad.net/kolla-ansible/+bug/1938285 too, making the existing patch there unnecessary.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/802743

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/802743
Committed: https://opendev.org/openstack/kolla-ansible/commit/8d5dde3723c75958b7583d30b525667e99ec2496
Submitter: "Zuul (22348)"
Branch: master

commit 8d5dde3723c75958b7583d30b525667e99ec2496
Author: Skylar Kelty <email address hidden>
Date: Wed Jul 28 16:39:05 2021 +0100

    Update Manila deploy steps for Wallaby

    Manila has changed from using subfolders to subvolumes.
    We need a bit of a tidy up to prevent deploy errors.
    This change also adds the ability to specify the ceph FS
    Manila uses instead of relying on the default "first found".

    Closes-Bug: #1938285
    Closes-Bug: #1935784
    Change-Id: I1d0d34919fbbe74a4022cd496bf84b8b764b5e0f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/814665

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/814666

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/814665
Committed: https://opendev.org/openstack/kolla-ansible/commit/45b0b87f8c697fd76de35c61f1b17d22dd0054eb
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 45b0b87f8c697fd76de35c61f1b17d22dd0054eb
Author: Skylar Kelty <email address hidden>
Date: Wed Jul 28 16:39:05 2021 +0100

    Update Manila deploy steps for Wallaby

    Manila has changed from using subfolders to subvolumes.
    We need a bit of a tidy up to prevent deploy errors.
    This change also adds the ability to specify the ceph FS
    Manila uses instead of relying on the default "first found".

    Closes-Bug: #1938285
    Closes-Bug: #1935784
    Change-Id: I1d0d34919fbbe74a4022cd496bf84b8b764b5e0f
    (cherry picked from commit 8d5dde3723c75958b7583d30b525667e99ec2496)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/814666
Committed: https://opendev.org/openstack/kolla-ansible/commit/94627f1c819cd09d4aa13c2c1d81e825b3c3895c
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 94627f1c819cd09d4aa13c2c1d81e825b3c3895c
Author: Skylar Kelty <email address hidden>
Date: Wed Jul 28 16:39:05 2021 +0100

    Update Manila deploy steps for Wallaby

    Manila has changed from using subfolders to subvolumes.
    We need a bit of a tidy up to prevent deploy errors.
    This change also adds the ability to specify the ceph FS
    Manila uses instead of relying on the default "first found".

    Closes-Bug: #1938285
    Closes-Bug: #1935784
    Change-Id: I1d0d34919fbbe74a4022cd496bf84b8b764b5e0f
    (cherry picked from commit 8d5dde3723c75958b7583d30b525667e99ec2496)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 13.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 12.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc1 release candidate.