Permission denied Errno 13 when creating manila share

[David Coronel]

I'm seeing the following error when trying to create a CephFS NFS share, from /var/log/manila/manila-share.log on manila-ganesha-az1/0:

2020-10-26 13:51:09.792 55344 ERROR oslo_messaging.rpc.server [req-add78ae8-7146-475e-8aed-2d15829e5fcb f2ee9e8060e54d058060d220fab84088 86fcc0b3839b45029dd325641ddc2a09 - - -] Exception during message handling: cephfs.OSError: error in mkdir volumes: Permission denied [Errno 13]

I create the share this way, it has a status of 'error':

$ manila create nfs 1 --name newtestdavid7 --description "My share" --share-type cephfstype

$ manila list
+--------------------------------------+----------------+------+-------------+----------------+-----------+-----------------+----------------------------------------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+--------------------------------------+----------------+------+-------------+----------------+-----------+-----------------+----------------------------------------+-------------------+
| 5935a24c-537c-4b0f-ace2-11ff39d8dba7 | newtestdavid7 | 1 | NFS | error | False | cephfstype | juju-cf9afe-12-lxd-1@cephfsnfs1#cephfs | nova |

This is an OpenStack from the bionic-train cloud archive.

manila charm rev 92
manila-gasnesha charm rev 5

manila version 1:9.0.0-0ubuntu1~cloud0
nfs-ganesha version 2.6.0-2

[David Coronel (davecore)]

subscribed ~field-critical

[David Coronel (davecore)]

It looks like I'm hitting something similar to https://bugzilla.redhat.com/show_bug.cgi?id=1639823

The workaround from that bug resolves the manila create issue:

From a ceph-mon unit as root:

# mount -t ceph 172.31.252.11:6789:/ /mnt -o name=manila-ganesha-az2,secret="<key>"

# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
172.31.252.11:6789:/ 331T 0 331T 0% /mnt

# ls -ld /mnt
drwxr-xr-x 1 root root 0 Oct 3 04:08 /mnt

If I change the permissions of /mnt to 777, I can then create shares with manila.

Additional info:

We have two AZs in this environment. The names of our charms are:

manila
manila-ganesha-az1
manila-ganesha-az2

[David Coronel (davecore)]

I also had to give extra ceph auth caps permissions to the client.manila-ganesha-<az1/az2> clients. They were missing the mds permissions which are required to create the cephfs volumes:

Before:

client.manila-ganesha-az1
        key: <key>
        caps: [mon] allow r; allow command "osd blacklist"
        caps: [osd] allow rwx

Fix:

# ceph auth caps client.manila-ganesha-az1 mds 'allow *' mon 'allow r; allow command "osd blacklist"' osd 'allow rwx'

After:

client.manila-ganesha-az1
        key: <key>
        caps: [mds] allow *
        caps: [mon] allow r; allow command "osd blacklist"
        caps: [osd] allow rwx

[David Coronel (davecore)]

With the chmod 777 workaround in place, I can now create shares with manila create, but I can't allow access to an IP:

$ manila access-allow newtestdavid9 ip 172.24.4.229

$ manila access-list newtestdavid9
| id | access_type | access_to | access_level | state | access_key | created_at | updated_at |
| e1c4b913-913f-448b-9973-b497341d7314 | ip | 172.24.4.229 | rw | error | None | 2020-10-26T18:35:47.000000 | None |

I see this log in /var/log/manila/manila-share.log on manila-ganesha-az1/0:

2020-10-26 18:35:47.580 55344 ERROR oslo_messaging.rpc.server [req-38244b91-5fe2-475c-a693-128beb25057d f2ee9e8060e54d058060d220fab84088 86fcc0b3839b45029dd325641ddc2a09 - - -] Exception during message handling: cephfs.OSError: error in open /volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta: Permission denied [Errno 13]

When I mount the NFS root, here is what I see in volumes:

# df -h /mnt/volumes/
Filesystem Size Used Avail Use% Mounted on
172.31.250.17:6789:/ 332T 0 332T 0% /mnt

# ls -lan /mnt/volumes/
total 1
drwxr-xr-x 1 111 115 6 Oct 26 15:52 .
drwxrwxrwx 1 111 115 1 Oct 26 14:27 ..
-rwxr-xr-x 1 111 115 0 Oct 26 14:44 _:3b232b7e-1949-49c3-a742-7ca689f34f46.meta
-rwxr-xr-x 1 111 115 0 Oct 26 15:52 _:bfdb2ddf-c089-4947-a8b5-4245757e8794.meta
drwxr-xr-x 1 111 115 0 Oct 26 14:50 _deleting
---------- 1 111 115 196 Oct 26 15:52 '$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta'
drwxr-xr-x 1 111 115 2 Oct 26 14:50 _nogroup
-rwxr-xr-x 1 111 115 0 Oct 26 15:15 _None:bfdb2ddf-c089-4947-a8b5-4245757e8794.meta

[David Coronel (davecore)]

I also did a "chown -R 111:115 /mnt" on the host where I mounted the root NFS, where 111 is the uid of the manila user on manila/0 and 115 is its gid.

[David Coronel (davecore)]

The file '/mnt/volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta' contains the following:

# cat '/mnt/volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta'

{"dirty": true, "tenant_id": "86fcc0b3839b45029dd325641ddc2a09", "volumes": {"None/bfdb2ddf-c089-4947-a8b5-4245757e8794": {"access_level": "rw", "dirty": true}}, "compat_version": 1, "version": 4}

But the file has no permissions at all:

---------- 1 111 115 196 Oct 26 15:52 '/mnt/volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta'

[David Coronel (davecore)]

I tried changing the permissions of the file '/mnt/volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta' to 755 like the other files, but when I try to do another 'manila access-allow newtestdavid9 ip 172.24.4.230', the file reverts back to no permissions.

[Billy Olsen (billy-olsen)]

Can you provide logs for this environment? I'm a bit confused by the permissions in comment #3 as the permissions already should be requesting the the correct permissions per [0]. If you can provide a ceph auth ls (with keys redacted) that might help us see what the permissions are set as.

[0] - https://opendev.org/openstack/charm-manila-ganesha/src/branch/stable/20.08/src/lib/charm/openstack/manila_ganesha.py#L42

[David Coronel (davecore)]

'ceph auth ls' in canonical pastebin: https://pastebin.canonical.com/p/mRGJK4ysbj/

The comment #3 might be a workaround to a different bug I may need to file.

There are no mds permissions for the manila-ganesha clients and it looks like those are required to create the cephfs volumes.

[Dmitrii Shcherbakov (dmitriis)]

Deployed a bionic-train bundle with manila-ganesha and validated that the mds caps are present:

https://paste.ubuntu.com/p/p6KCNHkgQm/ (bundle)
https://paste.ubuntu.com/p/Yq2jNV6D7v/ (ceph auth ls)

client.manila-ganesha
 key: <somekey-2>
 caps: [mds] allow *
 caps: [mon] allow r, allow command "auth del", allow command "auth caps", allow command "auth get", allow command "auth get-or-create"
 caps: [osd] allow rw
client.ganesha-53b0af21-d7ba-496f-b2e5-b2209121ec79
 key: <somekey-1>
 caps: [mds] allow rw path=/volumes/_nogroup/53b0af21-d7ba-496f-b2e5-b2209121ec79
 caps: [mon] allow r
 caps: [osd] allow rw pool=ceph-fs_data namespace=fsvolumens_53b0af21-d7ba-496f-b2e5-b2209121ec79

[Dmitrii Shcherbakov (dmitriis)]

Looking at the bundle David provided, I can see that there are two units of manila-ganesha for each app + hacluster used:

  manila-ganesha-az1:
    charm: local:bionic/manila-ganesha-3
    num_units: 2

  manila-ganesha-az2:
    charm: local:bionic/manila-ganesha-4
    num_units: 2

  hacluster-ganesha-az1:
    charm: cs:hacluster-69

  hacluster-ganesha-az2:
    charm: cs:hacluster-69

So this bug will definitely be hit: https://bugs.launchpad.net/charm-manila-ganesha/+bug/1890401

Not having the logs from manila-ganesha units, I suspect that there will be "Cannot send after transport endpoint shutdown [Errno 108]" errors in the manila-share.log.

I started working on a fix for LP: #1890401 in https://bugs.launchpad.net/charms.openstack/+bug/1891160 but there is still some work to do on the manila-ganesha charm side itself. I think it can be done as a part of https://review.opendev.org/#/c/743212

Regarding permissions, I can see that the client name is hard-coded here to be "manila-ganesha":

https://github.com/openstack/charm-manila-ganesha/blob/64ee2a393bda3b7d64ada66d25a8ca2115c37bfb/src/lib/charm/openstack/manila_ganesha.py#L233

So this will likely to cause problems when two apps named differently are present which also needs to be addressed.

[David Coronel (davecore)]

I don't see that error (Cannot send after transport endpoint shutdown [Errno 108]) in manila-share.log on the manila-ganesha units.

The only error I see is this one when I try to use manila access-allow:

2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server [req-8914182e-b5d6-4550-865e-9a4806521357 f2ee9e8060e54d058060d220fab84088 86fcc0b3839b45029dd325641ddc2a09 - - -] Exception during message handling: cephfs.OSError: error in open /volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta: Permission denied [Errno 13]
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 187, in wrapped
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/utils.py", line 568, in wrapper
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 3554, in update_access
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server=share_server)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 283, in update_access_rules
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server=share_server)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 322, in _update_access_rules
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server=share_server
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/cephfs/driver.py", line 289, in update_access
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server=share_server)
20...

[David Coronel (davecore)]

I had to change my tenant-storage binding from internal-space to public-space on my manila-ganesha-az1 and manila-ganesha-az2 charms when I noticed the export paths were not on the right subnet. This changed the path of my new shares, so that looks good.

However I just noticed that the IP of the mount path doesn't exist on any of my manila-ganesha units. It's the vip that we configure in the bundle, but the corosync/pacemaker bits seem wrong on the manila-ganesha units (example from manila-ganesha-az2/0):

================================
$ sudo crm_mon -rf1
Stack: corosync
Current DC: juju-cf9afe-35-lxd-1 (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Fri Oct 30 21:51:28 2020
Last change: Fri Oct 16 21:22:07 2020 by hacluster via crmd on juju-cf9afe-34-lxd-1

2 nodes configured
0 resources configured

Online: [ juju-cf9afe-34-lxd-1 juju-cf9afe-35-lxd-1 ]

No resources

Migration Summary:
* Node juju-cf9afe-35-lxd-1:
* Node juju-cf9afe-34-lxd-1:
================================

So I have 2 major issues that I can see:
1) the vip isn't on any of my units
2) granting access to the shares still fails:

$ manila access-allow newtestdavidpublic4 ip 0.0.0.0/0 --access-level rw

$ manila access-list newtestdavidpublic4
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
| id | access_type | access_to | access_level | state | access_key | created_at | updated_at |
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
| 8273d442-610c-4ea8-af93-ba52bcaaa9a3 | ip | 0.0.0.0/0 | rw | error | None | 2020-10-30T21:25:14.000000 | None |
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+

And like I explained in comment #4, the files in the root nfs dir look wrong:

# ls -lan /mnt/volumes/
total 1
drwxr-xr-x 1 111 115 6 Oct 26 15:52 .
drwxrwxrwx 1 111 115 1 Oct 26 14:27 ..
-rwxr-xr-x 1 111 115 0 Oct 26 14:44 _:3b232b7e-1949-49c3-a742-7ca689f34f46.meta
-rwxr-xr-x 1 111 115 0 Oct 26 15:52 _:bfdb2ddf-c089-4947-a8b5-4245757e8794.meta
drwxr-xr-x 1 111 115 0 Oct 26 14:50 _deleting
---------- 1 111 115 196 Oct 26 15:52 '$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta'
drwxr-xr-x 1 111 115 2 Oct 26 14:50 _nogroup
-rwxr-xr-x 1 111 115 0 Oct 26 15:15 _None:bfdb2ddf-c089-4947-a8b5-4245757e8794.meta

And last thing: I didn't try the charm build with a fix in the customer build yet, will try on Monday.

[David Coronel (davecore)]

@Dmitrii: I switched (juju upgrade-charm --switch) to your charm build[1] for my manila-ganesha-az1 and -az2 apps.

The charms are active/idle, but I'm still seeing 2 issues :

1) the manila-ganesha VIPs are not running anywhere, so I'll never be able to mount those shares anywhere.

2) the granting of access to IP addresses returns an error

[1]https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-manila-ganesha/743212/11/19850/manila-ganesha-charm_build-36207.tar.bz2

[David Coronel (davecore)]

I was able to mount a share by redeploying with only 1 unit of manila and 1 unit of ganesha. I also added a missing policy routing relation and recreated the share type. I'm not sure which one of these things allowed me to mount the share inside the VM:

# Built charm with Dmitrii's fix
git clone https://github.com/openstack/charm-manila-ganesha charm-manila-ganesha-with-dmitrii
cd charm-manila-ganesha-with-dmitrii
git fetch https://review.opendev.org/openstack/charm-manila-ganesha refs/changes/12/743212/11
git checkout FETCH_HEAD
tox -e build

# Deploy charms with only 1 unit of Manila and Ganesha

juju deploy /home/ubuntu/preprod/config/charms/manila-ganesha-with-dmitrii --bind "internal-space admin=internal-space ceph=ceph-access-space cluster=internal-space ha=oam-space public=public-space tenant-storage=public-space" \
--config worker-multiplier=0.25 \
--config openstack-origin="cloud:bionic-train" \
--config use-internal-endpoints=True \
--config vip="172.31.248.29 172.31.246.33" \
--config os-admin-hostname="ganesha1-i.<domain name>.com" \
--config os-internal-hostname="ganesha1-i.<domain name>.com" \
--config os-public-hostname="ganesha1.<domain name>.com" -n 1 --to lxd:11 \
manila-ganesha

juju deploy cs:manila --bind "internal-space admin=internal-space cluster=internal-space ha=oam-space public=public-space" \
--config worker-multiplier=0.25 \
--config openstack-origin="cloud:bionic-train" \
--config region=<region name> \
--config vip="172.31.248.28 172.31.246.32" \
--config default-share-backend=cephfsnfs1 \
--config share-protocols=NFS \
--config use-internal-endpoints=True \
--config ssl_ca="$(base64 /home/ubuntu/preprod/secrets/certs/cacert.pem)" \
--config ssl_cert="$(base64 /home/ubuntu/preprod/secrets/certs/<domain name>.crt)" \
--config ssl_key="$(base64 /home/ubuntu/preprod/secrets/certs/<domain name>.key)" \
--config os-admin-hostname=manila-i.<domain name>.com \
--config os-internal-hostname=manila-i.<domain name>.com \
--config os-public-hostname=manila.<domain name>.com -n 1 --to lxd:0 \
manila

juju add-relation ceph-mon-az1:client manila-ganesha:ceph
juju add-relation manila-ganesha:shared-db mysql:shared-db
juju add-relation manila-ganesha:amqp rabbitmq-server:amqp
juju add-relation manila-ganesha:identity-service keystone:identity-credentials
juju add-relation manila:remote-manila-plugin manila-ganesha:manila-plugin
juju add-relation manila:amqp rabbitmq-server:amqp
juju add-relation manila:identity-service keystone:identity-service
juju add-relation manila:shared-db mysql:shared-db

juju deploy cs:hacluster --bind "internal-space" --config cluster_count=1 hacluster-manila
juju deploy cs:hacluster --bind "internal-space" --config cluster_count=1 hacluster-ganesha

juju add-relation manila:ha hacluster-manila:ha
juju add-relation manila-ganesha:ha hacluster-ganesha:ha

# Create type
manila type-create cephfsnfstype false
manila type-key cephfsnfstype set vendor_name=Ceph storage_protocol=NFS

# Create share
manila create --share-type cephfsnfstype --name cephnfsshare1 nfs 1

# Give access to VM floating IP
manila access-allow cephnfsshare1 ip 172.10.10.17 --access-level rw

# Add relation between manila-ganesha an...

[David Coronel (davecore)]

subscribed ~field-high instead of field-critical since I have a workaround

[Corey Bryant (corey.bryant)]

I think this patch is about to get merged, but please be aware of the following bug which will stil exist for the time being: https://bugs.launchpad.net/charm-manila-ganesha/+bug/1904921

[Corey Bryant (corey.bryant)]

Charm patch can be found at: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/743212

[Corey Bryant (corey.bryant)]

https://review.opendev.org/c/openstack/charm-manila-ganesha/+/743212 is now merged into master

[Corey Bryant (corey.bryant)]

It looks like we didn't add this bug to the commit message so it won't automatically get updated by gerrit.

[Vern Hart (vern)]

This fix appears to be Fix-Released in the current manila-ganesha charm.