Comment 12 for bug 1901570

I don't see that error (Cannot send after transport endpoint shutdown [Errno 108]) in manila-share.log on the manila-ganesha units.

The only error I see is this one when I try to use manila access-allow:

2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server [req-8914182e-b5d6-4550-865e-9a4806521357 f2ee9e8060e54d058060d220fab84088 86fcc0b3839b45029dd325641ddc2a09 - - -] Exception during message handling: cephfs.OSError: error in open /volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta: Permission denied [Errno 13]
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 187, in wrapped
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/utils.py", line 568, in wrapper
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 3554, in update_access
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server=share_server)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 283, in update_access_rules
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server=share_server)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 322, in _update_access_rules
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server=share_server
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/cephfs/driver.py", line 289, in update_access
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server share_server=share_server)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/ganesha/__init__.py", line 305, in update_access
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server 'FSAL': self._fsal_hook(None, share, None)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/cephfs/driver.py", line 563, in _fsal_hook
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server tenant_id=share['project_id'])
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 977, in authorize
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server with self._auth_lock(auth_id):
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server return next(self.gen)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 847, in fn
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server fd = self.fs.open(path, os.O_CREAT, 0o755)
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server File "cephfs.pyx", line 1026, in cephfs.LibCephFS.open
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server cephfs.OSError: error in open /volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta: Permission denied [Errno 13]
2020-10-28 14:37:12.392 55344 ERROR oslo_messaging.rpc.server

This is after manually changing the ceph auth caps to:

-in AZ1:
client.manila-ganesha-az1
        key: <redacted>
        caps: [mds] allow *
        caps: [mon] allow r; allow command "osd blacklist"
        caps: [osd] allow rwx

-in AZ2:
client.manila-ganesha-az2
        key: <redacted>
        caps: [mds] allow *
        caps: [mon] allow r; allow command "osd blacklist"
        caps: [osd] allow rwx

The name of that file (/volumes/$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta) looks wrong to me when comparing it to the files in the root NFS directory:

# ls -lan /mnt/volumes/
total 1
drwxr-xr-x 1 111 115 6 Oct 26 15:52 .
drwxrwxrwx 1 111 115 1 Oct 26 14:27 ..
-rwxr-xr-x 1 111 115 0 Oct 26 14:44 _:3b232b7e-1949-49c3-a742-7ca689f34f46.meta
-rwxr-xr-x 1 111 115 0 Oct 26 15:52 _:bfdb2ddf-c089-4947-a8b5-4245757e8794.meta
drwxr-xr-x 1 111 115 0 Oct 26 14:50 _deleting
---------- 1 111 115 196 Oct 26 15:52 '$ganesha-bfdb2ddf-c089-4947-a8b5-4245757e8794.meta'
drwxr-xr-x 1 111 115 2 Oct 26 14:50 _nogroup
-rwxr-xr-x 1 111 115 0 Oct 26 15:15 _None:bfdb2ddf-c089-4947-a8b5-4245757e8794.meta