I have configured manila with cephfs backend. I have enabled multiple-backend in the manila-share node. My config file in manila-share node be flowing:
[DEFAULT]
transport_url = rabbit://openstack:123@10.5.10.190
default_share_type = default_share_type
rootwrap_config = /etc/manila/rootwrap.conf
auth_strategy = keystone
my_ip = 10.5.10.197
enabled_share_protocols = NFS,CIFS,CEPHFS
enabled_share_backends = cephfs,cephfs1
...
[cephfs]
driver_handles_share_servers = False
share_backend_name = cephfs
share_driver = manila.share.drivers.cephfs.driver.CephFSDriver
cephfs_protocol_helper_type = NFS
cephfs_volume_path_prefix = /
cephfs_conf_path = /etc/ceph/ceph.conf
cephfs_auth_id = manila
cephfs_cluster_name = ceph
cephfs_enable_snapshots = False
cephfs_ganesha_server_is_remote = True
cephfs_ganesha_server_username = root
cephfs_ganesha_server_password = 1
cephfs_ganesha_server_ip = 10.5.10.192
[cephfs1]
driver_handles_share_servers = False
share_backend_name = cephfs1
share_driver = manila.share.drivers.cephfs.driver.CephFSDriver
cephfs_protocol_helper_type = NFS
cephfs_volume_path_prefix = /
cephfs_conf_path = /etc/ceph/ceph.conf
cephfs_auth_id = manila
cephfs_cluster_name = ceph
cephfs_enable_snapshots = False
cephfs_ganesha_server_is_remote = True
cephfs_ganesha_server_username = root
cephfs_ganesha_server_password = 1
cephfs_ganesha_server_ip = 10.5.10.201
I just create a new share successfully with one cephfs backend. When I create a new share with cpehfs1 backend, i got error log:
2018-12-04 15:25:19.618 3459 ERROR manila.share.manager [req-aadc5886-1931-4648-acaa-eda51bba50f0 396c8615aea9403daa392b47fa33f4a7 63011a3519e94cbc821ad6e4247a8025 - - -] Share instance a746281c-66a9-453b-875f-033a60efa956 failed on creation.: Error: (108, 'error in stat: /_nogroup/a746281c-66a9-453b-875f-033a60efa956: error code 108')
2018-12-04 15:25:19.618 3459 WARNING manila.share.manager [req-aadc5886-1931-4648-acaa-eda51bba50f0 396c8615aea9403daa392b47fa33f4a7 63011a3519e94cbc821ad6e4247a8025 - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: Error: (108, 'error in stat: /_nogroup/a746281c-66a9-453b-875f-033a60efa956: error code 108')
2018-12-04 15:25:19.647 3459 INFO manila.message.api [req-aadc5886-1931-4648-acaa-eda51bba50f0 396c8615aea9403daa392b47fa33f4a7 63011a3519e94cbc821ad6e4247a8025 - - -] Creating message record for request_id = req-aadc5886-1931-4648-acaa-eda51bba50f0
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server [req-aadc5886-1931-4648-acaa-eda51bba50f0 396c8615aea9403daa392b47fa33f4a7 63011a3519e94cbc821ad6e4247a8025 - - -] Exception during message handling: Error: (108, 'error in stat: /_nogroup/a746281c-66a9-453b-875f-033a60efa956: error code 108')
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 187, in wrapped
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/utils.py", line 589, in wrapper
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1767, in create_share_instance
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server exception=e)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1730, in create_share_instance
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server context, share_instance, share_server=share_server)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py", line 240, in create_share
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server cephfs_share_path(share), size=size, data_isolated=data_isolated)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 620, in create_volume
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server self._mkdir_p(path)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 589, in _mkdir_p
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server self.fs.stat(path)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server File "cephfs.pyx", line 1007, in cephfs.LibCephFS.stat (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/build/src/pybind/cephfs/pyrex/cephfs.c:10672)
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server Error: (108, 'error in stat: /_nogroup/a746281c-66a9-453b-875f-033a60efa956: error code 108')
2018-12-04 15:25:19.660 3459 ERROR oslo_messaging.rpc.server
I'm sorry to be slow getting to this one but you appear to be running two ganesha servers against the same ceph cluster. It may be that this can be fixed by setting the cluster up so that each ganesha server is working with a differenet ceph_volume_path (in your configuration they are both set to '/'). In any case, the error we're getting the error from the ceph software itself, not manila code or the manila ceph driver, when there is an attempt to 'stat' the path where by convention a directory would be created for the new share.
We'll likely need to involve the ceph development team, and perhaps ganesha team, but before that maybe you can see if working with separate volume paths for each back end resolves the issue. If not, then we'll need to get more detailed topology information from you about how you have configured ganesha and ceph outside of manila in order to pursue the matter.