Fail to use manila with cephfs NFS share backend

[Boxiang Zhu]

**Environment**:
* OS:
NAME="CentOS Linux"
VERSION="7 (Core)"

* Kernel:
3.10.0-1062.4.1.el7.x86_64

* Docker version:
Client: Docker Engine - Community
 Version: 19.03.4

* Kolla-Ansible version:
master

* Docker image Install type (source/binary):
source

* something enabled in globals.yml:
enable_ceph: "yes"
enable_ceph_mds: "yes"
enable_ceph_nfs: "yes"
enable_manila: "yes"
enable_manila_backend_cephfs_nfs: "yes"

* some configs in /etc/kolla/manila-share/manila.conf:
[DEFAULT]
enabled_share_protocols = NFS,CIFS
......
[cephfsnfs1]
driver_handles_share_servers = False
share_backend_name = CEPHFSNFS1
share_driver = manila.share.drivers.cephfs.driver.CephFSDriver
cephfs_protocol_helper_type = NFS
cephfs_conf_path = /etc/ceph/ceph.conf
cephfs_auth_id = manila
cephfs_cluster_name = ceph
cephfs_enable_snapshots = False
cephfs_ganesha_server_is_remote = False
cephfs_ganesha_server_ip = 172.16.60.84

I use CLI to create the nfs, some commands as followed:
    -> manila type-create cephfsnfstype false
    -> manila type-key cephfsnfstype set vendor_name=Ceph storage_protocol=NFS
    -> manila create --share-type cephfsnfstype --name cephnfsshare1 nfs 1
    -> manila share-export-location-list cephnfsshare1
    +-----------------------------------------------+------------------------------------------------------------------------------------+-----------+
    | ID | Path | Preferred |
    +-----------------------------------------------+------------------------------------------------------------------------------------+-----------+
    | b101bf59-4cd7-4a09-a12e-b6dd48a5bb18 | 172.16.60.84:/volumes/_nogroup/93b1e23d-0166-41a4-a12a-51bf4c3654a5 | False |
    +-----------------------------------------------+------------------------------------------------------------------------------------+-----------+
    -> manila access-allow cephnfsshare1 ip 172.16.60.119

But I have got some error messages from /var/lib/docker/volumes/kolla_logs/_data/manila/manila-share.log

2019-11-11 10:11:12.035 26 ERROR manila.share.drivers.ganesha.manager [req-673584f9-eaed-4464-819e-ace63dfa0f49 6f4bd93b18c546c880d5d4e526a0243e 70dba7786a8b4326a35e03d0ad8707f2 - - -] Error while executing management command on Ganesha node <no name>: dbus call exportmgr.AddExport.: ProcessExecutionError: Unexpected error while running command.
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server [req-673584f9-eaed-4464-819e-ace63dfa0f49 6f4bd93b18c546c880d5d4e526a0243e 70dba7786a8b4326a35e03d0ad8707f2 - - -] Exception during message handling: GaneshaCommandFailure: Ganesha management command failed.
Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-93b1e23d-0166-41a4-a12a-51bf4c3654a5.conf string:EXPORT(Export_Id=105)
Exit code: 1
Stdout: u''
Stderr: u'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n'
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/manager.py", line 187, in wrapped
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/utils.py", line 568, in wrapper
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/manager.py", line 3554, in update_access
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server=share_server)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/access.py", line 283, in update_access_rules
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server=share_server)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/access.py", line 322, in _update_access_rules
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server=share_server
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py", line 289, in update_access
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server=share_server)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/drivers/ganesha/__init__.py", line 308, in update_access
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server self.ganesha.add_export(share['name'], confdict)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/drivers/ganesha/manager.py", line 491, in add_export
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server cmd=e.cmd)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server GaneshaCommandFailure: Ganesha management command failed.
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-93b1e23d-0166-41a4-a12a-51bf4c3654a5.conf string:EXPORT(Export_Id=105)
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Exit code: 1
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Stdout: u''
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Stderr: u'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n'
2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server

I found some possible solutions[0][1][2][3] to try to fix this, but failed.

[0] https://github.com/nfs-ganesha/nfs-ganesha/issues/483
[1] https://github.com/nfs-ganesha/nfs-ganesha/issues/219
[2] https://github.com/gluster/storhaug/issues/14
[3] https://sourceforge.net/p/nfs-ganesha/mailman/message/32227132/

[Boxiang Zhu (bxzhu-5355)]

In "/var/lib/docker/volumes/kolla_logs/_data/ceph/ceph-nfs.log", an error msg like this:

ganesha.nfsd-6[main] gsh_dbus_pkginit :DBUS :CRIT :server bus reg failed (org.ganesha.nfsd, Connection ":1.206" is not allowed to own the service "org.ganesha.nfsd" due to security policies in the configuration file)

[Radosław Piliszek (yoctozepto)]

Boxiang via mail:

> BTW, in fact, before I met the problem, I found anther issue from ceph-nfs.log.
> The ganesha can not connect to the /run/dbus/system_bus_socket, So I change
> the kolla-ansible/share/kolla-ansible/ansible/roles/ceph/tasks/start_nfss.yml, add
> `` - "/run/:/run/:shared" `` under volumes section.

[Radosław Piliszek (yoctozepto)]

Kolla Ansible master is Train atm.

[Radosław Piliszek (yoctozepto)]

What release of Kolla images did you use? Please share the rest of globals.yml file and any other configs that may be related.

[Boxiang Zhu (bxzhu-5355)]

For this:

Radosław Piliszek (yoctozepto) wrote 17 hours ago: #2
Boxiang via mail:

> BTW, in fact, before I met the problem, I found anther issue from ceph-nfs.log.
> The ganesha can not connect to the /run/dbus/system_bus_socket, So I change
> the kolla-ansible/share/kolla-ansible/ansible/roles/ceph/tasks/start_nfss.yml, add
> `` - "/run/:/run/:shared" `` under volumes section.

I added another bug: https://bugs.launchpad.net/kolla-ansible/+bug/1852159

[Boxiang Zhu (bxzhu-5355)]

(.openstack) [root@zbx-allinone ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kolla/centos-source-nova-compute master efc2fa6c135d 20 hours ago 1.99GB
kolla/centos-source-cinder-volume master 3ddddc852ed6 20 hours ago 1.24GB
kolla/centos-source-cinder-backup master 16925efdd356 20 hours ago 1.24GB
kolla/centos-source-cinder-api master 2cbae0ac133f 20 hours ago 1.23GB
kolla/centos-source-neutron-server master 34e4f2401ad9 20 hours ago 1.17GB
kolla/centos-source-cinder-scheduler master 0fc10f23f6d3 21 hours ago 1.17GB
kolla/centos-source-neutron-l3-agent master 4be9df7d5e51 21 hours ago 1.19GB
kolla/centos-source-neutron-metadata-agent master 78fa894e237b 21 hours ago 1.15GB
kolla/centos-source-neutron-dhcp-agent master deee59abb186 21 hours ago 1.15GB
kolla/centos-source-neutron-openvswitch-agent master fd879e19507c 21 hours ago 1.15GB
kolla/centos-source-manila-api master 0c91267b2c9b 21 hours ago 1.07GB
kolla/centos-source-manila-data master 2abf47ad5f8b 21 hours ago 1.08GB
kolla/centos-source-heat-api master d8d2be9d86d0 21 hours ago 1.06GB
kolla/centos-source-heat-api-cfn master 090b9894f3b6 21 hours ago 1.06GB
kolla/centos-source-heat-engine master ef3cd2486319 21 hours ago 1.06GB
kolla/centos-source-manila-share master fa8140790bac 21 hours ago 1.14GB
kolla/centos-source-nova-api master 5c54c61ab35f 21 hours ago 1.23GB
kolla/centos-source-horizon master 58c4df7eee24 21 hours ago 1.18GB
kolla/centos-source-nova-ssh master d93090dbd161 21 hours ago 1.2GB
kolla/centos-source-manila-scheduler master ef963814febe 21 hours ago 1.01GB
kolla/centos-source-nova-conductor master fdf00c7f186b 21 hours ago 1.17GB
kolla/centos-source-nova-novncproxy master 2127672eecb8 21 hours ago 1.2GB
kolla/centos-source-placement-api master 090412a52034 21 hours ago 1.07GB
kolla/centos-source-nova-scheduler master 115ec3b9ab1d 21 hours ago 1.17GB
kolla/centos-source-keystone-fernet master 23cf1b6da3e3 21 hours ago 1.07GB
kolla/centos-source-keystone master 950b374ba646 21 hours ago 1.07GB
kolla/centos-source-keystone-ssh m...

[Michal Nasiadka (mnasiadka)]

Ceph deployment support in Kolla-Ansible has been removed in Ussuri release, please move to external ceph.

[Jacolex (jacolex)]

Hello
The bug is not related to the ceph deployment, but to CephFS over Ganesha in Manila. It's still not works.
The reason is that it's trying to reach ganesha, which in my opinion should be installed somewhere on host next to manila_share container (maybe another conatiner is needed or as workaround - installation on host, idk yet).