This happens with with manila 3.0.1dev27 (Newton release) on SLE12SP2.
# Problem
when starting manila-share (configured with cephfs as backend) and there is a problem in init_host(), the share service does not retry or crash which makes manila-share unusable.
# Reproducer
- configure a cephfs backend and set "cephfs_conf_path" to a file that does not exist (invalid config)
- start manila-share
Now it will be stuck (beside the periodic jobs that fill the logs slowly) with:
2017-05-11 14:43:24.870 31628 DEBUG oslo_service.service [req-5ff58f67-4813-41a3-9ee9-75cd9bcc7d97 - - - - -] database.use_db_reconnect = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2678
2017-05-11 14:43:24.871 31628 DEBUG oslo_service.service [req-5ff58f67-4813-41a3-9ee9-75cd9bcc7d97 - - - - -] ******************************************************************************** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2680
2017-05-11 14:43:25.107 31637 DEBUG manila.service [req-50a9c741-1ac5-4c35-896a-d43fc5f88626 - - - - -] Creating RPC server for service manila-share. start /usr/lib/python2.7/site-packages/manila/service.py:110
2017-05-11 14:43:25.132 31637 DEBUG manila.share.manager [req-50a9c741-1ac5-4c35-896a-d43fc5f88626 - - - - -] Start initialization of driver: 'CephFSNativeDriver@d52-54-77-77-01-01@backend-cephfs-0' init_host /usr/lib/python2.7/site-packages/manila/share/manager.py:266
2017-05-11 14:43:25.133 31637 INFO manila.share.drivers.cephfs.cephfs_native [req-50a9c741-1ac5-4c35-896a-d43fc5f88626 - - - - -] [ceph}] Ceph client found, connecting...
2017-05-11 14:43:25.133 31637 DEBUG ceph_volume_client [req-50a9c741-1ac5-4c35-896a-d43fc5f88626 - - - - -] Connecting to RADOS with config /etc/ceph/ceph.conf2... connect /usr/lib/python2.7/site-packages/ceph_volume_client.py:451
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager [req-50a9c741-1ac5-4c35-896a-d43fc5f88626 - - - - -] Error encountered during initialization of driver 'CephFSNativeDriver' on 'd52-54-77-77-01-01@backend-cephfs-0' host. error calling conf_read_file: error code 22
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager Traceback (most recent call last):
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 269, in init_host
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager self.driver.check_for_setup_error()
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 88, in check_for_setup_error
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager self.volume_client
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 156, in volume_client
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager self._volume_client.connect(premount_evict=premount_evict)
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 456, in connect
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager conf={}
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "rados.pyx", line 525, in rados.Rados.__init__ (/home/abuild/rpmbuild/BUILD/ceph-10.2.6+git.1490339825.57146d8/src/build/rados.c:6081)
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (/home/abuild/rpmbuild/BUILD/ceph-10.2.6+git.1490339825.57146d8/src/build/rados.c:4334)
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "rados.pyx", line 568, in rados.Rados.__setup (/home/abuild/rpmbuild/BUILD/ceph-10.2.6+git.1490339825.57146d8/src/build/rados.c:6966)
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (/home/abuild/rpmbuild/BUILD/ceph-10.2.6+git.1490339825.57146d8/src/build/rados.c:4334)
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager File "rados.pyx", line 631, in rados.Rados.conf_read_file (/home/abuild/rpmbuild/BUILD/ceph-10.2.6+git.1490339825.57146d8/src/build/rados.c:8104)
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager Error: error calling conf_read_file: error code 22
2017-05-11 14:43:25.134 31637 ERROR manila.share.manager
2017-05-11 14:43:25.136 31637 INFO ceph_volume_client [req-50a9c741-1ac5-4c35-896a-d43fc5f88626 - - - - -] disconnect
2017-05-11 14:45:17.140 31637 DEBUG oslo_service.periodic_task [req-63b66bec-d1c4-419b-9dd9-a698ffd70179 - - - - -] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2017-05-11 14:45:17.140 31637 DEBUG oslo_service.periodic_task [req-63b66bec-d1c4-419b-9dd9-a698ffd70179 - - - - -] Running periodic task ShareManager.migration_driver_continue run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2017-05-11 14:45:17.141 31637 ERROR oslo_service.periodic_task [req-63b66bec-d1c4-419b-9dd9-a698ffd70179 - - - - -] Error during ShareManager.migration_driver_continue
2017-05-11 14:45:17.141 31637 ERROR oslo_service.periodic_task Traceback (most recent call last):
2017-05-11 14:45:17.141 31637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2017-05-11 14:45:17.141 31637 ERROR oslo_service.periodic_task task(self, context)
2017-05-11 14:45:17.141 31637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/manila/utils.py", line 616, in wrapper
2017-05-11 14:45:17.141 31637 ERROR oslo_service.periodic_task raise exception.DriverNotInitialized(driver=driver_name)
2017-05-11 14:45:17.141 31637 ERROR oslo_service.periodic_task DriverNotInitialized: Share driver 'CephFSNativeDriver' not initialized.
2017-05-11 14:45:17.141 31637 ERROR oslo_service.periodic_task
2017-05-11 14:45:17.142 31637 DEBUG oslo_service.periodic_task [req-63b66bec-d1c4-419b-9dd9-a698ffd70179 - - - - -] Running periodic task ShareManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2017-05-11 14:45:17.142 31637 ERROR oslo_service.periodic_task [req-63b66bec-d1c4-419b-9dd9-a698ffd70179 - - - - -] Error during ShareManager._report_driver_status
2017-05-11 14:45:17.142 31637 ERROR oslo_service.periodic_task Traceback (most recent call last):
2017-05-11 14:45:17.142 31637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2017-05-11 14:45:17.142 31637 ERROR oslo_service.periodic_task task(self, context)
2017-05-11 14:45:17.142 31637 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/manila/utils.py", line 616, in wrapper
2017-05-11 14:45:17.142 31637 ERROR oslo_service.periodic_task raise exception.DriverNotInitialized(driver=driver_name)
2017-05-11 14:45:17.142 31637 ERROR oslo_service.periodic_task DriverNotInitialized: Share driver 'CephFSNativeDriver' not initialized.
2017-05-11 14:45:17.142 31637 ERROR oslo_service.periodic_task
For the record: the bugfix for https:/ /bugs.launchpad .net/manila/ +bug/1500964 introduced the problem.