ceph-nfs blocks on erroring nfs-ganesha

[Peter Sabaini]

We've seen repeated faults in CI when testing focal-quincy where the nfs-ganesha service would fail to come up.

The ganesha.log seems to indicate an issue in reading from the rados url mentioned in /etc/ganesha/ganesha.conf, rados://ceph-nfs/ganesha-export-index , it's unclear why that failed though.

03/07/2023 08:03:13 : epoch 64a280c1 : juju-df9c97-zaza-c36e477e7041-6 : ganesha.nfsd-83314[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 3.0.3
03/07/2023 08:03:13 : epoch 64a280c1 : juju-df9c97-zaza-c36e477e7041-6 : ganesha.nfsd-83315[main] cu_rados_url_fetch :CONFIG :EVENT :cu_rados_url_fetch: Failed reading ceph-nfs/ganesha-export-index Unknown error -2
03/07/2023 08:03:13 : epoch 64a280c1 : juju-df9c97-zaza-c36e477e7041-6 : ganesha.nfsd-83315[main] main :NFS STARTUP :CRIT :Error (token scan) while parsing (/etc/ganesha/ganesha.conf)
03/07/2023 08:03:13 : epoch 64a280c1 : juju-df9c97-zaza-c36e477e7041-6 : ganesha.nfsd-83315[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:75): new url (rados://ceph-nfs/ganesha-export-index) open error (Unknown error -2), ignored
03/07/2023 08:03:13 : epoch 64a280c1 : juju-df9c97-zaza-c36e477e7041-6 : ganesha.nfsd-83315[main] main :NFS STARTUP :FATAL :Fatal errors. Server exiting...

I've attempted to get more diagnostics by increasing the loglevel of the ganesha service, but this unfortunately ultimately failed

CI run:
https://openstack-ci-reports.ubuntu.com/artifacts/f73/881525/3/check/focal-quincy/f73ea4a/

SOS report from the failing unit:
https://private-fileshare.canonical.com/~sabaini/01e6ae40-be78-4c9b-bf9d-8d66ccd72642/sosreport-juju-df9c97-zaza-c36e477e7041-6-ceph-nfs-2023-07-03-zdebizc.tar.xz

[Corey Bryant (corey.bryant)]

Hi, I'm also hitting this on the following review: https://review.opendev.org/c/openstack/charm-ceph-nfs/+/888826