test_manila_share sometimes fails because the share is in error state

[Aurelien Lourot]

Visible here:
https://review.opendev.org/c/openstack/charm-manila-ganesha/+/807569
https://openstack-ci-reports.ubuntu.com/artifacts/9b3/807569/10/check/focal-ussuri/9b3dcd0/job-output.txt

2021-10-21 19:11:57 [INFO] test_manila_share (zaza.openstack.charm_tests.manila_ganesha.tests.ManilaGaneshaTests)
2021-10-21 19:11:57 [INFO] Test that a Manila share can be accessed on two instances.
2021-10-21 19:11:57 [INFO] ...
2021-10-21 19:12:00 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
2021-10-21 19:12:02 [INFO] Using keystone API V3 (or later) for overcloud auth
scp: /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt: No such file or directory
2021-10-21 19:12:04 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
2021-10-21 19:12:08 [INFO] Launching instance zaza-manilatests-ins-1
2021-10-21 19:12:11 [INFO] Checking instance is active
2021-10-21 19:12:12 [INFO] resource: resource 7e7b0190-101a-46ca-9312-8ba0414f28b3 in BUILD state, waiting for ACTIVE
2021-10-21 19:12:15 [INFO] resource: resource 7e7b0190-101a-46ca-9312-8ba0414f28b3 in BUILD state, waiting for ACTIVE
2021-10-21 19:12:17 [INFO] resource: resource 7e7b0190-101a-46ca-9312-8ba0414f28b3 in BUILD state, waiting for ACTIVE
2021-10-21 19:12:21 [INFO] resource: resource 7e7b0190-101a-46ca-9312-8ba0414f28b3 in BUILD state, waiting for ACTIVE
2021-10-21 19:12:30 [INFO] resource: resource 7e7b0190-101a-46ca-9312-8ba0414f28b3 in BUILD state, waiting for ACTIVE
2021-10-21 19:12:46 [INFO] resource: resource 7e7b0190-101a-46ca-9312-8ba0414f28b3 in ACTIVE state, waiting for ACTIVE
2021-10-21 19:12:46 [INFO] Checking cloud init is complete
2021-10-21 19:14:58 [INFO] Assigning floating ip.
2021-10-21 19:14:58 [INFO] Creating floatingip
2021-10-21 19:15:01 [INFO] Assigned floating IP 172.16.196.210 to zaza-manilatests-ins-1
2021-10-21 19:15:06 [INFO] Testing ssh access.
2021-10-21 19:15:06 [INFO] Attempting to ssh to zaza-manilatests-ins-1(172.16.196.210)
2021-10-21 19:15:06 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2021-10-21 19:15:06 [INFO] Authentication (publickey) successful!
2021-10-21 19:15:06 [INFO] Running uname -n on zaza-manilatests-ins-1
2021-10-21 19:15:08 [INFO] SSH to zaza-manilatests-ins-1(172.16.196.210) successful
2021-10-21 19:15:10 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
2021-10-21 19:15:13 [INFO] Using keystone API V3 (or later) for overcloud auth
scp: /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt: No such file or directory
2021-10-21 19:15:15 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
2021-10-21 19:15:19 [INFO] Launching instance zaza-manilatests-ins-2
2021-10-21 19:15:21 [INFO] Checking instance is active
2021-10-21 19:15:21 [INFO] resource: resource 5066d62e-fb42-4972-be92-0d1f0d6128af in BUILD state, waiting for ACTIVE
2021-10-21 19:15:22 [INFO] resource: resource 5066d62e-fb42-4972-be92-0d1f0d6128af in BUILD state, waiting for ACTIVE
2021-10-21 19:15:24 [INFO] resource: resource 5066d62e-fb42-4972-be92-0d1f0d6128af in BUILD state, waiting for ACTIVE
2021-10-21 19:15:29 [INFO] resource: resource 5066d62e-fb42-4972-be92-0d1f0d6128af in BUILD state, waiting for ACTIVE
2021-10-21 19:15:37 [INFO] resource: resource 5066d62e-fb42-4972-be92-0d1f0d6128af in BUILD state, waiting for ACTIVE
2021-10-21 19:15:53 [INFO] resource: resource 5066d62e-fb42-4972-be92-0d1f0d6128af in ACTIVE state, waiting for ACTIVE
2021-10-21 19:15:53 [INFO] Checking cloud init is complete
2021-10-21 19:17:00 [INFO] Assigning floating ip.
2021-10-21 19:17:00 [WARNING] A floating IP already exists but ports do not match Potentially creating more than one.
2021-10-21 19:17:00 [INFO] Creating floatingip
2021-10-21 19:17:04 [INFO] Assigned floating IP 172.16.196.200 to zaza-manilatests-ins-2
2021-10-21 19:17:09 [INFO] Testing ssh access.
2021-10-21 19:17:09 [INFO] Attempting to ssh to zaza-manilatests-ins-2(172.16.196.200)
2021-10-21 19:17:09 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2021-10-21 19:17:09 [INFO] Authentication (publickey) successful!
2021-10-21 19:17:09 [INFO] Running uname -n on zaza-manilatests-ins-2
2021-10-21 19:17:10 [INFO] SSH to zaza-manilatests-ins-2(172.16.196.200) successful
2021-10-21 19:17:18 [INFO] Waiting for a share to become available: resource 3cb39d6c-b953-4081-ae93-db577f5ad30f in error state, waiting for available
2021-10-21 19:17:21 [INFO] Waiting for a share to become available: resource 3cb39d6c-b953-4081-ae93-db577f5ad30f in error state, waiting for available
2021-10-21 19:17:21 [INFO] FAIL

I couldn't find the root cause in the logs so far.

[Aurelien Lourot (aurelien-lourot)]

It seems like in some occurrences [1][2] the logs show:

INFO ceph_volume_client [-] disconnect
INFO manila.share.drivers.cephfs.driver [req-c6f21b2d-a1e3-4d6f-9585-964c11cfed99 - - - - -] [CEPHFSNFS1}] Ceph client found, connecting...
INFO ceph_volume_client [req-c6f21b2d-a1e3-4d6f-9585-964c11cfed99 - - - - -] evict clients with auth_name=manila-ganesha
INFO ceph_volume_client [req-c6f21b2d-a1e3-4d6f-9585-964c11cfed99 - - - - -] evict: joined all
ERROR ceph_volume_client [req-c6f21b2d-a1e3-4d6f-9585-964c11cfed99 - - - - -] Failed to evict client with auth_name=manila-ganesha from mds 0/4911: Error -13 ("") while Sending evict to mds.4911
ERROR manila.share.manager [req-c6f21b2d-a1e3-4d6f-9585-964c11cfed99 - - - - -] Error encountered during initialization of driver CephFSDriver@172.16.0.163@cephfsnfs1: ceph_volume_client.EvictionError: Failed to evict client with auth_name=manila-ganesha from mds 0/4911: Error -13 ("") while Sending evict to mds.4911
ERROR manila.share.manager Traceback (most recent call last):
ERROR manila.share.manager File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 311, in _driver_setup
ERROR manila.share.manager self.driver.do_setup(ctxt)
ERROR manila.share.manager File "/usr/lib/python3/dist-packages/manila/share/drivers/cephfs/driver.py", line 155, in do_setup
ERROR manila.share.manager ceph_vol_client=self.volume_client)
ERROR manila.share.manager File "/usr/lib/python3/dist-packages/manila/share/drivers/cephfs/driver.py", line 232, in volume_client
ERROR manila.share.manager self._volume_client.connect(premount_evict=premount_evict)
ERROR manila.share.manager File "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 526, in connect
ERROR manila.share.manager self._connect(premount_evict)
ERROR manila.share.manager File "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 496, in _connect
ERROR manila.share.manager self.evict(premount_evict)
ERROR manila.share.manager File "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 468, in evict
ERROR manila.share.manager raise EvictionError(msg)
ERROR manila.share.manager ceph_volume_client.EvictionError: Failed to evict client with auth_name=manila-ganesha from mds 0/4911: Error -13 ("") while Sending evict to mds.4911

But not always, so we may have two different issues.

[1] https://review.opendev.org/c/openstack/charm-manila-ganesha/+/814768
[2] https://openstack-ci-reports.ubuntu.com/artifacts/22a/814768/1/check/ganesha-focal-ussuri-non-ha/22ad9ed/