Ceph/NFS driver hits ganesha export ID limit

[Goutham Pacha Ravi]

Description
===========
It is not possible to create new access rules for the Manila shares backed by CephFS/NFS when the driver hits the export ID limit:

We create a new access rule:

```
$ manila access-allow testnfs1 ip 0.0.0.0
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| id | 84d1c08d-cb3a-4578-9886-a1e6be3db13f |
| share_id | 38ffd41c-18d3-4578-abdf-d93660996f38 |
| access_level | rw |
| access_to | 0.0.0.0 |
| access_type | ip |
| state | queued_to_apply |
| access_key | None |
| created_at | 2023-02-27T12:24:07.000000 |
| updated_at | None |
| metadata | {} |
+--------------+--------------------------------------+
```
But its in error state and not possible to access to the new nfs filesystem:

```
# manila access-list testnfs1
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
| id | access_type | access_to | access_level | state | access_key | created_at | updated_at |
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
| 84d1c08d-cb3a-4578-9886-a1e6be3db13f | ip | 0.0.0.0 | rw | error | None | 2023-02-27T12:24:07.000000 | None |
+--------------------------------------+-------------+-----------+--------------+-------+------------+----------------------------+------------+
```

Error from Ganesha logs:

```
...
...
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.nfsd-57[dbus_heartbeat] reclaim_reset :FSAL :EVENT :start_reclaim failed: -2
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.nfsd-57[dbus_heartbeat] gsh_export_addexport :EXPORT :CRIT :0 export entries in /etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r added because (invalid param value) errors
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.nfsd-57[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (AddExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidFileContent), message = (0 export entries in /etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r added because (invalid param value) errors. Details:
Config File (/etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r:2): (65618) is out of range
Config File (/etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r:1): 1 errors while processing parameters for EXPORT
Config File (/etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r:1): Errors found in configuration block EXPORT
```

Specifically:

 Config File (/etc/ganesha/export.d/share-fd001db5-0be5-4aa4-aade-9a2450581512.conf.bSLM9r:2): (65618) is out of range

RCA:
"Exporting" (creating a share and allowing access to it) more than 64536 NFS shares with CephFS/NFS can cause this issue. A single ganesha gateway would possibly be overwhelmed (and overwhelm the hardware it is running on) with those many shares. We don't have to keep the shares around, we could export and delete them over a period of time. The bug here is that the driver never reclaims old export IDs.

Steps to reproduce
==================
An easy way to reproduce the failure is to set the export index counter rados object to a very high number

*
echo "65536" | sudo tee -a /etc/ganesha/ganesha-export-counter
sudo rados -n client.manila -p manila_data put ganesha-export-counter /etc/ganesha/ganesha-export-counter
sudo rm /etc/ganesha/ganesha-export-counter
* Create a new share and allow-access to it

Expected result
===============
Access rules are successfully applied

Actual result
=============
Access rules transition to "error" state and the share is not exported via NFS-Ganesha.

Environment
===========
1. Version of OpenStack Manila: Train, the issue persists with master however

2. Storage backend: CephFS (NFS)

[Vida Haririan (vhariria)]

This issue has a workaround ..

Additional related discussions are at https://meetings.opendev.org/meetings/manila/2023/manila.2023-03-16-15.00.log.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/891508

[Vida Haririan (vhariria)]

See related comments in manila-dalmatian-m2-bugsquash ethercal https://ethercalc.net/o2qsmjpagsfb