[doc] Stale pools data not removed from cache after timeout

[Vida Haririan]

Stale pools data is not removed from cache after timeout period. The old aggregate entry still shows in manila.log file.

Steps:
- Change an existing aggregate name from aggrx_n1 to aggry_n1
- Wait for cache to refresh, 60 sec
- Show manila.log
- Expect: Manila log to only show the new aggregate, aggry_n1
- Actual: Both aggrx_n1 to aggry_n1 are still stored in cache

Reproduced 100%

[Vida Haririan (vhariria)]

Additional comments http://eavesdrop.openstack.org/meetings/manila/2021/manila.2021-02-11-15.00.log.html

[Carlos Eduardo (silvacarlose)]

Nahim will help trying to reproduce this in a NetApp storage.

[Nahim Alves de Souza (nahimsouza)]

Hi Vida, we were able to reproduce this bug. Using a NetApp backend, when the aggregate name is changed, nothing changes on the pool-list. The old aggregate remains on the list and the new aggregate is not shown, unless manila service is restarted.

However, we need to discuss this behavior before making any changes. Do you know any customer that reported this bug?

Our concern is the impact that this change can bring to the driver behavior, because renaming the aggregate could break the existing shares on the pool that was renamed and today customers normally restart the service whenever they need to change the backend configuration.

I would like to hear your opinion about that.
And also, Goutham/Carlos, what do you think?

[Vida Haririan (vhariria)]

Hi Nahim, Thank you for clarifying the reproduce steps. I am not aware of an incident, and will discuss with Goutham/Carlos and provide feedback.

[Carlos Eduardo (silvacarlose)]

Thank you for getting back with more information, Nahim. I am currently confused with all of the pool renaming thing. I was checking the database and realized that the host is stored in a share as a string [1]. In case the pool name changes and there are shares created, we would need to modify that host information on the shares as well, considering that some operations will get the host and pool information.
If we take as an example the shares migration or replication: we forward to the driver the current host and pool that hosts the share, and the driver might need to connect to the pool or the host. In that case, I believe that could break some operations as well. Did you try replication, share migration or creating shares from snapshots in different pools/backends?
I am not aware of any issues hit by customers in the recent past, but if the issue I mentioned above occurs, then it might be a big issue that will block some operations with shares. Renaming a pool is not something people would do very often, so this might be the reason why we have not seen some cases like this in the past.
It is good to hear that the restart of the service gets the new information though :) - I'd say we'd need to be aware if it will break other operations then.

[Goutham Pacha Ravi (gouthamr)]

IMHO, renaming a backend resource is a breaking operation for the reasons you mentioned - i.e., the pool info is stored on manila and used by the driver to locate resources. One way the driver can support this in an automated manned is through ensure_shares - and that currently is kicked off only with bouncing the manila-share service. A challenge here though is something needs to tell the driver that the pool names have changed from underneath it - the driver has so far been stateless in this regard.

I think the decent solution for folks that may attempt renaming NetApp storage aggregates is by documenting this procedure:

1) Rename aggregate
2) Use manila-manage to update share hosts (``manila-manage share update_host [-h] --currenthost CURRENTHOST --newhost NEWHOST [--force FORCE]``)
3) Restart the manila-share service

[Nahim Alves de Souza (nahimsouza)]

Thank you all for the inputs.

Goutham's suggestion sounds good to me - we could document the procedure to inform the user/customer how this renaming operation should be done.