slow replica snapshot create times

[Maurice Escher]

Hi,

I'm looking at slow transitions from 'creating' to 'available' for replica snapshots.

Even with an empty share (no data) it can take longer than two times the replica_state_update_interval (which I kept at the default of 5 minutes).

It seems the periodic task periodic_share_replica_snapshot_update is sometimes not executed: grepping "Updating status of share replica" at the debug output of manila-share,
e.g. https://paste.opendev.org/show/b1hbVAm08aL1tvSiW6Hu/
at '2023-02-24 03:42' 'Updating status of share replica snapshots' is missing. In most of the cases both log lines are aligned and appear every 5 minutes together.

Now I wonder if this is related to both methods using the same lock @locked_share_replica_operation further down on _share_replica_update and _update_replica_snapshot?

I hope that an easy workaround would be to first introduce a different interval - like replica_snapshot_state_update_interval, and set to a different value so the times are not always colliding.

Additionally I think that maybe _update_replica_snapshot does not really need that locking. I can imagine it was just copy pasted when it was introduced, or added 'to be on the safe side'...
Because looking at all the guards and errors that are handled, it should(tm) be safe to remove that locking or maybe introduce a new lock just for operations on replica snapshots?

What do you think? Am I misinterpreting something? How to speed up replica snapshot create times?

Cheers,
Maurice

[Maurice Escher (maurice-escher)]

The problem does not come from locking, periodic tasks itself seem flawed to me, see https://bugs.launchpad.net/oslo.service/+bug/2011587

[Vida Haririan (vhariria)]

Additional related discussions are at https://meetings.opendev.org/meetings/manila/2023/manila.2023-03-16-15.00.log.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/877927