slow replica snapshot create times

Bug #2008497 reported by Maurice Escher
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
In Progress
Medium
Unassigned

Bug Description

Hi,

I'm looking at slow transitions from 'creating' to 'available' for replica snapshots.

Even with an empty share (no data) it can take longer than two times the replica_state_update_interval (which I kept at the default of 5 minutes).

It seems the periodic task periodic_share_replica_snapshot_update is sometimes not executed: grepping "Updating status of share replica" at the debug output of manila-share,
e.g. https://paste.opendev.org/show/b1hbVAm08aL1tvSiW6Hu/
at '2023-02-24 03:42' 'Updating status of share replica snapshots' is missing. In most of the cases both log lines are aligned and appear every 5 minutes together.

Now I wonder if this is related to both methods using the same lock @locked_share_replica_operation further down on _share_replica_update and _update_replica_snapshot?

I hope that an easy workaround would be to first introduce a different interval - like replica_snapshot_state_update_interval, and set to a different value so the times are not always colliding.

Additionally I think that maybe _update_replica_snapshot does not really need that locking. I can imagine it was just copy pasted when it was introduced, or added 'to be on the safe side'...
Because looking at all the guards and errors that are handled, it should(tm) be safe to remove that locking or maybe introduce a new lock just for operations on replica snapshots?

What do you think? Am I misinterpreting something? How to speed up replica snapshot create times?

Cheers,
Maurice

Tags: replication
Vida Haririan (vhariria)
tags: added: replication
Revision history for this message
Maurice Escher (maurice-escher) wrote :

The problem does not come from locking, periodic tasks itself seem flawed to me, see https://bugs.launchpad.net/oslo.service/+bug/2011587

Vida Haririan (vhariria)
Changed in manila:
importance: Undecided → Medium
Revision history for this message
Vida Haririan (vhariria) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/877927

Changed in manila:
status: New → In Progress
Revision history for this message
Vida Haririan (vhariria) wrote :

See related comments in manila-dalmatian-m2-bugsquash ethercal https://ethercalc.net/o2qsmjpagsfb

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.