Rsync replicator handoff partition cleanup delete can race

[jinwen]

At the same time, another object was uploaded successfully in handoff device, but this time the service object_replicator completed the sync operation and the partition handoff folder is to be deleted.
as result, the another new object will be deleted and object not found.

[jinwen (zwen45)]

1024 partitions, 1.000000 replicas, 1 regions, 1 zones, 2 devices, 0.55 balance, 0.00 dispersion
The minimum number of hours before a partition can be reassigned is 0 (0:00:00 remaining)
The overload factor is 0.00% (0.000000)
Ring file /srv/swift_rings/object-1.ring.gz is up-to-date
Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta
            0 3232270448 3232270448 10.0.14.112:6200 10.0.15.112:6250 sdf 3725.00 959 -0.04
            1 3232270448 3232270448 10.0.14.112:6200 10.0.15.112:6250 sdh1 251.00 65 0.55

[clayg (clay-gerrard)]

I could imagine the handoff partition cleanup after an rsync being racy, and with a single replica ring there's not really any durability to lean on.

Can you reproduce the issue using ssync?

[jinwen (zwen45)]

Thank you for your reply. Although I did not reproduce this issue using ssync, I think it worked.
Will it only delete the object_hash[-3:] directory, not the partition directory using ssync?

[clayg (clay-gerrard)]

i think ssync delete's individual hashes:

https://github.com/openstack/swift/blob/master/swift/obj/replicator.py#L552