Object replicator report a failure when suffixes is empty

[Seongsoo Cho]

After upgrade from ocata to train, the object-replicator suddenly report a replication fail.
The failure does not always occur. Sometimes it success to replicate all partition and sometimes fail.

I try to trace the code that why the replicator report fails.

In update function, there are mismatched suffxies between local and remote.
( https://github.com/openstack/swift/blob/stable/train/swift/obj/replicator.py#L672 )

Ann then the replicator recalculate the mismatched suffixes's hash and compare it again, the result of suffixes is empty list.

So, the rsync function return False because of the empty suffixes ( https://github.com/openstack/swift/blob/stable/train/swift/obj/replicator.py#L451 ) and the replicator report it to failure_devs_info ( https://github.com/openstack/swift/blob/stable/train/swift/obj/replicator.py#L694 )

I don't know why this happens, but it's been happening since the upgrade.

Have you ever experienced a similar situation?

[Tim Burke (1-tim-z)]

Maybe confirmed in https://bugs.launchpad.net/swift/+bug/1886782 ?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741538
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=907942eb4743e5a4213b1f84fd57a58386594bb6
Submitter: Zuul
Branch: master

commit 907942eb4743e5a4213b1f84fd57a58386594bb6
Author: Tim Burke <email address hidden>
Date: Thu Jul 16 13:04:00 2020 -0700

    Stop syncing empty suffixes list

    Change-Id: I918ab4ccbf4d081b26f4117937410cdad1caf8d3
    Closes-Bug: #1862645
    Closes-Bug: #1886782

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/743279

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743279
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0bdbbc4a655fb4f74cd087636fc136c0f9d57754
Submitter: Zuul
Branch: stable/ussuri

commit 0bdbbc4a655fb4f74cd087636fc136c0f9d57754
Author: Tim Burke <email address hidden>
Date: Thu Jul 16 13:04:00 2020 -0700

    Stop syncing empty suffixes list

    Change-Id: I918ab4ccbf4d081b26f4117937410cdad1caf8d3
    Closes-Bug: #1862645
    Closes-Bug: #1886782
    (cherry picked from commit 907942eb4743e5a4213b1f84fd57a58386594bb6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/743614

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743614
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=69225ada606c00e5a68d82c297c7c5c100bf33ab
Submitter: Zuul
Branch: stable/train

commit 69225ada606c00e5a68d82c297c7c5c100bf33ab
Author: Tim Burke <email address hidden>
Date: Thu Jul 16 13:04:00 2020 -0700

    Stop syncing empty suffixes list

    Change-Id: I918ab4ccbf4d081b26f4117937410cdad1caf8d3
    Closes-Bug: #1862645
    Closes-Bug: #1886782
    (cherry picked from commit 907942eb4743e5a4213b1f84fd57a58386594bb6)