reconstructor may fail to rebuild frag if other frags have different *metadata* timestamps

[Alistair Coles]

When trying to rebuild a fragment missing on another node, the reconstructor makes requests for other fragments to other nodes and gathers responses in buckets according to the response X-Backend-Timestamp. A single bucket with sufficient responses is required to rebuild the missing fragment.

However, if objects have POSTed metadata then X-Backend-Timestamp is the timestamp of the .meta file, not the .data file. This can cause fragment responses with the same *data timestamp* to be placed in multiple buckets if some fragments have a .meta file in their hash dir while others do not (for example, because the POST may have landed on some handoffs).

Because frags are placed in different buckets, no single bucket may have enough frags to rebuild and so the reconstructor fails to rebuild the missing frag.

The situation would likely resolve itself once meta files have been replicated to all nodes, but there is a potential delay is increasing the durability of the data.

The buckets should be keyed by X-Backend-Data-Timestamp.

This unit test patch (against commit 46b8f941920) illustrates the bug:

    def test_reconstruct_fa_mixed_meta_timestamps_works(self):
        # verify scenario where all fragments have same data timestamp but some
        # have different meta timestamp
        job = {
            'partition': 0,
            'policy': self.policy,
        }
        part_nodes = self.policy.object_ring.get_part_nodes(0)
        node = part_nodes[4]
        node['backend_index'] = self.policy.get_backend_index(node['index'])

        test_data = (b'rebuild' * self.policy.ec_segment_size)[:-777]
        etag = md5(test_data, usedforsecurity=False).hexdigest()
        ec_archive_bodies = encode_frag_archive_bodies(self.policy, test_data)

        broken_body = ec_archive_bodies.pop(4)
        ts_data = next(self.ts_iter) # all frags .data timestamp
        ts_meta = next(self.ts_iter) # some frags .meta timestamp
        ts_cycle = itertools.cycle((ts_data, ts_meta))
        responses = list()
        for body in ec_archive_bodies:
            ts = next(ts_cycle) # vary timestamp between data and meta
            headers = get_header_frag_index(self, body)
            headers.update({'X-Object-Sysmeta-Ec-Etag': etag,
                            'X-Timestamp': ts.normal,
                            'X-Backend-Timestamp': ts.internal,
                            'X-Backend-Data-Timestamp': ts_data.internal,
                            'X-Backend-Durable-Timestamp': ts_data.internal})
            responses.append((200, body, headers))

        codes, body_iter, headers_iter = zip(*responses)
        with mocked_http_conn(*codes, body_iter=body_iter,
                              headers=headers_iter):
            df = self.reconstructor.reconstruct_fa(
                job, node, dict(self.obj_metadata))
            fixed_body = b''.join(df.reader())
            self.assertEqual(len(fixed_body), len(broken_body))
            self.assertEqual(
                md5(fixed_body, usedforsecurity=False).hexdigest(),
                md5(broken_body, usedforsecurity=False).hexdigest())

Test should pass but currently fails:

E
======================================================================
ERROR: test_reconstruct_fa_mixed_meta_timestamps_works (test.unit.obj.test_reconstructor.TestReconstructFragmentArchive)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/acoles/0dev/openstack/swift/test/unit/obj/test_reconstructor.py", line 4700, in test_reconstruct_fa_mixed_meta_timestamps_works
    job, node, dict(self.obj_metadata))
  File "/Users/acoles/0dev/openstack/swift/swift/obj/reconstructor.py", line 525, in reconstruct_fa
    raise DiskFileError('Unable to reconstruct EC archive')
swift.common.exceptions.DiskFileError: Unable to reconstruct EC archive

----------------------------------------------------------------------
Ran 1 test in 0.075s

FAILED (errors=1)
object-reconstructor ERROR: Unable to get enough responses (7/10) to reconstruct durable 10.0.0.4:1004/sde/0/a/c/o policy#0 frag#4 with ETag f0510d5479985e9523dc1688b0bc7d63 and timestamp 1620384286.00000
object-reconstructor ERROR: Unable to get enough responses (6/10) to reconstruct non-durable 10.0.0.4:1004/sde/0/a/c/o policy#0 frag#4 with ETag f0510d5479985e9523dc1688b0bc7d63 and timestamp 1620384287.00000
object-reconstructor INFO: Nothing reconstructed for 3.0994415283203125e-06 seconds.

Error
Traceback (most recent call last):
  File "/Users/acoles/.pyenv/versions/3.6.9/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/Users/acoles/.pyenv/versions/3.6.9/lib/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/Users/acoles/0dev/openstack/swift/test/unit/obj/test_reconstructor.py", line 4700, in test_reconstruct_fa_mixed_meta_timestamps_works
    job, node, dict(self.obj_metadata))
  File "/Users/acoles/0dev/openstack/swift/swift/obj/reconstructor.py", line 525, in reconstruct_fa
    raise DiskFileError('Unable to reconstruct EC archive')
swift.common.exceptions.DiskFileError: Unable to reconstruct EC archive

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/790235

[Alistair Coles (alistair-coles)]

https://review.opendev.org/c/openstack/swift/+/790235

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/swift/+/790235
Committed: https://opendev.org/openstack/swift/commit/eeaac713fd9d593336e636af19228e3c17ac8b0e
Submitter: "Zuul (22348)"
Branch: master

commit eeaac713fd9d593336e636af19228e3c17ac8b0e
Author: Alistair Coles <email address hidden>
Date: Fri May 7 11:32:48 2021 +0100

    reconstructor: gather rebuild fragments by x-data-timestamp

    Fix the reconstructor fragment rebuild to gather other fragments in
    buckets keyed by x-backend-data-timestamp rather than
    x-backend-timestamp. The former is the actual .data file timestamp;
    the latter can vary when .meta files have been written to some but not
    all fragment hash dirs, causing rebuild to fail.

    Change-Id: I8bbed8cb80b2796907492a39cd5b2d7069e1ca55
    Closes-Bug: 1927720

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift 2.28.0 release.