Sporadic test failures in test_reconstruct_fa_with_mixed_etag_with_same_timestamp

Bug #1686571 reported by Tim Burke
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

Since we don't know the order in which connections are served, we may miss the frag with the bad etag, leading to failures like

======================================================================
FAIL: test_reconstruct_fa_with_mixed_etag_with_same_timestamp (test.unit.obj.test_reconstructor.TestReconstructFragmentArchive)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/vagrant/swift/test/unit/obj/test_reconstructor.py", line 3128, in test_reconstruct_fa_with_mixed_etag_with_same_timestamp
    self.assertEqual(1, len(error_log_lines))
AssertionError: 1 != 0
-------------------- >> begin captured stdout << ---------------------
object-reconstructor DEBUG: Reconstruct frag #1 with frag indexes [0, 2, 3, 4, 5, 6, 7, 8, 9, 10]
object-reconstructor DEBUG: Reconstruct frag #1 with frag indexes [0, 2, 3, 4, 5, 6, 9, 10, 11, 12]

--------------------- >> end captured stdout << ----------------------

Alternatively, I guess we may get stuck *expecting* the bad etag, causing failures like

======================================================================
ERROR: test_reconstruct_fa_with_mixed_etag_with_same_timestamp (test.unit.obj.test_reconstructor.TestReconstructFragmentArchive)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/vagrant/swift/test/unit/obj/test_reconstructor.py", line 3120, in test_reconstruct_fa_with_mixed_etag_with_same_timestamp
    job, node, dict(self.obj_metadata))
  File "/vagrant/swift/swift/obj/reconstructor.py", line 383, in reconstruct_fa
    raise DiskFileError('Unable to reconstruct EC archive')
DiskFileError: Unable to reconstruct EC archive
-------------------- >> begin captured stdout << ---------------------
object-reconstructor DEBUG: Reconstruct frag #1 with frag indexes [0, 2, 3, 4, 5, 6, 7, 8, 9, 10]
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Mixed Etag (f0510d5479985e9523dc1688b0bc7d63, some garbage) for 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1
object-reconstructor ERROR: Unable to get enough responses (1/10) to reconstruct 10.0.0.1:1001/sdb/0/a/c/o policy#0 frag#1 with ETag some garbage

--------------------- >> end captured stdout << ----------------------

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.