Occassional 503 in test_GET_with_many_missed_overwrite_will_need_handoff

[Tim Burke]

Rarely (~0.3% of the time?) test_GET_with_many_missed_overwrite_will_need_handoff will go searching for obj1 frags instead of obj2 -- the failure looks like

======================================================================
FAIL: test_GET_with_many_missed_overwrite_will_need_handoff (test.unit.proxy.controllers.test_obj.TestECObjController)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 2571, in test_GET_with_many_missed_overwrite_will_need_handoff
    self.assertEqual(resp.status_int, 200)
AssertionError: 503 != 200
-------------------- >> begin captured stdout << ---------------------
proxy-server DEBUG: Loaded override config for (default): ProxyOverrideOptions({}, {'sorting_method': 'shuffle', 'read_affinity': '', 'write_affinity': '', 'write_affinity_node_count': '2 * replicas', 'write_affinity_handoff_delete_count': None}) (txn: txn1) (client_ip: 127.0.0.2)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: Problem with fragment response: ETag mismatch (txn: tx337e229ae79e4c9c8ef99-005993438f)
proxy-server ERROR: ERROR with Object server 10.0.0.15:1015/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.15':1015 at index 15 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.16:1016/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.16':1016 at index 16 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.17:1017/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.17':1017 at index 17 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.18:1018/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.18':1018 at index 18 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.19:1019/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.19':1019 at index 19 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.20:1020/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.20':1020 at index 20 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.21:1021/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.21':1021 at index 21 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.22:1022/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.22':1022 at index 22 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.23:1023/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.23':1023 at index 23 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.24:1024/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.24':1024 at index 24 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.25:1025/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.25':1025 at index 25 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.26:1026/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.26':1026 at index 26 - but only got 15 stub response nodes
proxy-server ERROR: ERROR with Object server 10.0.0.27:1027/sda re: Trying to GET /v1/a/c/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/base.py", line 1124, in _make_node_request
    possible_source = conn.getresponse()
  File "/vagrant/swift/test/unit/proxy/controllers/test_obj.py", line 1575, in getresponse
    self.resp = get_response(self.req)
  File "/vagrant/swift/test/unit/__init__.py", line 1264, in get_response
    len(node_frags)))
Exception: Found node '10.0.0.27':1027 at index 27 - but only got 15 stub response nodes
proxy-server ERROR: Object returning 503 for [] (txn: tx337e229ae79e4c9c8ef99-005993438f)

--------------------- >> end captured stdout << ----------------------
    '503 != 200' = '%s != %s' % (safe_repr(503), safe_repr(200))
    '503 != 200' = self._formatMessage('503 != 200', '503 != 200')
>> raise self.failureException('503 != 200')

----------------------------------------------------------------------