Write timeout bug in proxy server

[gholt]

The proxy server during object puts where a write timeout to an object server will catch the exception, log it, and then proceed to send the next chunk to that object server later. This behavior can cause the object server to return an odd etag causing a 422 later from the proxy. We should close that object server connection on a write timeout and return a 5xx when we have too few connections left (< replicas / 2).

[clayg (clay-gerrard)]

In [6]: mylist = [1,2,3]

in the proxy the stale connection is being "removed" on error, but it causes one of the remaining connections to miss a chunk.

In [7]: mybytes = ['a', 'b', 'c']

In [8]: for byte in mybytes:
   ...: for conn in mylist:
   ...: if conn == 1:
   ...: mylist.remove(conn)
   ...: print byte, conn
   ...:
   ...:
a 1
a 3
b 2
b 3
c 2
c 3

[clayg (clay-gerrard)]

$ bzr diff
=== modified file 'swift/proxy/server.py'
--- swift/proxy/server.py 2010-09-10 14:52:10 +0000
+++ swift/proxy/server.py 2010-10-04 20:20:31 +0000
@@ -692,7 +692,7 @@
                 req.bytes_transferred += len_chunk
                 if req.bytes_transferred > MAX_FILE_SIZE:
                     return HTTPRequestEntityTooLarge(request=req)
- for conn in conns:
+ for conn in conns[:]:
                     try:
                         with ChunkWriteTimeout(self.app.node_timeout):
                             if req.headers.get('transfer-encoding'):