Ok, I'd say there's something going on here - it's probably terrible. First I uploaded a 100MB object into a replicated storage policy: $ swift stat test !$ swift stat test big.test Account: AUTH_test Container: test Object: big.test Content Type: application/octet-stream Content Length: 104857600 Last Modified: Tue, 12 Apr 2016 18:28:47 GMT ETag: 2f282b84e7e608d5852449ed940bfc51 Meta Mtime: 1460485719.000000 Accept-Ranges: bytes X-Timestamp: 1460485726.32074 X-Trans-Id: tx155cddfa10d443f6a4018-00570d3e7a Then I started to download it - *slowly* [1] Then I sang that song about dem bones while looking at my script connected to the proxy and the proxy connected to the object server: $ sudo netstat -pt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:45703 localhost:44453 ESTABLISHED 2265/python tcp 0 0 localhost:59865 localhost:http-alt TIME_WAIT - tcp 6117196 0 localhost:51880 localhost:6040 ESTABLISHED 2524/python tcp 0 0 localhost:59077 localhost:43993 ESTABLISHED 2264/python tcp 0 0 saio:ssh 10.0.2.2:50402 ESTABLISHED 2420/sshd: vagrant tcp 0 0 saio:ssh 10.0.2.2:50398 ESTABLISHED 2273/sshd: vagrant tcp 0 0 localhost:39449 localhost:6022 TIME_WAIT - tcp 0 3971935 localhost:http-alt localhost:59867 ESTABLISHED 2524/python tcp 0 0 localhost:43993 localhost:59077 ESTABLISHED 2264/python tcp 0 0 localhost:56920 localhost:46044 ESTABLISHED 2263/python tcp 3172362 0 localhost:59867 localhost:http-alt ESTABLISHED 2553/python tcp 0 0 localhost:44453 localhost:45703 ESTABLISHED 2265/python tcp 0 0 localhost:46044 localhost:56920 ESTABLISHED 2263/python tcp 0 0 localhost:11211 localhost:39119 ESTABLISHED 914/memcached tcp 0 4097781 localhost:6040 localhost:51880 ESTABLISHED 2263/python tcp 0 0 localhost:55820 localhost:6021 TIME_WAIT - tcp 0 0 localhost:39119 localhost:11211 ESTABLISHED 2524/python I had previously looked at the pids for my object servers' so I knew that 2263 with all the bytes in the Send-Q was the object server apparently servicing this request. I killed it. $ kill -9 2263 $ sudo netstat -pt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:45703 localhost:44453 ESTABLISHED 2265/python tcp 0 0 localhost:59865 localhost:http-alt TIME_WAIT - tcp 6113698 0 localhost:51880 localhost:6040 ESTABLISHED 2524/python tcp 0 0 localhost:59077 localhost:43993 ESTABLISHED 2264/python tcp 0 0 saio:ssh 10.0.2.2:50402 ESTABLISHED 2420/sshd: vagrant tcp 0 0 saio:ssh 10.0.2.2:50398 ESTABLISHED 2273/sshd: vagrant tcp 0 0 localhost:39449 localhost:6022 TIME_WAIT - tcp 0 4102901 localhost:http-alt localhost:59867 ESTABLISHED 2524/python tcp 0 0 localhost:43993 localhost:59077 ESTABLISHED 2264/python tcp 3203274 0 localhost:59867 localhost:http-alt ESTABLISHED 2553/python tcp 0 0 localhost:44453 localhost:45703 ESTABLISHED 2265/python tcp 0 0 localhost:46044 localhost:56920 TIME_WAIT - tcp 0 0 localhost:11211 localhost:39119 ESTABLISHED 914/memcached tcp 0 2657156 localhost:6040 localhost:51880 FIN_WAIT1 - tcp 0 0 localhost:55820 localhost:6021 TIME_WAIT - tcp 0 0 localhost:39119 localhost:11211 ESTABLISHED 2524/python Voila - we're now in FIN_WAIT And the object server knew what's up: Apr 12 18:33:51 saio object-6040: Removing dead child 2263 Apr 12 18:33:51 saio object-6040: Started child 2559 The client was still happily eating bytes from out of it's plugged full Recv-Q and the proxy was happily shuffling bits into it's Send-Q slowly draining the dead object's buffers: $ sudo netstat -pt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:45703 localhost:44453 ESTABLISHED 2265/python tcp 0 0 localhost:59865 localhost:http-alt TIME_WAIT - tcp 6111366 0 localhost:51880 localhost:6040 ESTABLISHED 2524/python tcp 0 0 localhost:59077 localhost:43993 ESTABLISHED 2264/python tcp 0 0 saio:ssh 10.0.2.2:50402 ESTABLISHED 2420/sshd: vagrant tcp 0 0 saio:ssh 10.0.2.2:50398 ESTABLISHED 2273/sshd: vagrant tcp 0 0 localhost:39449 localhost:6022 TIME_WAIT - tcp 0 3928980 localhost:http-alt localhost:59867 ESTABLISHED 2524/python tcp 0 0 localhost:43993 localhost:59077 ESTABLISHED 2264/python tcp 3268447 0 localhost:59867 localhost:http-alt ESTABLISHED 2553/python tcp 0 0 localhost:44453 localhost:45703 ESTABLISHED 2265/python tcp 0 0 localhost:46044 localhost:56920 TIME_WAIT - tcp 0 0 localhost:11211 localhost:39119 ESTABLISHED 914/memcached tcp 0 0 localhost:6040 localhost:51880 FIN_WAIT2 - tcp 0 0 localhost:39119 localhost:11211 ESTABLISHED 2524/python Notice 2524 is the proxy, and it's connection on :51880 is now CLOSE_WAIT and draining vagrant@saio:~$ sudo netstat -pt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:45703 localhost:44453 ESTABLISHED 2265/python tcp 5886104 0 localhost:51880 localhost:6040 CLOSE_WAIT 2524/python tcp 0 0 localhost:59077 localhost:43993 ESTABLISHED 2264/python tcp 0 0 saio:ssh 10.0.2.2:50402 ESTABLISHED 2420/sshd: vagrant tcp 0 0 saio:ssh 10.0.2.2:50398 ESTABLISHED 2273/sshd: vagrant tcp 0 3077701 localhost:http-alt localhost:59867 ESTABLISHED 2524/python tcp 0 0 localhost:43993 localhost:59077 ESTABLISHED 2264/python tcp 3212624 0 localhost:59867 localhost:http-alt ESTABLISHED 2553/python tcp 0 0 localhost:44453 localhost:45703 ESTABLISHED 2265/python tcp 0 0 localhost:46044 localhost:56920 TIME_WAIT - tcp 0 0 localhost:11211 localhost:39119 ESTABLISHED 914/memcached tcp 0 0 localhost:6040 localhost:51880 FIN_WAIT2 - tcp 0 0 localhost:39119 localhost:11211 ESTABLISHED 2524/python ... all the way down to vagrant@saio:~$ sudo netstat -pt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:45703 localhost:44453 ESTABLISHED 2265/python tcp 1560728 0 localhost:51880 localhost:6040 CLOSE_WAIT 2524/python tcp 0 0 localhost:59077 localhost:43993 ESTABLISHED 2264/python tcp 0 0 saio:ssh 10.0.2.2:50402 ESTABLISHED 2420/sshd: vagrant tcp 0 0 saio:ssh 10.0.2.2:50398 ESTABLISHED 2273/sshd: vagrant tcp 0 2946735 localhost:http-alt localhost:59867 ESTABLISHED 2524/python tcp 0 0 localhost:43993 localhost:59077 ESTABLISHED 2264/python tcp 3266361 0 localhost:59867 localhost:http-alt ESTABLISHED 2553/python tcp 0 0 localhost:44453 localhost:45703 ESTABLISHED 2265/python tcp 0 0 localhost:11211 localhost:39119 ESTABLISHED 914/memcached tcp 0 0 localhost:6040 localhost:51880 FIN_WAIT2 - tcp 0 0 localhost:39119 localhost:11211 ESTABLISHED 2524/python ... zero vagrant@saio:~$ sudo netstat -pt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:45703 localhost:44453 ESTABLISHED 2265/python tcp 0 0 localhost:59077 localhost:43993 ESTABLISHED 2264/python tcp 0 0 saio:ssh 10.0.2.2:50402 ESTABLISHED 2420/sshd: vagrant tcp 0 0 saio:ssh 10.0.2.2:50398 ESTABLISHED 2273/sshd: vagrant tcp 0 2820327 localhost:http-alt localhost:59867 ESTABLISHED 2524/python tcp 0 0 localhost:43993 localhost:59077 ESTABLISHED 2264/python tcp 3255128 0 localhost:59867 localhost:http-alt ESTABLISHED 2553/python tcp 0 0 localhost:44453 localhost:45703 ESTABLISHED 2265/python tcp 0 0 localhost:11211 localhost:39119 ESTABLISHED 914/memcached tcp 0 0 localhost:39119 localhost:11211 ESTABLISHED 2524/python ... *eventually* our slow client drains the proxy's Send-Q out: $ sudo netstat -pt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:45703 localhost:44453 ESTABLISHED 2265/python tcp 0 0 localhost:59077 localhost:43993 ESTABLISHED 2264/python tcp 0 36 saio:ssh 10.0.2.2:50402 ESTABLISHED 2420/sshd: vagrant tcp 0 0 saio:ssh 10.0.2.2:50398 ESTABLISHED 2273/sshd: vagrant tcp 0 0 localhost:http-alt localhost:59867 ESTABLISHED 2524/python tcp 0 0 localhost:43993 localhost:59077 ESTABLISHED 2264/python tcp 2175455 0 localhost:59867 localhost:http-alt ESTABLISHED 2553/python tcp 0 0 localhost:44453 localhost:45703 ESTABLISHED 2265/python tcp 0 0 localhost:11211 localhost:39119 ESTABLISHED 914/memcached tcp 0 0 localhost:39119 localhost:11211 ESTABLISHED 2524/python ... and it's own Recv-Q $ sudo netstat -pt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:45703 localhost:44453 ESTABLISHED 2265/python tcp 0 0 localhost:59077 localhost:43993 ESTABLISHED 2264/python tcp 0 0 saio:ssh 10.0.2.2:50402 ESTABLISHED 2420/sshd: vagrant tcp 0 0 saio:ssh 10.0.2.2:50398 ESTABLISHED 2273/sshd: vagrant tcp 0 0 localhost:http-alt localhost:59867 ESTABLISHED 2524/python tcp 0 0 localhost:43993 localhost:59077 ESTABLISHED 2264/python tcp 0 0 localhost:59867 localhost:http-alt ESTABLISHED 2553/python tcp 0 0 localhost:44453 localhost:45703 ESTABLISHED 2265/python tcp 0 0 localhost:11211 localhost:39119 ESTABLISHED 914/memcached tcp 0 0 localhost:39119 localhost:11211 ESTABLISHED 2524/python And... here we sit. Still connected. But not filling any bytes or reading any bytes. The proxy's already logged: Apr 12 18:35:16 saio proxy-server: 127.0.0.1 127.0.0.1 12/Apr/2016/18/35/16 GET /v1/AUTH_test/test/big.test HTTP/1.0 200 - python-swiftclient-3.0.1.dev16 AUTH_tk413f78ca5... - 21090455 - txd6b4c07c23c4404e9c2ea-00570d3f71 - 115.2363 - - 1460486001.594772100 1460486116.831044912 0 So I don't think we're going to get out of this state - the proxy is treating the client (blocked on read) as if it's going to send another pipelined request or something? Need to look into why the resuming getter isn't doing it's thing. 1. https://gist.github.com/clayg/2458a4a7e1451c75fbc5a63fcae11635