seamless reload disconnects clients poorly

[clayg]

I'm seeing some ungraceful disconnects with seamless-reload on the proxy:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
    func(*args, **kwargs)
  File "/vagrant/swift-bench/swiftbench/bench.py", line 463, in _run
    container_name, name, http_conn=conn)
  File "/vagrant/python-swiftclient/swiftclient/client.py", line 1220, in get_object
    conn.request(method, path, '', headers)
  File "/vagrant/python-swiftclient/swiftclient/client.py", line 469, in request
    files=files, **self.requests_args)
  File "/vagrant/python-swiftclient/swiftclient/client.py", line 452, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))

I'm reloading like:

vagrant@saio:~$ swift-init reload-seamless proxy
Signal proxy-server pid: 9172 signal: 10
Signal proxy-server pid: 9173 signal: 10

It's not exactly clear to me what the expected behavior is during a proxy-server reload-seamless for existing connections that are presumably trying to make pipeline requests. I suppose ideally the server would send Connection: close in the response. But if the client keeps sending requests at somepoint I suppose it would close the connection - so it's possible this is a client side issue and swift-bench bug?

[Tim Burke (1-tim-z)]

I think this is going to be unavoidable -- it'd be *nice* if we could get a `Connection: close` header out, and we might even be able to do that for writes or even account/container reads. But any connections where we've already sent headers when the SIGUSR1 was received are out of luck -- the server's hanging up and there's no good way to signal that to the client.