Object server should reply 507, not 500, when disk fills up

[Tim Burke]

When the client includes a Content-Length, things work as expected:

Jul 21 21:02:50 saio object-6040: 127.0.0.1 - - [21/Jul/2020:21:02:50 +0000] "PUT /sdb4/998/AUTH_test/c2/o" 507 109 "PUT http://saio:8090/v1/AUTH_test/c2/o" "txa9da7df20c7c4eba85ee9-005f1757fa" "proxy-server 31226" 0.0008 "-" 31192 1
Jul 21 21:02:50 saio proxy-server: ERROR Insufficient Storage 127.0.0.4:6040/sdb4 (txn: txa9da7df20c7c4eba85ee9-005f1757fa) (client_ip: 127.0.0.1)
Jul 21 21:02:50 saio object-6020: 127.0.0.1 - - [21/Jul/2020:21:02:50 +0000] "PUT /sdb2/998/AUTH_test/c2/o" 507 109 "PUT http://saio:8090/v1/AUTH_test/c2/o" "txa9da7df20c7c4eba85ee9-005f1757fa" "proxy-server 31226" 0.0009 "-" 31189 1
Jul 21 21:02:50 saio proxy-server: ERROR Insufficient Storage 127.0.0.2:6020/sdb2 (txn: txa9da7df20c7c4eba85ee9-005f1757fa) (client_ip: 127.0.0.1)
Jul 21 21:02:50 saio object-6030: 127.0.0.1 - - [21/Jul/2020:21:02:50 +0000] "PUT /sdb3/998/AUTH_test/c2/o" 507 109 "PUT http://saio:8090/v1/AUTH_test/c2/o" "txa9da7df20c7c4eba85ee9-005f1757fa" "proxy-server 31226" 0.0010 "-" 31191 1
Jul 21 21:02:50 saio proxy-server: ERROR Insufficient Storage 127.0.0.3:6030/sdb3 (txn: txa9da7df20c7c4eba85ee9-005f1757fa) (client_ip: 127.0.0.1)
Jul 21 21:02:50 saio object-6010: 127.0.0.1 - - [21/Jul/2020:21:02:50 +0000] "PUT /sdb1/998/AUTH_test/c2/o" 507 109 "PUT http://saio:8090/v1/AUTH_test/c2/o" "txa9da7df20c7c4eba85ee9-005f1757fa" "proxy-server 31226" 0.0009 "-" 31180 1
Jul 21 21:02:50 saio proxy-server: ERROR Insufficient Storage 127.0.0.1:6010/sdb1 (txn: txa9da7df20c7c4eba85ee9-005f1757fa)
Jul 21 21:02:50 saio proxy-server: Object PUT returning 503, 0/2 required connections (txn: txa9da7df20c7c4eba85ee9-005f1757fa) (client_ip: 127.0.0.1)
Jul 21 21:02:50 saio proxy-server: 127.0.0.1 127.0.0.1 21/Jul/2020/21/02/50 PUT /v1/AUTH_test/c2/o HTTP/1.0 503 - curl/7.58.0 - - 118 - txa9da7df20c7c4eba85ee9-005f1757fa - 0.0767 - - 1595365370.436378002 1595365370.513077021 1

But when the client sends it chunked...

Jul 21 21:08:37 saio object-6040: ERROR __call__ error with PUT /sdb4/998/AUTH_test/c2/o : [Errno 28] No space left on device (txn: tx896c3280e49242e9bba6c-005f175954)
Jul 21 21:08:37 saio object-6030: ERROR __call__ error with PUT /sdb3/998/AUTH_test/c2/o : [Errno 28] No space left on device (txn: tx896c3280e49242e9bba6c-005f175954)
Jul 21 21:08:37 saio object-6020: ERROR __call__ error with PUT /sdb2/998/AUTH_test/c2/o : [Errno 28] No space left on device (txn: tx896c3280e49242e9bba6c-005f175954)
Jul 21 21:08:37 saio object-6020: 127.0.0.1 - - [21/Jul/2020:21:08:37 +0000] "PUT /sdb2/998/AUTH_test/c2/o" 500 617 "PUT http://saio:8090/v1/AUTH_test/c2/o" "tx896c3280e49242e9bba6c-005f175954" "proxy-server 31799" 0.4824 "-" 31794 1
Jul 21 21:08:37 saio object-6030: 127.0.0.1 - - [21/Jul/2020:21:08:37 +0000] "PUT /sdb3/998/AUTH_test/c2/o" 500 617 "PUT http://saio:8090/v1/AUTH_test/c2/o" "tx896c3280e49242e9bba6c-005f175954" "proxy-server 31799" 0.4771 "-" 31793 1
Jul 21 21:08:37 saio object-6040: 127.0.0.1 - - [21/Jul/2020:21:08:37 +0000] "PUT /sdb4/998/AUTH_test/c2/o" 500 617 "PUT http://saio:8090/v1/AUTH_test/c2/o" "tx896c3280e49242e9bba6c-005f175954" "proxy-server 31799" 0.4736 "-" 31795 1
Jul 21 21:08:37 saio proxy-server: ERROR with Object server 127.0.0.4:6040/sdb4 re: Trying to write to /AUTH_test/c2/o:
Traceback (most recent call last):
  File "/vagrant/swift/swift/proxy/controllers/obj.py", line 1670, in _send_chunk
    self.conn.send(to_send)
  File "/usr/lib/python2.7/httplib.py", line 870, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe (txn: tx896c3280e49242e9bba6c-005f175954) (client_ip: 127.0.0.1)
... ^^^ a couple more times ...
Jul 21 21:08:37 saio proxy-server: Object PUT exceptions during send, 0/2 required connections (txn: tx896c3280e49242e9bba6c-005f175954) (client_ip: 127.0.0.1)
Jul 21 21:08:37 saio proxy-server: 127.0.0.1 127.0.0.1 21/Jul/2020/21/08/37 PUT /v1/AUTH_test/c2/o HTTP/1.0 503 - curl/7.58.0 - 17301504 118 - tx896c3280e49242e9bba6c-005f175954 - 0.5738 - - 1595365716.821777105 1595365717.395591021 1

We should be able to catch that ENOSPC and return 507.

[Tim Burke (1-tim-z)]

This also has implications with the framing issues seen in https://bugs.launchpad.net/swift/+bug/1709185 and https://github.com/eventlet/eventlet/pull/578; I was just uploading from /dev/zero like

$ curl http://saio:8090/v1/AUTH_test/c2/o -T /dev/zero

and would see some logs like

Jul 21 21:08:37 saio proxy-server: ERROR WSGI: code 400, message Bad request syntax ('\x00\x00...

afterwards -- followed by a badly-formatted traceback coming up out of eventlet:

Jul 21 21:08:37 saio proxy-server: STDERR: Traceback (most recent call last):
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 461, in fire_timers
Jul 21 21:08:37 saio proxy-server: STDERR: timer()
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
Jul 21 21:08:37 saio proxy-server: STDERR: cb(*args, **kw)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 175, in _do_send
Jul 21 21:08:37 saio proxy-server: STDERR: waiter.switch(result)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 221, in main
Jul 21 21:08:37 saio proxy-server: STDERR: result = function(*args, **kwargs)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 818, in process_request
Jul 21 21:08:37 saio proxy-server: STDERR: proto.__init__(conn_state, self)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/vagrant/swift/swift/common/wsgi.py", line 421, in __init__
Jul 21 21:08:37 saio proxy-server: STDERR: wsgi.HttpProtocol.__init__(self, *args, **kwargs)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 357, in __init__
Jul 21 21:08:37 saio proxy-server: STDERR: self.handle()
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 390, in handle
Jul 21 21:08:37 saio proxy-server: STDERR: self.handle_one_request()
Jul 21 21:08:37 saio proxy-server: STDERR: File "/vagrant/swift/swift/common/wsgi.py", line 541, in handle_one_request
Jul 21 21:08:37 saio proxy-server: STDERR: got = wsgi.HttpProtocol.handle_one_request(self)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 431, in handle_one_request
Jul 21 21:08:37 saio proxy-server: STDERR: if not self.parse_request():
Jul 21 21:08:37 saio proxy-server: STDERR: File "/vagrant/swift/swift/common/wsgi.py", line 482, in parse_request
Jul 21 21:08:37 saio proxy-server: STDERR: return wsgi.HttpProtocol.parse_request(self)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/lib/python2.7/BaseHTTPServer.py", line 286, in parse_request
Jul 21 21:08:37 saio proxy-server: STDERR: self.send_error(400, "Bad request syntax (%r)" % requestline)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/usr/lib/python2.7/BaseHTTPServer.py", line 384, in send_error
Jul 21 21:08:37 saio proxy-server: STDERR: self.wfile.write(content)
Jul 21 21:08:37 saio proxy-server: STDERR: File "/u...