503 incomplete transfer While processing manifest of EC s3 MPU logged at 200 success
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
In Progress
|
Undecided
|
Unassigned |
Bug Description
When I added a sleep to my object servers just before returning their response:
diff --git a/swift/
index 5a883e4c1.
--- a/swift/
+++ b/swift/
@@ -27,6 +27,7 @@ import socket
import math
from swift import gettext_ as _
from hashlib import md5
+import random
from eventlet import sleep, wsgi, Timeout, tpool
from eventlet.
@@ -1138,6 +1139,8 @@ class ObjectControlle
headers[
resp = HTTPNotFound(
condition
+ if obj_size > 1000000 and random.random() > 0.7:
+ sleep(2.0)
return resp
@public
I observed a similar set of log lines from an transaction we observed on our production cluster:
Sep 15 15:09:45 saio proxy-server: ERROR with Object server 127.0.0.4:6040/sdb8 re: Trying to GET /v1/AUTH_
Sep 15 15:27:32 saio proxy-server: ERROR with Object server 127.0.0.3:6030/sdb7 re: Trying to GET /v1/AUTH_
Sep 15 15:27:37 saio proxy-server: ERROR with Object server 127.0.0.3:6030/sdb3 re: Trying to GET /v1/AUTH_
Sep 15 15:28:49 saio proxy-server: ERROR with Object server 127.0.0.3:6030/sdb7 re: Trying to GET /v1/AUTH_
Sep 15 15:28:50 saio proxy-server: ERROR with Object server 127.0.0.2:6020/sdb2 re: Trying to GET /v1/AUTH_
Sep 15 15:28:51 saio proxy-server: ERROR with Object server 127.0.0.3:6030/sdb3 re: Trying to GET /v1/AUTH_
Sep 15 15:28:51 saio proxy-server: Object returning 503 for [] (txn: txc9adaa0c1e394
Sep 15 15:28:51 saio proxy-server: - - 15/Sep/
Sep 15 15:28:51 saio proxy-server: While processing manifest /v1/AUTH_
Sep 15 15:28:51 saio proxy-server: 127.0.0.1 127.0.0.1 15/Sep/
This was with an s3 client making a request for a MPU/SLO stored in an EC bucket
Even though we already sent a 200 to the client, I think we could (should?) log (and emit stats) for the access line as a server error for better accountability and tracking.
It seems timeouts can also manifest like:
2020-09- 17T16:02: 29.965585+ 00:00 ss0206 proxy-server: err InternalError: 500 Internal Server Error (unexpected status code 409): lib/python2. 7/site- packages/ swift/common/ middleware/ s3api/s3api. py", line 303, in __call__ request( req) lib/python2. 7/site- packages/ swift/common/ middleware/ s3api/s3api. py", line 341, in handle_request lib/python2. 7/site- packages/ swift/common/ middleware/ s3api/controlle rs/obj. py", line 134, in GET lib/python2. 7/site- packages/ swift/common/ middleware/ s3api/controlle rs/obj. py", line 100, in GETorHEAD response( self.app, query=query) lib/python2. 7/site- packages/ swift/common/ middleware/ s3api/s3request .py", line 1419, in get_response lib/python2. 7/site- packages/ swift/common/ middleware/ s3api/s3request .py", line 1403, in _get_response 'unexpected status code %d' % status) 5378a97a- 005f63888b)
Traceback (most recent call last):
File "/opt/ss/
resp = self.handle_
File "/opt/ss/
res = handler(req)
File "/opt/ss/
return self.GETorHEAD(req)
File "/opt/ss/
resp = req.get_
File "/opt/ss/
headers, body, query)
File "/opt/ss/
raise InternalError(
InternalError: InternalError: 500 Internal Server Error (unexpected status code 409) (txn: tx821f2f8a806c4