EC: ECPutter doesn't close backend HTTPConnection when exception raised
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
I'm not sure if this has *really* security risk yet but for safety, wrote down as private one.
For Erasure Code (EC) case, current Swift uses ECPutter to handle PUT object requests and it works like as wrapper of HTTPConnection. ECPutter keeps a HTTPConnection instance to communicate with an object-server and it will be closed at the end of request.
However, ECPutter doesn't close the HTTPConnection instance when an error (e.g. 422 Unprocessable Entity) raised during PUT sequence. This makes ECPutter to leave non-closed HTTPConnection inside and object-server will wait the next chunk because the connection is still open there. After that, object-server will give up to read the next chunk due to ChunkReadTimeou
The reproduce procedure is too simple:
- create a container with ec-policy
- create a local file (e.g. "touch empty_object")
- upload object into the container with invalid etag like as 'swift upload ec_container empty_object --header "Etag: invalid"'
After that, you can get UnprocessableEntity immediately. *BUT AFTER 60 seconds*, you can see a log output line at object-server like as:
Jun 21 01:13:39 ubuntu object-server: 127.0.0.1 - - [21/Jun/
ADME.rst" 408 - "PUT http://
server 4760" 60.1765 "-" 4833 1
This is an evidence that object-server was waiting in 60 secs *AFTER* proxy-server return 422 immediately.
The attachment file is the code possibly fixes this issue but it needs probably further work, reviews because it has only one unit test yet.
Changed in swift: | |
status: | New → Confirmed |
information type: | Private → Public |
Confirmed using the method Kota describes to reproduce.
The problem can also be observed by running this functional test with an EC policy:
nosetests test/functional /tests. py:TestFile. testFileSizeLim it
which does PUTs with content-length >0 but then sends no data and the client disconnects - the object servers will report 408 after 60 second timeout.
If fallocate is enabled then I believe that the bytes fallocate'd for the tmp file that the object server never writes into will be reserved until the connection is closed i.e. 60 secs by default, which is unfortunate if the declared content-length is large.
Consequently that func test may fail on an SAIO where the available filesystem space is less than that required to fallocate space for 4 max file size objects.