GET EC policy object includes connection:close header which causes external LB returns 502 bad gateway to client

Bug #1680731 reported by Hugo Kou
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Medium
Unassigned

Bug Description

The `connection: close` cause problem if there's LB infront of Swift proxy servers.

```
swift@vm-16:~/anc-swift$ python src/disconnecting_client.py AUTH_tk9bf859c51d3c443dabc0d310ee3a3917
Using token AUTH_tk9bf859c51d3c443dabc0d310ee3a3917

Resp status 200
content-length: 10485760
content-type: application/octet-stream
accept-ranges: bytes
last-modified: Thu, 06 Apr 2017 17:59:19 GMT
connection: close
etag: f1c9645dbc14efddc7d8a322685f26eb
x-timestamp: 1491501558.61049
x-trans-id: txbb4e9f8007f84fb69054c-0058e69b1d
date: Thu, 06 Apr 2017 19:46:37 GMT
x-object-meta-mtime: 1491498549.279092
x-openstack-request-id: txbb4e9f8007f84fb69054c-0058e69b1d
```

Revision history for this message
Charles Hsu (charles0126) wrote :

It seems like this issue only happens on object in EC policy,

# EC policy

$ curl -i http://192.168.190.21/v1/AUTH_demo/ec64/a -X GET -H "X-Auth-Token: AUTH_tk383654937f2d4fa3a1cb26fbaaf82b40"
HTTP/1.1 200 OK
Content-Length: 0
X-Object-Meta-Mtime: 1491541780.000000
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 05:09:45 GMT
Connection: close
Etag: d41d8cd98f00b204e9800998ecf8427e
X-Timestamp: 1491541784.48274
Date: Fri, 07 Apr 2017 06:52:24 GMT
Content-Type: application/octet-stream
X-Trans-Id: tx476d99bf575f41e48e8ba-0058e73728
X-Openstack-Request-Id: tx476d99bf575f41e48e8ba-0058e73728

# Replica policy

$ curl -i http://192.168.190.21/v1/AUTH_demo/gw/20MB -I -H "X-Auth-Token: AUTH_tk383654937f2d4fa3a1cb26fbaaf82b40"
HTTP/1.1 200 OK
Content-Length: 20971520
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 06:51:27 GMT
Etag: 8f4e33f3dc3e414ff94e5fb6905cba8c
X-Timestamp: 1491547886.25414
X-Object-Meta-Mtime: 1491547877.000000
X-Trans-Id: tx63c97a2b94e34ae2a5830-0058e7372c
X-Openstack-Request-Id: tx63c97a2b94e34ae2a5830-0058e7372c
Date: Fri, 07 Apr 2017 06:52:28 GMT

Revision history for this message
clayg (clay-gerrard) wrote :

We should be able to test for this in a functest, i'm worried there may be other backend headers that leak out.

Changed in swift:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
clayg (clay-gerrard) wrote :

FWIW no gateway should translate this response to 502 - or any response that returns connection: close at any time (in in fact that may not even be related to this bug).

However, any client or gateway that gets a swift EC GET response tho will see this header and should close the connection - although FIWI I have noticed you *can* pipeline EC get requests just fine - this isn't eventlet.wsgi server setting close and shutting down the socket - it's just a left over backend header. But the semantic is wrong.

We've always returned it too - since when Sam & Paul kicked around the first versions of EC GET on the feature branch till right now as best I can tell.

Revision history for this message
clayg (clay-gerrard) wrote :

ZOMGBBQ - you know what it *might* be - if a client *sends* 'Connection: keep-alive' (which is redundant in http 1.1 but not invalid) the most insane thing comes back:

ubuntu@saio:~$ curl -H 'x-auth-token: AUTH_tkbda4caa25b4f4f60a50ec26207487741' http://saio:8080/v1/AUTH_test/ec-test/deleteme -H 'connection: keep-alive' -v
* Trying 127.0.0.1...
* Connected to saio (127.0.0.1) port 8080 (#0)
> GET /v1/AUTH_test/ec-test/deleteme HTTP/1.1
> Host: saio:8080
> User-Agent: curl/7.47.0
> Accept: */*
> x-auth-token: AUTH_tkbda4caa25b4f4f60a50ec26207487741
> connection: keep-alive
>
< HTTP/1.1 200 OK
< Content-Length: 3145728
< X-Object-Meta-Mtime: 1491470291.015581
< Accept-Ranges: bytes
< Last-Modified: Fri, 07 Apr 2017 08:01:02 GMT
< Connection: close <-*****close****
< Etag: d1dd210d6b1312cb342b56d02bd5e651
< X-Timestamp: 1491552061.98489
< Date: Fri, 07 Apr 2017 08:04:21 GMT
< Content-Type: application/octet-stream
< X-Trans-Id: tx11ec5191d0304934b174d-0058e74805
< X-Openstack-Request-Id: tx11ec5191d0304934b174d-0058e74805
< Connection: keep-alive <-*****keep-alive****
<
* Closing connection 0

That is interesting and could very well invalid, and almost definitely could cause problem for otherwise reasonably behaved clients...

without the raw dump curl seems to sort out the correct answer:

ubuntu@saio:~$ curl -H 'x-auth-token: AUTH_tkbda4caa25b4f4f60a50ec26207487741' http://saio:8080/v1/AUTH_test/ec-test/deleteme -H 'connection: keep-alive' -I
HTTP/1.1 200 OK
Content-Length: 3145728
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:01:02 GMT
Etag: d1dd210d6b1312cb342b56d02bd5e651
X-Timestamp: 1491552061.98489
X-Object-Meta-Mtime: 1491470291.015581
X-Trans-Id: tx3c00e2a73c444b4798050-0058e74869
X-Openstack-Request-Id: tx3c00e2a73c444b4798050-0058e74869
Date: Fri, 07 Apr 2017 08:06:01 GMT
Connection: keep-alive

swiftclient has a slightly different opinion

DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'3145728', u'Content-Type': u'application/octet-stream', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Fri, 07 Apr 2017 08:01:02 GMT', u'Connection': u'close, keep-alive', u'Etag': u'd1dd210d6b1312cb342b56d02bd5e651', u'X-Timestamp': u'1491552061.98489', u'X-Trans-Id': u'tx9bbeda8da5914148b6228-0058e7487e', u'Date': u'Fri, 07 Apr 2017 08:06:22 GMT', u'X-Object-Meta-Mtime': u'1491470291.015581', u'X-Openstack-Request-Id': u'tx9bbeda8da5914148b6228-0058e7487e'}

fascinating

Revision history for this message
Alistair Coles (alistair-coles) wrote :

EC GET responses (and only EC GET, not HEAD, not replicated policy) seem to have always incuded the connection: close header, logs below from 2.9.0 but I saw similar on Mitaka (2.7.1):

swift@vm-16:~/junk$ cd ../swift; git status
HEAD detached at 2.9.0
nothing to commit, working directory clean

Replicated policy:

swift@vm-16:~/junk$ curl -i http://localhost:8080/v1/AUTH_test/c2/10MB_file -X GET -H "X-Auth-Token: AUTH_tk5005e34fa7f84e9b85dd698467496a11"
HTTP/1.1 200 OK
Content-Length: 10485760
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:14:08 GMT
Etag: f1c9645dbc14efddc7d8a322685f26eb
X-Timestamp: 1491552847.85534
X-Object-Meta-Mtime: 1491498549.279092
X-Trans-Id: txf617173c4682472eacaed-0058e74b36
Date: Fri, 07 Apr 2017 08:17:58 GMT

EC policy:

swift@vm-16:~/junk$ curl -i http://localhost:8080/v1/AUTH_test/c1/10MB_file -X GET -H "X-Auth-Token: AUTH_tk5005e34fa7f84e9b85dd698467496a11"
HTTP/1.1 200 OK
Content-Length: 10485760
X-Object-Meta-Mtime: 1491498549.279092
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:14:16 GMT
Connection: close
Etag: f1c9645dbc14efddc7d8a322685f26eb
X-Timestamp: 1491552855.87341
Date: Fri, 07 Apr 2017 08:18:02 GMT
Content-Type: application/octet-stream
X-Trans-Id: txf6518213957f499cb39ba-0058e74b3a

EC HEAD:

swift@vm-16:~/swift$ curl -i http://localhost:8080/v1/AUTH_test/c1/10MB_file -I -H "X-Auth-Token: AUTH_tk5005e34fa7f84e9b85dd698467496a11"
HTTP/1.1 200 OK
Content-Length: 10485760
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:14:16 GMT
Etag: f1c9645dbc14efddc7d8a322685f26eb
X-Timestamp: 1491552855.87341
X-Object-Meta-Mtime: 1491498549.279092
X-Trans-Id: txff4b34d61f0b4323a8dc1-0058e74bae
Date: Fri, 07 Apr 2017 08:19:58 GMT

Revision history for this message
Alistair Coles (alistair-coles) wrote :

@Clay in your comment #4 you say that curl sorts out the connection headers, but actually I think you have issued a HEAD request by having the -I option:

"
curl -H 'x-auth-token: AUTH_tkbda4caa25b4f4f60a50ec26207487741' http://saio:8080/v1/AUTH_test/ec-test/deleteme -H 'connection: keep-alive' -I
HTTP/1.1 200 OK
Content-Length: 3145728
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:01:02 GMT
Etag: d1dd210d6b1312cb342b56d02bd5e651
X-Timestamp: 1491552061.98489
X-Object-Meta-Mtime: 1491470291.015581
X-Trans-Id: tx3c00e2a73c444b4798050-0058e74869
X-Openstack-Request-Id: tx3c00e2a73c444b4798050-0058e74869
Date: Fri, 07 Apr 2017 08:06:01 GMT
Connection: keep-alive
"

I see both Connection headers reported for a GET:

swift@vm-16:~/junk$ curl -i http://localhost:8080/v1/AUTH_test/c1/10MB_file -X GET -H "X-Auth-Token: AUTH_tk5005e34fa7f84e9b85dd698467496a11" -H "Connection: keep-alive"
HTTP/1.1 200 OK
Content-Length: 10485760
X-Object-Meta-Mtime: 1491498549.279092
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:14:16 GMT
Connection: close <----
Etag: f1c9645dbc14efddc7d8a322685f26eb
X-Timestamp: 1491552855.87341
Date: Fri, 07 Apr 2017 09:11:07 GMT
Content-Type: application/octet-stream
X-Trans-Id: tx72f5f1ca261c42a6a98a1-0058e757ab
Connection: keep-alive <----

Revision history for this message
clayg (clay-gerrard) wrote :

the -I request is definitely a HEAD request, and need to remember *we don't leak backend headers on HEAD

Thank you for correcting me.

Revision history for this message
Matthew Oliver (matt-0) wrote :

So we already have a function in the swift/proxy/controllers/base.py that strips the connection from the backend called update_headers. this is why HEAD and replication work.

We don't call it on the EC side, so maybe something like this is all we need to do.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/456706
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=6c320b2908e5821e2f398e0dae8a67774e110603
Submitter: Jenkins
Branch: master

commit 6c320b2908e5821e2f398e0dae8a67774e110603
Author: Alistair Coles <email address hidden>
Date: Thu Apr 13 11:16:54 2017 +0100

    Stop including Connection header in EC GET responses

    Currently, EC GET responses from proxy to clients, unlike any other
    response, include a "Connection: close" header. If the client has sent
    a "Connection: keep-alive" header then eventlet.wsgi appends this to
    the client response, so clients can receive a response with both
    headers:

    Connection: close
    Connection: keep-alive

    This patch fixes the proxy EC GET path to remove any Connection header
    from it's response, but does not change the behaviour of eventlet.wsgi
    with respect to returning any client provided 'Connection: keep-alive'
    header.

    Change-Id: I43cd27c978edb4a1a587f031dbbee26e9acdc920
    Co-Authored-By: Matthew Oliver <email address hidden>
    Closes-Bug: #1680731

Changed in swift:
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.15.0

This issue was fixed in the openstack/swift 2.15.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers