Mirantis OpenStack

Radosgw does not handle Transfer-Encoding: chunked

Bug #1505273 reported by Daniil Lapshin
24
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
High
Albert Syriy
Mirantis OpenStack 8.0
6.1.x
Won't Fix
High
Denis Puchkin
Mirantis OpenStack 6.1-updates
7.0.x
Won't Fix
High
Denis Puchkin
Mirantis OpenStack 7.0-updates
8.0.x
Fix Released
High
Albert Syriy
Mirantis OpenStack 8.0
9.x
Fix Released
High
MOS Linux
Mirantis OpenStack 9.0

Bug Description

Hi,

When you try to use "Transfer-Encoding: chunked" header in request to Swift API, if ceph is used as object storage backend, you are getting this error.

# curl -k -i http://172.16.23.2:8080/swift/v1/test/test -X PUT -H "X-Auth-Token: $TOKEN" -H "Transfer-Encoding: chunked" -H "Content-Length: 5274" -T test
HTTP/1.1 411 Length Required
Date: Mon, 12 Oct 2015 15:11:21 GMT
Server: Apache/2.4.7 (Ubuntu)
Content-Length: 238
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>411 Length Required</title>
</head><body>
<h1>Length Required</h1>
<p>A request of the requested method PUT requires a valid Content-length.<br />
</p>
</body></html>

This header should be understood by radosgw as stated in documentation (http://docs.ceph.com/docs/master/radosgw/swift/objectops/#create-update-an-object)
After a bit of researching I've found that apache uses mod_fastcgi to handle requests to radosgw, while, as stated in documentation (http://docs.ceph.com/docs/master/install/install-ceph-gateway) and in this bug (http://tracker.ceph.com/issues/3297), it should use mod_proxy_fcgi.

Issue is reproducible in fresh deployment of 6.1.

Roman Podoliaka (rpodolyaka)
Changed in mos:
assignee: nobody → MOS Ceph (mos-ceph)
Revision history for this message
Denis Klepikov (dklepikov) wrote :

High due to customer-found

tags: added: customer-found
Oleksandr Liemieshko (oliemieshko)
tags: added: support
Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

To fix this the following is required:
- use Apache2 2.4.9+ (I have tested with 2.4.17)
- enable mod_proxy and mod_proxy_fcgi, optionally disable mod_fastcgi since it is not needed anymore
- edit /etc/apache2/sites-available/rgw.conf:
--- remove/comment all fcgi stuff
--- disable s3gw RewriteRule, use generic:
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
--- add ProxyPassMatch with syntax like below:
ProxyPassPatch "^.*$" "unix:/tmp/radosgw.sock|fcgi://localhost/var/www/radosgw/"

Side note - this may break S3 API (Swift API will work and chunked upload will work), so this is not a complete solution - RewriteRule needs to be customized somehow (or ProxyPassMatch parameters)

Revision history for this message
Roman Rufanov (rrufanov) wrote :

Customer found on 6.1 - need fix.

Revision history for this message
Denis Ipatov (dipatov) wrote :

This bug https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1411030 has a patch. Can you include it in an upcoming update?

Revision history for this message
Radoslaw Zarzynski (rzarzynski) wrote :

Through debugging the issue with Alexander Lemeshko, it has been observed
that the failed HTTP request isn't even delivered to radosgw instance - Apache
simply rejects it before passing through FastCGI.

radosgw does support chunked upload [1].

[1] https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest_swift.cc#L532

Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

We don't deliver apache2 in MOS6.1+. It comes from upstream's repos for ubuntu.

Revision history for this message
Albert Syriy (asyriy) wrote :

Guys, did I understand correct, that you asked to have our own appach package?

Revision history for this message
Albert Syriy (asyriy) wrote :

@dipatov, @dlapshin, @dmeltsaykin: waiting for your answer

Revision history for this message
Andrey Shestakov (ashestakov) wrote :

related bug: https://bugs.launchpad.net/glance/+bug/1518431
can works via mod_proxy_fcgi and radosgw as cgi server.

Revision history for this message
Albert Syriy (asyriy) wrote :

New apache2 package 2.4.10-9ubuntu1.1 from vivid has dependency on newer version of packages.
We can't update version apache2 without updating other packages.

Revision history for this message
Andrey Shestakov (ashestakov) wrote :

proposed fix https://review.openstack.org/#/c/258521/

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Fuel w/a https://review.openstack.org/#/c/258521

Changed in fuel:
milestone: none → 8.0
importance: Undecided → High
status: New → In Progress
assignee: nobody → Andrey Shestakov (ashestakov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/258521
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e8567ee17b6f013d8d0f1460415523af084adcc7
Submitter: Jenkins
Branch: master

commit e8567ee17b6f013d8d0f1460415523af084adcc7
Author: Andrey Shestakov <email address hidden>
Date: Wed Dec 16 16:05:18 2015 +0200

    Change apache2 CGI module for RadosGW

    In apache-2.4.7 fastcgi module is not support chunked transfer.
    It breaks uploads of objects to RadosGW.
    Fix available since apache-2.4.10 (vivid) and upgrade apache in MOS is not
    possible, so this workaround proposed.
    RadosGW runs with native CGI server on 127.0.0.1:9000 and apache uses
    mod_proxy_fcgi to proxy user requests to rados.

    Closes-bug: #1521623
    Closes-bug: #1505273

    Change-Id: Ia3959c4009c6c673074a9b7536b1cc87834c301d

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/261353

Ivan Ponomarev (ivanzipfer)
tags: added: area-mos
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/261353
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b092234605defaf6e41b15dc949472bd4b2d04ee
Submitter: Jenkins
Branch: stable/8.0

commit b092234605defaf6e41b15dc949472bd4b2d04ee
Author: Andrey Shestakov <email address hidden>
Date: Wed Dec 16 16:05:18 2015 +0200

    Change apache2 CGI module for RadosGW

    In apache-2.4.7 fastcgi module is not support chunked transfer.
    It breaks uploads of objects to RadosGW.
    Fix available since apache-2.4.10 (vivid) and upgrade apache in MOS is not
    possible, so this workaround proposed.
    RadosGW runs with native CGI server on 127.0.0.1:9000 and apache uses
    mod_proxy_fcgi to proxy user requests to rados.

    Closes-bug: #1521623
    Closes-bug: #1505273

    Change-Id: Ia3959c4009c6c673074a9b7536b1cc87834c301d
    (cherry picked from commit e8567ee17b6f013d8d0f1460415523af084adcc7)

Albert Syriy (asyriy)
Changed in mos:
status: In Progress → New
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/265966

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/265966
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a2d391a76082c7b00740b11a36e698006f185808
Submitter: Jenkins
Branch: master

commit a2d391a76082c7b00740b11a36e698006f185808
Author: Alex Schultz <email address hidden>
Date: Mon Jan 11 11:12:01 2016 -0700

    Add apache mod_proxy for ceph radosgw

    With the switch to fcgi for radosgw we need to include mod_proxy as
    mod_proxy_fcgi requires it. This change pulls in the proxy module to
    resolve apache2 startup failures when ceph is enabled.

    Change-Id: I7044b58066e70b11ed9358579a2f369ce761ee06
    Related-Bug: #1521623
    Related-Bug: #1505273
    Closes-Bug: #1532860

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/8.0)

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/266370

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/266370
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ac654f2a5bac67ee8bd8330cb4b8443a84b3ea28
Submitter: Jenkins
Branch: stable/8.0

commit ac654f2a5bac67ee8bd8330cb4b8443a84b3ea28
Author: Alex Schultz <email address hidden>
Date: Mon Jan 11 11:12:01 2016 -0700

    Add apache mod_proxy for ceph radosgw

    With the switch to fcgi for radosgw we need to include mod_proxy as
    mod_proxy_fcgi requires it. This change pulls in the proxy module to
    resolve apache2 startup failures when ceph is enabled.

    Change-Id: I7044b58066e70b11ed9358579a2f369ce761ee06
    Related-Bug: #1521623
    Related-Bug: #1505273
    Closes-Bug: #1532860
    (cherry picked from commit a2d391a76082c7b00740b11a36e698006f185808)

Timur Nurlygayanov (tnurlygayanov)
Changed in mos:
status: New → Won't Fix
no longer affects: fuel
Revision history for this message
Albert Syriy (asyriy) wrote :

The fixes for the master/9.0 have been committed and merged
https://review.openstack.org/#/c/258521/6
https://review.openstack.org/#/c/265966/

Due this, the status for master, MOS 9.0 was changed to fix_committed.

Maksym Strukov (unbelll)
tags: added: on-verification
Revision history for this message
Maksym Strukov (unbelll) wrote :

Verified as fixed in 8.0-570

[root@nailgun ~]# curl -s -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}' -H 'Content-type: application/json' http://10.109.25.2:5000/v2.0/tokens
{"access": {"token": {"issued_at": "2016-02-19T12:21:41.798504", "expires": "2016-02-20T12:21:41Z", "id": "8beb2c1f6b2c4745ac78e2296416f179", "tenant": {"description": "", "enabled": true, "id": "13382b34fa8e4e7ab1f6695d80c2d595", "name": "admin"}, "audit_ids": ["7TDFCu8cQt2bLeQdXTNlWQ"]}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://10.109.25.2:8000/ostf", "region": "RegionOne", "internalURL": "http://10.109.25.2:8000/ostf", "id": "dacfb57b4ec74aa1899d2e51213c78eb", "publicURL": "http://10.109.25.2:8000/ostf"}], "endpoints_links": [], "type": "ostf", "name": "ostf"}, {"endpoints": [{"adminURL": "http://10.109.25.2:8000/api", "region": "RegionOne", "internalURL": "http://10.109.25.2:8000/api", "id": "f68271098315479d8e2152a303bf426c", "publicURL": "http://10.109.25.2:8000/api"}], "endpoints_links": [], "type": "fuel", "name": "nailgun"}, {"endpoints": [{"adminURL": "http://10.109.25.2:35357/v2.0/v2.0", "region": "RegionOne", "internalURL": "http://10.109.25.2:5000/v2.0/v2.0", "id": "e43b006835be4ec588f1367cce2c7c14", "publicURL": "http://10.109.25.2:5000/v2.0/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "7be419bf1d554f34a877f564ab132489", "roles": [{"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["b999d49c009d4e6493f3aa65a3968fbe"]}}}

[root@nailgun ~]# TOKEN=8beb2c1f6b2c4745ac78e2296416f179

[root@nailgun ~]# curl -k -i http://10.109.25.2:8080/swift/v1/test/test -X PUT -H "X-Auth-Token: $TOKEN" -H "Transfer-Encoding: chunked" -H "Content-Length: 5274" HTTP/1.1 405 Not Allowed
Server: nginx
Date: Fri, 19 Feb 2016 12:21:59 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 166
Connection: keep-alive

<html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>

tags: removed: on-verification
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Release notes:

RadosGW now properly handles requests with chunked encoding. Previously it used to fail with "411 Length Required"

tags: added: release-notes
Olga Gusarenko (ogusarenko)
tags: added: 8.0 release-notes-done
removed: release-notes
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/trusty/ceph (6.1)

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/19379

Revision history for this message
Alexander Gubanov (ogubanov) wrote :

Verified on MOS 9.0 (build 248) - works!
Proof: http://pastebin.com/kYcYUV3q

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Fix for 6.1 needs additional work - moved out of 6.1-mu-6 scope

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

We are not going to update ceph in 6.1 and 7.0 MUs - Won't Fix.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.