Chunked transfer encoding is not being used

Bug #1056220 reported by Stuart McLaren on 2012-09-25
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
python-glanceclient
High
Stuart McLaren

Bug Description

"Transfer-Encoding: chunked" isn't being used. This leads to problems when the size of the file/stream being uploaded is not known in advance (as may be the case when nova takes a snapshot for example).

This is demonstrated here using named pipes, which are shown to work for the old client but not the new cilent. A patch to use transfer encoding for uploads allows the named pipe example to work.

In the commands below 'glance' is the old glance client (circa Diablo), 'glance2'
is the latest python-glanceclient:

'Old client' image upload using named pipe:

Terminal 1:

$ mkfifo pipe
$ glance -A xxxx -U https://x.x.x.x:9292 add name='x1' disk_format=raw container_format=bare < pipe

Terminal 2:

$ echo hi > pipe

Terminal 1 output:

Added new image with ID: 0931df72-2b0d-49c1-af4f-991ccce5cbf5

Verify image uploaded ok:

$ glance2 --os-image-url https://localhost:9292 --os-auth-token xxxx --insecure image-show 0931df72-2b0d-49c1-af4f-991ccce5cbf5
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 764efa883dda1e11db47671c4a3bbd9e |
| container_format | bare |
| created_at | 2012-09-25T12:19:58.392020 |
| deleted | False |
| disk_format | raw |
| id | 0931df72-2b0d-49c1-af4f-991ccce5cbf5 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | x1 |
| owner | 53648642643910 |
| protected | False |
| size | 3 |
| status | active |
| updated_at | 2012-09-25T12:19:59.127150 |
+------------------+--------------------------------------+

$ glance2 --os-image-url https://localhost:9292 --os-auth-token HPAuth_50616483e4b090b995592508 --insecure image-download 0931df72-2b0d-49c1-af4f-991ccce5cbf5
hi

HTTP exchange:

POST /v1/images HTTP/1.1.
Host: x.x.x.x:9292.
Accept-Encoding: identity.
x-image-meta-container_format: bare.
x-image-meta-min_disk: 0.
x-auth-token: xxxxx.
x-image-meta-is_public: False.
x-image-meta-min_ram: 0.
content-type: application/octet-stream.
x-image-meta-disk_format: raw.
x-image-meta-name: x1.
Transfer-Encoding: chunked.
.

##
T x.x.x.x:44638 -> x.x.x.x:9292 [AP]
3.
hi
.
0.
.

HTTP/1.1 201 Created.
Content-Type: application/json.
Content-Length: 440.
Location: http://x.x.x.x:9292/v1/images/779855b6-9e8a-43e1-b613-eed3d86d217c.
Etag: 764efa883dda1e11db47671c4a3bbd9e.
X-Openstack-Request-Id: req-bda4951d-b199-42d2-8cee-8ab549498e03.
Date: Tue, 25 Sep 2012 13:03:37 GMT.
.

'New client' image upload using named pipe:

Terminal 1:

$ mkfifo pipe
$ glance2 --os-image-url https://localhost:9292 --os-auth-token xxxx -d -v --insecure image-create --disk-format raw --container-format bare < pipe

Terminal 2:
$ echo hi > pipe

Terminal 1 output:

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2012-09-25T12:37:38.781688 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | 180777a0-a449-4105-8d8b-f882c64c416a |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | 53648642643910 |
| protected | False |
| size | 0 |
| status | queued |
| updated_at | 2012-09-25T12:37:38.781697 |
+------------------+--------------------------------------+

Verify image uploaded (fails):

$ glance2 --os-image-url https://localhost:9292 --os-auth-token xxxx --insecure image-download 180777a0-a449-4105-8d8b-f882c64c416a
Request returned failure status.
HTTPNotFound (HTTP 404)

HTTP exchange:

POST /v1/images HTTP/1.1.
Host: x.x.x.x:9292.
Accept-Encoding: identity.
x-image-meta-container_format: bare.
User-Agent: python-glanceclient.
x-image-meta-is_public: False.
X-Auth-Token: xxxxx.
Content-Type: application/octet-stream.
x-image-meta-disk_format: raw.
.

HTTP/1.1 201 Created.
Content-Type: application/json.
Content-Length: 410.
Location: http://x.x.x.x:9292/v1/images/15f2961f-536b-4dc3-afe6-fb5c86ed1dab.
Etag: None.
X-Openstack-Request-Id: req-029da29f-c9b9-4619-92bd-bcdc68cd6037.
Date: Tue, 25 Sep 2012 13:02:17 GMT.
.

Repeat with new client patched to enable chunked transfer encoding:

Terminal 1:

$ mkfifo pipe
$ glance2 --os-image-url https://localhost:9292 --os-auth-token HPAuth_50616483e4b090b995592508 -d -v --insecure image-create --disk-format raw --container-format bare < pipe

Terminal 2:
$ echo hi > pipe

Terminal 1 output:

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 764efa883dda1e11db47671c4a3bbd9e |
| container_format | bare |
| created_at | 2012-09-25T13:23:21.421992 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | ed5f01ff-c8f7-48f1-8333-ce17946c3b8d |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | 53648642643910 |
| protected | False |
| size | 3 |
| status | active |
| updated_at | 2012-09-25T13:23:22.201230 |
+------------------+--------------------------------------+

Verify image uploaded (succeeds):

$ glance2 --os-image-url http://localhost:9292 --os-auth-token xxxxx --insecure image-download ed5f01ff-c8f7-48f1-8333-ce17946c3b8d
hi

HTTP exchange:

POST /v1/images HTTP/1.1.
Host: x.x.x.x:9292.
Accept-Encoding: identity.
x-image-meta-container_format: bare.
User-Agent: python-glanceclient.
x-image-meta-is_public: False.
X-Auth-Token: HPAuth_50616483e4b090b995592508.
Content-Type: application/octet-stream.
x-image-meta-disk_format: raw.
Transfer-Encoding: chunked.
.

#
3.
hi
.

##
0.
.

##

HTTP/1.1 201 Created.
Content-Type: application/json.
Content-Length: 440.
Location: http://x.x.x.x:9292/v1/images/ed5f01ff-c8f7-48f1-8333-ce17946c3b8d.
Etag: 764efa883dda1e11db47671c4a3bbd9e.
X-Openstack-Request-Id: req-954b6210-5090-4317-b825-87398fbb39a1.
Date: Tue, 25 Sep 2012 13:23:22 GMT.

Changed in python-glanceclient:
assignee: nobody → Stuart McLaren (stuart-mclaren)
Stuart McLaren (stuart-mclaren) wrote :

I'd say so. I hadn't spotted that one. Its likely that the fix I'm working on would also fix the echo "1234" case.

Fix proposed to branch: master
Review: https://review.openstack.org/13709

Changed in python-glanceclient:
status: New → In Progress

Reviewed: https://review.openstack.org/13709
Committed: http://github.com/openstack/python-glanceclient/commit/727aadbc257ec3c99dd1621202948d288d45c8cc
Submitter: Jenkins
Branch: master

commit 727aadbc257ec3c99dd1621202948d288d45c8cc
Author: Stuart McLaren <email address hidden>
Date: Wed Sep 26 12:56:51 2012 +0000

    Handle create/update of images with unknown size

    It may not be possible to know in advance the total
    size of image data which is to be uploaded, for example
    if the data is being piped to stdin.

    To handle this we use HTTP Transfer-Encoding: chunked
    and do not set any image size headers.

    Various subtly different cases needed to be handled for
    both image-create and image-update, including:

     * input from named pipe
     * piped input of zero size
     * regular file of zero length

    Fix for bug 1056220.

    Change-Id: I0c7f0a64d883e058993b954a1c465c5b057f2bcf

Changed in python-glanceclient:
status: In Progress → Fix Committed
Brian Waldon (bcwaldon) on 2012-10-24
Changed in python-glanceclient:
importance: Undecided → High
milestone: none → v0.6.0
Brian Waldon (bcwaldon) on 2012-11-20
Changed in python-glanceclient:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers