Chunked transfer encoding is not being used

[Stuart McLaren]

"Transfer-Encoding: chunked" isn't being used. This leads to problems when the size of the file/stream being uploaded is not known in advance (as may be the case when nova takes a snapshot for example).

This is demonstrated here using named pipes, which are shown to work for the old client but not the new cilent. A patch to use transfer encoding for uploads allows the named pipe example to work.

In the commands below 'glance' is the old glance client (circa Diablo), 'glance2'
is the latest python-glanceclient:

'Old client' image upload using named pipe:

Terminal 1:

$ mkfifo pipe
$ glance -A xxxx -U https://x.x.x.x:9292 add name='x1' disk_format=raw container_format=bare < pipe

Terminal 2:

$ echo hi > pipe

Terminal 1 output:

Added new image with ID: 0931df72-2b0d-49c1-af4f-991ccce5cbf5

Verify image uploaded ok:

$ glance2 --os-image-url https://localhost:9292 --os-auth-token xxxx --insecure image-show 0931df72-2b0d-49c1-af4f-991ccce5cbf5
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 764efa883dda1e11db47671c4a3bbd9e |
| container_format | bare |
| created_at | 2012-09-25T12:19:58.392020 |
| deleted | False |
| disk_format | raw |
| id | 0931df72-2b0d-49c1-af4f-991ccce5cbf5 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | x1 |
| owner | 53648642643910 |
| protected | False |
| size | 3 |
| status | active |
| updated_at | 2012-09-25T12:19:59.127150 |
+------------------+--------------------------------------+

$ glance2 --os-image-url https://localhost:9292 --os-auth-token HPAuth_50616483e4b090b995592508 --insecure image-download 0931df72-2b0d-49c1-af4f-991ccce5cbf5
hi

HTTP exchange:

POST /v1/images HTTP/1.1.
Host: x.x.x.x:9292.
Accept-Encoding: identity.
x-image-meta-container_format: bare.
x-image-meta-min_disk: 0.
x-auth-token: xxxxx.
x-image-meta-is_public: False.
x-image-meta-min_ram: 0.
content-type: application/octet-stream.
x-image-meta-disk_format: raw.
x-image-meta-name: x1.
Transfer-Encoding: chunked.
.

##
T x.x.x.x:44638 -> x.x.x.x:9292 [AP]
3.
hi
.
0.
.

HTTP/1.1 201 Created.
Content-Type: application/json.
Content-Length: 440.
Location: http://x.x.x.x:9292/v1/images/779855b6-9e8a-43e1-b613-eed3d86d217c.
Etag: 764efa883dda1e11db47671c4a3bbd9e.
X-Openstack-Request-Id: req-bda4951d-b199-42d2-8cee-8ab549498e03.
Date: Tue, 25 Sep 2012 13:03:37 GMT.
.

'New client' image upload using named pipe:

Terminal 1:

$ mkfifo pipe
$ glance2 --os-image-url https://localhost:9292 --os-auth-token xxxx -d -v --insecure image-create --disk-format raw --container-format bare < pipe

Terminal 2:
$ echo hi > pipe

Terminal 1 output:

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2012-09-25T12:37:38.781688 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | 180777a0-a449-4105-8d8b-f882c64c416a |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | 53648642643910 |
| protected | False |
| size | 0 |
| status | queued |
| updated_at | 2012-09-25T12:37:38.781697 |
+------------------+--------------------------------------+

Verify image uploaded (fails):

$ glance2 --os-image-url https://localhost:9292 --os-auth-token xxxx --insecure image-download 180777a0-a449-4105-8d8b-f882c64c416a
Request returned failure status.
HTTPNotFound (HTTP 404)

HTTP exchange:

POST /v1/images HTTP/1.1.
Host: x.x.x.x:9292.
Accept-Encoding: identity.
x-image-meta-container_format: bare.
User-Agent: python-glanceclient.
x-image-meta-is_public: False.
X-Auth-Token: xxxxx.
Content-Type: application/octet-stream.
x-image-meta-disk_format: raw.
.

HTTP/1.1 201 Created.
Content-Type: application/json.
Content-Length: 410.
Location: http://x.x.x.x:9292/v1/images/15f2961f-536b-4dc3-afe6-fb5c86ed1dab.
Etag: None.
X-Openstack-Request-Id: req-029da29f-c9b9-4619-92bd-bcdc68cd6037.
Date: Tue, 25 Sep 2012 13:02:17 GMT.
.

Repeat with new client patched to enable chunked transfer encoding:

Terminal 1:

$ mkfifo pipe
$ glance2 --os-image-url https://localhost:9292 --os-auth-token HPAuth_50616483e4b090b995592508 -d -v --insecure image-create --disk-format raw --container-format bare < pipe

Terminal 2:
$ echo hi > pipe

Terminal 1 output:

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 764efa883dda1e11db47671c4a3bbd9e |
| container_format | bare |
| created_at | 2012-09-25T13:23:21.421992 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | ed5f01ff-c8f7-48f1-8333-ce17946c3b8d |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | 53648642643910 |
| protected | False |
| size | 3 |
| status | active |
| updated_at | 2012-09-25T13:23:22.201230 |
+------------------+--------------------------------------+

Verify image uploaded (succeeds):

$ glance2 --os-image-url http://localhost:9292 --os-auth-token xxxxx --insecure image-download ed5f01ff-c8f7-48f1-8333-ce17946c3b8d
hi

HTTP exchange:

POST /v1/images HTTP/1.1.
Host: x.x.x.x:9292.
Accept-Encoding: identity.
x-image-meta-container_format: bare.
User-Agent: python-glanceclient.
x-image-meta-is_public: False.
X-Auth-Token: HPAuth_50616483e4b090b995592508.
Content-Type: application/octet-stream.
x-image-meta-disk_format: raw.
Transfer-Encoding: chunked.
.

#
3.
hi
.

##
0.
.

##

HTTP/1.1 201 Created.
Content-Type: application/json.
Content-Length: 440.
Location: http://x.x.x.x:9292/v1/images/ed5f01ff-c8f7-48f1-8333-ce17946c3b8d.
Etag: 764efa883dda1e11db47671c4a3bbd9e.
X-Openstack-Request-Id: req-954b6210-5090-4317-b825-87398fbb39a1.
Date: Tue, 25 Sep 2012 13:23:22 GMT.

[Brian Waldon (bcwaldon)]

Is this related to https://bugs.launchpad.net/python-glanceclient/+bug/1035250 ?

[Stuart McLaren (stuart-mclaren)]

I'd say so. I hadn't spotted that one. Its likely that the fix I'm working on would also fix the echo "1234" case.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/13709

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/13709
Committed: http://github.com/openstack/python-glanceclient/commit/727aadbc257ec3c99dd1621202948d288d45c8cc
Submitter: Jenkins
Branch: master

commit 727aadbc257ec3c99dd1621202948d288d45c8cc
Author: Stuart McLaren <email address hidden>
Date: Wed Sep 26 12:56:51 2012 +0000

    Handle create/update of images with unknown size

    It may not be possible to know in advance the total
    size of image data which is to be uploaded, for example
    if the data is being piped to stdin.

    To handle this we use HTTP Transfer-Encoding: chunked
    and do not set any image size headers.

    Various subtly different cases needed to be handled for
    both image-create and image-update, including:

     * input from named pipe
     * piped input of zero size
     * regular file of zero length

    Fix for bug 1056220.

    Change-Id: I0c7f0a64d883e058993b954a1c465c5b057f2bcf