glance upload to rbd creates zero length image

[Edward Hope-Morley]

If using upload() from V2 Glance API to write data to an RBD image the following problem occurs:

A length of zero can be used which means an image of size 0 is created thus resulting in the following exception:

2013-08-19 12:09:36.044 31475 ERROR glance.api.v2.image_data [e9a89f79-d9ed-40b7-a1a0-157f6eb2b3d1 5cdd3419b10e4459a12593282f483331 9f93fd6845f04c6c9b51dcc9e24008a1] Failed to upload image data due to internal error
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data Traceback (most recent call last):
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/v2/image_data.py", line 53, in upload
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data image.set_data(data, size)
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/domain/proxy.py", line 126, in set_data
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data self.base.set_data(data, size)
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier/__init__.py", line 201, in set_data
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data self.image.set_data(data, size)
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/domain/proxy.py", line 126, in set_data
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data self.base.set_data(data, size)
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/store/__init__.py", line 647, in set_data
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data self.image.image_id, utils.CooperativeReader(data), size)
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/store/__init__.py", line 363, in add_to_backend
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store)
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/store/__init__.py", line 341, in store_add_to_backend
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data (location, size, checksum, metadata) = store.add(image_id, data, size)
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/store/rbd.py", line 287, in add
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data offset += image.write(chunk, offset)
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/rbd.py", line 648, in write
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data raise make_ex(ret, "error writing to %s" % (self.name,))
2013-08-19 12:09:36.044 31475 TRACE glance.api.v2.image_data InvalidArgument: error writing to 4c3712f8-e6ef-4b39-aaf0-975f046252f4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/42640

[Edward Hope-Morley (hopem)]

The python-glanceclient always does chunked transfers so we cannot set content-length (since that would disable chunking). The v2 API upload() command does not inspect image metadata so, therefore, it looks like the correct solution here is to behave like the swift backend and support ChunkedReads thus allowing for multiple chunks of data of unknown size so be written to a single rbd image.

[Edward Hope-Morley (hopem)]

Ok so having explored this a bit further, it now transpires that writing images to an rbd backend fails for both v1 and v2 api if (a) the image size is not supplied to the client and (b) the image size cannot be determined by the client by checking the input (file). So, I think the best way to go about this is to do for v2 like we have in v1 i.e. allow the client to accept image_size as metadata and send it as x-image-meta-size so that glance.api.v2.ImageDataController.upload can get image size from either the request content-length or x-image-meta-size.

The glanceclient will need to be modified to allow for these field to be set (https://bugs.launchpad.net/python-glanceclient/+bug/1220197) and the v2 glance api upload() will need to check for these fields. In the case where image-size is still zero, we wil be force to use the non-optimal action of resize-and-write for each chunk.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/42640
Committed: http://github.com/openstack/glance/commit/1414c3fa8e12604a35e2c299dcac13830a419aaf
Submitter: Jenkins
Branch: master

commit 1414c3fa8e12604a35e2c299dcac13830a419aaf
Author: Edward Hope-Morley <email address hidden>
Date: Mon Aug 19 14:22:11 2013 +0100

    Add rbd store support for zero size image

    If glanceclient does not provide image size to RBD store add() it
    will create a zero-size image and fail when writing to it. If
    provided with zero size we now work out the size of each chunk
    being written and resize the rbd image prior to each write. This
    will be much slower than if new the image size on create but it
    will at least work.

    There is a corresponding glanceclient fix to ensure content-length
    is set if known by the glanceclient, see bug 1220197

    Change-Id: I014c4b71c77c92b9876786d1ba438cdb90f83233
    Fixes: bug #1213880