Should swift backend compute size if none is specified?

[Tom Hancock]

From an email I posted to the mailing list yesterday
===========================================

We ran into a backward compat issue closely related to this last week.

If the newer Glance code in Diablo is running on a server configured with a Swift backend and interacting with an older Glance client on an older Nova system that doesn't set content-length then (even assuming the image is < 5GB) it gets stored in the backend using the usual means, but the image size in the registry is never set correctly.
These few lines were in the pre-diablo code to do a HEAD on the object in swift if the size was unknown and if they're re-added then the registry gets updated with the correct length. It looks like this still applies to essex.

regards,
Tom

---------------------------- glance/store/swift.py ---------------------------- @@ -339,6 +339,12 @@ class Store(glance.store.base.Store):
                 # best...
                 obj_etag = swift_conn.put_object(self.container, obj_name,
                                                  image_file)
+ if image_size == 0:
+ resp_headers = swift_conn.head_object(self.container, obj_name)
+ # header keys are lowercased by Swift
+ if 'content-length' in resp_headers:
+ image_size = int(resp_headers['content-length'])
+ logger.debug(_("Zero image size so setting
+ object size to %s") % image_size)
             else:
                 # Write the image into Swift in chunks. We cannot
                 # stream chunks of the webob.Request.body_file, unfortunately,

[Jay Pipes (jaypipes)]

Tom, I'm going to have to get you used to proposing patches in Gerrit ;)

Thanks for the patch. I'll assign this to someone, and also, Ewan's idea about if image_size = 0, then do the segmented manifest upload make a lot of sense. Not sure why that didn't occur to me earlier, but oh well!

-jay

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/1819
Committed: http://github.com/openstack/glance/commit/1f28b6ca7efb9bed12052c10af71e57773693970
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 1f28b6ca7efb9bed12052c10af71e57773693970
Author: Paul Bourke <email address hidden>
Date: Mon Nov 21 16:50:23 2011 +0000

    Fix bug 891738

    Compute image size for the registry in the case where none is specified.

    Change-Id: Ic21adf4865d0f481db9bd362cfaeeebd0942c974

[Paul Bourke (pauldbourke)]

All, while the fix for this has been deemed valid, there is a secondary improvement.

Images that have no size specified *and* are larger than 5G will still fail to be added to Swift. I'm going to push a topic branch called bug/891738-improvement that addresses this.

Hope it's ok to post here, let me know if you'd like me to open a new bug for this.

[Jay Pipes (jaypipes)]

https://review.openstack.org/#change,2072

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/2072
Committed: http://github.com/openstack/glance/commit/98b1ef337ee63f239829b44aec4d05cea99f98ad
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 98b1ef337ee63f239829b44aec4d05cea99f98ad
Author: Paul Bourke <email address hidden>
Date: Mon Dec 5 15:56:18 2011 +0000

    Secondary iteration of fix for bug 891738

    A problem exists that images larger than large_object_size without a
    known image_size will fail to be segmented and hence fail to be added
    to swift.

    This patch alters the chunking process slightly:

    - Instead of reading min(large_object_chunk_size, bytes_left) from the
      stream, simply read large_object_chunk_size until we reach EOL(None).
      This works even when there is less than large_object_chunk_size bytes
      amount available.

    - The image_size can be determined from the combined length of each
      chunk read. If we are passed an unknown image_size this value can be
      used for the registry.

    Test Changes:

    - Fixed bug in fake_put_object which was highlighted by Jay when
      reviewing this changeset.
      The put of the manifest object needed to be moved inside the
      check for previous existence of a key.

    - Added test_add_large_object_zero_size to
      glance.tests.unit.test_swift_store:TestStore.

      This unit test demonstrates the bug and will test for regressions. It
      mirrors test_add_large_object but specifies an image_size of 0 to
      excercise the new code path.

    Change-Id: Ic8b78be5dce2281c80372ed446499f6d0bc07f40

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/diablo
Review: https://review.openstack.org/2863

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/diablo
Review: https://review.openstack.org/2864

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/2863
Committed: http://github.com/openstack/glance/commit/34aa8969fadf6b2eac463d13e2c3b1cde7e9d4bc
Submitter: Jenkins
Branch: stable/diablo

commit 34aa8969fadf6b2eac463d13e2c3b1cde7e9d4bc
Author: Paul Bourke <email address hidden>
Date: Mon Nov 21 16:50:23 2011 +0000

    Fix bug 891738

    Compute image size for the registry in the case where none is specified.

    (cherry picked from commit 1f28b6ca7efb9bed12052c10af71e57773693970)

    Change-Id: Ic21adf4865d0f481db9bd362cfaeeebd0942c974