Should swift backend compute size if none is specified?

Bug #891738 reported by Tom Hancock on 2011-11-17
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Medium
Paul Bourke
Diablo
Undecided
Unassigned

Bug Description

From an email I posted to the mailing list yesterday
===========================================

We ran into a backward compat issue closely related to this last week.

If the newer Glance code in Diablo is running on a server configured with a Swift backend and interacting with an older Glance client on an older Nova system that doesn't set content-length then (even assuming the image is < 5GB) it gets stored in the backend using the usual means, but the image size in the registry is never set correctly.
These few lines were in the pre-diablo code to do a HEAD on the object in swift if the size was unknown and if they're re-added then the registry gets updated with the correct length. It looks like this still applies to essex.

regards,
Tom

---------------------------- glance/store/swift.py ---------------------------- @@ -339,6 +339,12 @@ class Store(glance.store.base.Store):
                 # best...
                 obj_etag = swift_conn.put_object(self.container, obj_name,
                                                  image_file)
+ if image_size == 0:
+ resp_headers = swift_conn.head_object(self.container, obj_name)
+ # header keys are lowercased by Swift
+ if 'content-length' in resp_headers:
+ image_size = int(resp_headers['content-length'])
+ logger.debug(_("Zero image size so setting
+ object size to %s") % image_size)
             else:
                 # Write the image into Swift in chunks. We cannot
                 # stream chunks of the webob.Request.body_file, unfortunately,

Jay Pipes (jaypipes) wrote :

Tom, I'm going to have to get you used to proposing patches in Gerrit ;)

Thanks for the patch. I'll assign this to someone, and also, Ewan's idea about if image_size = 0, then do the segmented manifest upload make a lot of sense. Not sure why that didn't occur to me earlier, but oh well!

-jay

Changed in glance:
status: New → Triaged
importance: Undecided → Medium
milestone: none → essex-2

Reviewed: https://review.openstack.org/1819
Committed: http://github.com/openstack/glance/commit/1f28b6ca7efb9bed12052c10af71e57773693970
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 1f28b6ca7efb9bed12052c10af71e57773693970
Author: Paul Bourke <email address hidden>
Date: Mon Nov 21 16:50:23 2011 +0000

    Fix bug 891738

    Compute image size for the registry in the case where none is specified.

    Change-Id: Ic21adf4865d0f481db9bd362cfaeeebd0942c974

Changed in glance:
status: Triaged → Fix Committed
Paul Bourke (pauldbourke) wrote :

All, while the fix for this has been deemed valid, there is a secondary improvement.

Images that have no size specified *and* are larger than 5G will still fail to be added to Swift. I'm going to push a topic branch called bug/891738-improvement that addresses this.

Hope it's ok to post here, let me know if you'd like me to open a new bug for this.

Jay Pipes (jaypipes) wrote :
Changed in glance:
assignee: nobody → Paul Bourke (pauldbourke)
status: Fix Committed → In Progress

Reviewed: https://review.openstack.org/2072
Committed: http://github.com/openstack/glance/commit/98b1ef337ee63f239829b44aec4d05cea99f98ad
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 98b1ef337ee63f239829b44aec4d05cea99f98ad
Author: Paul Bourke <email address hidden>
Date: Mon Dec 5 15:56:18 2011 +0000

    Secondary iteration of fix for bug 891738

    A problem exists that images larger than large_object_size without a
    known image_size will fail to be segmented and hence fail to be added
    to swift.

    This patch alters the chunking process slightly:

    - Instead of reading min(large_object_chunk_size, bytes_left) from the
      stream, simply read large_object_chunk_size until we reach EOL(None).
      This works even when there is less than large_object_chunk_size bytes
      amount available.

    - The image_size can be determined from the combined length of each
      chunk read. If we are passed an unknown image_size this value can be
      used for the registry.

    Test Changes:

    - Fixed bug in fake_put_object which was highlighted by Jay when
      reviewing this changeset.
      The put of the manifest object needed to be moved inside the
      check for previous existence of a key.

    - Added test_add_large_object_zero_size to
      glance.tests.unit.test_swift_store:TestStore.

      This unit test demonstrates the bug and will test for regressions. It
      mirrors test_add_large_object but specifies an image_size of 0 to
      excercise the new code path.

    Change-Id: Ic8b78be5dce2281c80372ed446499f6d0bc07f40

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2011-12-14
Changed in glance:
status: Fix Committed → Fix Released

Fix proposed to branch: stable/diablo
Review: https://review.openstack.org/2864

Reviewed: https://review.openstack.org/2863
Committed: http://github.com/openstack/glance/commit/34aa8969fadf6b2eac463d13e2c3b1cde7e9d4bc
Submitter: Jenkins
Branch: stable/diablo

commit 34aa8969fadf6b2eac463d13e2c3b1cde7e9d4bc
Author: Paul Bourke <email address hidden>
Date: Mon Nov 21 16:50:23 2011 +0000

    Fix bug 891738

    Compute image size for the registry in the case where none is specified.

    (cherry picked from commit 1f28b6ca7efb9bed12052c10af71e57773693970)

    Change-Id: Ic21adf4865d0f481db9bd362cfaeeebd0942c974

tags: added: in-stable-diablo
Thierry Carrez (ttx) on 2012-04-05
Changed in glance:
milestone: essex-2 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers