ChunkReader has no len() Swiftclient + Glance

Bug #1297465 reported by Michael Petersen
36
This bug affects 6 people
Affects Status Importance Assigned to Milestone
glance_store
Fix Released
Undecided
Unassigned
python-swiftclient
Fix Released
Undecided
Unassigned

Bug Description

On CentOS 6.5

Name : openstack-glance
Arch : noarch
Version : 2013.2.2
Release : 2.el6
Size : 52 k
Repo : installed
From repo : openstack-havana

When uploading an image that is larger than swifts chunk size I receive a ChunkReader error:

2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils TypeError: object of type 'ChunkReader' has no len()

2014-03-25 18:29:16.916 21092 TRACE glance.store.swift TypeError: object of type 'ChunkReader' has no len()

Full Traceback:

2014-03-25 18:29:16.916 21092 ERROR glance.store.swift [34633528-d86d-4055-bdc9-1bcdf872fc2b OpenStack Admin 57813631e9e5420589216b33925ef6a3] Error during chunked upload to backend, deleting stale chunks
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift Traceback (most recent call last):
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/glance/store/swift.py", line 384, in add
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift content_length=content_length)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1318, in put_object
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift response_dict=response_dict)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1192, in _retry
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift rv = func(self.url, self.token, *args, **kwargs)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 943, in put_object
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift conn.putrequest(path, headers=headers, data=contents)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 197, in putrequest
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift return self.request('PUT', full_path, data, headers, files)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 187, in request
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift files=files, **self.requests_args)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 176, in _request
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift return requests.request(*arg, **kwarg)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift return session.request(method=method, url=url, **kwargs)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 276, in request
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift prep = req.prepare()
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/requests/models.py", line 224, in prepare
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift p.prepare_body(self.data, self.files)
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift File "/usr/lib/python2.6/site-packages/requests/models.py", line 384, in prepare_body
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift self.headers['Content-Length'] = str(len(body))
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift TypeError: object of type 'ChunkReader' has no len()
2014-03-25 18:29:16.916 21092 TRACE glance.store.swift
2014-03-25 18:29:16.977 21092 ERROR glance.api.v1.upload_utils [34633528-d86d-4055-bdc9-1bcdf872fc2b OpenStack Admin 57813631e9e5420589216b33925ef6a3] Failed to upload image 375a5784-8911-433a-a2c2-56ea0c621eda
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 101, in upload_data_to_store
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils store)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/glance/store/__init__.py", line 333, in store_add_to_backend
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils (location, size, checksum, metadata) = store.add(image_id, data, size)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/glance/store/swift.py", line 384, in add
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils content_length=content_length)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1318, in put_object
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils response_dict=response_dict)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1192, in _retry
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils rv = func(self.url, self.token, *args, **kwargs)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 943, in put_object
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils conn.putrequest(path, headers=headers, data=contents)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 197, in putrequest
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils return self.request('PUT', full_path, data, headers, files)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 187, in request
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils files=files, **self.requests_args)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 176, in _request
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils return requests.request(*arg, **kwarg)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils return session.request(method=method, url=url, **kwargs)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 276, in request
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils prep = req.prepare()
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/requests/models.py", line 224, in prepare
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils p.prepare_body(self.data, self.files)
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/requests/models.py", line 384, in prepare_body
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils self.headers['Content-Length'] = str(len(body))
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils TypeError: object of type 'ChunkReader' has no len()
2014-03-25 18:29:16.977 21092 TRACE glance.api.v1.upload_utils

tags: added: havana
Revision history for this message
Michael Petersen (mpetason) wrote :

As an update: The size doesn't seem to matter.

If I use this and try to upload a 7G Windows image then I get the same error as above:

swift_store_large_object_size = 5120

If I use this and upload a Centos 300M Image I get the same error as above:

swift_store_large_object_size = 250

Then this setting is left at 200:

swift_store_large_object_chunk_size = 200

I have tried to change the object chunk size to different values as well, lower and higher. 50, 500, 1024. Each produces the same error: TypeError: object of type 'ChunkReader' has no len().

no longer affects: swift
Revision history for this message
bewiwi (bewiwi-9) wrote :

I have the same problem and it affects snapshotting system too

Revision history for this message
Charles Shultz (drcms) wrote :

Same problem here. As soon as the image file size exceeds swift_store_large_object_size (default of 5GB) and the image shall be chunked into partitions, glance will fail.
This has been working for me with the initial rollout of Havanna, so this may have been introduced with an update.

Revision history for this message
Jakub Pavlik (pavlk-jakub) wrote :

Hello,
I have exactly same problem. Can anybody help us? I tried to update at new version of stable Havana 2013.2.3, but it does not help.

Revision history for this message
Vlastimil Mikes (v-mikes) wrote :

Same problem here, any help would be appreciated.

Revision history for this message
david martin (dmartls1) wrote :

Looks like a bug in pyrthon-requests that was fixed a while ago: https://github.com/sigmavirus24/requests/commit/1d5c4f3f0f49e04165303952ccc5fd1c70aee7c1

I was able to work around the issue by hacking /usr/lib/python2.6/site-packages/requests/models.py
-self.headers['Content-Length'] = str(len(body))
+self.headers['Content-Length'] = super_len(body)

Revision history for this message
Robert van Leeuwen (rovanleeuwen) wrote :

Thx for the workaround David.
Note that you (also?) need this on the compute nodes.
Just doing the glance api node is not enough.

Revision history for this message
Jakub Pavlik (pavlk-jakub) wrote :

I updated at version python-swiftclient-2.0.3-1.el6.noarch and now it is OK. It works also with version 1.8.0-1.el6.noarch, but 2.0.2-1.el6.noarch seems to be broken.

Revision history for this message
HouMing Wang (houming-wang) wrote :

I encountered the same problem, and I can fix this by the following 2 steps:
1.Update the version of python-requests from python-requests-1.1.0-4.el6.noarch to python-requests-1.1.0-4.el6.noarch

2. Change the "registry_client_timeout = 0" in /etc/glance/glance-api.conf

Dolph Mathews (dolph)
tags: added: havana-backport-potential
removed: glance havana swiftclient
Revision history for this message
Erno Kuvaja (jokke) wrote :

Moved to glance-store as the swift driver lives there now.

affects: glance → glance-store
Revision history for this message
Louis Taylor (kragniz) wrote :

Marking as fix released, since the underlying bug in requests has been released.

Changed in glance-store:
status: New → Fix Released
Changed in python-swiftclient:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.