Kilo: I/O error uploading image

[Tom Verdaat]

Using a Ceph backend. Same configuration works just fine on Juno.

Glance image creation from file using the API works. Horizon image creation from URL works too, but using file upload does not.

Apache error.log:

Unhandled exception in thread started by <function image_update at 0x7fc97aeed320>
Traceback (most recent call last):
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/glance.py", line 112, in image_update
    exceptions.handle(request, ignore=True)
  File "/usr/lib/python2.7/dist-packages/horizon/exceptions.py", line 364, in handle
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/glance.py", line 110, in image_update
    image = glanceclient(request).images.update(image_id, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 329, in update
    resp, body = self.client.put(url, headers=hdrs, data=image_data)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 265, in put
    return self._request('PUT', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 206, in _request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 350, in send
    for i in request.body:
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 170, in chunk_body
    chunk = body.read(CHUNKSIZE)
ValueError: I/O operation on closed file

Horizon log:

[req-0e69bfd9-c6ab-4131-b445-aa57c1a455f7 87d1da7fba6f4f5a9d4e7f78da344e91 ba35660ba55b4a5283c691a4c6d99f23 - - -] Failed to upload image 90a30bfb-946c-489d-9a04-5f601af0f821
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store
    context=req.context)
  File "/usr/lib/python2.7/dist-packages/glance_store/backend.py", line 339, in store_add_to_backend
    context=context)
  File "/usr/lib/python2.7/dist-packages/glance_store/capabilities.py", line 226, in op_checker
    return store_op_fun(store, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/rbd.py", line 384, in add
    self._delete_image(loc.image, loc.snapshot)
  File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/rbd.py", line 290, in _delete_image
    with conn.open_ioctx(target_pool) as ioctx:
  File "/usr/lib/python2.7/dist-packages/rados.py", line 667, in open_ioctx
    raise make_ex(ret, "error opening pool '%s'" % ioctx_name)
ObjectNotFound: error opening pool '90a30bfb-946c-489d-9a04-5f601af0f821'

[reachlin (reachlin)]

I saw this bug too.

[reachlin (reachlin)]

I checked the glanceclient code, and the error happens when determining the file size:
https://github.com/openstack/python-glanceclient/blob/master/glanceclient/common/utils.py#L300

I don't know why they don't use file_obj.size instead

[reachlin (reachlin)]

file_obj.size has the correct value when Horzion is uploading an image from a file.

[reachlin (reachlin)]

After I fixed that problem, there's a new error:

  File "/home/lincai/osee/horizon/.venv/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/lincai/osee/horizon/.venv/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/lincai/osee/horizon/.venv/local/lib/python2.7/site-packages/requests/adapters.py", line 390, in send
    for i in request.body:
  File "/home/lincai/osee/horizon/.venv/local/lib/python2.7/site-packages/glanceclient/common/http.py", line 171, in chunk_body
    chunk = body.read(CHUNKSIZE)
ValueError: I/O operation on closed file.

I suspect there's some sort of threads sync. problem.

[reachlin (reachlin)]

https://groups.google.com/forum/#!msg/django-users/Ni1slAnsfLE/e0e7ztkt2twJ

[reachlin (reachlin)]

This bug is caused by the change in Django 1.7, which is InMemoryUploadedFile is closed automatically. The new thread always has a closed file to read.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/184935

[reachlin (reachlin)]

https://review.openstack.org/#/c/184935/

[Tom Verdaat (tom-verdaat)]

Thanks for the work on this bug. I'm afraid I have to disagree with the Liberty milestone. The Kilo release notes state that Horizon is compatible with Djanog 1.7, which obviously with this bug present it isn't. I therefore think this should be a Kilo bugfix, not just part of a future release.

[Matthias Runge (mrunge)]

Tom, the way it works is, it get's fixed in liberty, then we can backport it to kilo. A backport is not acceptable, if it's not fixed in liberty, too.

[Tom Verdaat (tom-verdaat)]

Great. Thanks for the explanation and tagging it for backporting :-)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184935
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=810ad30607fff86fdb8c9b78fe0ac32d09259b78
Submitter: Jenkins
Branch: master

commit 810ad30607fff86fdb8c9b78fe0ac32d09259b78
Author: lincai <email address hidden>
Date: Fri May 22 10:54:42 2015 +0800

    I/O error uploading image

    Fixed two errors when upload an image from a file.
    One is the file name doesn't exist error.
    Another is file got closed in a new thread since Django 1.7.

    Change-Id: Ia64c385d89e0d938b14182f2744251f76b892eac
    Closes-Bug: #1451429

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/187928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/187928
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=78e9a997e4c6faf393d3bdaa3a043f1796aaa1bd
Submitter: Jenkins
Branch: stable/kilo

commit 78e9a997e4c6faf393d3bdaa3a043f1796aaa1bd
Author: lincai <email address hidden>
Date: Fri May 22 10:54:42 2015 +0800

    I/O error uploading image

    Fixed two errors when upload an image from a file.
    One is the file name doesn't exist error.
    Another is file got closed in a new thread since Django 1.7.

    Change-Id: Ia64c385d89e0d938b14182f2744251f76b892eac
    Closes-Bug: #1451429
    (cherry picked from commit 810ad30607fff86fdb8c9b78fe0ac32d09259b78)

[Anario (anit)]

Could you please explain me what to download and install to fix dashboard kilo ?