Glance with swift backend - upload of an image from horizon is not streaming

[kanimozhis]

Glance is configured to use swift as its backend.
While creating image from horizon using image source as image file, the image is uploaded in the controller occupying the disk space of the controller instead of streaming. A tmp file is created in /tmp directory and the owner of this file horizon user.
Image create fails if there is no enough space in controller to hold the full image.
But when the same image is uploaded using glance CLI (from a remote Linux macine) streaming happens as - remote machine to glance to swift.
Streaming also happens in the case of upload using URL from Horizon.
Horizon should stream in the case of image source as image file

[kanimozhis (kanimozhis)]

It is not about cleaning of /tmp directory.
Instead of creating a tmp file, streaming of file data is required as it happens while uploading image from glance CLI.

[Timur Sufiev (tsufiev-x)]

Storing uploaded files in /tmp is an inherent property of the Django file-uploading machinery: https://docs.djangoproject.com/en/1.6/topics/http/file-uploads/#where-uploaded-data-is-stored

While streaming file contents directly to glance would certainly free up some space on controller, it would also raise the question on how to perform form validation in this case.

[Jyoti Ranjan (jyoti-ranjan)]

The defect directly influence consumption of Horizon for Glance, especially concurrency. If user wants to upload 3 images of large size say 40 GB, he needs to plan for a machine having 120 GB to run horizon. For windows images, 40 GB is common size. Because, all images need to be copied to /tmp directory. For a cloud, 3 concurrent request is very low end and number of concurrent request is likely to be significantly higher.

Considering this aspect, I think that it is a critical defect and streaming functionality is crucial. Otherwise, usage of Glance in horizon will get limited. Or, operator will be forced to use cli which does not look good for Horizon.

[mattfarina (mattfarina)]

This is a problem of scalability and a problem for large uploads, through Horizon, to swift.

I wonder if an easy solution is to change the form so the action points to the API endpoint for glance/swift and have the browser push directly into the service.

[Timur Sufiev (tsufiev-x)]

Nikunj, are you working on this? I have one idea regarding this bug that I'd like to check...

[Nikunj Aggarwal (nikunj2512)]

Timur, you can say that i am thinking how to solve this problem. But if you have an idea on how to solve it, go head and give it a shot.

[Timur Sufiev (tsufiev-x)]

Okay, I'm going to try the luck with the Django Custom Storage https://docs.djangoproject.com/en/1.7/howto/custom-file-storage/

[Timur Sufiev (tsufiev-x)]

Raising to High since it seriously limits the concurrent image upload by different users.

[Timur Sufiev (tsufiev-x)]

The patch under work is at https://review.openstack.org/#/c/166969/

[Matthias Runge (mrunge)]

cool, good to know.

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Lyle (<email address hidden>) on branch: master
Review: https://review.openstack.org/166969
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/230434

[OpenStack Infra (hudson-openstack)]

Change abandoned by Timur Sufiev (<email address hidden>) on branch: master
Review: https://review.openstack.org/166969
Reason: More lightweight solution is here: https://review.openstack.org/230434

[Timur Sufiev (tsufiev-x)]

Update: waiting for Glance team to implement multipart/form-data parser on server-side, to enable 'more lightweight solution' ^^^

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/317365

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/317365
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=7e65af5f135a272eed975edd0a797c3bdf086b82
Submitter: Jenkins
Branch: master

commit 7e65af5f135a272eed975edd0a797c3bdf086b82
Author: Timur Sufiev <email address hidden>
Date: Thu May 19 18:15:24 2016 +0300

    Embed support for external data sinks into api.glance

    In case 'data' image attribute is a base string (instead of in-memory
    or on-disk file), api.glance sends back an image wrapper with a redirect
    url and a token to its caller, so the caller could upload the file to
    that url directly.

    Provide a unit test for api.glance behavior when an external upload
    location is used. That also requires to fix glance stub endpoint data
    in keystone_data.py since it didn't reflect the reality.

    Also document the new HORIZON_IMAGES_UPLOAD_MODE setting that will
    govern direct images upload and the define approach to deprecating the
    old HORIZON_IMAGES_ALLOW_UPLOAD setting. The old setting is deprecated
    as of Newton release and planned to be removed in P. 'Removing' means
    that it will no longer be used / referenced at all in code, not the
    actual presence in settings.py (it is removed from settings.py in this
    commit). What really matters is if the customized value of
    HORIZON_IMAGES_ALLOW_UPLOAD in local_settings.py will be still
    considered during the deprecation period.

    Help text in Django Create Image form in case if local file upload was
    enabled was wrong, fixed that.

    Related-Bug: #1403129
    Partially implements blueprint: horizon-glance-large-image-upload

    Change-Id: I24ff55e0135514fae89c20175cf9c764e871969b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230434
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=93af461e40fca76eae58066cc1f9de83d7b716ea
Submitter: Jenkins
Branch: master

commit 93af461e40fca76eae58066cc1f9de83d7b716ea
Author: Timur Sufiev <email address hidden>
Date: Tue May 17 13:16:15 2016 +0300

    [Django] Allow to upload the image directly to Glance service

    Since large Glance images even temporarily stored on dashboard side
    tend to fill up Web Server filesystem, it is desirable to route image
    payload directly to Glance service (which usually streams it to
    storage backend, which in turn has plenty of space).

    To make it possible we need to trick Django into thinking that a file
    was selected inside FileInput, while its contents are not actually
    transferred to Django server. Then, once image is created client-side
    code needs to know the exact url the image payload needs to be
    transferred to. Both tasks are solved via using ExternalFileField /
    ExternalUploadMeta classes which allow to work around the usual Django
    form processing workflow with minimal changes to CreateImage form
    business logic.

    The client-side code relies on CORS being enabled for Glance service
    (otherwise browser would forbid the PUT request to a location
    different from the one form content came from). In a Devstack setup
    you'll need to edit [cors] section of glance-api.conf file, setting
    `allowed_origin` setting to the full hostname of the web server (say,
    http://<HOST_IP>/dashboard) and restart glance-api process.

    A progress bar is implemented to track the progress of a file upload,
    in case a really huge image is transferred.

    The new machinery could be easily switched on/off with a single
    setting `HORIZON_IMAGES_UPLOAD_MODE` set to 'direct' / 'legacy'.

    Related-Bug: #1467890
    Closes-Bug: #1403129
    Implements blueprint: horizon-glance-large-image-upload
    Change-Id: I01d02f75268186b43066df6fd966aa01c08e01d7

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/horizon 10.0.0.0b3 development milestone.

[SUYASH KARMARKAR (skarmarkar)]

Is there a workaround to get past this issue till we have a fix?

[Timur Sufiev (tsufiev-x)]

Yes, disable the local file upload in Create Image is the well-known workaround for this issue :).

[anurag jindal (anurag-jindal29)]

Hi Suyash

Did you get he workaround ? If yes then please share.