FileField need clean up temp file and and get file size without loading file

[Gloria Gu]

We are using forms.FileField to upload a client local file.

We noticed that if the file size is big, for example 7 gb, Horizon dumps a temp file in /tmp dir where horizon server runs. If the file loaded is different each time, the temp files are left in the /tmp. Over the long run, it could overflow the disk spaces.

We would like to have a way to clean up the tmp file once file loading is done.

The way that we use to get the file handle in the form's clean or handle method is :

f = self.request.FILES['file"]

If we use this to get the file size in the clean method to validate, the whole file gets loaded either into memory or to temp dir, it would be nice to get the file size without loading the whole file.

[zhaoyim (yyyiming)]

Openstack forms.FileField inherited from Django, and when the upload file<2.5M, the file will write into memory, if >2.5M, it will write into the system temp folder. The attribute FILE_UPLOAD_MAX_MEMORY_SIZE is the file size threshold value whether write to memory or system temp folder. Seems we can set FILE_UPLOAD_MAX_MEMORY_SIZE to control.

[zhaoyim (yyyiming)]

After did some investigation and found already support upload tmp file will be deleted after the upload done.

And for the get file size without loading file, after the file uploaded, we can get the file size, because all the code is running in the httpd server side.

So I think this issue is an invalid defect.

If you want to use more memory instead of disk, you can set FILE_UPLOAD_MAX_MEMORY_SIZE=size(byte) in the /usr/share/openstack-dashboard/openstack_dashboard/settings.py

FILE_UPLOAD_MAX_MEMORY_SIZE
The maximum size, in bytes, for files that will be uploaded into memory. Files larger than FILE_UPLOAD_MAX_MEMORY_SIZE will be streamed to disk.

Defaults to 2.5 megabytes.

@Gloria Gu (gloria-gu) @Lin Hua Cheng (lin-hua-cheng) could you help double confirm? Thanks!

[zhaoyim (yyyiming)]

Din NOT get any response, so mark to invalid. Anyone think it is not right, please change the status and give some comments. Thanks!

[Andrew (openstackdotby)]

Yes, I can confirm problem.

I use both service in workable state: glance and horizon (via apache 2.4 wsgi)

local_settings.py has variable for storing big file for uploading in django level - FILE_UPLOAD_TEMP_DIR = '/opt/openstack/tmp'

In web interface, in section of uploading images, I am doing uploading of image file on 500Mb (raw format) , process is going without problem. WSGI process is making upload of this file in temporary directory and after this putting of image into 'glance'.
It is OK. User is log out from web interface.
Image in glance backend, but in temporary directory I see this temporary file.
'
total 578196
-rw------- 1 homitaka homitaka 592066560 Nov 15 18:19 tmpO4djfP.upload

'

It is real bug and problem. After few uploading I lost 2-3 Gbyte of space. Rebooting of httpd server - file still in tmp. Could you say, where in code of interaction between wsgi process and service of glance need to insert code for removing this file, after creating of image?

[Andrew (openstackdotby)]

After few day envestigation code I created patch:

https://upwork.link/tips-and-tricks/openstack-horizon-bug-of-filefield-need-clean-up-temp-file-or-how-to-remove-supefluos-files-in-horizondjango-after-uploading/

Now all temporary files of images (in context Django/Horizon), that were uploaded via browser, will be removed from temporary place after uploading in backend storage of glance (I use ceph and restriction of size of images up to 512Mb). Everything is OK and work

[Akihiro Motoki (amotoki)]

which Django version is used when this bug occurs?

As of the development version (for Stein release), I cannot reproduce this bug. Django 1.11 is used for python 2.7 environment. When I uploaded an image file larger than 2.5MB, I saw a corresponding temporary file under FILE_UPLOAD_TEMP_DIR directory during being uploaded, but the temporary file was deleted successfully.

[Akihiro Motoki (amotoki)]

Clearing the assignee as it was assigned two years ago and no activity since then.

[Akihiro Motoki (amotoki)]

Note that as of Rocky (and Stein) release the image panel is implemented by Angular version by default but FileField is still used in openstack_dashboard/api/rest/glance.py. If FileField still has this problem, the bug should occur even now.

[Akihiro Motoki (amotoki)]

Marking this bug as Incomplete from the following reasons.

- Django version is unclear
- It seems the bug no longer happens with later Django versions like 1.11 and 2.0

If you still see this, feel free to leave comments and change the status.

[XiaojueGuan (xiaojuegaun)]

[root@localhost system]# ls -lsh /tmp/
total 48K
   0 -rw------- 1 root root 0 Feb 4 18:24 prl_nettool.lock
4.0K drwxr-xr-x 2 root root 4.0K Feb 4 21:03 rootwrap-bkAjrb
4.0K drwxr-xr-x 2 root root 4.0K Feb 4 20:25 rootwrap-J_rK0o
4.0K drwxr-xr-x 2 root root 4.0K Feb 4 21:03 rootwrap-kTBp5q
4.0K drwxr-xr-x 2 root root 4.0K Feb 4 21:03 rootwrap-OpO5Y0
4.0K drwxr-xr-x 2 root root 4.0K Feb 4 20:24 rootwrap-wf1efQ
4.0K drwxr-xr-x 2 root root 4.0K Feb 4 20:24 rootwrap-XPvPoa
4.0K drwx------ 3 root root 4.0K Feb 4 20:28 systemd-private-d8960234f35f4826843726b66a466e5f-chronyd.service-mL1UzQ
4.0K drwx------ 3 root root 4.0K Feb 4 20:29 [e-mail]-m9WrSq
4.0K drwx------ 3 root root 4.0K Feb 4 20:31 systemd-private-d8960234f35f4826843726b66a466e5f-httpd.service-x1OrJ5
4.0K drwx------ 3 root root 4.0K Feb 4 20:29 systemd-private-d8960234f35f4826843726b66a466e5f-mariadb.service-a60zsS
4.0K drwx------ 3 root root 4.0K Feb 4 20:28 systemd-private-d8960234f35f4826843726b66a466e5f-systemd-machined.service-Aj1Tb1
4.0K -rw------- 1 root root 235 Feb 4 14:50 yum_save_tx.2019-02-04.14-50.P4GoUK.yumtx