Horizon image upload failed - Danger error submitting form

[Tzach Shefi]

Description of problem: When uploading an image (Fedora-Server-DVD-x86_64-21_Alpha.iso ~1.9G) to glance from Horizon it fails, Horizon error message:
Danger: There was an error submitting the form. Please try again.

Version-Release number of selected component (if applicable):
rhel7
python-django-horizon-2014.2.1-4.el7ost.noarch

How reproducible:
Tried it a few times same error, also tested other file types and sizes, including qcow2 ~300mb.
Tested on a second setup, CEPH as Glance backend, same error.

Steps to Reproduce:
1. Upload from Horizon large image, in my case 1.9Giga ISO file, get error message
2.Try to upload same file from CLI works fine.
3. FYI a small Cirros image is created successfully.

Actual results:
Failed to upload image from Horizon:
Danger: There was an error submitting the form. Please try again.

Expected results:
Image should upload successfully without error.

Additional info:
Attaching Horizon httpd glance logs.
Done from Firefox browser.

[Tzach Shefi (tshefi)]

FYI
Did some more testing, it looks like this is a local client (laptop) issue.
Still can't explain what causes it or how to reproduce.

From my laptop to two deployments still fails just rechecked again, then from another laptop to both deployments works fine.

Guess it's a case of "close not a bug", no point in troubleshooting.

[Wu Wenxiang (wu-wenxiang)]

Thanks, maybe you can collect horizon error log while repo this issue.

[Tzach Shefi (tshefi)]

Here you go horizon error log.
My laptops's IP is 10.35.2.129

Maybe this is a tip.

[Tue Jan 27 14:07:38.624319 2015] [:error] [pid 26471] [remote 10.35.2.129:4292] mod_wsgi (pid=26471): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Tue Jan 27 14:07:38.624396 2015] [:error] [pid 26471] [remote 10.35.2.129:4292] IOError: failed to write data

[Wu Wenxiang (wu-wenxiang)]

Agree with you, IOError, seems not a horizon bug.

[Matthias Runge (mrunge)]

There still seem to be an issue with uploading images. We should make sure, there is enough space to store the file.

[Timur Sufiev (tsufiev-x)]

This bug appears to be a consequence of bug 1403129 - because Horizon first stores the image at the controller node instead of streaming data directly to Glance. The following lines from horizon_error.log have lead me to that conclusion:

[Sun Jan 25 13:38:57.438590 2015] [:error] [pid 10292] [remote 10.35.2.129:188] mod_wsgi (pid=10292): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Sun Jan 25 13:38:57.438663 2015] [:error] [pid 10292] [remote 10.35.2.129:188] IOError: failed to write data

[Tzach Shefi (tshefi)]

Adding logs form another deployment, same problem, Image create fails with Danger error.
Uploading failed for 1gb as well as for 17gb, odd Horizon traceback happened with 17GB image, I didn't see it with 1Gb image which I tired to upload later.

Adding info about deployment:
Juno HA deployment over RHEL7.1
VIP of both Glance and Horizon were on same controller at time of the problem.

Deployment will remain running for a while, if any one needs more debugging info let me know what.

[Timur Sufiev (tsufiev-x)]

Tzach, could you please run 'df -h' on controller node to make sure that the error was due to insufficient space?

[Tzach Shefi (tshefi)]

Timur, wasn't sure if to replay to your Q here or answer on #1403129.
Glance mounted on an NFS share (EMC).
Doesn't look like it's a limited disk space issue, other controllers more or less the same.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root 465G 94G 371G 21% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 39M 16G 1% /dev/shm
tmpfs 16G 1.2G 15G 8% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda2 477M 124M 328M 28% /boot
10.....:/OpenStack/glance 2.4T 265G 2.1T 12% /var/lib/glance/images

[Matthias Runge (mrunge)]

I don't think the issue is limited space on glance side. But the image needs to be buffered somewhere (on horizon host).

[Tzach Shefi (tshefi)]

Just happen on another another deployment, this time a simple all in one not glance over local dir.
Kilo release, over rhel 7.1
python-django-horizon-2015.1.0-5.el7ost.noarch

3.7G image created via CLI , from horizon same image failed with Danger..
This time Horizon log shows

2015-05-13 13:03:54,012 3498 INFO openstack_auth.forms Login successful for user "admin".
2015-05-13 13:03:55,675 3497 ERROR horizon.middleware Total Cookie size for user_id: 7f87e3b15f33424e89ede54ccb8d3d02 is 4211B >= 4093B. You need to configure file-based or database-backed sessions instead of cookie-based sessions: http://docs.openstack.org/developer/horizon/topics/deployment.html#session-storage
2015-05-13 13:04:00,228 3498 ERROR horizon.middleware Total Cookie size for user_id: 7f87e3b15f33424e89ede54ccb8d3d02 is 4179B >= 4093B. You need to configure file-based or database-backed sessions instead of cookie-based sessions: http://docs.openstack.org/developer/horizon/topics/deployment.html#session-storage
2015-05-13 13:04:01,072 3499 ERROR horizon.middleware Total Cookie size for user_id: 7f87e3b15f33424e89ede54ccb8d3d02 is 4187B >= 4093B. You need to configure file-based or database-backed sessions instead of cookie-based sessions: http://docs.openstack.org/developer/horizon/topics/deployment.html#session-storage
2015-05-13 13:15:32,289 3498 WARNING django.request Forbidden (CSRF token missing or incorrect.): /dashboard/admin/images/create/

[Timur Sufiev (tsufiev-x)]

Tzach, the last issue IMO is obviously connected with cookies storage per-domain exhaustion, since in your deployment they are also used as a storage for sessions. CSRF token is missing because browser couldn't add its cookie to a request (again, IMO).

Regarding comment 10, where in this configuration the /tmp folder is located? It's the common place where django usually stores the uploaded files (see https://github.com/django/django/blob/master/django/core/files/uploadedfile.py#L64)

[Tzach Shefi (tshefi)]

Timur thanks, I'm not a Horizon guy, cookie exhaustion or CSRF token don't mean much to me.
If there is a default setting I should change on kilo config I'd be happy to try again, please point me in the right direction.

Point being Kilo also has a problem, from end-user perspective result looks the same, can't upload large images.
If it's the same bug, or new/different one for Kilo it's still a problem which needs attention.

As per comment 10, doesn't look like a limited space problem on tmp
# grep -ir tmp /etc/openstack-dashboard/*
/etc/openstack-dashboard/local_settings:FILE_UPLOAD_TEMP_DIR = '/tmp'

# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root 465G 94G 371G 21% /