Creating glance image size 4.5GB triggers to an error 500

[Sergii Turivnyi]

ISO:
{"build_id": "2015-07-30_16-01-07", "build_number": "113", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-07-30_16-01-07", "build_number": "113", "api": "1.0", "fuel-library_sha": "d1291ae75680818e715608814422075049a10ce8", "nailgun_sha": "21ba6e2606a056883734392187845c172ecf99aa", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "1512b9af6b41cc95c4d891c593aeebe0faca5a63", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "dee9f2eb7e2822e89f6253f500f0c2e376a5b824", "production": "docker", "python-fuelclient_sha": "71bb8fa87ee25f0c1bb84317884da7c917902a63", "astute_sha": "488db988a1f2e18f99decf417371c50b2a7fb794", "fuel-ostf_sha": "92cdab6c6829be0d2d0c561fe56346dac8708d95", "release": "7.0", "fuelmain_sha": "de5b333815f8541224c6726dc8446ffc7fb18b5b"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "d1291ae75680818e715608814422075049a10ce8", "nailgun_sha": "21ba6e2606a056883734392187845c172ecf99aa", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "1512b9af6b41cc95c4d891c593aeebe0faca5a63", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "dee9f2eb7e2822e89f6253f500f0c2e376a5b824", "production": "docker", "python-fuelclient_sha": "71bb8fa87ee25f0c1bb84317884da7c917902a63", "astute_sha": "488db988a1f2e18f99decf417371c50b2a7fb794", "fuel-ostf_sha": "92cdab6c6829be0d2d0c561fe56346dac8708d95", "release": "7.0", "fuelmain_sha": "de5b333815f8541224c6726dc8446ffc7fb18b5b"}

Preconditions:
Download Windows server 2008 iso to controller node
9600.17050.WINBLUE_REFRESH.140317-1640_X64FRE_SERVER_EVAL_EN-US-IR3_SSS_X64FREE_EN-US_DV9.ISO

Steps to reproduce:
Navigate to controller node using ssh
Execute: glance image-create --file 9600.17050.WINBLUE_REFRESH.140317-1640_X64FRE_SERVER_EVAL_EN-US-IR3_SSS_X64FREE_EN-US_DV9.ISO --name "win" --container-format bare --disk-format iso

Actual result:
HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
http://paste.openstack.org/show/406771/

https://bugs.launchpad.net/glance/+bug/1481320

Expected result:
Image status is active

[Mike Fedosin (mfedosin)]

From your paste:
BackendException: Failed to add object to Swift.
Got error from Swift: put_object('glance', '4a7d39b8-a716-4cc9-a90d-900ef2f200eb', ...)

Seems like there was something wrong inside Swift...

[Evgeny Sikachev (esikachev)]

Mike, if we try upload image via Horizon - uploading is correct. This bug reproduce only from CLI

[Darja Shakhray (dshakhray)]

I uploaded image with size 3GB, bug reproduced

[Evgeny Sikachev (esikachev)]

When we uploaded image big size, glance raise BackendException: Failed to add object to Swift. After this exception glance does not load any images from CLI. Via Horizon we can upload any images(less 3GB)

[Evgeny Sikachev (esikachev)]

i think, we have next problem: if you deploy environment with mos-components-ci scripts, then for "/"(root) we have 20G. When we upload images via glance+swift, swift use root disk and chunks use all memory on node. I think is swift problem. Need cleaning chunk after correct/incorrect upload. if it done manually, image upload correct

[Timur Nurlygayanov (tnurlygayanov)]

The root of the issue in swift, which tries to store all chunks on the disk and doesn't remove chunk in upload of file fails. Swift should remove all chunks if upload finished successfully or with errors.

Possible workaround:
we need to allocate more hard disk space on controller nodes to upload files and manually clear directory when swift stored the data.

[Alexey Galkin (agalkin)]

We must make sure that the Swift removes chunks, but now it's just a guess. I'll check it.

[Sergey Yudin (tsipa740)]

>We must make sure that the Swift removes chunks, but now it's just a guess. I'll check it.

I believe this was done in 2012 https://bugs.launchpad.net/glance/+bug/1019421

[Timur Nurlygayanov (tnurlygayanov)]

Aleksey, could you please chack the issue again of the latest build? for example, on MOS 7.0 test ISO #219?

Thank you!

[Alexey Galkin (agalkin)]

Verified on MOS 7.0 in ISO #228.

root@node-1:~# glance image-create --file 9600.17050.WINBLUE_REFRESH.140317-1640_X64FRE_SERVER_EVAL_EN-US-IR3_SSS_X64FREE_EN-US_DV9.ISO --name "win" --container-format bare --disk-format iso --progress

[=============================>] 100%

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 5b5e08c490ad16b59b1d9fab0def883a |
| container_format | bare |
| created_at | 2015-08-25T14:39:34.000000 |
| deleted | False |
| deleted_at | None |
| disk_format | iso |
| id | 54dc8ba0-52a5-473f-aefd-0f9a187c5b62 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | win |
| owner | daa51074d1074cfda7aed0105d56e7ad |
| protected | False |
| size | 4542291968 |
| status | active |
| updated_at | 2015-08-25T14:42:12.000000 |
| virtual_size | None |
+------------------+--------------------------------------+
root@node-1:~#