Upload image with glance crash in CentOS HA for large images

[Aviram Bar-Haim]

After a successful CentOS HA installation, and Cirros image upload, glance image-create failed with image of 800MB.

log:

[root@node-1 ~]# glance image-create --name MLNX_VM --disk-format=qcow2 --container-format=bare --file /tmp/VM_CentOS6.4x64_img.qcow2

Request returned failure status.
500 Internal Server Error
Failed to upload image 40b3fd76-1ff2-4981-9611-2571e191a739
    (HTTP 500)

in syslog:
<43>Jul 30 07:38:12 node-2 swift-proxy-server: ERROR with Object server 192.168.1.3:6000/2 re: Trying to write to /v1/AUTH_437bd4ea513a489f8f31a2ef75c0b0bd/glance/ac676806-7721-4a72-859b-79fd4905553a: ChunkWriteTimeout (10s)

and

2014-07-30 07:38:16.939 5547 TRACE glance.api.v1.upload_utils BackendException: Failed to add object to Swift.
2014-07-30 07:38:16.939 5547 TRACE glance.api.v1.upload_utils Got error from Swift: put_object('glance', 'ac676806-7721-4a72-859b-79fd4905553a', ...) failure and no ability to reset contents for reupload.

After changing in all controllers the following lines in /etc/haproxy/haproxy.cfg:
  timeout http-request 1m
  timeout queue 1m
  timeout connect 1m
  timeout client 1m
  timeout server 1m
  timeout check 1m

And restart haproxy with: "crm resource restart clone_p_haproxy"

I managed to upload:

[root@node-2 ~]# glance image-create --name MLNXVM --disk-format=qcow2 --container-format=bare --file /tmp/VM_CentOS6.4x64_img.qcow2

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 7b9884a9ab42715f78b6541cb5c2e00d |
| container_format | bare |
| created_at | 2014-07-30T10:52:17 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | c5cd2d5e-2dd6-4d4d-92a0-e409f00e893e |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | MLNXVM |
| owner | e182a5603ab54bc6bcac5e5a6ba8f1f1 |
| protected | False |
| size | 861536256 |
| status | active |
| updated_at | 2014-07-30T10:52:49 |
| virtual_size | None |
+------------------+--------------------------------------+

[Nastya Urlapova (aurlapova)]

Amiram, also provide iso version and diagnostic snapshot.

[Aviram Bar-Haim (aviramb)]

from http://<master_node>:8000/api/version:
{"build_id": "2014-07-27_05-27-27", "ostf_sha": "8c328521b1444f22c50463b9432193e20ed33813", "build_number": "44", "auth_required": true, "api": "1.0", "nailgun_sha": "83cc9ed44ebc8dd97248483b6d414ebbc4cff3c0", "production": "docker", "fuelmain_sha": "9adfbf5a52cedbdd16ec1a74f6c44c5b3419b87c", "astute_sha": "aa5aed61035a8dc4035ab1619a8bb540a7430a95", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "d1c7f67b3cf51978d3178c8666ea398f2477dcb5"}

diag snapshot here: https://drive.google.com/file/d/0B26TG8UdpL-EazQ5N2VkZ0d0eTQ/edit?usp=sharing

snapshot is missing glance logs, so I've added them in the bug description

[Dmitry Borodaenko (angdraug)]

Looks like a duplicate of bug #1348701, but the error message glance gets from swift is slightly different, lets keep them separate until we know more.

[Sergii Golovatiuk (sgolovatiuk)]

I was not able to reproduce this issue neither on LVM nor Ceph storage backends. It looks like its related to iSer configuration or was already fixed.

[Aviram Bar-Haim (aviramb)]

Checked and it's working on 2014-08-06_11-23-14, probably fixed.