failed to load image via glance cli

[Andrey Shestakov]

OS: Centos
Mode: HA

Steps to reproduce:
1. load small image via glance image-create - success
2. load big image via glance image-create - fail

My case:

[root@node-2 F17-x86_64-cfntools]# dd if=/dev/zero of=/tmp/moo100.img bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.105806 s, 991 MB/s

[root@node-2 F17-x86_64-cfntools]# glance -d -v image-create --is-public true --disk-format qcow2 --container-format bare --file /tmp/moo100.img --name moo100
/usr/lib/python2.6/site-packages/eventlet/hubs/__init__.py:8: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
curl -i -X POST -H 'x-image-meta-container_format: bare' -H 'Transfer-Encoding: chunked' -H 'User-Agent: python-glanceclient' -H 'x-image-meta-size: 104857600' -H 'x-image-meta-is_public: True' -H 'X-Auth-Token: 52fd6840e659419ab667a7ce39aca8a4' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: qcow2' -H 'x-image-meta-name: moo100' -d '<open file u'/tmp/moo100.img', mode 'rb' at 0x1b104b0>' http://198.11.227.34:9292/v1/images

HTTP/1.1 201 Created
content-length: 479
etag: 2f282b84e7e608d5852449ed940bfc51
location: http://198.11.227.34:9292/v1/images/1deeed8d-2b09-4789-b86e-d0f9eb9529ee
date: Fri, 25 Jul 2014 15:30:29 GMT
content-type: application/json
x-openstack-request-id: req-6e14bc19-c595-4aa3-8b96-90a86635b83b

{"image": {"status": "active", "deleted": false, "container_format": "bare", "min_ram": 0, "updated_at": "2014-07-25T15:30:29", "owner": "3e8a4979b9c24899b17cbd94c5c1ebdf", "min_disk": 0, "is_public": true, "deleted_at": null, "id": "1deeed8d-2b09-4789-b86e-d0f9eb9529ee", "size": 104857600, "virtual_size": null, "name": "moo100", "checksum": "2f282b84e7e608d5852449ed940bfc51", "created_at": "2014-07-25T15:30:19", "disk_format": "qcow2", "properties": {}, "protected": false}}

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 2f282b84e7e608d5852449ed940bfc51 |
| container_format | bare |
| created_at | 2014-07-25T15:30:19 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 1deeed8d-2b09-4789-b86e-d0f9eb9529ee |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | moo100 |
| owner | 3e8a4979b9c24899b17cbd94c5c1ebdf |
| protected | False |
| size | 104857600 |
| status | active |
| updated_at | 2014-07-25T15:30:29 |
| virtual_size | None |
+------------------+--------------------------------------+

[root@node-2 F17-x86_64-cfntools]# dd if=/dev/zero of=/tmp/moo1000.img bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.89716 s, 1.2 GB/s

[root@node-2 F17-x86_64-cfntools]# glance -d -v image-create --is-public true --disk-format qcow2 --container-format bare --file /tmp/moo1000.img --name moo1000
/usr/lib/python2.6/site-packages/eventlet/hubs/__init__.py:8: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
curl -i -X POST -H 'x-image-meta-container_format: bare' -H 'Transfer-Encoding: chunked' -H 'User-Agent: python-glanceclient' -H 'x-image-meta-size: 1048576000' -H 'x-image-meta-is_public: True' -H 'X-Auth-Token: 381c48fab79d4e89b66be5fc2379850c' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: qcow2' -H 'x-image-meta-name: moo1000' -d '<open file u'/tmp/moo1000.img', mode 'rb' at 0x1a824b0>' http://198.11.227.34:9292/v1/images

HTTP/1.1 500 Internal Server Error
date: Fri, 25 Jul 2014 15:31:13 GMT
content-length: 91
content-type: text/plain; charset=UTF-8
x-openstack-request-id: req-8679dc20-e982-484e-b861-79aa4197d9c8

500 Internal Server Error

Failed to upload image 7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2

Request returned failure status.
500 Internal Server Error
Failed to upload image 7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2
    (HTTP 500)

glance-all.log:
<151>Jul 25 15:30:52 node-2 glance-swiftclient DEBUG: REQ: curl -i http://198.11.227.34:8080/v1/AUTH_3bf99c76b10344aab4484f524f09005a/glance -I -H "X-Auth-Token: e50f3a0138084efbb4321e967db3c6f7"
<151>Jul 25 15:30:52 node-2 glance-swiftclient DEBUG: RESP STATUS: 204
<151>Jul 25 15:30:52 node-2 glance-glance.store.swift DEBUG: Adding image object '7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2' to Swift
<151>Jul 25 15:31:12 node-2 glance-swiftclient DEBUG: REQ: curl -i http://198.11.227.34:8080/v1/AUTH_3bf99c76b10344aab4484f524f09005a/glance/7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2 -X PUT -H "X-Auth-Token: e50f3a0138084efbb4321e967db3c6f7"
<151>Jul 25 15:31:12 node-2 glance-swiftclient DEBUG: RESP STATUS: 503
<151>Jul 25 15:31:12 node-2 glance-swiftclient DEBUG: RESP BODY: <html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>
<147>Jul 25 15:31:13 node-2 glance-glance.store.swift ERROR: Failed to add object to Swift.
Got error from Swift: put_object('glance', '7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2', ...) failure and no ability to reset contents for reupload.
<147>Jul 25 15:31:13 node-2 glance-glance.api.v1.upload_utils ERROR: Failed to upload image 7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 99, in upload_data_to_store
    store)
  File "/usr/lib/python2.6/site-packages/glance/store/__init__.py", line 380, in store_add_to_backend
    (location, size, checksum, metadata) = store.add(image_id, data, size)
  File "/usr/lib/python2.6/site-packages/glance/store/swift.py", line 503, in add
    raise glance.store.BackendException(msg)
BackendException: Failed to add object to Swift.
Got error from Swift: put_object('glance', '7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2', ...) failure and no ability to reset contents for reupload.
<151>Jul 25 15:31:13 node-2 glance-glance.registry.client.v1.api DEBUG: Updating image metadata for image 7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2...
<151>Jul 25 15:31:13 node-2 glance-glance.common.client DEBUG: Constructed URL: http://192.168.0.4:9191/images/7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2
<151>Jul 25 15:31:13 node-2 glance-iso8601.iso8601 DEBUG: Parsed 2014-07-25T16:30:47.000000Z into {'tz_sign': None, 'second_fraction': u'000000', 'hour': u'16', 'daydash': u'25', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'47', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'07', 'day': None, 'minute': u'30'} with default timezone <iso8601.iso8601.Utc object at 0x2bec650>

syslog:
<43>Jul 25 15:31:08 node-2 swift-proxy-server: ERROR with Object server 192.168.0.6:6000/1 re: Trying to write to /v1/AUTH_3bf99c76b10344aab4484f524f09005a/glance/7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2: ChunkWriteTimeout (10s)
<43>Jul 25 15:31:08 node-2 swift-proxy-server: ERROR with Object server 192.168.0.6:6000/2 re: Trying to write to /v1/AUTH_3bf99c76b10344aab4484f524f09005a/glance/7fcbd9d4-9c28-4387-8bb7-af5a361dc4e2: ChunkWriteTimeout (10s)
<43>Jul 25 15:31:08 node-2 swift-proxy-server: Object PUT exceptions during send, 0/2 required connections (txn: tx90782bd5c44e425187a37-0053d2782c) (client_ip: 192.168.0.4)

[Ryan Moe (rmoe)]

I can't reproduce this. I was able to upload 100MB, 1GB and 10GB images. Can you upload a diagnostic snapshot?

[Sergii Golovatiuk (sgolovatiuk)]

I was not able to reproduce this.

root@node-1:/var/lib/glance# glance image-create --name Test --disk-format=qcow2 --container-format=bare --file fedora-19.x86_64.qcow2
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 4a3579e6c647ca980d2ea62e0e74bf7c |
| container_format | bare |
| created_at | 2014-08-11T20:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 095d6dfa-0924-4dd8-b5f0-ed2b257d7b7c |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | Test |
| owner | 97583ed2dcaf4bd19ff669b444ca9b45 |
| protected | False |
| size | 8837595136 |
| status | active |
| updated_at | 2014-08-11T20:33:14 |
| virtual_size | None |
+------------------+--------------------------------------+
root@node-1:/var/lib/glance# ll fedora-19.x86_64.qcow2
-rw-r--r-- 1 root root 8837595136 Aug 11 20:24 fedora-19.x86_64.qcow2
root@node-1:/var/lib/glance#

[Tytus Kurek (tkurek7)]

I've just experienced the same issue when uploading 245M Ubuntu cloud image. Uploading 14M CirrOS image works fine. It may be an issue with the image size.

[Egor Kotko (ykotko)]

{u'build_id': u'2014-09-17_21-40-34', u'ostf_sha': u'64cb59c681658a7a55cc2c09d079072a41beb346', u'build_number': u'11', u'auth_required': True, u'nailgun_sha': u'eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d', u'production': u'docker', u'api': u'1.0', u'fuelmain_sha': u'8ef433e939425eabd1034c0b70e90bdf888b69fd', u'astute_sha': u'f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13', u'feature_groups': [u'mirantis'], u'release': u'5.1', u'release_versions': {u'2014.1.1-5.1': {u'VERSION': {u'build_id': u'2014-09-17_21-40-34', u'ostf_sha': u'64cb59c681658a7a55cc2c09d079072a41beb346', u'build_number': u'11', u'api': u'1.0', u'nailgun_sha': u'eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d', u'production': u'docker', u'fuelmain_sha': u'8ef433e939425eabd1034c0b70e90bdf888b69fd', u'astute_sha': u'f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13', u'feature_groups': [u'mirantis'], u'release': u'5.1', u'fuellib_sha': u'd9b16846e54f76c8ebe7764d2b5b8231d6b25079'}}}, u'fuellib_sha': u'd9b16846e54f76c8ebe7764d2b5b8231d6b25079'}