Kilo glance image create stuck in saving and queued state

Bug #1464034 reported by Alfred Shen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Invalid
Undecided
Alfred Shen

Bug Description

In Kilo release, glance image-create with or without --os-image-api-version caused "queued" status with following debug output. No error log on keystone-api/registry.log. Othe glance CLIs work OK.

Similar symptom was reported in https://bugs.launchpad.net/bugs/1146830 but seemed to be with diff cause.

# dpkg -l | grep glance
ii glance 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Image Registry and Delivery Service - Daemons
ii glance-api 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Image Registry and Delivery Service - API
ii glance-common 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Image Registry and Delivery Service - Common
ii glance-registry 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Image Registry and Delivery Service - Registry
ii python-glance 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Image Registry and Delivery Service - Python library
ii python-glance-store 0.4.0-0ubuntu1~cloud0 all OpenStack Image Service store library - Python 2.x
ii python-glanceclient 1:0.15.0-0ubuntu1~cloud0 all Client library for Openstack glance server.

$ glance --debug --os-image-api-version 2 image-create --file /tmp/cirros-0.3.4-x86_64-disk.img --disk-format qcow2 --container-format bare --progress
curl -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}252d465682ed3a3d092b0ce05954601afb56c7df' -H 'Content-Type: application/octet-stream' http://3.39.89.230:9292/v2/schemas/image

HTTP/1.0 200 OK
content-length: 3867
via: 1.0 sjc1intproxy01 (squid/3.1.10)
x-cache: MISS from sjc1intproxy01
x-cache-lookup: MISS from sjc1intproxy01:8080
connection: keep-alive
date: Wed, 10 Jun 2015 21:41:24 GMT
content-type: application/json; charset=UTF-8
x-openstack-request-id: req-req-d0430e7a-5e36-466d-8d79-afefe9737695

{"additionalProperties": {"type": "string"}, "name": "image", "links": [{"href": "{self}", "rel": "self"}, {"href": "{file}", "rel": "enclosure"}, {"href": "{schema}", "rel": "describedby"}], "properties": {"status": {"enum": ["queued", "saving", "active", "killed", "deleted", "pending_delete"], "type": "string", "description": "Status of the image (READ-ONLY)"}, "tags": {"items": {"type": "string", "maxLength": 255}, "type": "array", "description": "List of strings related to the image"}, "kernel_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "ID of image stored in Glance that should be used as the kernel when booting an AMI-style image.", "is_base": false}, "container_format": {"enum": [null, "ami", "ari", "aki", "bare", "ovf", "ova"], "type": ["null", "string"], "description": "Format of the container"}, "min_ram": {"type": "integer", "description": "Amount of ram (in MB) required to boot image."}, "ramdisk_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.", "is_base": false}, "locations": {"items": {"required": ["url", "metadata"], "type": "object", "properties": {"url": {"type": "string", "maxLength": 255}, "metadata": {"type": "object"}}}, "type": "array", "description": "A set of URLs to access the image file kept in external store"}, "visibility": {"enum": ["public", "private"], "type": "string", "description": "Scope of image accessibility"}, "updated_at": {"type": "string", "description": "Date and time of the last image modification (READ-ONLY)"}, "owner": {"type": ["null", "string"], "description": "Owner of the image", "maxLength": 255}, "file": {"type": "string", "description": "(READ-ONLY)"}, "min_disk": {"type": "integer", "description": "Amount of disk space (in GB) required to boot image."}, "virtual_size": {"type": ["null", "integer"], "description": "Virtual size of image in bytes (READ-ONLY)"}, "id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "An identifier for the image"}, "size": {"type": ["null", "integer"], "description": "Size of image file in bytes (READ-ONLY)"}, "instance_uuid": {"type": "string", "description": "ID of instance used to create this image.", "is_base": false}, "os_distro": {"type": "string", "description": "Common name of operating system distribution as specified in http://docs.openstack.org/trunk/openstack-compute/admin/content/adding-images.html", "is_base": false}, "name": {"type": ["null", "string"], "description": "Descriptive name for the image", "maxLength": 255}, "checksum": {"type": ["null", "string"], "description": "md5 hash of image contents. (READ-ONLY)", "maxLength": 32}, "created_at": {"type": "string", "description": "Date and time of image registration (READ-ONLY)"}, "disk_format": {"enum": [null, "ami", "ari", "aki", "vhd", "vmdk", "raw", "qcow2", "vdi", "iso"], "type": ["null", "string"], "description": "Format of the disk"}, "os_version": {"type": "string", "description": "Operating system version as specified by the distributor", "is_base": false}, "protected": {"type": "boolean", "description": "If true, image will not be deletable."}, "architecture": {"type": "string", "description": "Operating system architecture as specified in http://docs.openstack.org/trunk/openstack-compute/admin/content/adding-images.html", "is_base": false}, "direct_url": {"type": "string", "description": "URL to access the image file kept in external store (READ-ONLY)"}, "self": {"type": "string", "description": "(READ-ONLY)"}, "schema": {"type": "string", "description": "(READ-ONLY)"}}}

curl -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}252d465682ed3a3d092b0ce05954601afb56c7df' -H 'Content-Type: application/octet-stream' http://3.39.89.230:9292/v2/schemas/image

HTTP/1.0 200 OK
content-length: 3867
via: 1.0 sjc1intproxy01 (squid/3.1.10)
x-cache: MISS from sjc1intproxy01
x-cache-lookup: MISS from sjc1intproxy01:8080
connection: keep-alive
date: Wed, 10 Jun 2015 21:41:24 GMT
content-type: application/json; charset=UTF-8
x-openstack-request-id: req-req-535be2e3-66a4-4cd9-b88f-183a2acb30c0

{"additionalProperties": {"type": "string"}, "name": "image", "links": [{"href": "{self}", "rel": "self"}, {"href": "{file}", "rel": "enclosure"}, {"href": "{schema}", "rel": "describedby"}], "properties": {"status": {"enum": ["queued", "saving", "active", "killed", "deleted", "pending_delete"], "type": "string", "description": "Status of the image (READ-ONLY)"}, "tags": {"items": {"type": "string", "maxLength": 255}, "type": "array", "description": "List of strings related to the image"}, "kernel_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "ID of image stored in Glance that should be used as the kernel when booting an AMI-style image.", "is_base": false}, "container_format": {"enum": [null, "ami", "ari", "aki", "bare", "ovf", "ova"], "type": ["null", "string"], "description": "Format of the container"}, "min_ram": {"type": "integer", "description": "Amount of ram (in MB) required to boot image."}, "ramdisk_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.", "is_base": false}, "locations": {"items": {"required": ["url", "metadata"], "type": "object", "properties": {"url": {"type": "string", "maxLength": 255}, "metadata": {"type": "object"}}}, "type": "array", "description": "A set of URLs to access the image file kept in external store"}, "visibility": {"enum": ["public", "private"], "type": "string", "description": "Scope of image accessibility"}, "updated_at": {"type": "string", "description": "Date and time of the last image modification (READ-ONLY)"}, "owner": {"type": ["null", "string"], "description": "Owner of the image", "maxLength": 255}, "file": {"type": "string", "description": "(READ-ONLY)"}, "min_disk": {"type": "integer", "description": "Amount of disk space (in GB) required to boot image."}, "virtual_size": {"type": ["null", "integer"], "description": "Virtual size of image in bytes (READ-ONLY)"}, "id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "An identifier for the image"}, "size": {"type": ["null", "integer"], "description": "Size of image file in bytes (READ-ONLY)"}, "instance_uuid": {"type": "string", "description": "ID of instance used to create this image.", "is_base": false}, "os_distro": {"type": "string", "description": "Common name of operating system distribution as specified in http://docs.openstack.org/trunk/openstack-compute/admin/content/adding-images.html", "is_base": false}, "name": {"type": ["null", "string"], "description": "Descriptive name for the image", "maxLength": 255}, "checksum": {"type": ["null", "string"], "description": "md5 hash of image contents. (READ-ONLY)", "maxLength": 32}, "created_at": {"type": "string", "description": "Date and time of image registration (READ-ONLY)"}, "disk_format": {"enum": [null, "ami", "ari", "aki", "vhd", "vmdk", "raw", "qcow2", "vdi", "iso"], "type": ["null", "string"], "description": "Format of the disk"}, "os_version": {"type": "string", "description": "Operating system version as specified by the distributor", "is_base": false}, "protected": {"type": "boolean", "description": "If true, image will not be deletable."}, "architecture": {"type": "string", "description": "Operating system architecture as specified in http://docs.openstack.org/trunk/openstack-compute/admin/content/adding-images.html", "is_base": false}, "direct_url": {"type": "string", "description": "URL to access the image file kept in external store (READ-ONLY)"}, "self": {"type": "string", "description": "(READ-ONLY)"}, "schema": {"type": "string", "description": "(READ-ONLY)"}}}

curl -i -X POST -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}252d465682ed3a3d092b0ce05954601afb56c7df' -H 'Content-Type: application/json' -d '{"container_format": "bare", "disk_format": "qcow2"}' http://3.39.89.230:9292/v2/images

HTTP/1.0 201 Created
content-length: 547
via: 1.0 sjc1intproxy01 (squid/3.1.10)
x-cache: MISS from sjc1intproxy01
x-cache-lookup: MISS from sjc1intproxy01:8080
connection: keep-alive
location: http://3.39.89.230:9292/v2/images/383b51bd-2ca1-429d-ac70-49187bc685c6
date: Wed, 10 Jun 2015 21:41:25 GMT
content-type: application/json; charset=UTF-8
x-openstack-request-id: req-req-76eae7da-a2fd-4037-aa5d-0c66e9b515b6

{"status": "queued", "name": null, "tags": [], "container_format": "bare", "created_at": "2015-06-10T21:41:25Z", "size": null, "disk_format": "qcow2", "updated_at": "2015-06-10T21:41:25Z", "visibility": "private", "self": "/v2/images/383b51bd-2ca1-429d-ac70-49187bc685c6", "min_disk": 0, "protected": false, "id": "383b51bd-2ca1-429d-ac70-49187bc685c6", "file": "/v2/images/383b51bd-2ca1-429d-ac70-49187bc685c6/file", "checksum": null, "owner": "a3b62a7ed7154ea5bfe189c90b553690", "virtual_size": null, "min_ram": 0, "schema": "/v2/schemas/image"}

curl -i -X PUT -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}252d465682ed3a3d092b0ce05954601afb56c7df' -H 'Content-Type: application/octet-stream' http://3.39.89.230:9292/v2/images/383b51bd-2ca1-429d-ac70-49187bc685c6/file
[==> ] 5%+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2015-06-10T21:41:25Z |
| disk_format | qcow2 |
| id | 383b51bd-2ca1-429d-ac70-49187bc685c6 |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | a3b62a7ed7154ea5bfe189c90b553690 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2015-06-10T21:41:25Z |
| virtual_size | None |
| visibility | private |
+------------------+--------------------------------------+
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/glanceclient/shell.py", line 637, in main
    args.func(client, args)
  File "/usr/lib/python2.7/dist-packages/glanceclient/v2/shell.py", line 73, in do_image_create
    do_image_upload(gc, args)
  File "/usr/lib/python2.7/dist-packages/glanceclient/v2/shell.py", line 258, in do_image_upload
    gc.images.upload(args.id, image_data, args.size)
  File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 138, in upload
    self.http_client.put(url, headers=hdrs, data=body)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 266, in put
    return self._request('PUT', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 215, in _request
    raise exc.CommunicationError(message=message)
CommunicationError: Error finding address for http://3.39.89.230:9292/v2/images/383b51bd-2ca1-429d-ac70-49187bc685c6/file: [Errno 32] Broken pipe
Error finding address for http://3.39.89.230:9292/v2/images/383b51bd-2ca1-429d-ac70-49187bc685c6/file: [Errno 32] Broken pipe

Tags: ge-contrail
Alfred Shen (alfredcs)
tags: added: ge-contrail
Revision history for this message
Alfred Shen (alfredcs) wrote :

Further investigation found that the upload failure was due to http_proxy. It prevented glanceclient from accessing API port which happen to be on a public IP. Using private IPs might mask this issue.

Changed in glance:
assignee: nobody → Alfred Shen (alfredcs)
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.