Installer Hangs in Uploading Images

[James Michael DuPont]

dev stack hangs here :
openstack --os-token 16076b8008eb4892ba7036c75e498e03 --os-url http://192.168.1.3:9292 image create cirros-0.3.2-x86_64-uec-kernel --public --container-format aki --disk-format aki

+ local 'command=/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf'
+ local group=
+ is_service_enabled g-api
++ set +o
++ grep xtrace
+ local 'xtrace=set -o xtrace'
+ set +o xtrace
+ return 0
+ [[ True = \T\r\u\e ]]
+ screen_process g-api '/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf' ''
+ local name=g-api
+ local 'command=/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf'
+ local group=
+ SCREEN_NAME=stack
+ SERVICE_DIR=/mnt/newdrive2/home/mdupont/experiments/stack/status
++ trueorfalse True True
+++ set +o
+++ grep xtrace
++ local 'xtrace=set -o xtrace'
++ set +o xtrace
+ USE_SCREEN=True
+ screen_rc g-api '/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf'
+ SCREEN_NAME=stack
+ SCREENRC=/mnt/data/home/mdupont/new3/devstack/stack-screenrc
+ [[ ! -e /mnt/data/home/mdupont/new3/devstack/stack-screenrc ]]
+ grep g-api /mnt/data/home/mdupont/new3/devstack/stack-screenrc
++ echo -ne '\015'
+ NL=$'\r'
+ echo 'screen -t g-api bash'
"'echo 'stuff "/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf
+ [[ -n '' ]]
+ screen -S stack -X screen -t g-api
+ [[ -n '' ]]
+ sleep 3
++ echo -ne '\015'
+ NL=$'\r'
+ [[ -n '' ]]
+ screen -S stack -p g-api -X stuff '/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf & echo $! >/mnt/newdrive2/home/mdupont/experiments/stack/status/stack/'-api.pid; fg || echo "g-api failed to start" | tee "/mnt/newdrive2/home/mdupont/experiments/stack/status/stack/g-api.failure"
+ echo 'Waiting for g-api (192.168.1.3:9292) to start...'
Waiting for g-api (192.168.1.3:9292) to start...
+ wait_for_service 60 http://192.168.1.3:9292
+ local timeout=60
+ local url=http://192.168.1.3:9292
+ timeout 60 sh -c 'while ! curl -k --noproxy '\''*'\'' -s http://192.168.1.3:9292 >/dev/null; do sleep 1; done'
+ is_service_enabled g-reg
++ set +o
++ grep xtrace
+ local 'xtrace=set -o xtrace'
+ set +o xtrace
+ return 0
++ keystone token-get
++ grep ' id '
++ get_field 2
++ local data field
++ read data
++ '[' 2 -lt 0 ']'
++ field='$3'
++ echo '| id | 16076b8008eb4892ba7036c75e498e03 |'
++ awk '-F[ \t]*\\|[ \t]*' '{print $3}'
++ read data
+ TOKEN=16076b8008eb4892ba7036c75e498e03
+ die_if_not_set 1205 TOKEN 'Keystone fail to get token'
+ local exitcode=0
++ set +o
++ grep xtrace
+ local 'xtrace=set -o xtrace'
+ set +o xtrace
+ is_baremetal
+ [[ g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-net,n-cond,n-sch,n-novnc,n-xvnc,n-cauth,c-sch,c-api,c-vol,h-eng,h-api,h-api-cfn,h-api-cw,horizon,rabbit,tempest,mysql =~ baremetal ]]
+ return 1
+ echo_summary 'Uploading images'
+ [[ -t 3 ]]
+ [[ True != \T\r\u\e ]]
+ echo -e Uploading images
+ [[ -n '' ]]
+ for image_url in '${IMAGE_URLS//,/ }'
+ upload_image http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz 16076b8008eb4892ba7036c75e498e03
2014-10-31 21:02:43.132 | Uploading images
+ local image_url=http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz
+ local token=16076b8008eb4892ba7036c75e498e03
+ local image image_fname image_name
+ mkdir -p /mnt/data/home/mdupont/new3/devstack/files/images
++ basename http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz
+ image_fname=cirros-0.3.2-x86_64-uec.tar.gz
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz != file* ]]
+ [[ ! -f /mnt/data/home/mdupont/new3/devstack/files/cirros-0.3.2-x86_64-uec.tar.gz ]]
++ stat -c %s /mnt/data/home/mdupont/new3/devstack/files/cirros-0.3.2-x86_64-uec.tar.gz
+ [[ 8655821 = \0 ]]
+ image=/mnt/data/home/mdupont/new3/devstack/files/cirros-0.3.2-x86_64-uec.tar.gz
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz =~ openvz ]]
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz =~ \.vmdk ]]
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz =~ \.vhd\.tgz ]]
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz =~ \.xen-raw\.tgz ]]
+ local kernel=
+ local ramdisk=
+ local disk_format=
+ local container_format=
+ local unpack=
+ local img_property=
+ case "$image_fname" in
+ '[' cirros-0.3.2-x86_64-uec '!=' cirros-0.3.2-x86_64-uec.tar.gz ']'
+ image_name=cirros-0.3.2-x86_64-uec
+ local xdir=/mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec
+ rm -Rf /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec
+ mkdir /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec
+ tar -zxf /mnt/data/home/mdupont/new3/devstack/files/cirros-0.3.2-x86_64-uec.tar.gz -C /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec
++ for f in '"$xdir/"*-vmlinuz*' '"$xdir/"aki-*/image'
++ '[' -f /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-vmlinuz ']'
++ echo /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-vmlinuz
++ break
++ true
+ kernel=/mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-vmlinuz
++ for f in '"$xdir/"*-initrd*' '"$xdir/"ari-*/image'
++ '[' -f /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-initrd ']'
++ echo /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-initrd
++ break
++ true
+ ramdisk=/mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-initrd
++ for f in '"$xdir/"*.img' '"$xdir/"ami-*/image'
++ '[' -f /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-blank.img ']'
++ echo /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-blank.img
++ break
++ true
+ image=/mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-blank.img
+ [[ -z cirros-0.3.2-x86_64-uec ]]
+ is_arch ppc64
++ uname -m
+ [[ x86_64 == \p\p\c\6\4 ]]
+ '[' '' = bare ']'
+ local kernel_id= ramdisk_id=
+ '[' -n /mnt/data/home/mdupont/new3/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-vmlinuz ']'
++ openstack --os-token 16076b8008eb4892ba7036c75e498e03 --os-url http://192.168.1.3:9292 image create cirros-0.3.2-x86_64-uec-kernel --public --container-format aki --disk-format aki
++ grep ' id '
++ get_field 2
++ local data field
++ read data

--------------------
I stopped it here after a few hours.

^C++ exit_trap
Traceback (most recent call last):
  File "/usr/local/bin/openstack", line 10, in <module>
    sys.exit(main())
++ local r=0
  File "/mnt/newdrive2/home/mdupont/experiments/stack/python-openstackclient/openstackclient/shell.py", line 348, in main
    return OpenStackShell().run(argv)
  File "/mnt/newdrive2/home/mdupont/experiments/stack/python-openstackclient/openstackclient/shell.py", line 178, in run
    return super(OpenStackShell, self).run(argv)
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 215, in run
    result = self.run_subcommand(remainder)
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 280, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/display.py", line 91, in run
    column_names, data = self.take_action(parsed_args)
  File "/mnt/newdrive2/home/mdupont/experiments/stack/python-openstackclient/openstackclient/image/v1/image.py", line 246, in take_action
    image = image_client.images.create(**kwargs)
  File "/mnt/newdrive2/home/mdupont/experiments/stack/python-glanceclient/glanceclient/v1/images.py", line 288, in create
+++ jobs -p
    data=image_data)
  File "/mnt/newdrive2/home/mdupont/experiments/stack/python-glanceclient/glanceclient/common/http.py", line 255, in post
    return self._request('POST', url, **kwargs)
  File "/mnt/newdrive2/home/mdupont/experiments/stack/python-glanceclient/glanceclient/common/http.py", line 199, in _request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
++ jobs=
++ [[ -n '' ]]
++ kill_spinner
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 350, in send
++ '[' '!' -z '' ']'
++ [[ 0 -ne 0 ]]
    ++ exit 0
low_conn.send(i)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
KeyboardInterrupt

[mmisc (0-mischa)]

I am also affected by this bug.
stack.sh hangs while uploading Images on my Debian unstable System.

+ echo_summary 'Uploading images'
+ [[ -t 3 ]]
+ [[ True != \T\r\u\e ]]
+ echo -e Uploading images
+ [[ -n '' ]]
+ for image_url in '${IMAGE_URLS//,/ }'
+ upload_image http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz 272bdea55ad54148b663e65be09c408c
2014-11-11 15:25:23.067 | Uploading images
+ local image_url=http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz
+ local token=272bdea55ad54148b663e65be09c408c
+ local image image_fname image_name
+ mkdir -p /home/mischa/devstack/files/images
++ basename http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz
+ image_fname=cirros-0.3.2-x86_64-uec.tar.gz
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz != file* ]]
+ [[ ! -f /home/mischa/devstack/files/cirros-0.3.2-x86_64-uec.tar.gz ]]
++ stat -c %s /home/mischa/devstack/files/cirros-0.3.2-x86_64-uec.tar.gz
+ [[ 8655821 = \0 ]]
+ image=/home/mischa/devstack/files/cirros-0.3.2-x86_64-uec.tar.gz
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz =~ openvz ]]
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz =~ \.vmdk ]]
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz =~ \.vhd\.tgz ]]
+ [[ http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz =~ \.xen-raw\.tgz ]]
+ local kernel=
+ local ramdisk=
+ local disk_format=
+ local container_format=
+ local unpack=
+ local img_property=
+ case "$image_fname" in
+ '[' cirros-0.3.2-x86_64-uec '!=' cirros-0.3.2-x86_64-uec.tar.gz ']'
+ image_name=cirros-0.3.2-x86_64-uec
+ local xdir=/home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec
+ rm -Rf /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec
+ mkdir /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec
+ tar -zxf /home/mischa/devstack/files/cirros-0.3.2-x86_64-uec.tar.gz -C /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec
++ for f in '"$xdir/"*-vmlinuz*' '"$xdir/"aki-*/image'
++ '[' -f /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-vmlinuz ']'
++ echo /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-vmlinuz
++ break
++ true
+ kernel=/home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-vmlinuz
++ for f in '"$xdir/"*-initrd*' '"$xdir/"ari-*/image'
++ '[' -f /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-initrd ']'
++ echo /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-initrd
++ break
++ true
+ ramdisk=/home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-initrd
++ for f in '"$xdir/"*.img' '"$xdir/"ami-*/image'
++ '[' -f /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-blank.img ']'
++ echo /home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-blank.img
++ break
++ true
+ image=/home/mischa/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-blank.img
+ [[ -z cirros-0.3.2-x86_64-uec ]]
+ is_arch ppc64
++ uname -m
+ [[ x86_64 ==...

[mmisc (0-mischa)]

Starting the task by hand with additional verbose flags returns this:

mischa@hostname:~/devstack$ openstack --os-token fbe47be2549b45e08a1fbb9a238eb76e --os-url http://192.168.1.100:9292 image create cirros-0.3.2-x86_64-uec-kernel --public --container-format aki --disk-format aki -vv
DEBUG: openstackclient.shell compute API version 2, cmd group openstack.compute.v2
DEBUG: openstackclient.shell network API version 2, cmd group openstack.network.v2
DEBUG: openstackclient.shell image API version 1, cmd group openstack.image.v1
DEBUG: openstackclient.shell volume API version 1, cmd group openstack.volume.v1
DEBUG: openstackclient.shell identity API version 2.0, cmd group openstack.identity.v2_0
DEBUG: openstackclient.shell object_store API version 1, cmd group openstack.object_store.v1
DEBUG: openstackclient.api.auth Auth plugin token_endpoint selected
DEBUG: openstackclient.api.auth auth_type: token_endpoint
DEBUG: openstackclient.api.auth fetching option os_url
DEBUG: openstackclient.api.auth fetching option os_token
INFO: openstackclient.common.clientmanager Using auth plugin: token_endpoint
INFO: openstackclient.shell command: openstackclient.image.v1.image.CreateImage
DEBUG: openstackclient.image.v1.image.CreateImage take_action(Namespace(checksum=None, columns=[], container_format='aki', copy_from=None, disk_format='aki', file=None, force=False, formatter='table', id=None, location=None, max_width=0, min_disk=None, min_ram=None, name='cirros-0.3.2-x86_64-uec-kernel', owner=None, prefix='', private=False, properties=None, protected=False, public=True, size=None, store=None, unprotected=False, variables=[], volume=None))
DEBUG: openstackclient.image.client Instantiating image client: <class 'openstackclient.image.client.Client_v1'>
DEBUG: openstackclient.common.clientmanager Get auth_ref
DEBUG: glanceclient.common.http curl -i -X HEAD -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}d02b0123fbe1f07234001000b847a6cb03676c8c' -H 'Content-Type: application/octet-stream' http://192.168.1.100:9292/v1/images/cirros-0.3.2-x86_64-uec-kernel
INFO: urllib3.connectionpool Starting new HTTP connection (1): 192.168.1.100
DEBUG: urllib3.connectionpool "HEAD /v1/images/cirros-0.3.2-x86_64-uec-kernel HTTP/1.1" 404 0
DEBUG: glanceclient.common.http Request returned failure status 404.
DEBUG: glanceclient.common.http curl -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}d02b0123fbe1f07234001000b847a6cb03676c8c' -H 'Content-Type: application/octet-stream' http://192.168.1.100:9292/v1/images/detail?limit=20
DEBUG: urllib3.connectionpool "GET /v1/images/detail?limit=20 HTTP/1.1" 200 471
DEBUG: glanceclient.common.http
HTTP/1.1 200 OK
date: Fri, 14 Nov 2014 16:02:30 GMT
connection: keep-alive
content-type: application/json; charset=UTF-8
content-length: 471
x-openstack-request-id: req-d4cb6cfa-a9f0-4946-a686-c96efeaaa42c

{"images": [{"status": "queued", "deleted_at": null, "name": "cirros-0.3.2-x86_64-uec-kernel", "deleted": false, "container_format": "aki", "created_a...

[Terry Wilson (otherwiseguy)]

I was experiencing this and found that my devstack VM was out of diskspace. Deleting a couple of gigs of logs fixed the issue.

[Martin (ff1)]

Hi Terry,

Could you give more information about how you have fixed that issue ?

How many MB you had free when you faced the issue and how many MB you had after the logs cleaning ?

[Terry Wilson (otherwiseguy)]

Martin,

Before (when it wasn't working) I think I remember having somewhere around 500MB free. Afterwards I had around 2GB free.

[Martin (ff1)]

Thanks Terry,

The issue is solved now. I had 600MB and after adding more hdd everything went fine.

[Sean Dague (sdague)]

This devstack bug was last updated over 180 days ago, as devstack
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.