VMware: deleting instance snapshot too soon leaves instance in Image Uploading state

[Ryan Hsu]

branch: stable/havana
driver: VMwareVCDriver

When using the nova VMwareVCDriver, the following scenario will cause an instance to be stuck in "Image Uploading" state:

1. Create an instance
2. Snapshot the instance
3. While the image is in "Queuing" state, immediately delete the image

The screen-n-cpu.log shows:

2013-11-08 14:45:29.334 DEBUG glanceclient.common.http [-] curl -i -X PUT -H 'X-Service-Catalog: [{"endpoints": [{"adminURL": "http://172.30.0.3:8776/v1/61df65834f494153af76939ffbf5e1a0", "region": "RegionOne", "internalURL": "http://172.30.0.3:8776/v1/61df65834f494153af76939ffbf5e1a0", "id": "536fa98e69a544bca1086f07acdf7663", "publicURL": "http://172.30.0.3:8776/v1/61df65834f494153af76939ffbf5e1a0"}], "endpoints_links": [], "type": "volume", "name": "cinder"}]' -H 'X-Identity-Status: Confirmed' -H 'X-Auth-Token: bd579bd82586cca09e43c975944ef24d' -H 'x-image-meta-property-owner_id: 61df65834f494153af76939ffbf5e1a0' -H 'x-image-meta-container_format: bare' -H 'Transfer-Encoding: chunked' -H 'x-glance-registry-purge-props: true' -H 'X-Tenant-Id: 61df65834f494153af76939ffbf5e1a0' -H 'User-Agent: python-glanceclient' -H 'x-image-meta-property-vmware_image_version: 1' -H 'x-image-meta-property-vmware_adaptertype: lsiLogic' -H 'X-Roles: admin' -H 'X-User-Id: 65aedf8343994f329508a502518a7a0f' -H 'x-image-meta-is_public: false' -H 'x-image-meta-property-vmware_ostype: otherGuest' -H 'x-image-meta-size: 41125888' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: vmdk' -H 'x-image-meta-name: ax1_snap' -d '<ThreadSafePipe maxsize=10>' http://172.30.0.3:9292/v1/images/6cf638ec-844e-41f1-8597-c696a2d946da from (pid=15600) log_curl_request /opt/stack/python-glanceclient/glanceclient/common/http.py:142
2013-11-08 14:45:30.138 DEBUG nova.openstack.common.rpc.amqp [-] Making synchronous call on conductor ... from (pid=15600) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:553
2013-11-08 14:45:30.138 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is 354436bd11ce4e07af07157381ff6147 from (pid=15600) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:556
2013-11-08 14:45:30.139 DEBUG nova.openstack.common.rpc.amqp [-] UNIQUE_ID is 940f2aa02d024f5abecaefec16d44fc5. from (pid=15600) _add_unique_id /opt/stack/nova/nova/openstack/common/rpc/amqp.py:341
2013-11-08 14:45:30.141 DEBUG amqp [-] Closed channel #1 from (pid=15600) _do_close /usr/local/lib/python2.7/dist-packages/amqp/channel.py:95
2013-11-08 14:45:30.141 DEBUG amqp [-] using channel_id: 1 from (pid=15600) __init__ /usr/local/lib/python2.7/dist-packages/amqp/channel.py:71
2013-11-08 14:45:30.142 DEBUG amqp [-] Channel open from (pid=15600) _open_ok /usr/local/lib/python2.7/dist-packages/amqp/channel.py:429
2013-11-08 14:45:39.350 DEBUG glanceclient.common.http [-]
HTTP/1.1 403 Forbidden
date: Fri, 08 Nov 2013 22:45:29 GMT
content-length: 54
content-type: text/plain; charset=UTF-8
x-openstack-request-id: req-7d82c3c6-4d70-4506-8380-fe9e58b34801

403 Forbidden

Forbidden to update deleted image.

 from (pid=15600) log_http_response /opt/stack/python-glanceclient/glanceclient/common/http.py:152
2013-11-08 14:45:39.351 ERROR glanceclient.common.http [-] Request returned failure status.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/queue.py", line 107, in switch
    self.greenlet.switch(value)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/virt/vmwareapi/io_util.py", line 106, in _inner
    data=self.input)
  File "/opt/stack/nova/nova/image/glance.py", line 395, in update
    _reraise_translated_image_exception(image_id)
  File "/opt/stack/nova/nova/image/glance.py", line 393, in update
    image_id, **image_meta)
  File "/opt/stack/nova/nova/image/glance.py", line 212, in call
    return getattr(client.images, method)(*args, **kwargs)
  File "/opt/stack/python-glanceclient/glanceclient/v1/images.py", line 291, in update
    'PUT', url, headers=hdrs, body=image_data)
  File "/opt/stack/python-glanceclient/glanceclient/common/http.py", line 288, in raw_request
    return self._http_request(url, method, **kwargs)
  File "/opt/stack/python-glanceclient/glanceclient/common/http.py", line 248, in _http_request
    raise exc.from_response(resp, body_str)
ImageNotAuthorized: Not authorized for image 6cf638ec-844e-41f1-8597-c696a2d946da.

[Tracy Jones (tjones-i)]

Ryan - do you still get this issue? I just tried to repo and i did not get a task stuck issue

[Ryan Hsu (rhsu)]

Tracy, I'm still able to reproduce the issue on my machine. This is on a brand new devstack machine that I just stacked today so the master is still affected.

[Ryan Hsu (rhsu)]

Screenshot of Horizon displaying stuck "Image Uploading" status

[Kieran Spear (kspear)]

We've had a similar report from a user. We use libvirt/KVM, so I don't think this issue is specific to VMware, at least in Havana.

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/90168

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/90168
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=56607f9af4495e76edf6675a984a66da907c89f8
Submitter: Jenkins
Branch: master

commit 56607f9af4495e76edf6675a984a66da907c89f8
Author: Eric Brown <email address hidden>
Date: Thu Apr 24 09:21:04 2014 -0700

    VMware: uncaught exception during snapshot deletion

    During the image upload of the snapshot to glance, an exception is
    thrown that ends up getting ignored. As a result, the instance gets
    stuck in the "Image Uploading" state.

    Change-Id: Id2cb3c244edd831939794da9dcc1fd9fce59991c
    Closes-bug: #1249519

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/95359

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95359
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a2da9ce769fa6a99f64eddf78e1fef3725caa551
Submitter: Jenkins
Branch: stable/icehouse

commit a2da9ce769fa6a99f64eddf78e1fef3725caa551
Author: Eric Brown <email address hidden>
Date: Thu Apr 24 09:21:04 2014 -0700

    VMware: uncaught exception during snapshot deletion

    During the image upload of the snapshot to glance, an exception is
    thrown that ends up getting ignored. As a result, the instance gets
    stuck in the "Image Uploading" state.

    Change-Id: Id2cb3c244edd831939794da9dcc1fd9fce59991c
    Closes-bug: #1249519
    (cherry picked from commit 56607f9af4495e76edf6675a984a66da907c89f8)