VMware: Volume stuck in uploading state even after upload to image failure

[Vipin Balachandran]

Uploading a 2Gi volume to image service resulted in volume being stuck in uploading state for a long time.
As per the logs, the image service raised an internal server error. In this case, the volume should return back to available state.

Cinder volume logs:

2014-08-14 17:56:27.846 DEBUG glanceclient.common.http [-]
HTTP/1.1 500 Internal Server Error
date: Thu, 14 Aug 2014 12:26:27 GMT
content-length: 91
content-type: text/plain; charset=UTF-8
x-openstack-request-id: req-f741d884-6cdd-4304-b809-63469b12aaa8

500 Internal Server Error

Failed to upload image 3cc9f1f9-8749-4556-81f5-8a0c2bfa4f1f

 from (pid=35027) log_http_response /opt/stack/python-glanceclient/glanceclient/common/http.py:167
2014-08-14 17:56:27.846 DEBUG glanceclient.common.http [-] Request returned failure status: 500 from (pid=35027) _http_request /opt/stack/python-glanceclient/glanceclien
t/common/http.py:263
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/cinder/cinder/volume/drivers/vmware/io_util.py", line 112, in _inner
    data=self.input_file)
  File "/opt/stack/cinder/cinder/image/glance.py", line 320, in update
    _reraise_translated_image_exception(image_id)
  File "/opt/stack/cinder/cinder/image/glance.py", line 318, in update
    **image_meta)
  File "/opt/stack/cinder/cinder/image/glance.py", line 167, in call
    return getattr(client.images, method)(*args, **kwargs)
  File "/opt/stack/python-glanceclient/glanceclient/v1/images.py", line 332, in update
    'PUT', url, headers=hdrs, body=image_data)
  File "/opt/stack/python-glanceclient/glanceclient/common/http.py", line 328, in raw_request
    return self._http_request(url, method, **kwargs)
  File "/opt/stack/python-glanceclient/glanceclient/common/http.py", line 264, in _http_request
    raise exc.from_response(resp, body_str)
HTTPInternalServerError: 500 Internal Server Error
Failed to upload image 3cc9f1f9-8749-4556-81f5-8a0c2bfa4f1f
    (HTTP 500)
Removing descriptor: 10

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114981

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114981
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7ac4fc76f2d777fd4e0ec4245eaf989f99917be2
Submitter: Jenkins
Branch: master

commit 7ac4fc76f2d777fd4e0ec4245eaf989f99917be2
Author: Vipin Balachandran <email address hidden>
Date: Thu Aug 14 20:36:20 2014 +0530

    VMware: Handle exceptions raised by image update

    Exceptions raised by glance image update are not handled while writing
    to image. This results in volume being stuck in uploading state for
    a long time even after image transfer failure. This patch fixes the
    exception handling.

    Change-Id: I7651c96968caa49c5421d68ee4dbe9e55f7f53b1
    Closes-Bug: #1357239
    Partial-Bug: #1269345