VMware NSX CI failing on image create with "AttributeError: HTTPResponse instance has no attribute 'body'"

[Matt Riedemann]

http://208.91.1.172/logs/246635/1/1447801224/g-api.log.gz?level=TRACE

2015-11-17 15:00:08.355 3080 ERROR glance_store._drivers.vmware_datastore [req-37328c23-1207-4677-b5f6-1e6210968521 9185707de0264b2eb605b6c593a49383 ff0f37f25e8440638b8fe1fce9bba98b - - -] Communication error sending http PUT requestto the url /folder/openstack_glance/18e93048-fb8b-41df-be0b-2ee21bf365a9%3FdcPath%3Ddc%26dsName%3Ddatastore1.
Got IOError [Errno 32] Broken pipe
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data [req-37328c23-1207-4677-b5f6-1e6210968521 9185707de0264b2eb605b6c593a49383 ff0f37f25e8440638b8fe1fce9bba98b - - -] Failed to upload image data due to internal error
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data Traceback (most recent call last):
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/opt/stack/glance/glance/api/v2/image_data.py", line 91, in upload
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data image.set_data(data, size)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/opt/stack/glance/glance/domain/proxy.py", line 166, in set_data
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data self.base.set_data(data, size)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 431, in set_data
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data _send_notification(notify_error, 'image.upload', msg)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 378, in set_data
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data self.repo.set_data(data, size)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/opt/stack/glance/glance/api/policy.py", line 194, in set_data
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/opt/stack/glance/glance/quota/__init__.py", line 296, in set_data
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data self.image.set_data(data, size=size)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/opt/stack/glance/glance/location.py", line 389, in set_data
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data context=self.context)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 365, in add_to_backend
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store, context)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 340, in store_add_to_backend
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data context=context)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py", line 226, in op_checker
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data return store_op_fun(store, *args, **kwargs)
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/vmware_datastore.py", line 534, in add
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data 'body': res.body})
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data AttributeError: HTTPResponse instance has no attribute 'body'
2015-11-17 15:00:08.363 3080 ERROR glance.api.v2.image_data

[Matt Riedemann (mriedem)]

Looks like the root failure is a broken pipe, and the code isn't handling that and is trying to get the response body, but there isn't one because it failed.

[Matt Riedemann (mriedem)]

Fix is here: https://review.openstack.org/#/c/246714/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/247579
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=2ecc714c335ad2175f23743885cc3b9e356f1ba8
Submitter: Jenkins
Branch: stable/liberty

commit 2ecc714c335ad2175f23743885cc3b9e356f1ba8
Author: Matt Riedemann <email address hidden>
Date: Tue Nov 17 17:54:00 2015 -0800

    vmware: check for response body in error conditions

    The response may not have a body in certain error conditions, like
    a 500 on a socket error as seen in the VMware NSX CI. This fixes
    an AttributeError when trying to access the response body in those
    cases.

    Change-Id: I086bb5bdb59813d326d45b5f7545da2ebb12aa95
    Closes-Bug: #1517275
    (cherry picked from commit b709dd62c16ffb0d924bf25328e29a3fbec0f36a)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in openstack/glance_store 0.10.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/glance_store 0.9.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 0.9.2 release.