Broken Pipe in vsphere store due to inactive session

[Sabari Murugesan]

We are seeing the following error in glance-api configured with vsphere store. This happens after letting glance-api run for a while.

2014-12-03 15:06:08.641 3005 DEBUG glance.api.v1.images [-] Uploading image data for image 10908257-eb9a-47cf-aaba-afa203c3e9f0 to vsphere store _upload /opt/stack/glance/glance/api/v1/images.py:630
send: 'PUT /folder/openstack_glance/10908257-eb9a-47cf-aaba-afa203c3e9f0%3FdsName%3Dstore2%26dcPath%3DDatacenter1 HTTP/1.1\r\nHost: 10.20.116.124\r\nAccept-Encoding: identity\r\nContent-Length: 41126400\r\nCookie: vmware_soap_session="523fa07a-301c-5312-a35b-7fb99d822720"\r\n\r\n'
send: <glance_store._drivers.vmware_datastore._Reader object at 0x7f238c1d74d0>
sendIng a read()able
2014-12-03 15:06:08.677 3005 ERROR glance_store._drivers.vmware_datastore [-] Failed to upload content of image 10908257-eb9a-47cf-aaba-afa203c3e9f0
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore Traceback (most recent call last):
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/opt/stack/glance_store/glance_store/_drivers/vmware_datastore.py", line 351, in add
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore content=image_file)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/opt/stack/glance_store/glance_store/_drivers/vmware_datastore.py", line 502, in _get_http_conn
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore conn.request(method, url, content, headers)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 973, in request
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore self._send_request(method, url, body, headers)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore self.endheaders(body)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore self._send_output(message_body)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 833, in _send_output
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore self.send(message_body)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 802, in send
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore self.sock.sendall(datablock)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 137, in sendall
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore v = self.send(data[count:])
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 113, in send
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore super(GreenSSLSocket, self).send, data, flags)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 80, in _call_trampolining
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore return func(*a, **kw)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/ssl.py", line 298, in send
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore v = self._sslobj.write(data)
2014-12-03 15:06:08.677 3005 TRACE glance_store._drivers.vmware_datastore error: [Errno 32] Broken pipe

[Sabari Murugesan (smurugesan)]

I figured out the root cause of this issue I have been seeing for a while.

This happens whenever my vSphere session is expired. I can easily reproduce this issue by the following :-

1. Start g-api with vSphere store backend.2
2. Note the partial session id in the logs and terminate the session from vCenter (you can use the Managed Object Browser for terminating the session.)
3. Try glance image-create. You will hit the above mentioned broken-pipe error.

[Sabari Murugesan (smurugesan)]

Glance vSphere store has the logic to handle unauthorized 401 http response. But unfortunately the code breaks before that due to an issue with httplib and ssl sockets. Check this out http://bugs.python.org/issue5542 (at the bottom a note regarding the behavior with https).

I am attempting to fix this by checking if the session is active and then do a http PUT.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/141665

[Aleksandr Kolinko (akolinko)]

There is one more defect, caused by expired vSphere session:

3. Instead of "glance image-create" run "nova image-create --show --poll <instance_name> <snapshot_name>"

# nova image-create --show --poll testim1 testim1-snap

Server snapshotting... 25% complete
Error snapshotting server
ERROR (AttributeError): fault

You will hit the above mentioned broken-pipe error.

The question is: does the proposed solution fix this issue also?

[Sabari Murugesan (smurugesan)]

@akolinko: I tested and it fixes snapshot creation as well. Thanks for bringing this up. If you get a chance can you help reviewing the patch :)