Broken pipe error when copying image from glance to vSphere

Bug #1240355 reported by Ryan Hsu
42
This bug affects 9 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Medium
Yang Yu
glance_store
Confirmed
Low
Unassigned

Bug Description

Using the VMwareVCDriver on the latest nova (6affe67067) from master, launching an image for the first time is failing when copying image from Glance to vSphere. The error that shows in the nova log is:

Traceback (most recent call last):
  File "/opt/stack/nova/nova/virt/vmwareapi/io_util.py", line 176, in _inner
    self.output.write(data)
  File "/opt/stack/nova/nova/virt/vmwareapi/read_write_util.py", line 143, in write
    self.file_handle.send(data)
  File "/usr/lib/python2.7/httplib.py", line 790, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 131, in sendall
    v = self.send(data[count:])
  File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 107, in send
    super(GreenSSLSocket, self).send, data, flags)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 77, in
    return func(*a, **kw)
  File "/usr/lib/python2.7/ssl.py", line 198, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe

To reproduce, launch an instance using an image that has not yet been uploaded to vSphere. I have attached the full log here: http://paste.openstack.org/show/48536/

Tags: vmware
Ryan Hsu (rhsu)
affects: cloudkeep → nova
Tracy Jones (tjones-i)
Changed in nova:
assignee: nobody → Tracy Jones (tjones-i)
Revision history for this message
Tracy Jones (tjones-i) wrote :

ryan - is this still occurring? Im not able to repo

Revision history for this message
Ryan Hsu (rhsu) wrote :

Forgot to record the information on the bug after our chat in IRC:

- I'm not able to reproduce on the latest code from master.
- Gary has experienced this before so it seems to be intermittent
- Shawn has said he may want to look at this to see if he can at least write a patch to catch the Broken pipe error.

Tracy Jones (tjones-i)
Changed in nova:
assignee: Tracy Jones (tjones-i) → nobody
Changed in nova:
importance: Undecided → High
importance: High → Undecided
Revision history for this message
Gary Kotton (garyk) wrote :

I have seen this occasionaly on my setup but have not been able to reproduce.

Changed in nova:
status: New → Confirmed
Changed in nova:
importance: Undecided → Medium
Revision history for this message
Zoltan Arnold Nagy (zoltan) wrote :
Download full text (4.6 KiB)

I can reproduce this just by uploading an image:
root@x:~# glance image-create --name cirros --is-public=True --container-format=bare --disk-format=vmdk < ~/cirros-0.3.2-i386-disk.vmdk
Request returned failure status.
400 Bad Request
Client disconnected before sending all data to backend
    (HTTP 400)
root@x:~#

and in the log:
2014-05-22 02:12:16.655 27041 DEBUG glance.common.client [6f89c589-745f-42e7-9da8-3c78d7abe6a5 a315717e20174b10a39db36b722325d6 76d25b1928e7407392a69735a894c7fc - - -] Constructed URL: http://0.0.0.0:9191/images/9fcadd95-c1b0-452a-9516-7eb2c3d5ae68 _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:411
2014-05-22 02:12:16.689 27041 DEBUG glance.registry.client.v1.client [6f89c589-745f-42e7-9da8-3c78d7abe6a5 a315717e20174b10a39db36b722325d6 76d25b1928e7407392a69735a894c7fc - - -] Registry request PUT /images/9fcadd95-c1b0-452a-9516-7eb2c3d5ae68 HTTP 200 request id req-e5ec0e6d-2b95-4d57-b600-f6f52882e0bc do_request /usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py:114
2014-05-22 02:12:16.689 27041 DEBUG glance.api.v1.images [6f89c589-745f-42e7-9da8-3c78d7abe6a5 a315717e20174b10a39db36b722325d6 76d25b1928e7407392a69735a894c7fc - - -] Uploading image data for image 9fcadd95-c1b0-452a-9516-7eb2c3d5ae68 to vsphere store _upload /usr/lib/python2.7/dist-packages/glance/api/v1/images.py:595
2014-05-22 02:12:16.693 27041 ERROR glance.store.vmware_datastore [6f89c589-745f-42e7-9da8-3c78d7abe6a5 a315717e20174b10a39db36b722325d6 76d25b1928e7407392a69735a894c7fc - - -] Failed to upload content of image 9fcadd95-c1b0-452a-9516-7eb2c3d5ae68
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore Traceback (most recent call last):
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore File "/usr/lib/python2.7/dist-packages/glance/store/vmware_datastore.py", line 276, in add
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore content=image_file)
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore File "/usr/lib/python2.7/dist-packages/glance/store/vmware_datastore.py", line 408, in _get_http_conn
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore conn.request(method, url, content, headers)
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 973, in request
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore self._send_request(method, url, body, headers)
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore self.endheaders(body)
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore self._send_output(message_body)
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore File "/usr/lib/python2.7/httplib.py", line 833, in _send_output
2014-05-22 02:12:16.693 27041 TRACE glance.store.vmware_datastore self.send(message_body)
2014-05-22 02:12:16...

Read more...

Changed in nova:
assignee: nobody → Arnaud Legendre (arnaudleg)
Revision history for this message
Antonio Ken Iannillo (ak-iannillo) wrote :

I got the same problem over here. Tried quite everything but my bug seems to be persistent.

Revision history for this message
Antonio Ken Iannillo (ak-iannillo) wrote :

I got the same problem over here. Tried quite everything but my bug seems to be persistent.

When I installed glance services I tried to use vCenter data center as glance storage. I got the error when I send the "glance image-create" request. I tried with more than one vmdk image and they gave the same 404 error.

Revision history for this message
Arnaud Legendre (arnaudleg) wrote :

This bug might happen when you have hosts that are not responding and that are attached to the datastore where the VM will reside: this is a vCenter issue.
As a workaround, please make sure that all of your hosts (where datastores in datastore_regex are mounted) are connected.

Revision history for this message
Jaroslav Henner (jhenner) wrote :

I think this may be some problem with the client or some problem with getting/sending the file size, because when I upload from stdin, it fails, but when I use --file, the file gets uploaded.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

from Jaroslav's last comment it seems like an issue with glance image-create with --file and not Nova per se

Changed in nova:
status: Confirmed → Incomplete
Revision history for this message
Sean Dague (sdague) wrote :

Closing as invalid, I think the glance direction is more fruitful

Changed in nova:
status: Incomplete → Invalid
Ian Cordasco (icordasc)
Changed in glance:
status: New → Confirmed
importance: Undecided → Low
Erno Kuvaja (jokke)
affects: glance → glance-store
Yang Yu (yuyangbj)
Changed in nova:
assignee: Arnaud Legendre (arnaudleg) → Yang Yu (yuyangbj)
Revision history for this message
Yang Yu (yuyangbj) wrote :

We also find this error. Same as the bug description.

Revision history for this message
Yang Yu (yuyangbj) wrote :

Hi Ryan,

Did you use the VSAN? After I changed the VSAN to local datastore, the glance is working well, so I guess the problem is VSAN configuration.

Revision history for this message
Ryan Hsu (rhsu) wrote :

@Yang, I was not using VSAN for this bug.

Revision history for this message
Julian (jbarnett-p) wrote :

Same problem here. This worked previously when I set default_store=vsphere and attempted a glance image-create (without --store) but if I set default_store=file and now try to run glance image-create WITH --store vsphere as a parameter, I'm seeing the exact same issue as reported. No VSAN, local datastore from NFS mount.

Commands I used were:
glance image-create --name Debian --store vsphere --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" --file debian-2.6.32-i686.vmdk

yields the following log entries: http://paste.openstack.org/show/166417/

and
glance image-create --name Debian --store vsphere --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

different log entries: http://paste.openstack.org/show/166418/

Revision history for this message
Chinmaya Bharadwaj (acbharadwaj) wrote :

Even i'm able to reproduce the exact same issue in my set up. with the bigger sized images. With Cirros boot was successful.

Compute Trace back:

Cause: [Errno 32] Broken pipe
2015-02-27 15:02:59.386 18682 TRACE nova.virt.vmwareapi.io_util Traceback (most recent call last):
2015-02-27 15:02:59.386 18682 TRACE nova.virt.vmwareapi.io_util File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/vmwareapi/io_util.py", line 179, in _inner
2015-02-27 15:02:59.386 18682 TRACE nova.virt.vmwareapi.io_util self.output.write(data)
2015-02-27 15:02:59.386 18682 TRACE nova.virt.vmwareapi.io_util File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo_vmware/rw_handles.py", line 281, in write
2015-02-27 15:02:59.386 18682 TRACE nova.virt.vmwareapi.io_util raise exceptions.VimException(excep_msg, excep)
2015-02-27 15:02:59.386 18682 TRACE nova.virt.vmwareapi.io_util VimException: Error occurred while writing data to https://10.1.214.24:443/folder/vmware_temp/72ca7f49-1db2-4502-b0ff-61cd339a6c73/675c6f2d-12f3-44de-a178-0838bbba01bb/675c6f2d-12f3-44de-a178-0838bbba01bb-flat.vmdk?dsName=SharedDS&dcPath=Dhruv.
2015-02-27 15:02:59.386 18682 TRACE nova.virt.vmwareapi.io_util Cause: [Errno 32] Broken pipe

VPXD log trace:

 error 'HTTP server /folder'] Reader failure: Operation timed out
2015-02-27T15:03:14.956Z [7F19E7060700 error 'HTTP server /folder'] Error while handling the PUT request at path /folder/vmware_temp/72ca7f49-1db2-4502-b0ff-61cd339a6c73/675c6f2d-12f3-44de-a178-0838bbba01bb/675c6f2d-12f3-44de-a178-0838bbba01bb-flat.vmdk?dsName=SharedDS&dcPath=Tintn: N7Vmacore23NotInitializedExceptionE(Stream is not open/connected)
2015-02-27T15:03:14.956Z [7F19E7AF5700 warning 'Proxy Req 64848'] Server named pipe /var/run/vmware/vpxd-webserver-pipe disconnected with a response pending

It could be because, glance server is not responding, when trying to fetch the image from it.

Revision history for this message
Jaroslav Henner (jhenner) wrote :
Download full text (4.8 KiB)

It seems like at first it worked, after some time (more than a hour) it stopped working and I have seen

2015-03-10 18:38:03.389 20064 DEBUG glance.registry.client.v1.api [-] Updating image metadata for image 99369f93-7412-4e25-a16a-003f6d6fb8c4... update_image_metadata /usr/lib/python2.7/site-packages/glance/registry/client/v1/api.py:166
2015-03-10 18:38:03.389 20064 DEBUG glance.common.client [-] Constructed URL: http://0.0.0.0:9191/images/99369f93-7412-4e25-a16a-003f6d6fb8c4 _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:398
2015-03-10 18:38:03.424 20064 DEBUG glance.registry.client.v1.client [-] Registry request PUT /images/99369f93-7412-4e25-a16a-003f6d6fb8c4 HTTP 200 request id req-6412d49d-c254-4454-96b1-7de9716a23ad do_request /usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py:122
2015-03-10 18:38:03.424 20064 DEBUG glance.api.v1.images [-] Uploading image data for image 99369f93-7412-4e25-a16a-003f6d6fb8c4 to vsphere store _upload /usr/lib/python2.7/site-packages/glance/api/v1/images.py:623
2015-03-10 18:38:04.137 20064 ERROR glance_store._drivers.vmware_datastore [-] Failed to upload content of image 99369f93-7412-4e25-a16a-003f6d6fb8c4
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore Traceback (most recent call last):
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 346, in add
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore content=image_file)
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 489, in _get_http_conn
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore conn.request(method, url, content, headers)
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 973, in request
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore self._send_request(method, url, body, headers)
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 1007, in _send_request
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore self.endheaders(body)
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore self._send_output(message_body)
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 833, in _send_output
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore self.send(message_body)
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 802, in send
2015-03-10 18:38:04.137 20064 TRACE glance_store._drivers.vmware_datastore self.sock.sendall(datablock)
2015-03-10 18:38:04.137 20064 TRACE glance_s...

Read more...

Revision history for this message
Jaroslav Henner (jhenner) wrote :

seems like dup of LP#1402354

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.