library users can't upload image

[Cao ShuFeng]

I meet this problem in my devstack in ubuntu 14.04 and ubuntu 15.10.

But, I can't reproduce this problem in Fedora 23.

Environment info:
glance (deployed in 2016-03-11)
glanceclient (clone from git in 2016-03-16)

reproduce step:
1. create a image, but not upload a file
glance image-create --name caoshu.img --disk-format ari --container-format ari
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | ari |
| created_at | 2016-03-16T09:49:09Z |
| disk_format | ari |
| id | 732a8198-7ca0-482a-80e8-c507191529f0 |
| min_disk | 0 |
| min_ram | 0 |
| name | caoshu.img |
| owner | a97cd30647614587b1588a7d3d399f8a |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2016-03-16T09:49:09Z |
| virtual_size | None |
| visibility | private |
+------------------+--------------------------------------+
2. use library to upload the image
>>> from glanceclient import Client
>>> glance = Client('2', endpoint='http://10.167.225.40:9292', token='d452914e14ae422391fbcb7037d9dc35')
>>> f = open('/home/fujitsu/cirros.img','rb')
>>> glance.images.upload("732a8198-7ca0-482a-80e8-c507191529f0",f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 217, in upload
    self.http_client.put(url, headers=hdrs, data=body)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py", line 287, in put
    return self._request('PUT', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py", line 262, in _request
    raise exc.CommunicationError(message=message)
glanceclient.exc.CommunicationError: Error finding address for http://10.167.225.40:9292/v2/images/732a8198-7ca0-482a-80e8-c507191529f0/file: [Errno 32] Broken pipe

Something may be helpful:
A. This bug can't be reproduced with HttpSessionClient, like this:
http://paste.openstack.org/show/490669/

B. If I use the glance client to do some *other* request to the server-side before the upload request, this can't be reproduced too.This is quite strange but may be the key to solve this problem : (
http://paste.openstack.org/show/490665/

Thanks in advance for anyone that help me with this issue.

[Cao ShuFeng (caosf-fnst)]

In cinder, when I run this command:
"cinder upload-to-image abc abc-image"
It gets such error[1] message in cinder-volume's log.

[1]: http://paste.openstack.org/show/495266/

[wangxiyuan (wangxiyuan)]

Can't reproduce with master branch.
http://paste.openstack.org/show/495301/
Did I miss something?

[wangxiyuan (wangxiyuan)]

And this is the test for cinder:
http://paste.openstack.org/show/495302/

I just setup a new devstack today. So Maybe it's your env's probelm?

[Cao ShuFeng (caosf-fnst)]

Hi, wangxiyuan:
Thanks for helping me with this problem.
I have meet this problem several times.
In my devstack deployed in 2015-04-23 I meet this again.

This is my local.conf[1], I can't think of anything else that may be helpful.

[1]: http://paste.openstack.org/show/495376/

Thanks.

[wangxiyuan (wangxiyuan)]

So it looks like a socket problem.
Broken Pipe occurs when one end of the connection tries sending data while the other end has already closed the connection.

So I guess the glance close the connection sometimes.

Not sure why. Need the glance log to dig it into.

[Cao ShuFeng (caosf-fnst)]

Hi, wang:
This is the glance log[1] when I upload a volume from cinder to glance failed.

PS: I meet this problem *randomly* in ubuntu 14.04 and 15.10.
I am trying to figure out whether Fedora 23 get the same problem.

[1]: http://paste.openstack.org/show/495378/
Thanks.

[wangxiyuan (wangxiyuan)]

I saw a DELETE request in the log which is strange. Did you mix some request together?

[Cao ShuFeng (caosf-fnst)]

Hi, wang:
I think the DELETE request is posted by cinder itself.
When uploading fails, cinder will delete that image from glance.

https://github.com/openstack/cinder/blob/ad5170a9e12ab7e14aa18c34a6e828055ef8dbe6/cinder/volume/manager.py#L1196

[wangxiyuan (wangxiyuan)]

It's hard to say which component goes wrong. Since you meet the problem both in v1 and v2. And from the LOG you pasted, I can only say it maybe an DB connection error? But it's not a glance or client bug which I'm pretty sure.

You can ping me (wxy) through IRC to get a quick replay.

[Cao ShuFeng (caosf-fnst)]

Hi, wang:
I paste some more log for failed command"cinder upload-to-image cde cde-image"

c-api.log: http://paste.openstack.org/show/495493/
c-sch.log: http://paste.openstack.org/show/495494/
c-vol.log: http://paste.openstack.org/show/495495/
g-api.log: http://paste.openstack.org/show/495496/
g-reg.log: http://paste.openstack.org/show/495497/

Thanks!!!

[wangxiyuan (wangxiyuan)]

It looks like that cinder failed to send request to glance because the connection between cinder-volume and glance client is down.
Could you please set the config option "glance_num_retries" in cinder.conf to a new number,such as 5, and try again?

[Cao ShuFeng (caosf-fnst)]

Hi, wang:
The "glance_num_retries = 5" make some more error logs in c-vol.log :(

c-api.log: http://paste.openstack.org/show/495603/
c-sch.log: http://paste.openstack.org/show/495605/
c-vol.log: http://paste.openstack.org/show/495604/
g-api.log: http://paste.openstack.org/show/495606/
g-reg.log: http://paste.openstack.org/show/495607/

Thanks

[wangxiyuan (wangxiyuan)]

Talked with Cao on IRC. The reason is that cinder can't send the data to glance.

[Sean McGinnis (sean-mcginnis)]

Was there an issue on the Cinder side? Or was this due to some configuration error?

[sapd (saphi070)]

I have same issue with Cao ShuFeng, How to fix it?