High latency in SSL connect causes WantReadError

[Phil Neal]

We ran into an issue where the Ceilometer central agent was bubbling up errors ("WantReadError") from the Glance client when trying to poll the Glance api over an https connection. The target Glance instance is outside a firewall on a separate node.

Based on the debug trace, I've tracked this down to the common http library in glanceclient/common:

        try:
           .....
            conn_url = utils.ensure_str(conn_url)
            if kwargs['headers'].get('Transfer-Encoding') == 'chunked':
                conn.putrequest(method, conn_url)
           .....
            else:
                conn.request(method, conn_url, **kwargs)
            resp = conn.getresponse()

The issue occurs when conn.request() is made. On slow connections the slow process of the data transfer/ssl-decoding seems to take long enough that the data is not yet ready when conn.getresponse() follows immediately after, hence no data to read. I can reproduce this pretty consistently in our network setup, but it might be difficult for others to simulate: inserting a brief sleep between the two operations eliminates the issue.

[Phil Neal (nealph)]

Working on a fix for this using a select option to force a wait to occur.

[Phil Neal (nealph)]

Could not get to this...can someone on the Glance team review?

[Stuart McLaren (stuart-mclaren)]

Which version of python-glanceclient are you using?

[Jesse Pretorius (jesse-pretorius)]

Duplicate of https://bugs.launchpad.net/python-glanceclient/+bug/1157864 ?

[Feilong Wang (flwang)]

I think this is not a dup of bug/1157864, since we are still running into this problem on stable/icehouse release.

[Feilong Wang (flwang)]

In our prod, we can reproduce this issue. Now, we have to hack the code to sleep a while. But I'm looking for a way to upstream it.

2015-04-23 23:52:40,450 10583 ERROR django.request Internal Server Error: /horizon/project/instances/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/views.py", line 92, in get_data
    exceptions.handle(self.request, ignore=True)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/views.py", line 89, in get_data
    images, more = api.glance.image_list_detailed(self.request)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/glance.py", line 93, in image_list_detailed
    images = list(images_iter)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/glance.py", line 82, in <genexpr>
    images_iter = (i for i in images_iter if ((i.is_public and '-OLD-'
  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 174, in paginate
    images = self._list(url, "images")
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/base.py", line 53, in _list
    resp, body = self.api.json_request('GET', url)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 266, in json_request
    resp, body_iter = self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 226, in _http_request
    resp = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/...

[Kairat Kushaev (kkushaev)]

IceHouse is EOL, the same is applicable for Folsom (when the issue was created).
So looks like it was really a dupe.

[Kairat Kushaev (kkushaev)]

Please reopen a bug if it applicable to master.