keystoneclient.exceptions.RequestTimeout with requests==2.8.0

[Yves-Gwenael Bourhis]

With requests==2.8.0 ospurge gives this traceback:

Traceback (most recent call last):
  File "ospurge.py", line 982, in <module>
    main()
  File "ospurge.py", line 960, in main
    action, args.insecure)
  File "ospurge.py", line 799, in perform_on_project
    endpoint_type, region_name, insecure)
  File "ospurge.py", line 177, in __init__
    client.roles.list() # Only admins are allowed to do this
  File "/home/yves/openstack/ospurge/.venv/local/lib/python2.7/site-packages/keystoneclient/v2_0/roles.py", line 47, in list
    return self._list("/OS-KSADM/roles", "roles")
  File "/home/yves/openstack/ospurge/.venv/local/lib/python2.7/site-packages/keystoneclient/base.py", line 124, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/home/yves/openstack/ospurge/.venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 170, in get
    return self.request(url, 'GET', **kwargs)
  File "/home/yves/openstack/ospurge/.venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/home/yves/openstack/ospurge/.venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/home/yves/openstack/ospurge/.venv/local/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/home/yves/openstack/ospurge/.venv/local/lib/python2.7/site-packages/keystoneclient/session.py", line 386, in request
    resp = send(**kwargs)
  File "/home/yves/openstack/ospurge/.venv/local/lib/python2.7/site-packages/keystoneclient/session.py", line 427, in _send_request
    raise exceptions.RequestTimeout(msg)
keystoneclient.exceptions.RequestTimeout: Request to http://identity.fr1.cloudwatt.net:35357/v2.0/OS-KSADM/roles timed out (HTTP 408)

Downgrading to previous requests verion works fine

[Flavio Percoco (flaper87)]

FWIW, we're hitting an issue with requests 2.8.0 in glanceclient:

======================================================================
FAIL: glanceclient.tests.unit.v1.test_shell.ShellInvalidEndpointandParameterTest.test_members_list_invalid_endpoint
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "glanceclient/tests/unit/v1/test_shell.py", line 293, in test_members_list_invalid_endpoint
    self.run_command, 'member-list --image-id fake')
  File "/home/flaper87/workspace/openstack/python-glanceclient/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 422, in assertRaises
    self.assertThat(our_callable, matcher)
  File "/home/flaper87/workspace/openstack/python-glanceclient/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat
    mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
  File "/home/flaper87/workspace/openstack/python-glanceclient/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 483, in _matchHelper
    mismatch = matcher.match(matchee)
  File "/home/flaper87/workspace/openstack/python-glanceclient/.tox/py27/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
    mismatch = self.exception_matcher.match(exc_info)
  File "/home/flaper87/workspace/openstack/python-glanceclient/.tox/py27/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
    mismatch = matcher.match(matchee)
  File "/home/flaper87/workspace/openstack/python-glanceclient/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 414, in match
    reraise(*matchee)
  File "/home/flaper87/workspace/openstack/python-glanceclient/.tox/py27/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
    result = matchee()
  File "/home/flaper87/workspace/openstack/python-glanceclient/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 969, in __call__
    return self._callable_object(*self._args, **self._kwargs)
  File "glanceclient/tests/unit/v1/test_shell.py", line 263, in run_command
    self.shell.main(cmd.split())
  File "glanceclient/shell.py", line 695, in main
    args.func(client, args)
  File "glanceclient/v1/shell.py", line 405, in do_member_list
    members = gc.image_members.list(**kwargs)
  File "glanceclient/v1/image_members.py", line 51, in list
    out.extend(self._list_by_image(image))
  File "glanceclient/v1/image_members.py", line 63, in _list_by_image
    resp, body = self.client.get(url)
  File "glanceclient/common/http.py", line 280, in get
    return self._request('GET', url, **kwargs)
  File "glanceclient/common/http.py", line 257, in _request
    raise exc.InvalidEndpoint(message=message)
glanceclient.exc.InvalidEndpoint: Error communicating with http://is.invalid/v1/images/fake/members: HTTPConnectionPool(host='is.invalid', port=80): Max retries exceeded with url: /v1/images/fake/members (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f5e9d66de50>: Failed to establish a new connection: [Errno -2] Name or service not known',))
...

[Jesse Pretorius (jesse-pretorius)]

Is this perhaps a duplicate of https://bugs.launchpad.net/python-glanceclient/+bug/1476770 ?

[Cory Benfield (lukasaoz)]

It's entirely possible this issue is related to https://github.com/kennethreitz/requests/issues/2811, which reports that we accidentally changed one of our thrown exception types.

[Flavio Percoco (flaper87)]

https://review.openstack.org/#/c/213310/4

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIGVzdGFibGlzaCBhIG5ldyBjb25uZWN0aW9uOiBbRXJybm8gLTJdIE5hbWUgb3Igc2VydmljZSBub3Qga25vd24nLCkpXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0NDQzNDY3Njg4MTN9

[Matt Riedemann (mriedem)]

Actually:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIGVzdGFibGlzaCBhIG5ldyBjb25uZWN0aW9uOiBbRXJybm8gLTJdIE5hbWUgb3Igc2VydmljZSBub3Qga25vd24nLCkpXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIgQU5EIHByb2plY3Q6XCJvcGVuc3RhY2svcHl0aG9uLWdsYW5jZWNsaWVudFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDQ0MzQ2NzY4ODEzLCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

[Matt Riedemann (mriedem)]

https://review.openstack.org/#/c/232503/ has been restored while we work out my change wrt requests/urllib3. So this will block 2.8.0 and sync to glanceclient and whatever else is using it.

We should think about getting something into the gate-tempest-dsvm-full job that can hit whatever failed here since we didn't see this in the integrated gate.

[Phil Hopkins (phil-hopkins-a)]

This also affects Kilo and the fix needs to be backported. I ran into this with a Kilo source install yesterday (Oct. 8, 2015)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/233756
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=81a750da5ee5cd2b63eae5f13d37372dce6f0cf3
Submitter: Jenkins
Branch: master

commit 81a750da5ee5cd2b63eae5f13d37372dce6f0cf3
Author: Jesse Pretorius <email address hidden>
Date: Mon Oct 12 19:09:58 2015 +0100

    Block/cap incompatible libraries

    This updates the global requirements to block requests 2.8.0 due to:
      https://launchpad.net/bugs/1476770 and
      https://launchpad.net/bugs/1503768 and
      https://launchpad.net/bugs/1505326

    And also blocks oslo.messaging 2.6.0 temporarily due to:
      https://launchpad.net/bugs/1505295

    And also blocks oslo.versionedobjects 0.11.0 temporarily due to:
      https://launchpad.net/bugs/1505677

    And also blocks WebOb<1.5.0 temporarily due to:
      https://launchpad.net/bugs/1505153

    Related-Bug: #1476770
    Related-Bug: #1503768
    Related-Bug: #1505326
    Related-Bug: #1505295
    Related-Bug: #1505153
    Related-Bug: #1505677
    Change-Id: I3aabbf717ef21a41c7bb9d21957df838642926f0

[Vladislav Belogrudov (vlad-belogrudov)]

please redo bugfix to have requests < 2.8.0
2.8.1 is as buggy as previous one. Also backport to stable/kilo stable/liberty is needed.

[Jesse Pretorius (jesse-pretorius)]

For stable/kilo I think this has been resolved with the release of keystonemiddleware 1.5.3 as per https://review.openstack.org/247553

[Steve Martinelli (stevemar)]

this no longer affects ospurge, with its unspecified requirements it lays down requests==2.11.0.

py27 installed: alabaster==0.7.9,appdirs==1.4.0,Babel==2.3.4,bashate==0.5.1,chardet==2.3.0,cliff==2.2.0,cmd2==0.6.9,debtcollector==1.9.0,doc8==0.7.0,docutils==0.12,extras==1.0.0,fixtures==3.0.0,flake8==2.5.5,funcsigs==1.0.2,functools32==3.2.3.post2,futures==3.0.5,hacking==0.12.0,httpretty==0.8.14,iso8601==0.1.11,Jinja2==2.8,jsonpatch==1.14,jsonpointer==1.10,jsonschema==2.5.1,keystoneauth1==2.15.0,linecache2==1.0.0,MarkupSafe==0.23,mccabe==0.2.1,monotonic==1.2,msgpack-python==0.4.8,netaddr==0.7.18,netifaces==0.10.5,nose==1.3.7,openstackdocstheme==1.5.0,os-client-config==1.22.0,osc-lib==1.2.0,oslo.config==3.19.0,oslo.i18n==3.10.0,oslo.serialization==2.14.0,oslo.utils==3.18.0,-e git+http://github.com/openstack/ospurge@f36e4e8a0a755d28aea3b3d2e3dbf45ffb394b94#egg=ospurge,pbr==1.10.0,pep8==1.5.7,positional==1.1.1,prettytable==0.7.2,pyflakes==0.8.1,Pygments==2.1.3,pyparsing==2.1.10,python-ceilometerclient==2.7.0,python-cinderclient==1.9.0,python-glanceclient==2.5.0,python-heatclient==1.5.0,python-keystoneclient==3.6.0,python-mimeparse==1.6.0,python-neutronclient==6.0.0,python-novaclient==6.0.0,python-subunit==1.2.0,python-swiftclient==3.1.0,pytz==2016.7,PyYAML==3.12,requests==2.11.1,requestsexceptions==1.1.3,restructuredtext-lint==0.17.2,rfc3986==0.4.1,simplejson==3.10.0,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.3.6,sphinx-rtd-theme==0.1.9,stevedore==1.18.0,testrepository==0.0.20,testtools==2.2.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,warlock==1.2.0,wrapt==1.10.8

[Cyril Roelandt (cyril-roelandt)]

This is no longer an issue in glanceclient, and we are now using a much more recent version of requests.