Request of large files with --debug raises a MemoryError

[Tobias Diaz]

Working with images and logging in DEBUG level causes a MemoryError because _http_log_response() calls response.text which in turn tries to read all data before passing params to _remove_service_catalog:

  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/glanceclient/v1/images.py", line 148, in data
    % urlparse.quote(str(image_id)))
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in get
    return self.request(url, 'GET', **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/glanceclient/common/http.py", line 331, in request
    **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneclient/adapter.py", line 101, in request
    return self.session.request(url, method, **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneclient/session.py", line 404, in request
    resp = send(**kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneclient/session.py", line 464, in _send_request
    self._http_log_response(resp, logger)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneclient/session.py", line 219, in _http_log_response
    text = _remove_service_catalog(response.text)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/models.py", line 773, in text
    encoding = self.apparent_encoding
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/models.py", line 647, in apparent_encoding
    return chardet.detect(self.content)['encoding']
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/packages/chardet/__init__.py", line 30, in detect
    u.feed(aBuf)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/packages/chardet/universaldetector.py", line 128, in feed
    if prober.feed(aBuf) == constants.eFoundIt:
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/packages/chardet/charsetgroupprober.py", line 64, in feed
    st = prober.feed(aBuf)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/packages/chardet/sjisprober.py", line 54, in feed
    for i in range(0, aLen):
MemoryError

[kamalhussain (hussain-kamal)]

What is the exact command that was run? Could you also tell me the OpenStack version that is used?

[Tobias Diaz (int-0)]

I'm using the python API, not the CLI interface. The installed python API is:

python-glanceclient (2.4.0)
python-heatclient (1.3.0)
python-keystoneclient (3.3.0)
python-novaclient (5.0.0)
python-swiftclient (3.0.0)

I'm trying to upload a simple patch (just catch the MemoryError and set an alternate text for this situation) but I cannot send a review...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/359292

[Jamie Lennox (jamielennox)]

So keystoneclient is deprecated but this may be the sort of thing we want to fix anyway.

In keystoneauth we fixed this with [1] which basically ensures any text we print is ascii - I'm not sure if ascii is correct here because we would want to show unicode but that may raise some false positives.

Whilst this doesn't explicitly check size it will make sure we don't print any images which is the original bug.

I would like to do something more proactive than simply catch MemoryError because by the time you're at MemoryError you're kinda screwed.

Can we port terry's patch over to ksc? does that solve the issue?

[1] https://review.openstack.org/#/c/214380/

[Jamie Lennox (jamielennox)]

Also switch to using keystoneauth if you have the chance!

[Tobias Diaz (int-0)]

After migrating to keystoneath1 sessions the problem persist, the issue is the same: trying to get the body of the huge messages:

  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/glanceclient/v1/images.py", line 148, in data
    % urlparse.quote(str(image_id)))
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in get
    return self.request(url, 'GET', **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/glanceclient/common/http.py", line 331, in request
    **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneclient/adapter.py", line 101, in request
    return self.session.request(url, method, **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneauth1/session.py", line 555, in request
    resp = send(**kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneauth1/session.py", line 620, in _send_request
    self._http_log_response(response=resp, logger=logger)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneauth1/session.py", line 352, in _http_log_response
    text = self._remove_service_catalog(response.text)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/models.py", line 787, in text
    encoding = self.apparent_encoding
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/models.py", line 653, in apparent_encoding
    return chardet.detect(self.content)['encoding']
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/packages/chardet/__init__.py", line 30, in detect
    u.feed(aBuf)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/packages/chardet/universaldetector.py", line 128, in feed
    if prober.feed(aBuf) == constants.eFoundIt:
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/packages/chardet/charsetgroupprober.py", line 64, in feed
    st = prober.feed(aBuf)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/packages/chardet/sjisprober.py", line 54, in feed
    for i in range(0, aLen):
MemoryError

[Samuel de Medeiros Queiroz (samueldmq)]

Looking at the stacktrace posted in the description, this is not related to logging the response of the call, but to the call to the requests library in the response.text call.

Is this a bug in the requests library or are we not using it correctly ?

keystoneclient/session.py", line 464, in _send_request
    self._http_log_response(resp, logger)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/keystoneclient/session.py", line 219, in _http_log_response
    text = _remove_service_catalog(response.text)
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/requests/models.py", line 773, in text
    encoding = self.apparent_encoding
  File "/home/etobdia/.env/cloud-wrapper/lib/python2.7/site-packages/

[Kairat Kushaev (kkushaev)]

one of os users reproduced similar problem with the following scenario:
download glance image with --debug option

Image must be quite big.
If debug is turned on we got here:
https://github.com/openstack/keystoneauth/blob/459f3c0926b1cf62dadafb014b8e7a98e20dc6b9/keystoneauth1/session.py#L357
response.text tries to read all data before passing params to _remove_service_catalog.
After that we got:
https://zerobin.net/?bfd05129e9029bdf#hOFzRJUn1XdbbHLvDbxG45ufbGN50F0q8Esf+niph1M=
which indicates that requests tries to read all image as var in memory.
Apparently, we don't need to prematurely read response body in some cases.
Please also consider that glance is using chunked responses so we don't have overall response size in this case.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/416249

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/416260

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416249
Committed: https://git.openstack.org/cgit/openstack/keystoneauth/commit/?id=f345559a06c0128dcb7fede4b593487540da86ef
Submitter: Jenkins
Branch: master

commit f345559a06c0128dcb7fede4b593487540da86ef
Author: Samuel de Medeiros Queiroz <email address hidden>
Date: Tue Jan 3 11:04:59 2017 -0300

    Prevent MemoryError when logging response bodies

    Response bodies are loaded into memory prior to
    being logged.

    Loading huge response bodies may result in a
    MemoryError.

    This patch proposes that only JSON and TEXT
    responses be logged, i.e when the Content-Type
    header is application/json or application/text.

    Responses that do not include or have a different
    Content-Type header will have their body omitted.

    Closes-bug: 1616105

    Change-Id: I93b6fff73368c4f58bdebf8566c4948b50980cee

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/359292
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=3e56e0d7e5e1a76d806a3bc1f6d5ef9070f95771
Submitter: Jenkins
Branch: master

commit 3e56e0d7e5e1a76d806a3bc1f6d5ef9070f95771
Author: Tobias Diaz <email address hidden>
Date: Tue Aug 23 17:13:24 2016 +0200

    Prevent MemoryError when logging response bodies

    Response bodies are loaded into memory prior to
    being logged.

    Loading huge response bodies may result in a
    MemoryError.

    This patch proposes that only JSON and TEXT
    responses be logged, i.e when the Content-Type
    header is application/json or application/text.

    Responses that do not include or have a different
    Content-Type header will have their body omitted.

    This is a sort of backport of the fix for
    keystoneauth sessions, see
    I93b6fff73368c4f58bdebf8566c4948b50980cee

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>

    Closes-bug: 1616105

    Change-Id: I8f43eee3a0b35041c6cf672e476f8151cf2f8d14

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416260
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=af770f17b705a66bd4292b2a54df46ec5fdaa12b
Submitter: Jenkins
Branch: master

commit af770f17b705a66bd4292b2a54df46ec5fdaa12b
Author: Steve Martinelli <email address hidden>
Date: Tue Jan 3 09:55:09 2017 -0500

    Do not log binary data during request

    Do not log binary data during debug logging of a session.
    Replace the binary data with the string <binary_data> instead.

    sort of a backport of: I5184002f3a21c5e0ee510b21b9a7884c8dccd1e3

    Change-Id: I07ddbc3967f297597542f1975004d94c490f6e6b
    Related-Bug: 1616105

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/418090

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418091

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/418092

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418093

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/418094

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418095

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418095
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=e5bc019840c2dd08a4e48fd26a23bff8d1ab644c
Submitter: Jenkins
Branch: stable/mitaka

commit e5bc019840c2dd08a4e48fd26a23bff8d1ab644c
Author: Steve Martinelli <email address hidden>
Date: Tue Jan 3 09:55:09 2017 -0500

    Do not log binary data during request

    Do not log binary data during debug logging of a session.
    Replace the binary data with the string <binary_data> instead.

    sort of a backport of: I5184002f3a21c5e0ee510b21b9a7884c8dccd1e3

    Change-Id: I07ddbc3967f297597542f1975004d94c490f6e6b
    Related-Bug: 1616105
    (cherry picked from commit af770f17b705a66bd4292b2a54df46ec5fdaa12b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418094
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=ad4c0198efd4131f5b2a042fd8de7a076565ab74
Submitter: Jenkins
Branch: stable/newton

commit ad4c0198efd4131f5b2a042fd8de7a076565ab74
Author: Steve Martinelli <email address hidden>
Date: Tue Jan 3 09:55:09 2017 -0500

    Do not log binary data during request

    Do not log binary data during debug logging of a session.
    Replace the binary data with the string <binary_data> instead.

    sort of a backport of: I5184002f3a21c5e0ee510b21b9a7884c8dccd1e3

    Change-Id: I07ddbc3967f297597542f1975004d94c490f6e6b
    Related-Bug: 1616105
    (cherry picked from commit af770f17b705a66bd4292b2a54df46ec5fdaa12b)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/418194

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/418196

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418197

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418194
Committed: https://git.openstack.org/cgit/openstack/keystoneauth/commit/?id=d73fd3ee8438f760c37d9ef401ebf89a45a59f1d
Submitter: Jenkins
Branch: master

commit d73fd3ee8438f760c37d9ef401ebf89a45a59f1d
Author: Jamie Lennox <email address hidden>
Date: Tue Jan 10 14:45:35 2017 +1100

    Only log application/json in session to start

    When whitelisting content types to debug print from session we chose
    application/json and application/text. application/text is not a real
    mime type, text is typically text/plain.

    Rather than guess at mime types only print application/json to start
    with, but make it easy for additional types to be added later.

    Change-Id: Ica5fee076cdab8b1d5167161d28af7313fad9477
    Related-Bug: 1616105

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystoneauth 2.17.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/418653

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Martinelli (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/418090
Reason: see https://review.openstack.org/#/c/418196/ instead

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Martinelli (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/418091
Reason: see https://review.openstack.org/#/c/418197/ instead

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Martinelli (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/418092
Reason: i'll merge this with the follow up fix so it's a single patch

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Martinelli (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/418093
Reason: i'll merge this with the follow up fix so it's a single patch

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Martinelli (<email address hidden>) on branch: master
Review: https://review.openstack.org/418653
Reason: sigh, did this on master

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/418680

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418682

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418653
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=51d16fa344829aadf454faf5e0c4535a8f96a7c8
Submitter: Jenkins
Branch: master

commit 51d16fa344829aadf454faf5e0c4535a8f96a7c8
Author: Steve Martinelli <email address hidden>
Date: Tue Jan 10 21:58:36 2017 -0500

    Only log application/json in session to start

    When whitelisting content types to debug print from session we chose
    application/json and application/text. application/text is not a real
    mime type, text is typically text/plain.

    Rather than guess at mime types only print application/json to start
    with, but make it easy for additional types to be added later.

    Adapted from keystoneauth: Ica5fee076cdab8b1d5167161d28af7313fad9477
    Related-Bug: 1616105

    Change-Id: Ieaa8fb3ea8d25e09b89498f23b70b18c0f6153f1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418682
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=a0c67b860b3c5ff6f5fbb0e406dcbc3fd7aa9f88
Submitter: Jenkins
Branch: stable/mitaka

commit a0c67b860b3c5ff6f5fbb0e406dcbc3fd7aa9f88
Author: Tobias Diaz <email address hidden>
Date: Tue Aug 23 17:13:24 2016 +0200

    Only log application/json content type

    This is a combination of 2 commits. The first commit's
    message is:

    Prevent MemoryError when logging response bodies

    Response bodies are loaded into memory prior to
    being logged.

    Loading huge response bodies may result in a
    MemoryError.

    This patch proposes that only JSON and TEXT
    responses be logged, i.e when the Content-Type
    header is application/json or application/text.

    Responses that do not include or have a different
    Content-Type header will have their body omitted.

    This is a sort of backport of the fix for
    keystoneauth sessions, see
    I93b6fff73368c4f58bdebf8566c4948b50980cee

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>

    Closes-bug: 1616105

    Change-Id: I8f43eee3a0b35041c6cf672e476f8151cf2f8d14
    (cherry-picked from: 3e56e0d7e5e1a76d806a3bc1f6d5ef9070f95771)

    Only log application/json in session to start

    When whitelisting content types to debug print from session we chose
    application/json and application/text. application/text is not a real
    mime type, text is typically text/plain.

    Rather than guess at mime types only print application/json to start
    with, but make it easy for additional types to be added later.

    Adapted from keystoneauth: Ica5fee076cdab8b1d5167161d28af7313fad9477
    Related-Bug: 1616105

    Change-Id: Ieaa8fb3ea8d25e09b89498f23b70b18c0f6153f1
    (cherry-picked from: 51d16fa344829aadf454faf5e0c4535a8f96a7c8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418197
Committed: https://git.openstack.org/cgit/openstack/keystoneauth/commit/?id=13054e8b6d4e04c19ce24a3c8fcca320333c45fa
Submitter: Jenkins
Branch: stable/mitaka

commit 13054e8b6d4e04c19ce24a3c8fcca320333c45fa
Author: Samuel de Medeiros Queiroz <email address hidden>
Date: Tue Jan 3 11:04:59 2017 -0300

    Only log application/json content type

    This is a combination of 2 commits. The first commit's
    message is:

    Prevent MemoryError when logging response bodies

    Response bodies are loaded into memory prior to
    being logged.

    Loading huge response bodies may result in a
    MemoryError.

    This patch proposes that only JSON and TEXT
    responses be logged, i.e when the Content-Type
    header is application/json or application/text.

    Responses that do not include or have a different
    Content-Type header will have their body omitted.

    Closes-bug: 1616105

    Change-Id: I93b6fff73368c4f58bdebf8566c4948b50980cee
    (cherry picked from commit f345559a06c0128dcb7fede4b593487540da86ef)

    This is the 2nd commit message:

    Only log application/json in session to start

    When whitelisting content types to debug print from session we chose
    application/json and application/text. application/text is not a real
    mime type, text is typically text/plain.

    Rather than guess at mime types only print application/json to start
    with, but make it easy for additional types to be added later.

    Change-Id: Ica5fee076cdab8b1d5167161d28af7313fad9477
    Related-Bug: 1616105
    (cherry-picked from: d73fd3ee8438f760c37d9ef401ebf89a45a59f1d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418680
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=b69f3a1f0abcc13f65833d29036028bf3319a2cd
Submitter: Jenkins
Branch: stable/newton

commit b69f3a1f0abcc13f65833d29036028bf3319a2cd
Author: Tobias Diaz <email address hidden>
Date: Tue Aug 23 17:13:24 2016 +0200

    Only log application/json content type

    This is a combination of 2 commits. The first commit's
    message is:

    Prevent MemoryError when logging response bodies

    Response bodies are loaded into memory prior to
    being logged.

    Loading huge response bodies may result in a
    MemoryError.

    This patch proposes that only JSON and TEXT
    responses be logged, i.e when the Content-Type
    header is application/json or application/text.

    Responses that do not include or have a different
    Content-Type header will have their body omitted.

    This is a sort of backport of the fix for
    keystoneauth sessions, see
    I93b6fff73368c4f58bdebf8566c4948b50980cee

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>

    Closes-bug: 1616105

    Change-Id: I8f43eee3a0b35041c6cf672e476f8151cf2f8d14
    (cherry-picked from: 3e56e0d7e5e1a76d806a3bc1f6d5ef9070f95771)

    Only log application/json in session to start

    When whitelisting content types to debug print from session we chose
    application/json and application/text. application/text is not a real
    mime type, text is typically text/plain.

    Rather than guess at mime types only print application/json to start
    with, but make it easy for additional types to be added later.

    Adapted from keystoneauth: Ica5fee076cdab8b1d5167161d28af7313fad9477
    Related-Bug: 1616105

    Change-Id: Ieaa8fb3ea8d25e09b89498f23b70b18c0f6153f1
    (cherry-picked from: 51d16fa344829aadf454faf5e0c4535a8f96a7c8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418196
Committed: https://git.openstack.org/cgit/openstack/keystoneauth/commit/?id=5efe19ff7f7792bc35eb9e351d9aebb114a23b1e
Submitter: Jenkins
Branch: stable/newton

commit 5efe19ff7f7792bc35eb9e351d9aebb114a23b1e
Author: Samuel de Medeiros Queiroz <email address hidden>
Date: Tue Jan 3 11:04:59 2017 -0300

    Only log application/json content type

    This is a combination of 2 commits. The first commit's
    message is:

    Prevent MemoryError when logging response bodies

    Response bodies are loaded into memory prior to
    being logged.

    Loading huge response bodies may result in a
    MemoryError.

    This patch proposes that only JSON and TEXT
    responses be logged, i.e when the Content-Type
    header is application/json or application/text.

    Responses that do not include or have a different
    Content-Type header will have their body omitted.

    Closes-bug: 1616105

    Change-Id: I93b6fff73368c4f58bdebf8566c4948b50980cee
    (cherry picked from commit f345559a06c0128dcb7fede4b593487540da86ef)

    This is the 2nd commit message:

    Only log application/json in session to start

    When whitelisting content types to debug print from session we chose
    application/json and application/text. application/text is not a real
    mime type, text is typically text/plain.

    Rather than guess at mime types only print application/json to start
    with, but make it easy for additional types to be added later.

    Change-Id: Ica5fee076cdab8b1d5167161d28af7313fad9477
    Related-Bug: 1616105
    (cherry-picked from: d73fd3ee8438f760c37d9ef401ebf89a45a59f1d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-keystoneclient 3.9.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystoneauth 2.12.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystoneauth 2.4.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-keystoneclient 2.3.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-keystoneclient 3.5.1 release.