Broken compatibility with python-glanceclient

Bug #1766235 reported by Alexander Kurbatov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Glance Client
Fix Released
Critical
Brian Rosmaita
keystoneauth
Fix Released
Undecided
Monty Taylor

Bug Description

After the fix introduced in https://review.openstack.org/#/c/505764/ the keystoneauth cause TypeError: unorderable types: bytes() < str() for some requests executed by python-glanceclient e.g. when one tries to get the Glance task by id using SessionClient.

It is turned out that python-glanceclient encodes all the default headers before passing it to keystoneauth. The encoding is applied due to:
```
# NOTE(pumaranikar): To avoid bug #1641239, no modification of
# headers should be allowed after encode_headers() is called.
```
See https://github.com/openstack/python-glanceclient/blob/28c003dc1179ddb3124fd30c6f525dd341ae9213/glanceclient/common/http.py

While processing the request keystoneauth adds it's own headers (e.g. auth token and User-Agent) but doesn't encode or decode anything. As a result, we have a list of headers, some of which are plain strings while others are bytes.

Since v3.5.0 of keystoneauth all the received headers are sorted in _http_log_request (keystoneauth/keystoneauth1/session.py):
```
        if headers:
            # Sort headers so that testing can work consistently.
            for header in sorted(headers.items()):
                string_parts.append('-H "%s: %s"'
                                    % self._process_header(header))
```
The attempt to sort the headers cause TypeError: unorderable types: bytes() < str().

Revision history for this message
Alexander Kurbatov (alkurbatov) wrote :

For additional info:

The headers provided by python-glanceclient (token is replaced by a dummy):
```
dict_items([('User-Agent', 'python-glanceclient'), ('X-Auth-Token', 'xxx'), (b'Content-Type', b'application/octet-stream')])
```

The received traceback:
```
Traceback (most recent call last):
  File "/usr/libexec/vstorage-ui-backend/venv/lib64/python3.4/site-packages/python_glanceclient-2.9.0-py3.4.egg/glanceclient/common/utils.py", line 545, in inner
    return RequestIdProxy(wrapped(*args, **kwargs))
  File "/usr/libexec/vstorage-ui-backend/venv/lib64/python3.4/site-packages/python_glanceclient-2.9.0-py3.4.egg/glanceclient/v2/tasks.py", line 102, in get
    resp, body = self.http_client.get(url)
  File "/usr/libexec/vstorage-ui-backend/venv/lib64/python3.4/site-packages/keystoneauth1-3.5.0-py3.4.egg/keystoneauth1/adapter.py", line 304, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/libexec/vstorage-ui-backend/venv/lib64/python3.4/site-packages/python_glanceclient-2.9.0-py3.4.egg/glanceclient/common/http.py", line 335, in request
    **kwargs)
  File "/usr/libexec/vstorage-ui-backend/venv/lib64/python3.4/site-packages/keystoneauth1-3.5.0-py3.4.egg/keystoneauth1/adapter.py", line 189, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/libexec/vstorage-ui-backend/venv/lib64/python3.4/site-packages/keystoneauth1-3.5.0-py3.4.egg/keystoneauth1/session.py", line 732, in request
    logger=logger, split_loggers=split_loggers)
  File "/usr/libexec/vstorage-ui-backend/venv/lib64/python3.4/site-packages/keystoneauth1-3.5.0-py3.4.egg/keystoneauth1/session.py", line 393, in _http_log_request
    for header in sorted(headers.items()):
TypeError: unorderable types: bytes() < str()
```

description: updated
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

You can see from the discussion on https://bugs.launchpad.net/python-glanceclient/+bug/1641239 why we don't want to try to change this on our side.

Changed in python-glanceclient:
status: New → Won't Fix
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystoneauth (master)

Fix proposed to branch: master
Review: https://review.openstack.org/568365

Changed in keystoneauth:
assignee: nobody → Brian Rosmaita (brian-rosmaita)
status: New → In Progress
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Took a stab at fixing this, but no dice. There's still a problem with the py35 tests. Not sure if it's an artifact of the keystoneauth1 unit tests, or if it's a real problem. I could use some help. https://review.openstack.org/568365

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :
Changed in keystoneauth:
assignee: Brian Rosmaita (brian-rosmaita) → Monty Taylor (mordred)
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Monty has a new patch set up for the keystoneauth-side fix: https://review.openstack.org/568365

Looks like we do need a glanceclient-side fix also; see the commit message on this patch to the glanceclient for details: https://review.openstack.org/#/c/568698/

Changed in python-glanceclient:
status: Won't Fix → In Progress
assignee: nobody → Brian Rosmaita (brian-rosmaita)
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystoneauth (master)

Reviewed: https://review.openstack.org/568365
Committed: https://git.openstack.org/cgit/openstack/keystoneauth/commit/?id=35de6ebe93b94076964f4250bf3fa9b8ff1f8463
Submitter: Zuul
Branch: master

commit 35de6ebe93b94076964f4250bf3fa9b8ff1f8463
Author: Brian Rosmaita <email address hidden>
Date: Mon May 14 16:19:46 2018 -0400

    Fix logging of encoded headers

    A change introduced in 3.5.0 sorts headers, but runs into a problem
    when the headers are bytes, such as the headers provided by the
    python-glanceclient.

    requests expects headers to be str type in both python2 and python3.
    This means in python2 we need to encode unicode objects as ASCII (the
    encoding that should be used for HTTP headers) and in python3 we need to
    decode bytes as ASCII into str.

    Change-Id: Ib81497c3a873616c22ba68256c596a6fb113e11e
    Closes-bug: #1766235

Changed in keystoneauth:
status: In Progress → Fix Released
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Patch to release keystoneauth 3.6.2: https://review.openstack.org/#/c/568959/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystoneauth 3.6.2

This issue was fixed in the openstack/keystoneauth 3.6.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-glanceclient (master)

Reviewed: https://review.openstack.org/568698
Committed: https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=ee029a9b927a41c028427f8afc1821ed914e6d47
Submitter: Zuul
Branch: master

commit ee029a9b927a41c028427f8afc1821ed914e6d47
Author: Brian Rosmaita <email address hidden>
Date: Tue May 15 16:52:58 2018 -0400

    Handle HTTP headers per RFC 8187

    According to RFC 8187, HTTP headers should use 7-bit ASCII encoding.
    The glanceclient was encoding them as UTF-8, which can leave the 8th
    bit nonzero when representing unicode, and which presents problems
    for any recipient following the standard and decoding the headers as
    ASCII.

    This change requires keystoneauth1 3.6.2, which has a fix for a
    bug that made it unable to handle bytes in headers. The dependency
    is a patch bumping the keystoneauth1 version in upper-constraints.

    Depends-on: https://review.openstack.org/#/c/569138/

    Change-Id: I0d14974126fcb20e23a37347f4f1756c323cf2f5
    Closes-bug: #1766235

Changed in python-glanceclient:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-glanceclient 2.11.1

This issue was fixed in the openstack/python-glanceclient 2.11.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.