Bytes versus text issues on Python 3

Bug #1530888 reported by Victor Stinner on 2016-01-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tooz
Medium
Unassigned
python-tooz (Ubuntu)
High
James Page

Bug Description

It's unclear to me if group and member identifiers are supposed to be bytes or text strings in tooz. Try to run unit tests with "python3 -bb" to see BytesWarning errors.

It looks like bytes is preferred, but it's cause some conversion issues on Python 3.

Tooz errors have group_id and member_id attributes which are used to format error messages. Example:

---
class MemberAlreadyExist(ToozError):
    """Exception raised when the caller try to join a group but a member
    with the same identifier belongs to that group.
    """
    def __init__(self, group_id, member_id):
        self.group_id = group_id
        self.member_id = member_id
        super(MemberAlreadyExist, self).__init__(
            "Member %s has already joined %s" %
            (member_id, group_id))
---

If group_id and member_id must be bytes, we must decode them to format the error message. "%s" % b'bytes' raises a BytesWarning when python3 is used with -bb.

Julien Danjou (jdanjou) on 2016-01-04
Changed in python-tooz:
importance: Undecided → Medium
status: New → Triaged
Julien Danjou (jdanjou) wrote :

Since they're bytes, we can't really decode them safely I'd say. What's the best approach to decode them and print whatever it is (ascii or \xfoobar stuff)?

Luigi Toscano (ltoscano) wrote :
Download full text (4.1 KiB)

I hit a bug which looks like the same (or at least connected to this one).

While trying to run a full python3 Sahara job (https://review.openstack.org/#/c/576929/) I hit this (see

2018-07-17 18:26:16.262 18749 INFO sahara.main [-] Sahara engine started
2018-07-17 18:26:16.262 18749 DEBUG sahara.service.periodic [-] Starting periodic tasks with initial delay 57 seconds setup /opt/stack/sahara/sahara/service/periodic.py:253
2018-07-17 18:26:16.269 18749 CRITICAL sahara [-] Unhandled error: TypeError: can't concat bytes to str
2018-07-17 18:26:16.269 18749 ERROR sahara Traceback (most recent call last):
2018-07-17 18:26:16.269 18749 ERROR sahara File "/usr/local/bin/sahara-engine", line 10, in <module>
2018-07-17 18:26:16.269 18749 ERROR sahara sys.exit(main())
2018-07-17 18:26:16.269 18749 ERROR sahara File "/opt/stack/sahara/sahara/cli/sahara_engine.py", line 49, in main
2018-07-17 18:26:16.269 18749 ERROR sahara server.setup_sahara_engine()
2018-07-17 18:26:16.269 18749 ERROR sahara File "/opt/stack/sahara/sahara/main.py", line 109, in setup_sahara_engine
2018-07-17 18:26:16.269 18749 ERROR sahara periodic.setup()
2018-07-17 18:26:16.269 18749 ERROR sahara File "/opt/stack/sahara/sahara/service/periodic.py", line 262, in setup
2018-07-17 18:26:16.269 18749 ERROR sahara pt = _make_periodic_tasks()
2018-07-17 18:26:16.269 18749 ERROR sahara File "/opt/stack/sahara/sahara/service/periodic.py", line 137, in _make_periodic_tasks
2018-07-17 18:26:16.269 18749 ERROR sahara class SaharaPeriodicTasks(periodic_task.PeriodicTasks):
2018-07-17 18:26:16.269 18749 ERROR sahara File "/opt/stack/sahara/sahara/service/periodic.py", line 139, in SaharaPeriodicTasks
2018-07-17 18:26:16.269 18749 ERROR sahara CONF.periodic_coordinator_backend_url, 'sahara-periodic-tasks')
2018-07-17 18:26:16.269 18749 ERROR sahara File "/opt/stack/sahara/sahara/service/coordinator.py", line 98, in __init__
2018-07-17 18:26:16.269 18749 ERROR sahara super(HashRing, self).__init__(backend_url)
2018-07-17 18:26:16.269 18749 ERROR sahara File "/opt/stack/sahara/sahara/service/coordinator.py", line 51, in __init__
2018-07-17 18:26:16.269 18749 ERROR sahara self.coordinator.start()
2018-07-17 18:26:16.269 18749 ERROR sahara File "/usr/local/lib/python3.5/dist-packages/tooz/coordination.py", line 687, in start
2018-07-17 18:26:16.269 18749 ERROR sahara super(CoordinationDriverWithExecutor, self).start(start_heart)
2018-07-17 18:26:16.269 18749 ERROR sahara File "/usr/local/lib/python3.5/dist-packages/tooz/coordination.py", line 423, in start
2018-07-17 18:26:16.269 18749 ERROR sahara self._start()
2018-07-17 18:26:16.269 18749 ERROR sahara File "/usr/local/lib/python3.5/dist-packages/tooz/drivers/memcached.py", line 71, in wrapper
2018-07-17 18:26:16.269 18749 ERROR sahara return func(*args, **kwargs)
2018-07-17 18:26:16.269 18749 ERROR sahara File "/usr/local/lib/python3.5/dist-packages/tooz/drivers/memcached.py", line 277, in _start
2018-07-17 18:26:16.269 18749 ERROR sahara self.heartbeat()
2018-07-17 18:26:16.269 18749 ERROR sahara File "/usr/local/lib/python3.5/dist-packages/tooz/drivers/memcached.py", lin...

Read more...

Luigi Toscano (ltoscano) wrote :

On the sahara side, when changing the tooz backend from memcached to etcd3, instead of failing on _encode_member_id, the code fails on _encode_group_id, which is again probably relevant:

Traceback (most recent call last):
  File "/usr/local/bin/sahara-engine", line 10, in <module>
    sys.exit(main())
  File "/opt/stack/sahara/sahara/cli/sahara_engine.py", line 49, in main
    server.setup_sahara_engine()
  File "/opt/stack/sahara/sahara/main.py", line 109, in setup_sahara_engine
    periodic.setup()
  File "/opt/stack/sahara/sahara/service/periodic.py", line 262, in setup
    pt = _make_periodic_tasks()
  File "/opt/stack/sahara/sahara/service/periodic.py", line 137, in _make_periodic_tasks
    class SaharaPeriodicTasks(periodic_task.PeriodicTasks):
  File "/opt/stack/sahara/sahara/service/periodic.py", line 139, in SaharaPeriodicTasks
    CONF.periodic_coordinator_backend_url, 'sahara-periodic-tasks')
  File "/opt/stack/sahara/sahara/service/coordinator.py", line 99, in __init__
    self.join_group(group_id)
  File "/opt/stack/sahara/sahara/service/coordinator.py", line 69, in join_group
    self.coordinator.join_group(group_id).get()
  File "/usr/local/lib/python3.5/dist-packages/tooz/drivers/etcd3.py", line 218, in join_group
    encoded_group = self._encode_group_id(group_id)
  File "/usr/local/lib/python3.5/dist-packages/tooz/drivers/etcd3.py", line 163, in _encode_group_id
    return self.GROUP_PREFIX + group_id + b"/"
TypeError: can't concat bytes to str

James Page (james-page) on 2018-09-11
Changed in python-tooz (Ubuntu):
status: New → In Progress
importance: Undecided → High
milestone: none → ubuntu-18.10
assignee: nobody → James Page (james-page)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-tooz - 1.62.0-0ubuntu2

---------------
python-tooz (1.62.0-0ubuntu2) cosmic; urgency=medium

  * d/p/ensure-consistent-encoding-of-strings-for-ID.patch: Cherry
    pick inflight fix to resolve encoding issues under Python 3
    (LP: #1530888).

 -- James Page <email address hidden> Tue, 11 Sep 2018 11:14:22 -0600

Changed in python-tooz (Ubuntu):
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/601632
Committed: https://git.openstack.org/cgit/openstack/tooz/commit/?id=b5ebc60660f24e462ad5ee4aeb7028276f47d449
Submitter: Zuul
Branch: master

commit b5ebc60660f24e462ad5ee4aeb7028276f47d449
Author: James Page <email address hidden>
Date: Tue Sep 11 10:14:32 2018 -0600

    Ensure consistent encoding of strings for ID

    Ensure that ID's are correctly binary encoded when strings are
    provided, resolving compatibility issues with Python 3.

    Change-Id: Ia417920a7600aacd43f17b95ecd6a34b5faa226b
    Related-Bug: 1530888

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers