kolla vmtp image does not build on ubuntu bionic (and other Unicode issues)

[Radosław Piliszek]

tabulate has been updated in PyPI to 0.8.4 today and breaks builds:

INFO:kolla.common.utils.vmtp:Collecting tabulate>=0.7.3 (from vmtp==2.5.1.dev17)
INFO:kolla.common.utils.vmtp: Downloading http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/76/35/ae65ed1268d6e2a1be141723e5fffdf4a28e4f4e7c1e083709b308998f90/tabulate-0.8.4.tar.gz (45kB)
INFO:kolla.common.utils.vmtp: ERROR: Command errored out with exit status 1:
INFO:kolla.common.utils.vmtp: command: /var/lib/kolla/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-a0204_59/tabulate/setup.py'"'"'; __file__='"'"'/tmp/pip-install-a0204_59/tabulate/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
INFO:kolla.common.utils.vmtp: cwd: /tmp/pip-install-a0204_59/tabulate/
INFO:kolla.common.utils.vmtp: Complete output (5 lines):
INFO:kolla.common.utils.vmtp: Traceback (most recent call last):
INFO:kolla.common.utils.vmtp: File "<string>", line 1, in <module>
INFO:kolla.common.utils.vmtp: File "/tmp/pip-install-a0204_59/tabulate/setup.py", line 25, in <module>
INFO:kolla.common.utils.vmtp: f.write(LONG_DESCRIPTION)
INFO:kolla.common.utils.vmtp: UnicodeEncodeError: 'ascii' codec can't encode characters in position 5711-5728: ordinal not in range(128)
INFO:kolla.common.utils.vmtp: ----------------------------------------
INFO:kolla.common.utils.vmtp:
INFO:kolla.common.utils.vmtp:ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
INFO:kolla.common.utils.vmtp:

[Radosław Piliszek (yoctozepto)]

CentOS 7 not affected (python 2.7)

Ubuntu Bionic (18.04) has Python 3.6

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/684687

[Radosław Piliszek (yoctozepto)]

Debian Buster (10) not affected either (python 3.7) - weird

[Radosław Piliszek (yoctozepto)]

Seems to be something internal to kolla as tabulate installs fine on my Ubuntu workstation (wanted to debug but "it works for me" (TM) happened).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/684808

[Radosław Piliszek (yoctozepto)]

0.8.4 fixed by using Unicode locale in Ubuntu (somehow CentOS and Debian did not care)

[Radosław Piliszek (yoctozepto)]

0.8.5 released not so long ago seems to have fixed the issue for non-unicode too

[Will Szumski (willjs)]

Somewhat bizarrely the troublesome:

        with open("LONG_DESCRIPTION.md", "w") as f:
            f.write(LONG_DESCRIPTION)

only appears in the 0.8.4 python package from pypi. It is absent from the source in github and has since been removed in 0.8.5. I can only think it was added for debugging purposes and accidentally made it into the release.

[Radosław Piliszek (yoctozepto)]

Indeed, Will, I shared the same thought on IRC. More bizarrely it was never in the git repo of tabulate. Looks like local development dump made it into PyPI. Things happen. ;-)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: master
Review: https://review.opendev.org/684687
Reason: no longer an issue

[Radosław Piliszek (yoctozepto)]

Indeed looks like we need to modernize:
2019-10-02 07:21:00.045 17 ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file /etc/glance/metadefs/compute-hypervisor.json while populating metadata due to: 'ascii' codec can't decode byte 0xc2 in position 898: ordinal not in range(128): UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 898: ordinal not in range(128) 2019-10-02 07:21:00.656 17 ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file /etc/glance/metadefs/software-runtimes.json while populating metadata due to: 'ascii' codec can't decode byte 0xe2 in position 1367: ordinal not in range(128): UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1367: ordinal not in range(128) 2019-10-02 07:21:00.707 17 ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file /etc/glance/metadefs/compute-trust.json while populating metadata due to: 'ascii' codec can't decode byte 0xc2 in position 90: ordinal not in range(128): UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 90: ordinal not in range(128) 2019-10-02 07:39:48.431 16 ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file /etc/glance/metadefs/compute-hypervisor.json while populating metadata due to: 'ascii' codec can't decode byte 0xc2 in position 898: ordinal not in range(128): UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 898: ordinal not in range(128) 2019-10-02 07:39:48.472 16 ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file /etc/glance/metadefs/software-runtimes.json while populating metadata due to: 'ascii' codec can't decode byte 0xe2 in position 1367: ordinal not in range(128): UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1367: ordinal not in range(128) 2019-10-02 07:39:48.477 16 ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file /etc/glance/metadefs/compute-trust.json while populating metadata due to: 'ascii' codec can't decode byte 0xc2 in position 90: ordinal not in range(128): UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 90: ordinal not in range(128)

[Radosław Piliszek (yoctozepto)]

reason debian not affected: https://www.python.org/dev/peps/pep-0538/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/684808
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=a266023ab572f6dc728271a2f3936c385674c08a
Submitter: Zuul
Branch: master

commit a266023ab572f6dc728271a2f3936c385674c08a
Author: Radosław Piliszek <email address hidden>
Date: Wed Sep 25 18:28:38 2019 +0200

    Use C.UTF-8 locale

    This ought to fix problems with Unicode,
    including the recent vmtp issue with tabulate 0.8.4 failing
    due to non-Unicode locale (Ubuntu Bionic 18.04 Python 3.6).

    Change-Id: I8b55a6a260cab14c2292cac02a1a07729223c281
    Related-bug: #1845306
    Signed-off-by: Radosław Piliszek <email address hidden>