subunit2html failed with "UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position ..."

[Liron Kuchlani]

The exception is caused by the following code line:
https://github.com/openstack/os-testr/blob/master/os_testr/subunit2html.py#L685

683 script = TemplateData.REPORT_TEST_OUTPUT_TMPL % dict(
684 id=tid,
685 -> output=saxutils.escape(o + e),
686 )
687
688 row = tmpl % dict(
689 tid=tid,
690 Class=((n == 0 or n == 3) and 'hiddenRow' or 'none'),
##################################
For some reason the type of variable is different between o to e, as you can see below:
(Pdb) type(o)
<type 'unicode'>
(Pdb) type(e)
<type 'str'>
(Pdb) o + e
*** UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 123639: ordinal not in range(128)

Actual results:

Traceback (most recent call last):
  File "/usr/bin/subunit2html", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/os_testr/subunit2html.py", line 763, in main
    result.stopTestRun()
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 549, in stopTestRun
    sink.stopTestRun()
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 1775, in stopTestRun
    self.decorated.stopTestRun()
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 1529, in stopTestRun
    return self.decorated.stopTestRun()
  File "/usr/lib/python2.7/site-packages/os_testr/subunit2html.py", line 517, in stopTestRun
    report = self._generate_report()
  File "/usr/lib/python2.7/site-packages/os_testr/subunit2html.py", line 604, in _generate_report
    self._generate_report_test(rows, cid, tid, n, t, o, e)
  File "/usr/lib/python2.7/site-packages/os_testr/subunit2html.py", line 688, in _generate_report_test
    output=saxutils.escape(o + e),
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 123639: ordinal not in range(128)

[Liron Kuchlani (lkuchlan)]

This behavior only happens in python2

[Martin Kopec (mkopec)]

Related fix: https://review.opendev.org/#/c/700778/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/700778
Committed: https://git.openstack.org/cgit/openstack/os-testr/commit/?id=2c07d4e638122f845b3d0440f4864df8356d2a52
Submitter: Zuul
Branch: master

commit 2c07d4e638122f845b3d0440f4864df8356d2a52
Author: lkuchlan <email address hidden>
Date: Mon Dec 30 16:52:41 2019 +0200

    subunit2html dealing with incorrect data format

    The patch encodes a data string which contains
    non-ASCII characters to unicode type.

    In python2 there are two text types str and unicode.
    When we add two different text types, for example:
    >>> "Hello " + u"World"
    u'Hello World'

    The string on the left is decoded by using the default
    system encoding into a Unicode string.
    If it would contain non-ASCII characters(as in our case)
    this normally blow up with an UnicodeDecodeError because
    the default encoding is set to ASCII.

    Change-Id: I17e5aa6f45133d9d12aa38cec9cc42694913d4bd
    Closes-Bug: #1857901

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-testr 2.0.0 release.