UnicodeDecodeError when trying to display text summary to stdout

[Pierre Equoy]

When testing device 201701-25364, the session ends normally, but when checkbox wants to display the test summary to stdout, it crashes with the following error:

----------------------------------------------------------------------------
Reports will be saved to: /home/u/.local/share/checkbox-ng
ERROR:plainbox.bug:Undeclared exception UnicodeDecodeError raised from export_to_transport

                        Application Malfunction Detected

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 248, in eat
    return self._dispatch()
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 298, in _dispatch
    result = ingredient.dispatch(self.context)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 152, in dispatch
    return self._dispatch(context, 0)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 179, in _dispatch
    return self._dispatch_None(context, level, retval, command)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 208, in _dispatch_None
    return self._dispatch(context, level + 1)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 165, in _dispatch
    retval = command.invoked(context)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/subcommands.py", line 166, in invoked
    self._export_results()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/subcommands.py", line 559, in _export_results
    exporter_id, transport)
  File "/usr/lib/python3/dist-packages/plainbox/impl/decorators.py", line 104, in wrapper
    raise exc
  File "/usr/lib/python3/dist-packages/plainbox/impl/decorators.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/plainbox/impl/session/assistant.py", line 1398, in export_to_transport
    result = transport.send(exported_stream)
  File "/usr/lib/python3/dist-packages/plainbox/impl/transport.py", line 354, in send
    copyfileobj(data, translating_stream)
  File "/usr/lib/python3.5/shutil.py", line 76, in copyfileobj
    fdst.write(buf)
  File "/usr/lib/python3/dist-packages/plainbox/impl/exporter/__init__.py", line 389, in write
    return self.dest_stream.write(data.decode(self.encoding))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 16383: unexpected end of data

  Please report a bug including the information from the paragraph above. To
  report the bug visit https://bugs.launchpad.net/checkbox-ng/+filebug

  We are sorry for the inconvenience!
----------------------------------------------------------------------------

We can still manually export the session using the following command:

    plainbox session export -f xml -o /tmp/submission.xml <session_id>

[Pierre Equoy (pieq)]

This happens when Checkbox displays the text summary at the end of the test run.

This happens because the data stream seems to be truncated when it's about to display the checked mark sigil (☑ or, in utf-8, "0xE2 0x98 0x91") for job `suspend/bluetooth_obex_send_after_suspend`.

We could not determine the root cause yet.

Knowing that the text summary displayed on screen at the end of a run is for reference only, (1) it shouldn't prevent the creation of the actual submission file and (2) it's not a big deal if a check mark is missing from it.

To address (1), we could first create the submission files (xml, json, etc.), then display the text summary to stdout (currently it's the other way around). The part to upload the submission file would still be at the end of the whole process, but this part can be run manually.

For (2), we can just use, in plainbox/impl/exporter/__init__.py,

    return self.dest_stream.write(data.decode(self.encoding, errors='ignore'))

instead of

    return self.dest_stream.write(data.decode(self.encoding))

when the destination stream is the screen (stdout).

[Sylvain Pineau (sylvain-pineau)]

spineau: ePierre: I've found the root cause of https://bugs.launchpad.net/plainbox/+bug/1663166
spineau: ePierre: to render the text data we're using ByteStringStreamTranslator
spineau: ePierre: but we're using filling it with shutil.copyfileobj
spineau: ePierre: so there are (in our case) two calls to the write method
ePierre: spineau, \o/
spineau: ePierre: it mean two chunks, and guess what cut right in the middle of the sigil char
spineau: ePierre: copyfileobj(data, translating_stream, -1) in transport.py fixes the issue
ePierre: spineau, you mean stop using chunks and instead copying all of it at once?
spineau: ePierre: yes, or stop using the unicode checkboxes
ePierre: spineau, knowing that, it's kind of incredible this kind of issue doesn't happen more often...
ePierre: spineau, for your info, the session attached to the issue is a very legit, full test session
ePierre: spineau, which is 90% of what QA team tests
spineau: ePierre: indeed. I think we're using StreamTransport only for the ttext exporter so I'd assume we won't fill all the system memory if we use -1
ePierre: so I think it's fair to assume that if copying the whole thing at once works well with this session, it should work well with anything
ePierre: spineau, ok. Maybe it would be wise to check if this would have an inpact on ubuntu core or smalled devices that we've tested...