UnicodeDecodeError after test-suite has finished

[Martin Mrazik]

This is what the latest unity runs are producing:
--------------snip----------------
[...]
Checking system state for badly behaving test...
Test was well behaved.
Moving mouse to position 640,512 without animation.
Traceback (most recent call last):
  File "/usr/bin/autopilot", line 314, in <module>
    main()
  File "/usr/bin/autopilot", line 294, in main
    run_tests(args)
  File "/usr/bin/autopilot", line 161, in run_tests
    success = runner.run(test_suite).wasSuccessful()
  File "/usr/bin/autopilot", line 226, in run
    result.stopTestRun()
  File "/usr/lib/pymodules/python2.7/junitxml/__init__.py", line 169, in stopTestRun
    self._stream.write(''.join(self._results))
  File "/usr/lib/python2.7/codecs.py", line 691, in write
    return self.writer.write(data)
  File "/usr/lib/python2.7/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 454103: ordinal not in range(128)
Finishing test
Collecting any crash reports
Retracing linux-image-3.5.0-18-generic.0.crash...
Retracing _usr_bin_autopilot.1000.crash...
Done grabbing crashes
--------------snip----------------

As a result no results are recorded in jenkins:
--------------snip----------------
Archiving artifacts
Recording test results
ERROR: Failed to archive test reports
hudson.util.IOException2: Failed to read /var/lib/jenkins/jobs/ps-unity-autopilot-trunk/workspace/label/master/machine_name/dx-autopilot-intel/results/testresults/ap-test-result.xml
 at hudson.tasks.junit.TestResult.parse(TestResult.java:225)
 at hudson.tasks.junit.TestResult.parse(TestResult.java:163)
 at hudson.tasks.junit.TestResult.parse(TestResult.java:140)
 at hudson.tasks.junit.TestResult.<init>(TestResult.java:116)
 at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:114)
 at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:87)
 at hudson.FilePath.act(FilePath.java:758)
 at hudson.FilePath.act(FilePath.java:740)
 at hudson.tasks.junit.JUnitParser.parse(JUnitParser.java:83)
 at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:123)
 at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:135)
 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
 at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:692)
 at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:667)
 at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:645)
 at hudson.model.Build$RunnerImpl.post2(Build.java:161)
 at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:614)
 at hudson.model.Run.run(Run.java:1400)
 at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
 at hudson.model.ResourceController.execute(ResourceController.java:88)
 at hudson.model.Executor.run(Executor.java:175)
Caused by: org.dom4j.DocumentException: Error on line 2 of document file:///var/lib/jenkins/jobs/ps-unity-autopilot-trunk/workspace/label/master/machine_name/dx-autopilot-intel/results/testresults/ap-test-result.xml : XML document structures must start and end within the same entity. Nested exception: XML document structures must start and end within the same entity.
 at org.dom4j.io.SAXReader.read(SAXReader.java:482)
 at org.dom4j.io.SAXReader.read(SAXReader.java:264)
 at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:112)
 at hudson.tasks.junit.TestResult.parse(TestResult.java:208)
 ... 20 more
Caused by: org.xml.sax.SAXParseException; systemId: file:///var/lib/jenkins/jobs/ps-unity-autopilot-trunk/workspace/label/master/machine_name/dx-autopilot-intel/results/testresults/ap-test-result.xml; lineNumber: 2; columnNumber: 1; XML document structures must start and end within the same entity.
 at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
 at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
 at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:391)
 at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1404)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(XMLDocumentFragmentScannerImpl.java:882)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(XMLDocumentScannerImpl.java:582)
 at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(XMLEntityManager.java:1370)
 at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1740)
 at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1393)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2769)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:625)
 at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:116)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488)
 at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:819)
 at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:748)
 at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
 at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1208)
 at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:525)
 at org.dom4j.io.SAXReader.read(SAXReader.java:465)
 ... 23 more
Build step 'Publish JUnit test result report' changed build result to FAILURE
Notifying upstream projects of job completion
--------------snip----------------

[Francis Ginther (fginther)]

Some test is adding non-ascii characters to the test output. There isn't much to go on as the error occurs when the entire xml result file is being generated.

A hack to workaround this is to patch /usr/lib/python2.7/codecs.py StreamWriter.write to do this:
    def write(self, object):

        """ Writes the object's contents encoded to self.stream.
        """
        data, consumed = self.encode(unicode(object, errors=self.errors), self.errors)
        self.stream.write(data)

stream can then be opened with:
stream = open("/tmp/foo.xml", 'w', encoding='utf-8', errors='replace')

This will at least allow the output file to be generated with "??" characters replacing the non-ascii characters.

[Francis Ginther (fginther)]

Also considering this workaround, replacing:
    self.addDetail('test-log', text_content(self._log_buffer.getvalue()))
with:
    self.addDetail('test-log', unicode(text_content(self._log_buffer.getvalue()), errors='replace'))

[Francis Ginther (fginther)]

Oops, it should look like this:
    self.addDetail('test-log', text_content(unicode(self._log_buffer.getvalue(), errors='replace')))

[Francis Ginther (fginther)]

Linked a branch, lp:~fginther/autopilot/unicode-replace, with the unicode(blah, errors='replace') hack. I don't know if this will actually resolve the error as there are some other paths that can add content to the output.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autopilot - 1.2daily12.12.10-0ubuntu1

---------------
autopilot (1.2daily12.12.10-0ubuntu1) raring; urgency=low

  [ Didier Roche ]
  * debian/control:
    - update build-dep for latest deps
  * Automatic snapshot from revision 90 (bootstrap):
    - autopilot is checking for gtk2 gconf instead of real (LP: #1060973)
    - autopilot exits with 0 when a test error-ed (LP: #1050208)
    - Fixes a wrong import of make_proxy_object_from_service_name.
      (LP: #1053023)
    - autopilot crashed with AssertionError in _get_name_from_path(): Path must
      be within the project (LP: #1044701)
    - Adds a man page for autopilot (LP: #1037940)

  [ Thomi Richards ]
  * Autopilot ibus module should use Gir ibus module, not 'python-ibus'
    (LP: #1078917)
  * No timestamp in verbose log (LP: #1067161)
  * Less than (<) keypress emulates as a greater than keypress (>) (LP:
    #1081318)
  * Eventually() should have a variable time delay (LP: #1083435)

  [ Leo Arias ]
  * Typos in getting started page (LP: #1078692)
  * Typo in writing good tests page (LP: #1079129)
  * Typo in docs: responde (LP: #1078680)
  * Typo in docs: autopiolot (LP: #1078683)
  * Typo in running autopilot page (LP: #1078710)

  [ Francis Ginther ]
  * UnicodeDecodeError after test-suite has finished (LP: #1078732)

  [ Martin Mrazik ]
  * UnicodeDecodeError after test-suite has finished (LP: #1078732)

  [ Automatic PS uploader ]
  * Automatic snapshot from revision 111
 -- Automatic PS uploader <email address hidden> Mon, 10 Dec 2012 00:01:06 +0000