Coverage report generation crashes with UnicodeDecodeError

[Renier Morales]

System: Ubuntu 11.10, x86. Python 2.7.2+. Nova commit: f8aa96679552935e22bc47851a30e712beadd629
Running "./run_tests.sh -c" crashes on the coverage report generation step with:

Running PEP8 and HACKING compliance check...
ERROR: import 'import memcache' failed: No module named memcache
ERROR: import 'import memcache' failed: No module named memcache
ERROR: import 'import rabbit' failed: No module named rabbit
ERROR: import 'import win32console as win' failed: No module named win32console
ERROR: import 'import win32console' failed: No module named win32console
ERROR: import 'import pywintypes' failed: No module named pywintypes
ERROR: import 'import qpid' failed: No module named qpid
ERROR: import 'from nova.rpc import impl_qpid' failed: No module named qpid.messaging
ERROR: import 'import qpid.messaging' failed: No module named qpid.messaging
ERROR: import 'import qpid.messaging.exceptions' failed: No module named qpid.messaging.exceptions
ERROR: import 'import memcache' failed: No module named memcache
ERROR: import 'import memcache' failed: No module named memcache
ERROR: import 'import ldap' failed: No module named ldap
ERROR: import 'import bpython' failed: No module named bpython
ERROR: import 'import IPython' failed: No module named IPython
ERROR: import 'import XenAPI' failed: No module named XenAPI
ERROR: import 'import win32com.client' failed: No module named win32com.client
ERROR: import 'import XenAPIPlugin' failed: No module named XenAPIPlugin
ERROR: import 'import pluginlib_nova as pluginlib' failed: No module named XenAPI
ERROR: import 'import XenAPIPlugin' failed: No module named XenAPIPlugin
ERROR: import 'import xenstore' failed: No module named XenAPIPlugin
ERROR: import 'import XenAPI' failed: No module named XenAPI
ERROR: import 'import XenAPIPlugin' failed: No module named XenAPIPlugin
ERROR: import 'import XenAPI' failed: No module named XenAPI
ERROR: import 'import XenAPIPlugin' failed: No module named XenAPIPlugin
ERROR: import 'import pluginlib_nova as pluginlib' failed: No module named XenAPI
tools/hacking.py:122: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  __import__(mod, globals(), locals(), [], -1)

ERROR: import 'import XenAPIPlugin' failed: No module named XenAPIPlugin
Generating coverage report in covhtml/
Traceback (most recent call last):
  File "/opt/stack/nova/.venv/bin/coverage", line 9, in <module>
    load_entry_point('coverage==3.5.1', 'console_scripts', 'coverage')()
  File "/opt/stack/nova/.venv/local/lib/python2.7/site-packages/coverage/cmdline.py", line 657, in main
    status = CoverageScript().command_line(argv)
  File "/opt/stack/nova/.venv/local/lib/python2.7/site-packages/coverage/cmdline.py", line 549, in command_line
    directory=options.directory, **report_args)
  File "/opt/stack/nova/.venv/local/lib/python2.7/site-packages/coverage/control.py", line 599, in html_report
    reporter.report(morfs, config=self.config)
  File "/opt/stack/nova/.venv/local/lib/python2.7/site-packages/coverage/html.py", line 83, in report
    self.report_files(self.html_file, morfs, config, config.html_dir)
  File "/opt/stack/nova/.venv/local/lib/python2.7/site-packages/coverage/report.py", line 86, in report_files
    report_fn(cu, self.coverage._analyze(cu))
  File "/opt/stack/nova/.venv/local/lib/python2.7/site-packages/coverage/html.py", line 198, in html_file
    self.write_html(html_path, html)
  File "/opt/stack/nova/.venv/local/lib/python2.7/site-packages/coverage/html.py", line 103, in write_html
    write_encoded(fname, html, 'ascii', 'xmlcharrefreplace')
  File "/opt/stack/nova/.venv/local/lib/python2.7/site-packages/coverage/backward.py", line 137, in write_encoded
    f.write(text.decode('utf8'))
  File "/opt/stack/nova/.venv/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 28308: invalid continuation byte

[Renier Morales (renier-h)]

Even though we are passing --cover-package to nose (in run_tests.sh), it collects data on everything (seems this option is only to filter report output, not data collection). Problem is that when we use the coverage script directly to generate the report, we don't pass any report filtering option to it, so it goes and tries to generate report on all the data collected.

It happens that feedparser, used by nova/tests/api/openstack/compute/test_versions.py uses drv_libxml2 (python xml parser-- if installed on the system) and this file has a bad utf-8 character in it. So when coverage is generating the html report for this file, it crashes trying to decode parts of the file into utf8.

This problem is actually fixed upstream in the coverage tool, but this has not been put in an official release yet (see https://bitbucket.org/ned/coveragepy/issue/157/chokes-on-source-files-with-non-utf-8).

Solution seems to be to add filtering to the direct coverage tool call in run_tests.sh. After I did this in my working dir, things ran fine generating the right nova coverage reports. Putting a patch together on this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/6750

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6750
Committed: http://github.com/openstack/nova/commit/bcb53dfc6a149f94b4509ce90f89120b526f5582
Submitter: Jenkins
Branch: master

commit bcb53dfc6a149f94b4509ce90f89120b526f5582
Author: Renier Morales <email address hidden>
Date: Mon Apr 23 22:36:19 2012 -0400

    Scope coverage report generation to nova module.

    Coverage report generation was not using --include filters.
    This resulted in generating a report that included all external
    dependencies. Some of these may include problematic characters for
    utf8 decoding. In the odd case this was found, coverage would fail
    with a UnicodeDecodeError.

    Fixes bug #987077.

    Change-Id: I276698e65ac1037972209a406058168433cd826b