lp.scripts.tests.test_sphinxdocs.TestSphinxDocumentation.test_docs_build_without_error fails intermittently/rarely

[Gary Poster]

This happened in parallel testing but is another instance of generic fragility described in passing in bug 721166.

======================================================================
FAILURE: lp.scripts.tests.test_sphinxdocs.TestSphinxDocumentation.test_docs_build_without_error
worker ID: worker-18
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: Traceback (most recent call last):
  File "/var/lib/buildbot/slaves/slave/lucid-devel/build/lib/lp/scripts/tests/test_sphinxdocs.py", line 35, in test_docs_build_without_error
    self.assertEqual('Making output directory...\n', stderr)
MismatchError: !=:
reference = '''\
Making output directory...
'''
actual = '''\
Making output directory...
No handlers could be found for logger "root"
'''

The test can't handle anything else going to stdout or stderr. It should be made less fragile. The pertinent code is this:

        returncode, stdout, stderr = run_capturing_output(
            sphinx.main,
            ['sphinx-build', '-d', '%s/doctrees' % output_dir,
             '-aNq', doc_dir, '%s/html' % output_dir])
        self.assertEqual(0, returncode)
        self.assertEqual('Making output directory...\n', stderr)
        self.assertEqual('', stdout)

Benji pointed out that we could use the sphing-build command's -W option ("turn warnings into errors") and then simply verify that the returncode was 0, and not do a string check against stderr or stdout. They might be nice for addDetail when something goes wrong, but otherwise let's just do a simple returncode check.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r15408 (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15408) by a commit, but not testable.