autopilot exits with 0 when a test error-ed

[Martin Mrazik]

This is how my test run looks like:
------------snip------------
mmrazik@fry:/tmp/pep8izing$ autopilot run tests 2>/dev/null
Loading tests from: /tmp/pep8izing

Failed to import test module: tests.test_voteOnMergeProposal
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 252, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 230, in _get_module_from_name
    __import__(name)
  File "/tmp/pep8izing/tests/test_voteOnMergeProposal.py", line 172
    patch.object(self.LaunchpadTrigger.return_value, '
                                                     ^
SyntaxError: EOL while scanning string literal

.DEBUG: Going to trigger merge: <MagicMock name='mock.source_branch' id='152269196'>
[...]
----------------------------------------------------------------------
Ran 123 tests in 0.442s

OK
------------snip------------

There is no way to notice (e.g. in jenkins) that something went wrong as autopilot reports OK and exits with 0.

[Martin Mrazik (mrazik)]

btw. this is also making the unity smoke test job to pass:

+ /usr/bin/autopilot run unity.tests.test_dash.DashRevealTests.test_dash_reveal
No protocol specified
Loading tests from: /usr/lib/python2.7/dist-packages

Failed to import test module: unity.tests.test_spread
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 252, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 230, in _get_module_from_name
    __import__(name)
  File "/usr/lib/python2.7/dist-packages/unity/tests/__init__.py", line 13, in <module>
    from autopilot.emulators.bamf import BamfWindow
  File "/usr/lib/python2.7/dist-packages/autopilot/emulators/bamf.py", line 18, in <module>
    from gi.repository import Gdk
  File "/usr/lib/python2.7/dist-packages/gi/importer.py", line 76, in load_module
    dynamic_module._load()
  File "/usr/lib/python2.7/dist-packages/gi/module.py", line 244, in _load
    overrides_modules = __import__('gi.overrides', fromlist=[self._namespace])
  File "/usr/lib/python2.7/dist-packages/gi/overrides/Gdk.py", line 402, in <module>
    raise RuntimeError("Gdk couldn't be initialized")
RuntimeError: Gdk couldn't be initialized

[.../several more stacktraces later]
----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Finished: SUCCESS

[Martin Mrazik (mrazik)]

I believe this still isn't fixed:
http://s-jenkins:8080/job/unity-staging-daily-smoketest/distribution=quantal,flavor=amd64/95/console

At the very end:
---
Ran 0 tests in 0.002s
OK
----

According to the log this was installed:
python-autopilot_1.1+bzr82+pkg52~quantal1_all.deb

As far as I can see the bug should be fixed in bzr82.

[Thomi Richards (thomir-deactivatedaccount)]

Hi,

The patch only causes autopilot to exit with non-0 return code. Can you please confirm that that's not working? Testtools will still print 'OK' - to fix that requires a much deeper insertion into testtools code than I'm happy with at the moment.

In my local tests, autopilot exits non-0 when it encounters an ImportError...

[Martin Mrazik (mrazik)]

Yes, I'm fairly sure it exits with 0 because the job result was "passed". This is the excerpt of the code:

------------snip-----------------
#!/bin/bash
set -ex
...
/usr/bin/autopilot run unity.tests.test_dash.DashRevealTests.test_dash_reveal
------------snip-----------------

If autopilot would return non-zero the execution would just stop there and the job would fail (note the set -e).
Let me try to reproduce....

[Martin Mrazik (mrazik)]

I give up. I'm unable to reproduce with bzr83. Closing this again.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autopilot - 1.2daily12.12.10-0ubuntu1

---------------
autopilot (1.2daily12.12.10-0ubuntu1) raring; urgency=low

  [ Didier Roche ]
  * debian/control:
    - update build-dep for latest deps
  * Automatic snapshot from revision 90 (bootstrap):
    - autopilot is checking for gtk2 gconf instead of real (LP: #1060973)
    - autopilot exits with 0 when a test error-ed (LP: #1050208)
    - Fixes a wrong import of make_proxy_object_from_service_name.
      (LP: #1053023)
    - autopilot crashed with AssertionError in _get_name_from_path(): Path must
      be within the project (LP: #1044701)
    - Adds a man page for autopilot (LP: #1037940)

  [ Thomi Richards ]
  * Autopilot ibus module should use Gir ibus module, not 'python-ibus'
    (LP: #1078917)
  * No timestamp in verbose log (LP: #1067161)
  * Less than (<) keypress emulates as a greater than keypress (>) (LP:
    #1081318)
  * Eventually() should have a variable time delay (LP: #1083435)

  [ Leo Arias ]
  * Typos in getting started page (LP: #1078692)
  * Typo in writing good tests page (LP: #1079129)
  * Typo in docs: responde (LP: #1078680)
  * Typo in docs: autopiolot (LP: #1078683)
  * Typo in running autopilot page (LP: #1078710)

  [ Francis Ginther ]
  * UnicodeDecodeError after test-suite has finished (LP: #1078732)

  [ Martin Mrazik ]
  * UnicodeDecodeError after test-suite has finished (LP: #1078732)

  [ Automatic PS uploader ]
  * Automatic snapshot from revision 111
 -- Automatic PS uploader <email address hidden> Mon, 10 Dec 2012 00:01:06 +0000