Constant failure of test_swipe_out_application_started_by_url_dispatcher on smoketesting

[Łukasz Zemczak]

Since a longer time, smoketesting CI infrastructure shows a reproducible failure in the test_swipe_out_application_started_by_url_dispatcher autopilot integration test from the unity8 suite. The hardest thing about the failure is that it does not seem to be reproducible on local devices. Many testers tried running the whole suite on their mako devices and it all passes successfully on all runs.

The failure [1]:
Traceback (most recent call last):
  File "/home/phablet/autopilot/unity8/application_lifecycle/tests/test_url_dispatcher.py", line 76, in
test_swipe_out_application_started_by_url_dispatcher
    Eventually(Equals(application_name)))
  File "/usr/lib/python2.7/dist-packages/testtools/testcase.py", line 406, in assertThat
    raise mismatch_error
MismatchError: After 10.0 seconds test failed: 'tmpjLnQ24' != u''

We thought that the recent re-factoring of the test helped, but it sadly doesn't seem to be the case.

Quote from a status update e-mail regarding this issue:
"We might need to have some view on what's seen on the screen during the test-run in the lab... Right now it seems that url-dispatcher works correctly, starts the test-app correctly but autopilot cannot detect it as running. Debugging continues."

[1] http://ci.ubuntu.com/smokeng/trusty/touch/mako/241:20140317.2:20140304/7210/unity8/907112/

[Łukasz Zemczak (sil2100)]

So, it seems that the root cause was completely somewhere else. Thanks Leo for finding this out!

The problem was in how the test has been ran on CI. It seems that there IS a difference between how we run it and how it's ran in smoketesting - smoketesting doesn't install unity8-autopilot but configure it as if it was a click package. Therefore, since unity8-autopilot was not installed, all the required dependencies were not pulled in. And the url-dispatcher test requires url-dispatcher-tools to be installed.

We fixed this in the infra, so that now the dependency is installed as well.