Comment 38 for bug 1421009

The earlier simplified test case does not fail anymore or at least not easily. Whether on stock image or using the PPA, the problem with running the same test over and over is that Unity8 crashes or restarts sooner or later when running the command, for example that crasher bug noticed.

I ran a new baseline set of results on Saturday with image #153 to see how the situation looks like on mako after the Mir, UITK, glibc landings. It's still the same - UITK, webbrowser, calculator, ubuntu-system-settings, ubuntu-weather-app are good examples of suites that 100% pass on stock image. They gain respectively 4/9, 3/6, 1/1, 4/7, 3/4 tests failing based on two autopilot runs with the PPA 018 in use. The failures are random, ie the test failing changes depending on the run.

In other words, to reproduce the autopilot problem when using the PPA you can run any of the following on mako:
phablet-test-run ubuntuuitoolkit
phablet-test-run webbrowser_app
phablet-test-run ubuntu_calculator_app
phablet-test-run ubuntu_system_settings
phablet-test-run ubuntu_weather_app

Prepare the device by first running on device sudo apt install ubuntu-ui-toolkit-autopilot webbrowser-app-autopilot ubuntu-system-settings-autopilot python3-lxml