Test runs are not reproducible

[Jean-Baptiste Lallement]

current build number: 137
device name: mako
channel: ubuntu-touch/devel-proposed
alias: ubuntu-touch/vivid-proposed
last update: 2015-03-17 09:53:47
version version: 137
version ubuntu: 20150317
version device: 20150210
version custom: 20150317

ubuntu-sanity-suite r100

Runs are not reproducible and successive runs on the same device and reflashing the device between each run doesn't always report the same number of failures. On r100 and mako #137 the number of failures varies between 5 and 7.

Output of a run attached.

[Richard Huddie (rhuddie)]

Analysis of these failed test runs shows that many of the tests are failing due to completely blank scopes. So the tests are failing at the point of trying to launch an application from an icon, or open the app-store etc.

This could be related to the fact we restart unity8 for every test. This means that there is a delay before the scopes are usable again, and this delay could exceed the timeout for the wait_select method in the test. It would be useful to see if we can run the tests without re-starting unity8, as this is not something the user would be doing.

Other issues identified:
* The wifi password dialog in the settings wizard can be very slow to load, and this can cause a failure
* The username for the U1 app store is very long, I've seen failures where it fails to enter the remaining fields
* The 'All' category drop-down selector is not always visible in the apps scope (need to swipe down to display it)
* HTML5 test, cannot connect to chrome

[Jean-Baptiste Lallement (jibel)]

r105.

I executed several runs of the testsuite on latest devel-proposed/mako (165). The results are unstable and different each time.

On 7 runs here are the number of times the tests failed:
      5 test_014_webbrowser
      1 test_add_contact_with_name_and_number
      7 test_complete_settings_wizard
      2 test_discover_bluetooth_devices
      1 test_html5_application
      2 test_install_free_app
      4 test_sim_pin_lock_and_unlock
      7 test_take_picture_with_rear_camera
      2 test_uninstall_application

Full log http://people.canonical.com/~j-lallement/junk/ubuntu-sanity-tests.results.tgz