Test runs are not reproducible

Bug #1433029 reported by Jean-Baptiste Lallement on 2015-03-17
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-system-tests
Undecided
Unassigned

Bug Description

current build number: 137
device name: mako
channel: ubuntu-touch/devel-proposed
alias: ubuntu-touch/vivid-proposed
last update: 2015-03-17 09:53:47
version version: 137
version ubuntu: 20150317
version device: 20150210
version custom: 20150317

ubuntu-sanity-suite r100

Runs are not reproducible and successive runs on the same device and reflashing the device between each run doesn't always report the same number of failures. On r100 and mako #137 the number of failures varies between 5 and 7.

Output of a run attached.

Jean-Baptiste Lallement (jibel) wrote :
summary: - Test runs are unstable
+ Test runs are not reproducible
Richard Huddie (rhuddie) wrote :

Analysis of these failed test runs shows that many of the tests are failing due to completely blank scopes. So the tests are failing at the point of trying to launch an application from an icon, or open the app-store etc.

This could be related to the fact we restart unity8 for every test. This means that there is a delay before the scopes are usable again, and this delay could exceed the timeout for the wait_select method in the test. It would be useful to see if we can run the tests without re-starting unity8, as this is not something the user would be doing.

Other issues identified:
* The wifi password dialog in the settings wizard can be very slow to load, and this can cause a failure
* The username for the U1 app store is very long, I've seen failures where it fails to enter the remaining fields
* The 'All' category drop-down selector is not always visible in the apps scope (need to swipe down to display it)
* HTML5 test, cannot connect to chrome

Jean-Baptiste Lallement (jibel) wrote :

r105.

I executed several runs of the testsuite on latest devel-proposed/mako (165). The results are unstable and different each time.

On 7 runs here are the number of times the tests failed:
      5 test_014_webbrowser
      1 test_add_contact_with_name_and_number
      7 test_complete_settings_wizard
      2 test_discover_bluetooth_devices
      1 test_html5_application
      2 test_install_free_app
      4 test_sim_pin_lock_and_unlock
      7 test_take_picture_with_rear_camera
      2 test_uninstall_application

Full log http://people.canonical.com/~j-lallement/junk/ubuntu-sanity-tests.results.tgz

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers