Autopilot failures in webbrowser-app during smoketesting runs since image #72

[Łukasz Zemczak]

Since image #72 (which had a new Mir landing) we noticed some failures in the webbrowser-app test suites happening from time to time. They are not present in every image, but there are 1-3 failures happening every 1-2 images. Each time a different test seems to fail. The list of currently noticed test failures is as follows:

- test_chrome_hides_when_loaded
- test_clear_address_bar_dismisses_suggestions
- test_second_click_deselect_text
- test_error_only_for_current_tab
- test_close_tab

The smoketesting test results with the failures:
http://ci.ubuntu.com/smokeng/utopic/touch/mako/72:20140606.2:20140530/8449/webbrowser_app/ - 1 failure
http://ci.ubuntu.com/smokeng/utopic/touch/mako/75:20140609:20140530/8487/webbrowser_app/ - 1 failure
http://ci.ubuntu.com/smokeng/utopic/touch/mako/76:20140610:20140530/8503/webbrowser_app/ - 3 failures

Problems do not seem reproducible locally on the device.

[Olivier Tilloy (osomon)]

I can confirm that I can’t reproduce the failures locally.

As noted by Łukasz, a new version of Mir landed in image #72, and it seems to be the only relevant change in that image.

Prior to that, the latest change to webbrowser-app was in image #65 (corresponding to revision 556 of lp:webbrowser-app), and it really affected only the webapps container, not the webbrowser app itself.
Interestingly, the latest changes to autopilot were in image #65 as well.

The tests started failing at image #72, so that’s 7 images in a row with both webbrowser-app and autopilot changes that didn’t fail, which seems to indicate that those changes don’t have anything to do with the problem.

The failures we are seeing are varied, but one of the common patterns seems to be the insertion of random characters (mostly whitespaces, but not only) when typing a URL in the address bar. When that happens, the autopilot logs don’t show any extraneous input event being emitted, so there’s not enough information there to explain those extra characters.

All the above seems to point at the Mir changes in image #72 as the turning point.

Note that there hasn’t been failures in the smoke test runs for the last two images, #84 and #85.

[Olivier Tilloy (osomon)]

Closing as this test has been reliably passing for a long time now.