multiple StateNotFoundError failures in image 206

Bug #1284316 reported by Bill Filler
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Weather App
Fix Released
Martin Borho
autopilot (Ubuntu)
dialer-app (Ubuntu)
Fix Released

Bug Description

With build 206, multiple dialer-app tests are failing with StateNotFoundErrors

They all work without problem when run on the device, but not in the nightly smoke test

Related branches

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

I tried to revert AP and the toolkit, to previous versions known to create that issue, The StateNotFoundError still happens though. This isn't dialer-app specific, weather-app and a lot of others are impacted as well

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

One example (100% reproducible on runnning the whole weather-app testsuite on mako, having between 3 and 5 failures)

Traceback (most recent call last):
File "/home/phablet/autopilot/ubuntu_weather_app/tests/", line 193, in test_switch_scale_cancel
File "/home/phablet/autopilot/ubuntu_weather_app/tests/", line 50, in _check_units
'QQuickItem', focus=True)
File "/usr/lib/python2.7/dist-packages/autopilot/introspection/", line 322, in select_single
raise StateNotFoundError(type_name, **kwargs)
StateNotFoundError: State not found for class 'QQuickItem' and filters {'focus': True}.

Revision history for this message
Nicholas Skaggs (nskaggs) wrote :

I spoke with Bill briefly about the dialer app failures, and what I wrote on the terminal app bug applies here as well.

The TLDR, is tests need to be improved to fail well; that is it should be obvious what failed when a failure occurs.

A statenotfound error from Autopilot simply means the object in question didn't exist when you went to look for it. They often pop-up because of timing issues. Likely the test failed before this point and you didn't realize it. For instance a view didn't appear (app issue or platform /qmlscene issue), a click didn't register (possible app bug or platform bug, or potentially autopilot bug if click events aren't working), a UI element wasn't found (app bug, test bug, or platform bug), or a UI element wasn't able to be introspected (autopilot bug).

In the scenarios I mentioned above the issue is not not usually autopilot. And indeed I would venture the bugs we see are app, platform, or testcase bugs. Most certainly all of them have some sort of testcase bug as the test is likely not failing well as mentioned above, leaving this ambiguity on what went wrong and what the issue is.

So these issues are 1) Almost certainly not autopilot bugs, 2) Definitely are testcase bugs to the extent the tests aren't failing well, 3) Potentially the application, the test, or platform bugs.

Revision history for this message
Thomi Richards (thomir-deactivatedaccount) wrote :

The QA team spoke about this at length. The take-home points are:

1) This is almost certainly a bug in the test cases, rather than in autopilot itself. The fact that the issue exists independant of the autopilot version seems to confirm this. Unless anyone objects, I'm going to remove the bug task against autopilot, and add one for the actual applications that are failing.

2) The root cause here is that the tests are written poorly. Specifically, they don't fail well. Many tests are writtenn such that they have a bunch of actions, and then assertions at the end of the test case. This is best practise for *unit* tests, but does not work well in an acceptance test where the test code and applicaiton code are completely asynchronous of each other.

3) Leo is slowly converting existing acceptance test suites to the page object pattern ( which allows a greater level of abstraction, and should allow better failure messages. This process will likely help situations like this in the future.

4) balloons is going to blog about the page object conversion process, so hopefully it won't be just Leo doing the conversion, which should speed things up a bit.

Changed in autopilot (Ubuntu):
status: New → Invalid
Revision history for this message
Leo Arias (elopio) wrote :

I ran all the dialer app tests on my mako, rev 207 and got no failures. Also, it seems to be working nicely here:

Marking as incomplete there, waiting for more information. Bill or Didier, are you still seeing the errors on the dialer?

Changed in dialer-app (Ubuntu):
status: New → Incomplete
Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

Hey Leo, as told, the errors are flaky, so please don't take one local run as the source of truth. See as well my comments:
"One example (100% reproducible on runnning the whole weather-app testsuite on mako, having between 3 and 5 failures)". Did you try on weather-app?

You can look as well as the errors on

So, if I take last image:

I can see the state no found on: Only there.

On image 207 (which has NO change from 208):

on image 206 (which has NO change for the sdk, or autopilot or on the application itself):


So, that would mean that all those tests are not well written? Can you have a look at those?

Changed in dialer-app (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Martin Borho (martin-borho) wrote :

Regarding the failing weather app tests: the flaky tests are caused by not considering the newly added convergent layout. But a MP is already in the making and should (hopefully) land today.

Revision history for this message
Julien Funk (jaboing) wrote :

Just FYI, Omer and Chris Gagnon are actively investigating this.

Revision history for this message
Chris Gagnon (chris.gagnon) wrote :

I was not able to reproduce locally, I notice that the system settle tests are also failing.

Omer found running the dialer_app.tests.test_calls.TestCalls.test_outgoing_answer_remote_hangup test repeatedly in a loop would reproduce the failure.

It currently looks like the phone being under load is exposing flaky tests. I am working to better reproduce this now. The dialer-app definitely has a poorly written test.

Revision history for this message
Nicholas Skaggs (nskaggs) wrote :

I have seen tests fail when under load -- clicks have a tendency to be missed somewhere (app, platform?) when under load.

Changed in ubuntu-weather-app:
assignee: nobody → Martin Borho (martin-borho)
status: New → In Progress
Revision history for this message
Alan Pope 🍺🐧🐱 🦄 (popey) wrote :

Perhaps the bug I just filed against sudoku is a dupe of this? bug 1285279 - easily reproducible with r161 of sudoku.

Revision history for this message
Chris Gagnon (chris.gagnon) wrote :

I have this MP that fixes a number of flaky tests in the dialer app

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dialer-app - 0.1+14.04.20140226-0ubuntu1

dialer-app (0.1+14.04.20140226-0ubuntu1) trusty; urgency=low

  [ chris.gagnon ]
  * Use wait_select_single when getting HistoryDelegate0 instead of
    select_single, to fix a couple flaky tests (LP: #1284316)
 -- Ubuntu daily release <email address hidden> Wed, 26 Feb 2014 21:27:58 +0000

Changed in dialer-app (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

@Chris: the system settle test failures are false positive that we discussed at length on the phone mailing list. To be clear, the phone isn't under heavy load.

Revision history for this message
Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot) wrote :

Fix committed into lp:ubuntu-weather-app at revision 210, scheduled for release in ubuntu-weather-app, milestone alpha-1

Changed in ubuntu-weather-app:
status: In Progress → Fix Committed
Changed in ubuntu-weather-app:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.