Flaky test on ubuntu_terminal_app.tests.test_terminal.TestMainWindow

[Didier Roche-Tolomelli]

With image #206 (and 207 first run), we have ubuntu_terminal_app.tests.test_terminal.TestMainWindow sometimes failing (I get it locally, on mako 1 on 10 runs).

See some logs at http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224/6796/ubuntu_terminal_app/813070/.

[Alan Pope 🍺🐧🐱 🦄 (popey)]

Wonder if this is related to the fact that the font size is zero on first run - bug 1284602

[Alan Pope 🍺🐧🐱 🦄 (popey)]

I ran this 12 times on my mako and couldn't get it to fail.

[Łukasz Zemczak (sil2100)]

It shouldn't be related, as the moment it is failing is on showing the circle menu. This is independent of any settings, just a longpress. It also doesn't seem to be a time race, as a longpress requires holding for 800ms, while the AP test waits 2 seconds.

I ran this test in a loop for over an hour and could not reproduce. A few runs of the whole suite also didn't show the failure. Strange...

[Nicholas Skaggs (nskaggs)]

This test has a couple issues with it;

it uses sleep liberally -- these need to be removed
the test does not fail well; my guess is the tap event is not being captured (potential platform issue even!), but we have no way to know for sure as it is unclear where exactly the test failed.

If it's an app loading issue, we can add a check to wait for the mainview to appear (and ensure we don't consider it a platform issue)
If it's a tap event issue, we can add an assert to check for the event (and consider it as a potential platform issue)

If both are happening and the menu fails to load, it's certainly an app bug.

If the test itself isn't sending a tap event, it could be a test or autopilot bug.

As you can see, the test needs to be written to fail well so this is not ambiguous.

[Łukasz Zemczak (sil2100)]

So, I digged into to logs which we actually have and got a vague idea on where the failure is happening. If you look closely into the application logs, it mentions "Circle button clicked" as the last line. This means that the circle menu *did* appear. It also gives some additional info: the circle appeared earlier than expected, and the final press-and-hold (which generally we use to show the menu) clicks on the circle center and closes it instead.

What could this mean?
* In the tap action, the app might register the press but miss the release action
 -> Unlikely, as the button press callback is triggered on button press, and I'm not sure if this could happen if we send out another 'press' event while one 'press' is still taking place?
* In the tap action, autopilot does the tap in the way that there is a press event, sleep(0.1) and then a release event - maybe the sleep *sometimes* for some reasons takes longer (like 0.8, which is the longpress event)?
 -> A bit fishy since the logs show proper timings, i.e. we don't see a noticeable 0.8s longer wait anywhere
* Qt or any other responsible bits recognize the tap as a longpress invalidly after the platform change

Any thoughts?

[Nicholas Skaggs (nskaggs)]

I removed the pre-tap we were doing in the test, and then lengthened the press and release time, which should be more than enough. We'll go from there.

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

Fix committed into lp:ubuntu-terminal-app at revision 40, scheduled for release in ubuntu-terminal-app, milestone alpha-1