Timeout error in test cases

[Javier Collado]

Reviewing the test case results for the webbrowser application in maguro:
http://10.97.0.1:8080/job/saucy-touch-maguro-smoke-webbrowser-app-autopilot/13/console

I see that all the failures there are related to the autopilot's Eventually matcher.

This means one of:
- there's a race condition (unlikely because 10 seconds seems a reasonable timeout value)
- the application is misbehaving and not really performing the actions that autopilot asked it to do
- there's a communication problem between the application and the autopilot processes and the updates in the widgets aren't correctly communicated.

[Olivier Tilloy (osomon)]

Most (if not all) of those failures are due to a bug in autopilot whereby it’s unable to type colons when run on a device.
As a result, the URLs entered for the tests are incorrect, and the pages don’t load properly. This is already tracked by bug #1181216, I don’t think there is a need for this bug report (I would mark it as duplicate).

[Javier Collado (javier.collado)]

If the reason for those test failures is known. Please mark them as skipped in the code to avoid any confusion in the future since the autopilot error doesn't provide much information by itself. Once that bug is fixed, then the changes to skip them can be reverted. Please let me know what do you think about this.

[Olivier Tilloy (osomon)]

No, I don’t think we should skip those tests. They all pass on a desktop or in the VMs used by jenkins. They fail only when run on a device.

This bug in autopilot is critical (and has been marked as such for more than a month), and should be addressed asap (I’ve been discussing it with concerned parties today, hopefully we’ll get a solution soon). Skipping tests is not a solution, it only hides the problem.

[Javier Collado (javier.collado)]

Can the tests be skipped only when run in a device? If that's the case, I'd still be in favor of doing that. I agree on that skipping test cases won't fix the problem; but, as I said, it will be less confusing when reviewing results.

[Olivier Tilloy (osomon)]

If that was only one test or two, I’d tend to agree, but we’re talking about half the tests failing here (~ 20 tests), the bug has to be fixed urgently, there’s no point in pretending it’s not there by skipping tests. Besides, I don’t know if we can skip tests conditionally.