html5browser js test suites are too sensitive to test length
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
HTML5-browser |
Fix Released
|
Undecided
|
Gary Poster | ||
Launchpad itself |
Fix Released
|
High
|
Gary Poster |
Bug Description
We can only specify a per-suite timeout limit for html5browser tests. This means that test suites must be artificially divided, and it means that differences in machine speed aggregate per test making timeouts harder to predict. A real-world example is a set of tests that pass within the timeout on a developer box and on ec2, but take too long on buildbot.
As agreed with Curtis, we will make html5browser support incremental timeouts, and change our javascript testrunner to communicate when each test passes or fails. Then we can say, for instance, that a timeout for a yuixhr suite is 5 minutes (a limit we never expect to hit), but timeout for a single yuixhr test is somewhere between 6 and 10 seconds. We can do smaller or identical timeouts for pure-JS tests.
A related problem is that html5browser takes a long time to spin up on some boxes (64 bit, in Curtis' experience), adding 15 to 20 seconds per suite. Further work might be to solve this problem, and/or to make html5browser a service that LP starts with a layer and communicates with for each test. We won't try to solve that for this bug.
Related branches
- Curtis Hovey: Approve (code)
-
Diff: 332 lines (+176/-20)3 files modifiedhtml5browser/__init__.py (+81/-18)
html5browser/tests/test_browser.py (+94/-1)
setup.py (+1/-1)
- Ian Booth (community): Approve (code)
-
Diff: 134 lines (+55/-4)5 files modifiedlib/lp/app/javascript/server_fixture.js (+17/-0)
lib/lp/testing/__init__.py (+29/-4)
lib/lp/testing/yuixhr.py (+7/-0)
setup.py (+1/-0)
versions.cfg (+1/-0)
tags: |
added: qa-untestable removed: qa-needstesting |
Changed in launchpad: | |
status: | Fix Committed → Fix Released |
Changed in html5-browser: | |
status: | In Progress → Fix Released |
This blocks landing the work for high bug 872089 which itself blocks landing the work for critical bug 724609.