Request to do post build testing on webapps SRUs

[Francis Ginther]

From Robert Park:
We don't have the jenkins test setup for SRUs yet. we need to talk to victor and/or francis about the best way to get that set up.

I think what we need is to have a special -ci job for lp:unity-webapps-*/13.10 that specifically tests on saucy, builds packages, installs the packages, and then runs the unit tests. Then we'd need that log output hosted publicly and linked to from the SRU bug.

Victor/Francis: How do we go about setting this up? Any ideas?

[Robert Bruce Park (robru)]

For example, the SRU workflow will need to look like this:

1. Gmail.com experiences redesign, DOM changes/breaks

2. Fix is committed to lp:unity-webapps-gmail

3. Fix is tested in the development release by myself/alex abreu/victor/etc and also tested by the hourly jenkins tester that already exists.

4. Merge is proposed backporting the fix from lp:unity-webapps-gmail to lp:unity-webapps-gmail/13.10

5. -ci job comes in, builds the package, installs it in a saucy VM, runs the tests from Gmail.test.js, confirms that it is working.

6. -ci job posts the log at a public URL somewhere

7. -ci job approves the MP with a link to that URL

8. I copy that URL to the public SRU bug

9. SRU team sees this and approves the SRU without any further manual testing

So as far as I understand, everything in this workflow exists except steps 5 and 6. I think the current -ci jobs are just running jslint tests against the changes without running the unittests properly, which is something we'll need to fix. Justin is working on the ability to run the unittests with a locally-cached copy of the website, which will greatly reduce the flakiness of the tests, hopefully to the point that they're reliable enough to run in a -ci environment.