We need a coherent test suite in place before we can enable package autolanding.

[Robert Bruce Park]

Ken VanDine and I are responsible for getting webapps-applications autolanding in raring. Before we can do that, we need a testsuite in place with at least a minimum of unit/integration tests, eg, so that broken commits don't automatically get released into the distro without any human oversight.

I do see there is a tests/ directory, but it doesn't seem to run / has some issues / needs better coverage.

Somebody more familiar with webapps-applications than I will need to write some tests that a) are capable of detecting changes to the webservice that break our webapps, b) run in a minimal pbuilder / autopkgtest environment, and c) actually runs reliably when you type 'make check'.

Thanks!

[Ken VanDine (ken-vandine)]

Specifically what we need for this package is tests for default-apps, this wouldn't include testing all the individual userscripts.
Just what ends up in ubuntu-webapps-common, that really only includes the amazon userscript, a couple launchers and icons.

[Víctor R. Ruiz (vrruiz)]

I've linked to a branch with modifications to execute the tests inside pbuilder. Some caveats:

1. A credential file with user/password is required to actually run them. Not an issue in PS Jenkins, can be downloaded using a hook.
2. Our CI infrastructure don't have direct connection to the Internet, we use a special VM for site tests. Non-issue in PS Jenkins, can use it as slave.
3. This tests are prone to temporary errors. The experience tell us that sites sometimes report failures, so we decided to get notifications of non-temporary failures (+3 hours). Jenkins may not be so patient. Opinions?
4. All site tests are executed (we may get failures from the unmodified sites).

[Didier Roche-Tolomelli (didrocks)]

For the tests that requires internet connexion, there can be in issue if you assume some connexion to the Internet (even with special proxies).

The ppa/distro builders have no access to any Internet site other than the repositories. That's why we need a jenkins job for prevalidating that (they can have full Internet access), and not having those during build time.