ISO static validation non-critical tests

[Javier Collado]

As suggested in:
https://code.launchpad.net/~xnox/utah/wubi-still-exists/+merge/138851

a failure in the image regarding wubi is important, but the image might be
still testable. Hence, it would be interesting to split the non-critical test
cases in a separate jenkins job, so that we still get an alarm when something
goes wrong, but that doesn't prevent the smoke tests from being executed.

[Dimitri John Ledkov (xnox)]

Thinking further about this, I am not sure how best to implement this.

For some of the static validation, I might want to install extra packages but inherently I will not need any installation / provisioning of VMs nor hardware. I'd like to look more into static validation & extend the current one, but I not sure how to run this in jenkins without wasting resources.

[Max Brustkern (nuclearbob)]

I think we might be well served by splitting the existing tests into categories, at least including "critical" and "non-critical" where critical indicates that we don't need to do anything with the image if it fails, and non-critical indicates we have an issue, but running other smoke testing is still worthwhile. At the same time, we can probably fix lp:1087743 by implementing a flag to avoid checking the image against the mirrors.