Comment 3 for bug 1652329

On Jan 04, 2017, at 09:11 PM, Joshua Powers wrote:

>@smoser, before the holiday you mentioned how we have two targets: unit
>tests and style. For unit tests you mention wanting to move to flake8
>and have runs for each Ubuntu release. For the unit tests, you still
>want these to gate the build. So why not break them out as follows:

I don't think you *have* to have two separate tox.ini files for this. I
personally find it clearer to have a single tox.ini file. You don't have to
name all your environments in the default [tox]envlist; that's just the set
that gets run automatically when you don't provide an explicit set of
environments to run (e.g. through -e or $TOXENV.

The way I generally do this is to set up [testenv:FOO] sections for all the
tests I want, including various specifications for the generated environments,
but then I only name in envlist the ones I want to run by default during
development. Then in my CI file (e.g. .travis-ci.yml or whatever) I
explicitly invoke the environments I want to run for CI gating.

I don't have a great example of how this all hangs together with GitHub (since
e.g. ubuntu-image uses Ubuntu's autopkgtest infrastructure instead of Travis),
but for a GitLab example, see:

https://gitlab.com/mailman/mailman/blob/master/.gitlab-ci.yml

I'll just mention one other thing; this would all be much easier with these
two tox wishlists:

https://github.com/tox-dev/tox/issues/238

This would allow you to specify env groups so e.g. you could do something
like:

$ tox -g qa

and that would run the flake8 and any other style tests, while something like

$ tox -g ci

would run all the envs that you'd gate CI on.

https://github.com/tox-dev/tox/issues/418

Makes life easier for generated environments.

I certainly encourage more +1s on those issues if you agree with me! :)