run_tests should strictly be a wrapper for tox

[Morgan Fainberg]

run_tests (if kept) should strictly be a wrapper for tox

http://git.openstack.org/cgit/openstack/keystone/tree/run_tests.sh

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/103282

[Johannes Erdfelt (johannes.erdfelt)]

I disagree. run_tests.sh has features that tox doesn't have, like running using an existing virtual environment. Making it use tox will result in features being removed that people actively use.

See bug 1287814 for a similar bug and patch in nova that was abandoned because it ended up removing features that people depended on.

[Dolph Mathews (dolph)]

run_tests.sh is designed to make running tests as easy as possible for newcomers (hence the name). If you need the more powerful tools (like running without a new venv), then I'd suggest using the underlying test runner directly (testr, nosetests, etc).

There seems to be a misunderstanding of tox in bug 1287814.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Morgan Fainberg (<email address hidden>) on branch: master
Review: https://review.openstack.org/103282

[Johannes Erdfelt (johannes.erdfelt)]

I don't know where you got the impression that run_tests.sh is for newcomers.

The Testing wiki documents tox as the preferred method:

https://wiki.openstack.org/wiki/Testing

This makes sense since it *is* easier to use, on part because it is so opinionated. This is part of the reason why run_tests.sh still exists.

Perhaps you can take a moment and explain to me and bug 1287814 what we have misunderstood about tox.

[Morgan Fainberg (mdrnstm)]

@Johnannes,

run_tests.sh originally was the opinionated method of running tests, and I argue that it still is as opinionated as tox, just in different ways. Functionally, it supports a single core difference, the ability to run without a VENV (there are also a few minor differences but nothing to write home about).

Currently run_tests has been suffering from bitrot. This is because tox is the only method we use for gate/check.

While the transition to tox was occuring run_tests was for a large part documented in each project as the best way to run testing for development. I was meant at that time as a more friendly way than having to type-out: "tox -epy27 -- --subunit | subunit-2to1 | tools/colorizer.py" to get the old output. With the enhancements to tox and testr (and that there are odd edge cases such as needing to install keystone to use run_tests.sh) it doesn't make sense to provide a tool that will produce wildly different (in some cases) results from what gate and check will provide. -- This is what lends credence to the assertion that run_tests is meant for newcomers.

There is an argument as to which is easier to use, I'd say tox is *not* easier to use, it hides a lot of details away from the newcomers.

In either case, I've marked my original review as -2, before we sink too much more time into this (we have a lot to accomplish this cycle) lets figure out what we need to do to ensure we're not providing broken tools (or tools that produce wildly different results to newcomers) in-tree.

[Morgan Fainberg (mdrnstm)]

Or just plan to revisit run_tests somewhere down the line, perhaps with greatly expanded documentation on how to run tests sans VENV for those who need that.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Morgan Fainberg (<email address hidden>) on branch: master
Review: https://review.openstack.org/103282

[Morgan Fainberg (mdrnstm)]

Unassigning due to lack of activity.