selftest --randomize option

[Martin Pool]

Sometimes (as with bug 100010) we have a bug that occurs because of unintentional interference between tests. To track these down it would be nice to have an option to run the tests in a pseudorandom order. (Perhaps it should expose the random seed to make it reproducible.)

[Vincent Ladeuil (vila)]

If this is implemented by establishing the list of all tests, it may be possible to do more variations with that test list:

- prioritize some tests (say I want to run some specific tests before the rest of the test suite),

- create my own test lists,

- etc...

[Ian Clatworthy (ian-clatworthy)]

vila wrote:
> If this is implemented by establishing the list of all tests, it may be
> possible to do more variations with that test list:
>
> - prioritize some tests (say I want to run some specific tests before
> the rest of the test suite),
>
> - create my own test lists,
>
> - etc...
>
>
I'm actually doing something like that to test the --not option poolie
has asked for.

Ian C.

[Alexander Belchenko (bialix)]

vila пишет:
> If this is implemented by establishing the list of all tests, it may be
> possible to do more variations with that test list:
>
> - prioritize some tests (say I want to run some specific tests before
> the rest of the test suite),

IIUC, this one is already implemented by Martin as option '--first' of selftest command.

[µ]

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

vila wrote:
> If this is implemented by establishing the list of all tests, it may be
> possible to do more variations with that test list:
>
> - prioritize some tests (say I want to run some specific tests before
> the rest of the test suite),

The --first option does that.

I'd like to see the shortest, most error-prone tests to be run first
automatically. I've also got a plugin that determines which tests are
likely to have been broken by modifying a given file.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGE6fm0F+nu1YWqI0RAps+AJ4joHNba2gBmnjk7g9kUWZAVf/8hwCfSoN/
Oh+ZClPUX44p+2bFUFZjGzc=
=pp2l
-----END PGP SIGNATURE-----

[Vincent Ladeuil (vila)]

>>>>> "bialix" == Alexander Belchenko <email address hidden> writes:

    bialix> vila пишет:
    >> If this is implemented by establishing the list of all tests, it may be
    >> possible to do more variations with that test list:
    >>
    >> - prioritize some tests (say I want to run some specific tests before
    >> the rest of the test suite),

    bialix> IIUC, this one is already implemented by Martin as option '--first' of
    bialix> selftest command.

Yes.

But I had another idea in mind:

- record which tests are failing during one run,

- begin with these failing tests for the next run.

Being able to easily manipulate test lists (even if not disjoined
and leave the test suite remove the duplicates) can help.

On the other hand, working with Robert at NlSprint learned me to
always run selftest with a regexp covering what I need now (and
refine it when debugging), then finally run the whole for
validation.

So may be it's over-engineering.

In the same vein, there may be some value in decorating tests
with attributes related to the functionality under test and allow
selftest to select tests based on these attributes.

   Vincent

P.S.: Aaron also had an interesting experiment to automatically
correlate tests with code lines.

[Ian Clatworthy (ian-clatworthy)]

Fix commited as part of the patch for #102679.

[Martin Pool (mbp)]

On 4/4/07, vila <email address hidden> wrote:
> If this is implemented by establishing the list of all tests, it may be
> possible to do more variations with that test list:
>
> - prioritize some tests (say I want to run some specific tests before
> the rest of the test suite),

selftest --first branch

 it's really handy

--
Martin