"make check" running tests twice is inefficient

[Martin Pool]

"bzr check", run by pqm, runs the test suite once in the system default locale and once with LANG=C. This means the tests and pqm merges take roughly twice as long as running the tests themselves, which is already a considerable time.

The delay has a cost particularly when trying to make a release.

The second locale check seems to fairly rarely catch actual problems.

Our general approach is not to blindly multiply tests but to add specific tests or parameterizations. So we should do the same thing here.

[Robert Collins (lifeless)]

On Fri, 2009-06-12 at 01:38 +0000, Martin Pool wrote:
> Public bug reported:
>
> "bzr check", run by pqm, runs the test suite once in the system default
> locale and once with LANG=C. This means the tests and pqm merges take
> roughly twice as long as running the tests themselves, which is already
> a considerable time.
>
> The delay has a cost particularly when trying to make a release.

Perhaps we shouldn't be cramming as much stuff in and simple have the
release be 'tag and bag' ?

> The second locale check seems to fairly rarely catch actual problems.

Thats not my experience. When working on unicode/C stuff I quite
routinely have things fail.

> Our general approach is not to blindly multiply tests but to add
> specific tests or parameterizations. So we should do the same thing
> here.

I believe we are heavily handicapped by python in this regard. Once we
start doing gettext it may become even more difficult to change things
on the fly.

-Rob

[Martin Pool (mbp)]

Just removing the check is not felt to be a good tradeoff until we have some other place to regularly run those tests. When we have buildbot running, I would like to land this, make other machines test ascii and other locales, and remove them from the main pqm.

[Martin Pool (mbp)]

We had a list thread again "time to remove LANG=C form make check" and there were no objections, so I'll send this to pqm.

[Robert Collins (lifeless)]

On Wed, 2009-08-26 at 23:00 +0000, Martin Pool wrote:
> We had a list thread again "time to remove LANG=C form make check" and
> there were no objections, so I'll send this to pqm.

For the record, my objections are still standing. Lets do the experiment
though.

Vila: Can you please confirm that buildbot *is* running the ascii mode
tests explicitly (and not depending on 'make check' ... )

-Rob

[Vincent Ladeuil (vila)]

>>>>> "robert" == Robert Collins <email address hidden> writes:

    robert> On Wed, 2009-08-26 at 23:00 +0000, Martin Pool wrote:
    >> We had a list thread again "time to remove LANG=C form make check" and
    >> there were no objections, so I'll send this to pqm.

    robert> For the record, my objections are still standing. Lets do the experiment
    robert> though.

    robert> Vila: Can you please confirm that buildbot *is* running the ascii mode
    robert> tests explicitly (and not depending on 'make check' ... )

Since 2009-08-14, it runs:

class BzrSelftest(BzrTests):

    description = ['running selftest']
    descriptionDone = ['selftest']
    command = ['python', './bzr', 'selftest',
               # FIXME: we need a better way to define which plugins are tested
               '--no-plugins',
               '--parallel=fork',
               '-v',
               ]

# We don't specify LANG here, each slave has its own default value
tests_factory.addStep(BzrSelftest())
# Until we better control what plugins are taken into account (which mean not
# only defining a slave specific .bazaar/plugins directory but also being able
# to ignore the plugins installed system-wide), the best we can do is to run
# selftest --no-plugins :-/ -- vila 20090813
tests_factory.addStep(BzrSelftest(
        description=['running selftest no locale',],
        descriptionDone=['selftest no locale'],
        env=dict(LC_TYPE='', LC_ALL='', LANG='C'),
        ))

So it doesn't:
- depend on make check,
- test the core plugins.

  Vincent