Tests should pass using installer bzr versions and be run as part of the release process

[Martin Packman]

As demonstrated by bug 631350 it's possible to introduce pretty major breakage in the installer build process. We started discussing how the regression wasn't picked up by the test suite, in these cases the answer is "no one runs selftest with bzr.exe".

As the tests *mostly* work and it's not hard to add a skip to those that, for instance, rely on sys.executable pointing to a Python interpreter, doing that and then making sure `bzr.exe selftest` passes before releasing would catch a number problems.

[Martin Pool (mbp)]

I guess you're asking here both that we should fix them so they do pass, and also we should then add this to the release checklist. Perhaps Babune could automatically package and then run the tests.

[Alexander Belchenko (bialix)]

I should note that actual testing should be done on vanilla machine without build tools and Python 2.6 installed.

[Vincent Ladeuil (vila)]

>>>>> Alexander Belchenko <email address hidden> writes:

    > I should note that actual testing should be done on vanilla machine
    > without build tools and Python 2.6 installed.

Don't make testers life too hard either :)

We need a way to get the code to test and we want to run selftest (the
actual installer doesn't embed enough of the test suite for that if I
remember correctly). So we need bzr and its associated test tools
anyway.

Of course we should isolate the actual tests to ensure we don't grab any
hidden dependencies though.

A dedicated test suite for the installers sounds like a good plan anyway.

[Alexander Belchenko (bialix)]

Vincent Ladeuil пишет:
>>>>>> Alexander Belchenko <email address hidden> writes:
>
> > I should note that actual testing should be done on vanilla machine
> > without build tools and Python 2.6 installed.
>
> Don't make testers life too hard either :)

Actually, my point is to test entire bzr.exe + installer problems. E.g.
I suspect we might miss some dlls, or manifest or whatever. Only test on
clean achine can reveal this.

> We need a way to get the code to test and we want to run selftest (the
> actual installer doesn't embed enough of the test suite for that if I
> remember correctly). So we need bzr and its associated test tools
> anyway.

If we talking about bzr.exe then everything except python.exe should be
there.

> Of course we should isolate the actual tests to ensure we don't grab any
> hidden dependencies though.
>
> A dedicated test suite for the installers sounds like a good plan
> anyway.

Just skip tests if we run bzr.exe -- it's easy to create Feature. The
check for bzr.exe is very simple:

getattr(sys, "frozen", None) is not None

--
All the dude wanted was his rug back

[Vincent Ladeuil (vila)]

>>>>> Alexander Belchenko <email address hidden> writes:

<snip/>

    >> We need a way to get the code to test and we want to run selftest
    >> (the actual installer doesn't embed enough of the test suite for
    >> that if I remember correctly). So we need bzr and its associated
    >> test tools anyway.

    > If we talking about bzr.exe then everything except python.exe
    > should be there.

And bzr.exe will update itself ? I had bad experiences with such setups
for tests envs.

I'd strongly prefer working a good isolation when running the tests,
that's the point anyway: focus on the system under test instead of
tweaking the whole system.

    >> Of course we should isolate the actual tests to ensure we don't
    >> grab any hidden dependencies though.
    >>
    >> A dedicated test suite for the installers sounds like a good plan
    >> anyway.

    > Just skip tests if we run bzr.exe -- it's easy to create Feature. The
    > check for bzr.exe is very simple:

    > getattr(sys, "frozen", None) is not None

Sure. But adding a check to *all* existing tests ?

I've thought about that for other needs, and invariably the answer is:
creating a dedicated test suite (which could include exising tests) is
way easier than forcing new constraints on all existing tests.

And when I say dedicated test suite, it could be as simple as running:

  selftest -s bt.test_installer

And these tests could well use the feature you mentioned above to skip
in the normal case.

[Martin Pool (mbp)]

On 8 September 2010 18:12, Vincent Ladeuil <email address hidden> wrote:
>    >> Of course we should isolate the actual tests to ensure we don't
>    >> grab any hidden dependencies though.
>    >>
>    >> A dedicated test suite for the installers sounds like a good plan
>    >> anyway.
>
>    > Just skip tests if we run bzr.exe -- it's easy to create Feature. The
>    > check for bzr.exe is very simple:
>
>    > getattr(sys, "frozen", None) is not None
>
> Sure. But adding a check to *all* existing tests ?
>
> I've thought about that for other needs, and invariably the answer is:
> creating a dedicated test suite (which could include exising tests) is
> way easier than forcing new constraints on all existing tests.

I think gz meant: any tests that just cannot pass when run from a
frozen bzr can easily skip. Most of them should not need to. Some
such were mentioned recently.

>
> And when I say dedicated test suite, it could be as simple as running:
>
>  selftest -s bt.test_installer
>
> And these tests could well use the feature you mentioned above to skip
> in the normal case.

That'd be good.

--
Martin

[Alexander Belchenko (bialix)]

Martin Pool пишет:
> On 8 September 2010 18:12, Vincent Ladeuil <email address hidden> wrote:
>> >> Of course we should isolate the actual tests to ensure we don't
>> >> grab any hidden dependencies though.
>> >>
>> >> A dedicated test suite for the installers sounds like a good plan
>> >> anyway.
>>
>> > Just skip tests if we run bzr.exe -- it's easy to create Feature. The
>> > check for bzr.exe is very simple:
>>
>> > getattr(sys, "frozen", None) is not None
>>
>> Sure. But adding a check to *all* existing tests ?
>>
>> I've thought about that for other needs, and invariably the answer is:
>> creating a dedicated test suite (which could include exising tests) is
>> way easier than forcing new constraints on all existing tests.
>
> I think gz meant: any tests that just cannot pass when run from a
> frozen bzr can easily skip. Most of them should not need to. Some
> such were mentioned recently.

Yes, that's what I meant.

--
All the dude wanted was his rug back