'force-skiptest' should skip consideration of test results, but not suppress their output on update_excuses

[Steve Langasek]

I currently have a skiptest hint in place for php-defaults. update_excuses now shows:

php-defaults (32ubuntu1 to 35ubuntu3)

    Maintainer: Ubuntu Developers
    0 days old
    Should wait for tests relating to php-defaults 35ubuntu3, but forced by vorlon
    Valid candidate

Forcing the package to be considered as a candidate should not suppress the output from the tests; this is still important information even if the results a currently being disregarded.

(For the present instance, I'm only skipping the tests in order to get preliminary update_output results; so I would also really like to know the status of the tests so I can drop the hint again.)

[Martin Pitt (pitti)]

Note that we currently literally skip the tests with such a hint in place, i. e. britney doesn't even request them. This has helped us to actually land some perl or glibc fixes without having to wait for several thousand tests to finish.

php-defaults was actually fairly close to succeeding, there was just one remaining regression.

Anyway, for what you *really* want to do here: I think this sounds like this work item: "britney: produce update_output_notest.txt with a dry run ignoring test results, for easier transitions"? (Matthias asked for that on the last sprint too).

[Steve Langasek (vorlon)]

On Tue, Mar 15, 2016 at 09:31:28PM -0000, Martin Pitt wrote:
> Note that we currently literally skip the tests with such a hint in
> place, i. e. britney doesn't even request them. This has helped us to
> actually land some perl or glibc fixes without having to wait for
> several thousand tests to finish.

Ok, I don't think it's right that we skip requesting those tests as a result
of the hint. Nor do I think it's right that we skip displaying the results
of tests that have actually happened.

> php-defaults was actually fairly close to succeeding, there was just one
> remaining regression.

This transition has been ongoing for weeks, and there's a strong need to be
able to look at multiple layers and multiple branches of the tree in
parallel. Hence the work item about update_output_notest.txt, which you
mention.

> Anyway, for what you *really* want to do here: I think this sounds like
> this work item: "britney: produce update_output_notest.txt with a dry
> run ignoring test results, for easier transitions"? (Matthias asked for
> that on the last sprint too).

Yes, they are related. But even if force-skiptest is not needed for being
able to peek ahead to update_output, but is being used for some other
reason, I don't like having the tests suppressed to where I (or someone
else) can't see them and iterate on them in parallel to the hint.

[Martin Pitt (pitti)]

For the record, this was introduced in bug 1522893. But with some more manual operations we can get along with always requesting tests too.

[Martin Pitt (pitti)]

http://bazaar.launchpad.net/~ubuntu-release/britney/britney2-ubuntu/revision/588