Add "expect stderr" restriction

[Robie Basak]

The specification currently says:

    ...if a test exits nonzero, or prints to stderr, it is considered to have failed.

This is backwards. Plenty of programs write status to stderr and then exit with a zero status to indicate success. Examples: wget, curl. I'm sure there are many others. Unix convention is to use the exit status to indicate success/failure, so it would be far more consistent to make this behaviour the default.

This is a problem because now I have to catch every instance of this happening and redirect stderr to /dev/null. Worse, this violates the principle that when a test fails it should tell me why, since now all of stderr which might have told me has been redirected away.

A workaround might be to capture stderr to a temporary file and then only print it if the exit status was non-zero, but this is painful to do in every test and there is no good solution for logging stderr/stdout ordering (eg. annotate-output's manpage warns about this).

Before the specification is set in stone, IMHO the sense of this should be reversed. Tests should exit with a zero status on success. If they actually failed because some program failed but returned a zero exit status, then that's a bug that should be fixed. It might be sensible to define a "restriction" that changes this to the original behaviour of writes to stderr being treated as errors, but the default should be to use the exit status only.

If you don't agree, then at the very least please define and implement a "stderr-is-not-a-failure" restriction, so capturing stderr and handling the exit status does not have to be done in every test.

[Martin Pitt (pitti)]

Rodney and I discussed that yesterday, and a compromise was to add a new restriction (or feature) to not count stderr output as a failure.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autopkgtest - 2.3

---------------
autopkgtest (2.3) unstable; urgency=low

  * runner/adt-run.1: Fix bad "affect modify" wording. (LP: #1199500)
  * Drop hosts/chinstrap. We haven't actually used this in many years, it has
    been replaced with https://launchpad.net/auto-package-testing, Jenkins,
    and blocking migration to release until tests pass. (LP: #381210)
  * Promote apt-utils from Recommends to Depends:, to avoid test failures due
    to not finding apt-ftparchive. (LP: #1065874)
  * Promote pbuilder from Recommends: to Depends:, as we always call
    pbuilder-satisfydepends-classic. (LP: #1065874)
  * runner/adt-run: Stop explicitly checking for $HOME (with broken error
    message) when giving --gnupg-home, use os.path.expanduser() instead.
    (LP: #1065873)
  * runner/adt-run: Fix timeout --help output.
  * runner/adt-run: Fix bomb() to clean up the testbed before removing it.
    Afterwards (in cleanup()) we don't have a testbed object any more to run
    commands in. (LP: #1157292)
  * runner/adt-run: Fix UnicodeDecodeError crash if stderr contains non-ASCII
    characters. (LP: #1100283)
  * Add "allow-stderr" restriction for tests where re-routing or disabling
    stderr is impractical. (LP: #1197005)
  * runner-adt-run: Support architecture specifiers in Depends: field.
    (LP: #1165128)
  * runner/adt-run: Remove/fix some unused variables, spotted by pyflakes.

 -- Martin Pitt <email address hidden> Fri, 19 Jul 2013 11:58:15 +0200