Add "expect stderr" restriction
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
autopkgtest (Ubuntu) |
Fix Released
|
Medium
|
Martin Pitt |
Bug Description
The specification currently says:
...if a test exits nonzero, or prints to stderr, it is considered to have failed.
This is backwards. Plenty of programs write status to stderr and then exit with a zero status to indicate success. Examples: wget, curl. I'm sure there are many others. Unix convention is to use the exit status to indicate success/failure, so it would be far more consistent to make this behaviour the default.
This is a problem because now I have to catch every instance of this happening and redirect stderr to /dev/null. Worse, this violates the principle that when a test fails it should tell me why, since now all of stderr which might have told me has been redirected away.
A workaround might be to capture stderr to a temporary file and then only print it if the exit status was non-zero, but this is painful to do in every test and there is no good solution for logging stderr/stdout ordering (eg. annotate-output's manpage warns about this).
Before the specification is set in stone, IMHO the sense of this should be reversed. Tests should exit with a zero status on success. If they actually failed because some program failed but returned a zero exit status, then that's a bug that should be fixed. It might be sensible to define a "restriction" that changes this to the original behaviour of writes to stderr being treated as errors, but the default should be to use the exit status only.
If you don't agree, then at the very least please define and implement a "stderr-
Changed in autopkgtest (Ubuntu): | |
importance: | Undecided → Medium |
Changed in autopkgtest (Ubuntu): | |
status: | Triaged → In Progress |
assignee: | nobody → Martin Pitt (pitti) |
status: | In Progress → Fix Committed |
Rodney and I discussed that yesterday, and a compromise was to add a new restriction (or feature) to not count stderr output as a failure.