Retry /unknown results once before recording them as failures

[Matthias Klose]

as seen in
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/1835737/comments/7

- the email complains about "/unknown" packages. Please don't send emails for those, just give back stuff automatically.

- python-ltfatpy/1.0.12-1 (amd64) was run when the autopkg test queues were full. succeeded on a give back. Please give a failing test at least once before sending that email.

[Iain Lane (laney)]

 affects auto-package-testing
 status triaged
 importance medium
 summary "Retry /unknown results once before recording them as failures"
 affects britney
 status invalid

On Mon, Nov 04, 2019 at 12:01:40PM -0000, Matthias Klose wrote:
> Public bug reported:
>
> as seen in
> https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/1835737/comments/7
>
> - the email complains about "/unknown" packages. Please don't send
> emails for those, just give back stuff automatically.

I think we could do that, especially that we can't fix all of the causes
of those test failures apparently (some /unknown results are due to bugs
in the packages themselves so we can't blanket ignore or retry /unknown
results assuming they will magically become good).

I think we should simply retry all /unknown results one time. This could
be done quite without too much trouble in the worker, like how we
"internally" retry some types of tmpfail (e.g. kernel panics in some
packages). Then no changes in proposed-migration would be needed either.

> - python-ltfatpy/1.0.12-1 (amd64) was run when the autopkg test queues
> were full. succeeded on a give back. Please give a failing test at
> least once before sending that email.

No. Tests, just like package builds, should not assume that they will
run on an empty system. Please file bugs, ideally in Debian, when that
doesn't happen. If necessary I'll propose a change to the README to make
this clear.

Cheers,

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

[Iain Lane (laney)]

https://trello.com/c/I4HUnVEp/43-retry-unknown-results-once-before-recording-them-as-failures

[Matthias Klose (doko)]

> No. Tests, just like package builds, should not assume that they will
> run on an empty system. Please file bugs, ideally in Debian, when that
> doesn't happen. If necessary I'll propose a change to the README to make
> this clear.

this is a rather arrogant attitude.

- There is no documentation what autopkg tests can assume. Telling people what they cannot assume without telling them what they can assume is not a productive way forward.

- Debian only runs autopkg tests on amd64, the likeliness getting any patch accepted without telling expectations on the testbed and without having a way for Debian to reproduce an issue, is not very high.

Proposing just a README change without any other communication is no way forward.

Disappointed, Matthias

[Iain Lane (laney)]

On Mon, Nov 04, 2019 at 03:09:11PM -0000, Matthias Klose wrote:
> > No. Tests, just like package builds, should not assume that they will
> > run on an empty system. Please file bugs, ideally in Debian, when that
> > doesn't happen. If necessary I'll propose a change to the README to make
> > this clear.
>
> this is a rather arrogant attitude.

OK, apologies for being arrogant.

> - There is no documentation what autopkg tests can assume. Telling
> people what they cannot assume without telling them what they can assume
> is not a productive way forward.

What are the corresponding requirements for package builds and where are
they documented? If there's already some work done in this area then we
can build off that.

Thing is, I don't think it's even possible for us to guarantee that
things run on 'quiet' machines. Retrying to work around tests being
poorly written is no solution either (and it *will stop working* when we
are constantly retrying all tests because then there will never be a
quiet period). The best thing we can do IMO is to inform maintainers so
that they at least have the chance to fix their stuff.

> and without having a way for Debian to reproduce an issue, is not very
> high.

I don't see how it's very different from any other type of test failure.
Either the maintainer understands the failure or they don't, they can
run the test (on their machine or in a porter box) or they can't, they
can reproduce the problem or they can't. It doesn't really matter if
it's a flaky-under-load failure or any of the other crappy failures we
see all of the time.

In other words, I don't understand how this particular argument is
special to the case you're talking about here vs. any other failure we
see in Ubuntu but doesn't happen in Debian (VM vs lxc, proxy vs
no-proxy, non-amd64 arches). All of these failures should be reported to
the maintainer and they can judge when, whether, and with how much
effort to fix them according to their own priorities (as can people on
the Ubuntu side).

> Proposing just a README change without any other communication is no way
> forward.

README.package-tests.rst is the main documentation for how to write a
dep8 test that we have.

You're free to argue that simply changing that is not enough on its own.
If you have an additional suggestion for an appropriate place to
communicate expectations, please make it.

> Disappointed, Matthias

Sorry to have disappointed you.

Hopeful, Iain

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]