Comment 4 for bug 1851228

On Mon, Nov 04, 2019 at 03:09:11PM -0000, Matthias Klose wrote:
> > No. Tests, just like package builds, should not assume that they will
> > run on an empty system. Please file bugs, ideally in Debian, when that
> > doesn't happen. If necessary I'll propose a change to the README to make
> > this clear.
>
> this is a rather arrogant attitude.

OK, apologies for being arrogant.

> - There is no documentation what autopkg tests can assume. Telling
> people what they cannot assume without telling them what they can assume
> is not a productive way forward.

What are the corresponding requirements for package builds and where are
they documented? If there's already some work done in this area then we
can build off that.

Thing is, I don't think it's even possible for us to guarantee that
things run on 'quiet' machines. Retrying to work around tests being
poorly written is no solution either (and it *will stop working* when we
are constantly retrying all tests because then there will never be a
quiet period). The best thing we can do IMO is to inform maintainers so
that they at least have the chance to fix their stuff.

> and without having a way for Debian to reproduce an issue, is not very
> high.

I don't see how it's very different from any other type of test failure.
Either the maintainer understands the failure or they don't, they can
run the test (on their machine or in a porter box) or they can't, they
can reproduce the problem or they can't. It doesn't really matter if
it's a flaky-under-load failure or any of the other crappy failures we
see all of the time.

In other words, I don't understand how this particular argument is
special to the case you're talking about here vs. any other failure we
see in Ubuntu but doesn't happen in Debian (VM vs lxc, proxy vs
no-proxy, non-amd64 arches). All of these failures should be reported to
the maintainer and they can judge when, whether, and with how much
effort to fix them according to their own priorities (as can people on
the Ubuntu side).

> Proposing just a README change without any other communication is no way
> forward.

README.package-tests.rst is the main documentation for how to write a
dep8 test that we have.

You're free to argue that simply changing that is not enough on its own.
If you have an additional suggestion for an appropriate place to
communicate expectations, please make it.

> Disappointed, Matthias

Sorry to have disappointed you.

Hopeful, Iain

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]