packages that are only in -proposed cause stalled "in progress" tests

[Martin Pitt]

https://requests.ci-train.ubuntu.com/static/britney/landing-1931/yakkety/excuses.html currently shows:

qemu (1:2.6+dfsg-3ubuntu2 to 1:2.6.1+dfsg-0ubuntu1)
  autopkgtest for lava-dispatcher: amd64: Test in progress (always failed), armhf: Test in progress (always failed), i386: Test in progress (always failed), ppc64el: Test in progress (always failed), s390x: Test in progress (always failed)
  autopkgtest for lava-server: amd64: Test in progress, armhf: Test in progress (always failed), i386: Test in progress, ppc64el: Test in progress, s390x: Test in progress

These two packages got removed in xenial and now only exist in -proposed, where a silo test cannot see them.

These tests fail with code 12 "bad package":
W: Unable to locate package lava-server
autopkgtest [11:39:49]: ERROR: erroneous package: rules extract failed with exit code 1
blame: lava-server

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-yakkety-ci-train-ppa-service-landing-1931/yakkety/i386/l/lava-server/20160912_113959@/log.gz

[Martin Pitt (pitti)]

Britney should not trigger tests which only exist in -proposed. We try to minimize what we look at in -proposed, so only tests in -release should be relevant.

[Martin Pitt (pitti)]

Fixed: https://git.launchpad.net/~ubuntu-release/+git/britney2-ubuntu/commit/?id=61d7bacd4

[Martin Pitt (pitti)]

https://requests.ci-train.ubuntu.com/static/britney/landing-1931/yakkety/excuses.html still shows lava-server as bileto includes -proposed in the "testing" indexes for britney -- so britney has no chance of knowing that a package is only in -proposed.

IMHO bileto should not do that -- trying to land a silo into -release is cleaner and avoids breakage by unrelated packages in -proposed. There could theoretically be cases where a silo depends on a package which hasn't migrated to -release yet -- but then this is broken by definition and the silo must not/cannot land anyway.

[Robert Bruce Park (robru)]

I believe dobey has some strong opinions about this. How can it make sense to build against proposed but then not include proposed during testing? I was under the impression that you have to test against what you built against.

[Martin Pitt (pitti)]

Robert Bruce Park [2016-09-14 5:54 -0000]:
> I believe dobey has some strong opinions about this. How can it make
> sense to build against proposed but then not include proposed during
> testing? I was under the impression that you have to test against what
> you built against.

If there are versioned dependencies to these build dependencies in
-proposed (like library shlibs), then of course the tests will run
against those as well. We minimize the set of packages we test against
in -proposed to what dependencies prescribe; we don't test against the
entirety of -proposed as that involves a whole lot more than just your
build deps and would lead to a lot more (unrelated) failures.

[Robert Bruce Park (robru)]

One thing to consider is that quite often a bileto ticket will contain a library and an app, the library introducing new API, and then the app begins using that API, and nobody bothers to version the dependencies.

[Martin Pitt (pitti)]

Robert Bruce Park [2016-09-14 22:59 -0000]:
> One thing to consider is that quite often a bileto ticket will contain a
> library and an app, the library introducing new API, and then the app
> begins using that API, and nobody bothers to version the dependencies.

That's normally done automatically via dpkg-shlibdeps. I. e. you add a
new symbol to a library, add it to debian/libfoo1.symbols, and dpkg
will check which symbol your program uses and create a dependency high
enough to satisfy all these symbols.

It's really good practice to build packages with "dh_makeshlibs --
-c4" so that the package FTBFS if you forgot to update the symbols
(otherwise it's just a warning and you get a diff in the build log
which is easy to ignore).

Otherwise you'd just get a linker failure in CI, which is then
pointing out an actual packaging error. And we already run packages in
isolation during actual tests, which tells us that our packages are
fine.

[Robert Bruce Park (robru)]

I'm still not convinced that it makes sense to not include -proposed in bileto britney. Can you comment on this, steve?

[Martin Pitt (pitti)]

Let's keep the current "more strict than necessary" policy until we get too many complaints that this actually breaks things. If/once that happens, we can dial it down to run tests against release only (like we do for direct Ubuntu uploads).