"git ubuntu build-source" does not run debian/clean

[Robie Basak]

"usd build-source" uses -nc. Colin Watson helpfully pointed out that on occasion packages may use the "clean" debian/rules target to do things like generating debian/control from a control.in, as the least-worst place to put such a thing.

This also means that -d is "wrong" because the clean target is entitled to expect all build dependencies to be installed.

A way of solving this might be to do all build tasks inside sbuild or lxd or similar (or both). I favour lxd because of it's automatic dtrt image management.

Importance: High because I think correctness is important here. I don't want to have to give new contributors excuses about how our tooling is fundamentally broken and needs a workaround because they hit an edge case.

[Nish Aravamudan (nacc)]

This becomes more important in Artful, where we have these stray .buildinfo files and don't use -nc.

  - dpkg now produces .buildinfo files by default for all uploads,
    including source builds. This should, again, be mostly a no-op
    for you, but there is one gotcha, which is that multiple '-nc'
    runs of dpkg-buildpackage -S will produce a debian/files in
    your source that shouldn't be there. We've been discussing how
    to eradicate that upstream, but if you care deeply about cruft,
    watch out for that.

[Nish Aravamudan (nacc)]

In order to properly support this, it seems like we also need to support building in a schroot or lxd. I prefer a lxd as it means less configuration on the client (but does require a working lxd environment already). To that end, I think put it all in a try/except (I guess). I also think we should use pylxd or something as an abstraction layer.

Roughly:

git ubuntu build-source:
container = lxc launch -e ubuntu:<release> (needs to be ubuntu-daily for the development release)\
tempdir = lxc exec container mktemp -d
lxc file push . $container/tempdir
lxc exec container 'cd $tempdir; dpkg-buildpackage -S'
...
lxc file pull $container/tempdir/*.changes ../
for each file in changes:
    lxc file pull $container/tempdir/$file ../
if --sign:
    debsign

git ubuntu build same as above without -S.

At that point, I would consider dropping build-source, as it would always use a lxd.

[Robie Basak (racb)]

I would like:

Default to lxd, install build deps, do it all correctly. If that fails, don't try harder. Allow a lxc profile to be passed in so the user can specify a lxd configuration that works in a difficult environment.

A --fast-and-dangerous option (more sensibly named) that does a non-lxd local build with -nc and -d.

-nc and -d may be configurable (not sure) but I definitely want to be able to do --no-lxd --really-just-local -nc -d.

Eventually I'd like a dotfile so that I can set --fast-and-dangerous by default for myself personally.

This is for the two use cases:

1) Documentation and examples always work for new contributors - correctness first to smooth out edge cases.

2) Experienced devs accept the edge case gotchas for faster iteration when it is known that the edge cases will not bite (most source packages).

[Nish Aravamudan (nacc)]

Rough outline of commands that I think work:

1) get orig tarball(s) locally
  [ depends on pristine-tar branch for full functionality ]
2) git archive -o /tmp/<tarball> --prefix=<srcpkg>-<version> HEAD
3) lxc launch -e <ubuntu:release|ubuntu-daily:release> <containername>
4) lxc exec <containername> -- apt-get update
5) lxc exec <containername> -- apt-get install devscripts equivs
6) lxc file push /tmp/<tarball> <containername>/tmp/
7) containertmp = lxc exec <containername> -- mktemp -d
8) copy orig tarball(s) to container
9) lxc exec <containername> -- tar zxCf <containertmp> /tmp/<tarball>
10) lxc exec <containername> -- sh -c "cd <containertmp>/<srcpkg>-<version> && mk-build-deps -i -r debian/control && dpkg-buildpackage"
11) presuming success of 10), lxc file pull <containername> <containertmp>/*.{changes,dsc,deb} ../

[Nish Aravamudan (nacc)]

       --build=type
              Specifies the build type from a comma-separated list of
              components (since dpkg 1.18.5). Passed to dpkg-genchanges.

              The allowed values are:

              source Builds the source package. Note: when using this value
                     standalone and if what you want is simply to (re-)build
                     the source package, using dpkg-source is always better as
                     it does not require any build dependencies to be
                     installed to be able to call the clean target.

Interesting, --build=source is the same -S which is what build-source passes down to dpkg-buildpackage. I'm not sure how that last sentence is possible given the description of this bug?

[Nish Aravamudan (nacc)]

Talked with cjwatson on IRC:

09:19 < nacc> cjwatson: I was re-reviewing LP: #1687059, I believe you originally pointed this out to rbasak and myself. But then I'm reading `man dpkg-buildpackage`
              (http://paste.ubuntu.com/25528503/) and I'm wondering if a) we don't need actuall need the b-d for clean? or b) if `man dpkg-buildpackage` is wrong?
09:19 < ubottu> Launchpad bug 1687059 in usd-importer ""git ubuntu build-source" does not run debian/clean" [High,Triaged] https://launchpad.net/bugs/1687059
09:19 < nacc> slangasek: --^ you might also have an opinion/relevant knowledge
09:21 < cjwatson> nacc: I'm not sure where that assertion comes from. At a minimum, the clean target nearly always requires debhelper, which (regardless of one's opinion on
                  whether it should be) is not build-essential, and it often also requires other build-dependencies for extra helpers.
09:22 < nacc> cjwatson: yeah that's what I figured too, based upon what I've seen
09:22 < nacc> cjwatson: thanks for the input!
09:23 < cjwatson> nacc: I think this is a slightly mangled version of the assertion that if you have a tree that hasn't been built then you can use dpkg-source to construct a
                  source package *without* calling the clean target. (There exist packages that use the clean target to generate files that have to be in the source package,
                  but that wouldn't be a problem if the importer has taken a source package as its ...
09:23 < cjwatson> ... input.)
09:23 < nacc> cjwatson: right, I could see that
09:24 < cjwatson> Robie's suggestion there seems reasonable to me.