export-orig: Add `uscan -d` mechanism

[Bryce Harrington]

Git ubuntu's build.fetch_orig() routine iterates through a set of different mechanisms to try and obtain an upstream tarball for the current branch. However, in some cases all these options may unfortunately fail.

What I've been doing in these cases is to fallback to using `uscan -dd`. This relies on debian/watch, which is far from guaranteed to work, nor guaranteed to obtain the correct tarball version. Despite the caveats, my anecdotal experience has been good.

The idea I've had in mind is to add this as a last option for build.fetch_orig(), implemented as a routine in build.py named something like fetch_orig_from_debian_watch() that tests if os.path.exists('./debian/watch') and then invoke `uscan -d` (or something similar; uscan has a bunch of options to control behavior).

There's a few things I'm not sure about. First, what are the security implications of relying on this downloading random bits from the internet? Second, if upstream has releases up to 1.2, but we only have 1.0, and actually want 1.1, is there a way to hint uscan to grab the 1.1 tarball? And third, a lot of upstreams also publish hashsums of their tarballs for validation; is there a way to leverage these for validation?

[Robie Basak (racb)]

On Thu, Jul 08, 2021 at 10:07:12PM -0000, Bryce Harrington wrote:
> There's a few things I'm not sure about. First, what are the security
> implications of relying on this downloading random bits from the
> internet? Second, if upstream has releases up to 1.2, but we only have
> 1.0, and actually want 1.1, is there a way to hint uscan to grab the 1.1
> tarball? And third, a lot of upstreams also publish hashsums of their
> tarballs for validation; is there a way to leverage these for
> validation?

GPG validation is supported by uscan if a key is available in
debian/upstream/signing-key.asc. There may be other mechanisms as well.
Even just using an https: link in debian/watch can provide some
assurance.

However a key difference is that if an orig tarball is fetched from
pristine-tar or Launchpad or Debian, then it's a tarball that's been
"blessed" by a package maintainer we trust. It's OK to fall back to
uscan, but in that case I think it's important for the user to know and
acknowledge the lower level of trust. So I'm not sure about
_automatically_ dropping that trust level as a fallback. Providing a
separately selected option, or acknowledging use of a lower "trust
level" using a command line option would work, but would that harm the
usefulness of doing this?

Feedback appreciated.

[Bryce Harrington (bryce)]

On Fri, Jul 09, 2021 at 12:39:31PM -0000, Robie Basak wrote:
> On Thu, Jul 08, 2021 at 10:07:12PM -0000, Bryce Harrington wrote:
> > There's a few things I'm not sure about. First, what are the security
> > implications of relying on this downloading random bits from the
> > internet? Second, if upstream has releases up to 1.2, but we only have
> > 1.0, and actually want 1.1, is there a way to hint uscan to grab the 1.1
> > tarball? And third, a lot of upstreams also publish hashsums of their
> > tarballs for validation; is there a way to leverage these for
> > validation?
>
> GPG validation is supported by uscan if a key is available in
> debian/upstream/signing-key.asc. There may be other mechanisms as well.
> Even just using an https: link in debian/watch can provide some
> assurance.
>
> However a key difference is that if an orig tarball is fetched from
> pristine-tar or Launchpad or Debian, then it's a tarball that's been
> "blessed" by a package maintainer we trust. It's OK to fall back to
> uscan, but in that case I think it's important for the user to know and
> acknowledge the lower level of trust. So I'm not sure about
> _automatically_ dropping that trust level as a fallback. Providing a
> separately selected option, or acknowledging use of a lower "trust
> level" using a command line option would work, but would that harm the
> usefulness of doing this?
>
> Feedback appreciated.

It's a good point, and part of my original concern. In my own use, when
I've fallen back to using uscan it makes me worry a bit and doublecheck
upstream more than I would normally. That's good, but also a bit of a
hassle. I agree that making this work seamlessly might gloss over part
of the process that should be given some extra attention.

There's one other potential issue to consider that I forgot to mention.
While the uscan-obtained tarball from upstream almost always is what
Debian uses for packaging, it's not guaranteed to always be so; Debian
occassionally repacks or otherwise modifies the tarballs. It's rare,
and I've never run into this when I've used uscan, but I know it is a
non-zero possibility, so is another reason to give this extra
attention.

So, having this work automatically may be asking for trouble, as you
mention. But I think you also hit on a better question - what is going
to be most useful to the user?

The three main use cases are:

  1. Fix to an already merged package. IOW we're adding a *ubuntu2+
      release, and will *not* be uploading the tarball.

      In this case, if we use uscan and it gave us an incorrect tarball,
      we're going to get an error returned before anything enters the
      archive. The *security* risk is thus low, but we may be perplexed
      about what happened.

  2. Merge from upstream (ahead of Debian).

      This is where using uscan is actually the proper way to do it, and
      where having git ubuntu perform it could provide a really good user
      experience. OTOH, we do this far less frequently than the other
      two use cases here.

  3. Merge from debian.

      This would be the most problematic situation. We *will* be
      uploading the tarball so it is imperitive...

[Paride Legovini (paride)]

Hi,

I may be missing the obvious here, but can't we just rely on pristine-tar for cases (1) and (3)? My git-ubuntu clones have these pristine-tar branches:

  remotes/pkg/importer/debian/pristine-tar
  remotes/pkg/importer/ubuntu/pristine-tar

By doing for example

  git branch --track pristine-tar pkg/importer/debian/pristine-tar

pristine-tar can be used to checkout an unmerged orig tarball from Debian. Unfortunately the pristine-tar branch name is hardcoded in the pristine-tar cli, so this manual step is needed. There is a feature request to add support for specifying branch name to pristine-tar:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895377

which has been filed specifically for git-ubuntu by this Scott Moser guy :-)

As Bryce mentioned uscan can't be trusted to download the same .orig tarball which is in the archive. For example for the packages I maintain in Debian I use d/watch only for getting notified of new upstream releases, but then I package with a "pure git" workflow (using gbp-import-ref [1]), so the uscan downloaded tarball differs from what I upload. In general I think re-downloading the .orig tarballs where "blessed" ones exist is not a good idea.

Case (2) is certainly a valid use case for `uscan -d`, however I wonder if downloading those "ahead of Debian" orig tarballs belongs to git-ubuntu. [Maybe git-ubuntu could one day support merging from upstream in a "pure git" way, automatically setting up an `upstream` branch using the Vcs-Git field?]

[1] https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/man.gbp.import.ref.html