heuristic to determine compression-format from HEAD of upstream branch can lead to incorrect result when upstream is not merged to master

[Rolf Leggewie]

Binary package hint: git-buildpackage

gbp will create a new orig tar even when an orig.tar.bz2 is present. Please see attached file for test case.

[Guido Günther (agx)]

bz2 support support is there since quiet some time now. It detects the compression type from the pristine-tar branch, it can also be given in the config file or on the command line.

I'd be better if you could file the bugs in the Debian BTS since I usually don't check launchpad.

[Rolf Leggewie (r0lf)]

OK, let's hope the support trickles down to Ubuntu, then. I'd say it's something that someone (not saying it should be you, Guido) ought to cherry-pick into the lucid package.

I really hate dealing with the Debian BTS. E-Mail interface? give me a break! Even with reportbug it sucks (although something akin to reportbug for Launchpad would be really nice to have. I think there's a ticket open about this somewhere).

[Rolf Leggewie (r0lf)]

Just a reminder to self about where I found this

"isdnutils_3.9.20060704+dfsg.1.orig.tar.gz does not exist, creating from 'dfsg'"

[Rolf Leggewie (r0lf)]

I'm getting this with version 0.5.3, so "quite some time" seems to be relative. Maybe I'll get around to recompile the latest Debian package for lucid one of these days to test there.

[Rolf Leggewie (r0lf)]

I've rebuilt 0.5.4 with 88afa6117ea4dd6b5d8fd88936afffb11c25ec47 reverted ("Pass --multimaint-merge on to dch"). The devscripts package in lucid is not recent enough. The issue is still unchanged. Please refer to the test case above. bz2 support seems to be still at least incomplete as of today.

[Guido Günther (agx)]

On Mon, Sep 13, 2010 at 06:36:51AM -0000, Launchpad Bug Tracker wrote:
> You have been subscribed to a public bug by Rolf Leggewie (r0lf):
>
> Binary package hint: git-buildpackage
>
> gbp will create a new orig tar even when an orig.tar.bz2 is present
See the manpage:
    --git-compression-level=auto
Cheers,
 -- Guido

> TEST CASE:
> $ cd /tmp
> $ dget -ud http://ftp.de.debian.org/debian/pool/main/i/isdnutils/isdnutils_3.9.20060704+dfsg.1-2.dsc
> $ git-import-dsc ./isdnutils_3.9.20060704+dfsg.1-2.dsc
> $ cd isdnutils
> $ nice git-buildpackage --git-ignore-new
> [...]
> isdnutils_3.9.20060704+dfsg.1.orig.tar.gz does not exist, creating from 'upstream/3.9.20060704+dfsg.1'
> [...]
> dpkg-source: error: several orig.tar files found (./isdnutils_3.9.20060704+dfsg.1.orig.tar.bz2 and ./isdnutils_3.9.20060704+dfsg.1.orig.tar.gz) but only one is allowed
>
> ** Affects: git-buildpackage (Ubuntu)
> Importance: Undecided
> Status: New
>
> --
> gbp misses bz2 support
> https://bugs.edge.launchpad.net/bugs/615212
> You received this bug notification because you are a direct subscriber of the bug.
>

[Rolf Leggewie (r0lf)]

Thank you for your comment. Arguably, something isn't right here.

From the manpage of git-buildpackage: "--git-compression=type
              Specifies the upstream tarball compression type.[...] The default is auto which derives the compression type from the pristine-tar branch if available and falls back to gzip otherwise."

Here is my pristine-tar branch (isdnutils has become a bit of a complicated beast between Ubuntu and Debian over the years.

commit accbdec57c1b50b7be2359bc21d450ddab997488
Author: Rolf Leggewie <email address hidden>
Date: Wed Sep 8 18:35:28 2010 +0200

    pristine-tar data for isdnutils_3.12.20071127.orig.tar.gz

commit e72bfb99f94775bc93b065156943edea80d3ad32
Author: Rolf Leggewie <email address hidden>
Date: Wed Sep 8 18:32:12 2010 +0200

    pristine-tar data for isdnutils_3.10.20070306.orig.tar.gz

commit cbc61d6dab6430aa5553546b2e9752d453ed8926
Author: Rolf Leggewie <email address hidden>
Date: Wed Sep 8 17:55:47 2010 +0200

    pristine-tar data for isdnutils_3.9.20060704+dfsg.1.orig.tar.bz2

[...]

Master is following 3.9.20060704+dfsg.1, but there are newer upstream releases for the debian-branch of Ubuntu. I hope this makes sense.

[Rolf Leggewie (r0lf)]

I guess a test-case would be to import with --pristine-tar the latest releases of isdnutils from Debian stable unstable and Ubuntu maverick.

Debian stable: --upstream-branch=upstream debian-branch=master
Debian unstable: --upstream-branch=dsfg (branched off from upstream) --debian-branch=master
Maverick: --upstream-branch=upstream debian-branch=ubuntu

Then try to build the master branch against dfsg branch and you should run into this issue.

[Guido Günther (agx)]

On Mon, Sep 13, 2010 at 09:53:43AM -0000, Rolf Leggewie wrote:
> I guess a test-case would be to import with --pristine-tar the latest
> releases of isdnutils from Debian stable unstable and Ubuntu maverick.
>
> Debian stable: --upstream-branch=upstream debian-branch=master
> Debian unstable: --upstream-branch=dsfg (branched off from upstream) --debian-branch=master
> Maverick: --upstream-branch=upstream debian-branch=ubuntu
>
> Then try to build the master branch against dfsg branch and you should
> run into this issue.
Could you build a simple testcase please? Tarball detection works fine
here and I assume your last commit on the pristine-tar branch simply
doesn't give the correct compression type.
 -- Guido

[Guido Günther (agx)]

On Mon, Sep 13, 2010 at 01:35:12PM -0000, Rolf Leggewie wrote:
> ** Description changed:
>
> ** Description changed:
>
> Binary package hint: git-buildpackage
>
> - gbp will create a new orig tar even when an orig.tar.bz2 is present
> -
> - TEST CASE:
> - $ cd /tmp
> - $ dget -ud http://ftp.de.debian.org/debian/pool/main/i/isdnutils/isdnutils_3.9.20060704+dfsg.1-2.dsc
> - $ git-import-dsc ./isdnutils_3.9.20060704+dfsg.1-2.dsc
> - $ cd isdnutils
> - $ nice git-buildpackage --git-ignore-new
> - [...]
> - isdnutils_3.9.20060704+dfsg.1.orig.tar.gz does not exist, creating from 'upstream/3.9.20060704+dfsg.1'
> - [...]
> - dpkg-source: error: several orig.tar files found (./isdnutils_3.9.20060704+dfsg.1.orig.tar.bz2 and ./isdnutils_3.9.20060704+dfsg.1.orig.tar.gz) but only one is allowed
> + gbp will create a new orig tar even when an orig.tar.bz2 is present.
> + Please see attached file for test case.
This works here:

git-import-dsc --pristine-tar --download http://ftp.de.debian.org/debian/pool/main/i/isdnutils/isdnutils_3.9.20060704+dfsg.1-2.dsc
rm *.bz2
cd isdnutils
git-buildpackage -d
...
dpkg-source: info: using source format `3.0 (quilt)'
dpkg-source: info: building isdnutils using existing ./isdnutils_3.9.20060704+dfsg.1.orig.tar.bz2
...

The auto option was introduced in 0.5.0.
 -- Guido

[Rolf Leggewie (r0lf)]

Günther, thank you for trying to reproduce this issue. But your test is way too simplistic ;-) Try the script I have attached now (took a little longer due to bug 637286 throwing me off course for a while).

[Rolf Leggewie (r0lf)]

I think both this one and bug 637286 both stem from the same basic error in your assumption: that the upstream branch is always merged into master. isdnutils is a good example where this will not be the case. Ubuntu has gotten ahead of Debian and thus HEAD of the upstream branch is not yet merged with master (I expect that to take quite a while for me to achieve, isdnutils needs a lot of TLC).

[Guido Günther (agx)]

On Mon, Sep 13, 2010 at 04:14:08PM -0000, Rolf Leggewie wrote:
> Günther, thank you for trying to reproduce this issue. But your test is
> way too simplistic ;-) Try the script I have attached now (took a
> little longer due to bug 637286 throwing me off course for a while).
There no script at this mail. Please use the Debian BTS since.
 -- Guido

[Guido Günther (agx)]

Testcase fails with:

$ sh -e lp615212-test-case.sh
`isdnutils_3.12.20071127-0ubuntu6.dsc' -> `isdnutils.old/isdnutils_3.12.20071127-0ubuntu6.dsc'
`isdnutils_3.9.20060704+dfsg.1-2.dsc' -> `isdnutils.old/isdnutils_3.9.20060704+dfsg.1-2.dsc'
`isdnutils_3.9.20060704+dfsg.1-3.dsc' -> `isdnutils.old/isdnutils_3.9.20060704+dfsg.1-3.dsc'
`isdnutils_3.9.20060704-3.6.dsc' -> `isdnutils.old/isdnutils_3.9.20060704-3.6.dsc'
dget: retrieving http://ftp.de.debian.org/debian/pool/main/i/isdnutils/isdnutils_3.9.20060704+dfsg.1-1.dsc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404
dget: curl isdnutils_3.9.20060704+dfsg.1-1.dsc http://ftp.de.debian.org/debian/pool/main/i/isdnutils/isdnutils_3.9.20060704+dfsg.1-1.dsc failed

[Guido Günther (agx)]

Ignoring the error the outcome is expected:

commit 40a16b8c1523282b8b53c034b5fa86b25b93473e
Author: Guido Günther <email address hidden>
Date: Mon Sep 13 21:05:24 2010 +0200

    pristine-tar data for isdnutils_3.9.20060704.orig.tar.gz

:000000 100644 0000000... 8585744... A isdnutils_3.9.20060704.orig.tar.gz.delta
:000000 100644 0000000... 730e8af... A isdnutils_3.9.20060704.orig.tar.gz.id

so the last version is a tar.gz and so we assume this is this is the preferred format. So I'd say this is no bug.

[Rolf Leggewie (r0lf)]

> so the last version is a tar.gz and so we assume this is this is the preferred format.

yes, understood. But as I pointed out earlier, this assumption is obviously wrong. HEAD from upstream will not always be merged with master. Using HEAD from upstream branch to determine the compression format is not correct. I think the best way to go about this is to follow the commit graph from master back to upstream and use the commit you find this way for determining the compression method.

[Rolf Leggewie (r0lf)]

I've made a little ASCII graphic of this. The last two commits on the upstream branch are A and B. The last three commits on master are X, Y and Z. Y merges A into master. When building Z, the compression format should be determined by commit A, not commit B as is currently the case.

[Guido Günther (agx)]

I just noticed the issue after sending the last mail. The heuristic is to look at pristine-tar not upstream (basically since pristine-tar only supports a single branch) so we have to look at the commit messages.

[Rolf Leggewie (r0lf)]

yes, you are absolutely right. compression-format ought to be determined from pristine-tar branch. Currently, gbp always look at HEAD of that branch. What I meant to say is how to determine where in pristine-tar gbp should look. commit messages is another way, I guess.

[Guido Günther (agx)]

Attached (very lightly tested patch) fixes the issue for me.
 -- Guido

[Guido Günther (agx)]

On Tue, Sep 14, 2010 at 10:41:25AM +0200, Guido Günther wrote:
> Attached (very lightly tested patch) fixes the issue for me.
The fix for this is shipped in 0.5.9.
Cheers,
 -- Guido

[Guido Günther (agx)]

On Tue, Sep 14, 2010 at 10:41:25AM +0200, Guido Günther wrote:
> Attached (very lightly tested patch) fixes the issue for me.

The fix for this is shipped in 0.5.9.
Cheers,
 -- Guido

[Alessio Treglia (quadrispro)]

So, fixed in Natty (and Maverick too).
Closing.