merge-upstream does not repack zip archive

Bug #934096 reported by Maarten Bezemer on 2012-02-17
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
brz-debian
Medium
Unassigned
bzr-builddeb
Medium
Unassigned

Bug Description

Using merge-upstream it does not repack .zip into a .orig.tar.gz

$ bzr branch lp:ubuntu/jsch
$ cd jsch/
$ bzr merge-upstream
Using distribution maverick
Using version string 0.1.46.
Using uscan to look for the upstream tarball.
jsch: Version (0.1.46) available on remote site:
  http://qa.debian.org/watch/sf.php/jsch/jsch-0.1.46.zip
  (local version is 0.1.46)
jsch: Successfully downloaded updated package jsch-0.1.46.zip
the expected files generated by uscan could not be found in/tmp/tmpDWJ4vE.
bzr: ERROR: jsch 0.1.46 was not found in <bzrlib.plugins.builddeb.upstream.UScanSource object at 0x1a21d10>.

It results in an error as the .zip got downloaded but not repacked.
Manually using uscan --repack does result in the required .orig.tar.gz file

$ apt-cache policy bzr-builddeb
bzr-builddeb:
  Installed: 2.8.2
  Candidate: 2.8.2
  Version table:
 *** 2.8.2 0
         90 http://nl.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages
        100 /var/lib/dpkg/status
     2.7.8ubuntu0.1 0
        500 http://nl.archive.ubuntu.com/ubuntu/ oneiric-updates/universe amd64 Packages
     2.7.8 0
        500 http://nl.archive.ubuntu.com/ubuntu/ oneiric/universe amd64 Packages

Jelmer Vernooij (jelmer) on 2012-02-17
Changed in bzr-builddeb:
status: New → Triaged
importance: Undecided → Medium
Maarten Bezemer (veger) wrote :

I seem to have fixed the problem with this patch

tags: added: patch
Jelmer Vernooij (jelmer) wrote :

Thanks. Unfortunately using --repack will also repack .tar.bz2 files, which isn't correct for 3.0 (quilt) packages.

I suspect we could get gather_orig_files() to look for .zip files too.

Maarten Bezemer (veger) wrote :

Changing the prefix to %s-%s works, but I do not know whether one could assume that all zip archives are in the <package>-<version>.zip format...

On 02/17/2012 12:56 PM, Maarten Bezemer wrote:
> Changing the prefix to %s-%s works, but I do not know whether one could
> assume that all zip archives are in the<package>-<version>.zip
> format...
Yeah, that's a good point. I was hoping uscan would force the filename
for zip files too, but I guess that's not the case.

Another option would be to scan the uscan output for the filename,
perhaps with it XML output mode. That's more foolproof, but it's also a
fair bit more complex.

Cheers,

Jelmer

Maarten Bezemer (veger) wrote :

Ok, tried your idea and it seems to work nicely.

I have kept the original gather_original_files() call, in order to prevent breaking things. But I suppose it could be removed and use the provided uscan filename for all cases?

Maarten Bezemer (veger) wrote :

Is there anything left to in order to accept this patch? If so, please tell me and I'll try to fix/update the patch!

Jelmer Vernooij (jelmer) wrote :

Sorry for not looking at it yet - I usually just watch merge proposals rather than bug reports.

It seems you make assumptions of the local name based on the name the file has remotely (upstream-url). This isn't necessarily correct, in some cases the file gets renamed from what it is upstream.

Can you also add a test for this new behaviour?

Maarten Bezemer (veger) wrote :

I cannot get the testing environment working properly, I have some problems with requiring a version of the bzr test library which is only available in precise. Which is not a problem except it will update half my system if I install it. Manually building was also not working nicely. So, it is going to take some more time until I figure this thing out and have the testing system up-and-running.

About the assumptions of the filename changes. I have looked at the uscan code, and in some (exotic) situations the filename is indeed changed (to prevent some illegal filenames). I cannot see a way of getting notified about a filename change. Should we look into adding this information for the XML output (ie modifying uscan), or let bzr-builddeb check all know filename possibilities (in a correct order)? The first is most robust, but the latter is 'easier' (as we do not have to patch uscan and get it accepted)...

Jelmer Vernooij (jelmer) wrote :

If you're running oneiric you should be able to install a more recent bzr from the daily builds ppa:

https://launchpad.net/~bzr/+archive/daily

> About the assumptions of the filename changes. I have looked at the uscan code, and in some (exotic) situations the filename is
> indeed changed (to prevent some illegal filenames). I cannot see a way of getting notified about a filename change. Should we >
> look into adding this information for the XML output (ie modifying uscan), or let bzr-builddeb check all know filename possibilities > (in a correct order)? The first is most robust, but the latter is 'easier' (as we do not have to patch uscan and get it accepted)...
It looks like the <target> element can be used to find the filename that was created locally, so I don't think that would really require patching of uscan.

Another alternative would indeed be to create an empty directory somewhere and then assume that just make uscan put the file there. That seems fairly robust too, but is a bit more ugly than just looking at the --dehs output.

What do you think?

Maarten Bezemer (veger) wrote :

I'll try the daily builds!

I do not see a <target> element in the --dehs output. If have updated to the precise version of devscripts, but it still does not show the <target> element. It would be convenient to use.

Creating an empty directory and find the file that is downloaded into it, would also work. But, it seems a bit 'lame' to me... If there is not other (proper) solution, it is certainly usable (and quite robust for filename mangling).

Jelmer Vernooij (jelmer) wrote :

On 03/14/2012 01:48 PM, Maarten Bezemer wrote:
> I'll try the daily builds!
>
> I do not see a<target> element in the --dehs output. If have updated to
> the precise version of devscripts, but it still does not show the
> <target> element. It would be convenient to use.
>
> Creating an empty directory and find the file that is downloaded into
> it, would also work. But, it seems a bit 'lame' to me... If there is not
> other (proper) solution, it is certainly usable (and quite robust for
> filename mangling).
>
charis:/tmp/s4-experimental% uscan --dehs --force-download
<dehs>
<package>samba4</package>
<debian-uversion>4.0.0~alpha18.dfsg1</debian-uversion>
<debian-mangled-uversion>4.0.0~alpha18</debian-mangled-uversion>
<upstream-version>4.0.0~alpha18</upstream-version>
<upstream-url>ftp://ftp.samba.org/pub/samba/samba4/samba-4.0.0alpha18.tar.gz</upstream-url>
<status>Newer version available</status>
<target>samba4_4.0.0~alpha18.orig.tar.gz</target>
<messages>Successfully downloaded updated package
samba-4.0.0alpha18.tar.gz and symlinked samba4_4.0.0~alpha18.orig.tar.gz
to it</messages>
</dehs>

Cheers,

Jelmer

Maarten Bezemer (veger) wrote :

This is my output
$ uscan --dehs --force-download
<dehs>
<package>mc</package>
<debian-uversion>4.8.1</debian-uversion>
<debian-mangled-uversion>4.8.1</debian-mangled-uversion>
<upstream-version>4.8.1</upstream-version>
<upstream-url>http://www.midnight-commander.org/downloads/mc-4.8.1.tar.bz2</upstream-url>
<status>up to date</status>
</dehs>

(the archive is downloaded by uscan)

No <target> element available... For both the oneiric and precise versions of devscripts. Do you use some (conveniently) modified uscan version? :)

Furthermore, I tried to update sambe4 (after getting 4.4.0~alpha17 or something similar) and uscan gave an error that the watch file was incorrect, using the --debug shows no downloaded contenet (page), using wget results in a download and a generated index.html of the ftp site.

Something seems to be different between our uscan versions (or libraries that are used).

I am all pro, using the <target> element, but this difference needs to be 'solved' to be sure that others also have the <target> element (maybe we need to add/update package dependencies when we know the difference?)

Jelmer Vernooij (jelmer) wrote :

On Wed, Mar 14, 2012 at 01:47:24PM -0000, Maarten Bezemer wrote:
> This is my output
> $ uscan --dehs --force-download
> <dehs>
> <package>mc</package>
> <debian-uversion>4.8.1</debian-uversion>
> <debian-mangled-uversion>4.8.1</debian-mangled-uversion>
> <upstream-version>4.8.1</upstream-version>
> <upstream-url>http://www.midnight-commander.org/downloads/mc-4.8.1.tar.bz2</upstream-url>
> <status>up to date</status>
> </dehs>

> (the archive is downloaded by uscan)

> No <target> element available... For both the oneiric and precise
> versions of devscripts. Do you use some (conveniently) modified uscan
> version? :)
:-)

> Furthermore, I tried to update sambe4 (after getting 4.4.0~alpha17 or
> something similar) and uscan gave an error that the watch file was
> incorrect, using the --debug shows no downloaded contenet (page), using
> wget results in a download and a generated index.html of the ftp site.
This was with the Debian package, which is on alpha18. I don't recall
updating the watch file between alpha17 and 18, but perhaps I did.

> Something seems to be different between our uscan versions (or libraries
> that are used).

> I am all pro, using the <target> element, but this difference needs to
> be 'solved' to be sure that others also have the <target> element (maybe
> we need to add/update package dependencies when we know the difference?)
I'm running devscripts from precise, so I'm not entirely sure what's
going on here.

My suggestion would be to look at the uscan source code to see
why and if it spits out the target path it has downloaded a tarball
to. Of course, another good alternative is still to download into a
temporary directory and to use os.listdir(temp_dir) to find out what
the name of the downloaded tarball is. Not as nice as parsing the XML,
but also not the worst kludge ever :)

Cheers,

Jelmer

Jelmer Vernooij (jelmer) on 2017-06-17
Changed in brz-debian:
status: New → Triaged
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers