Unable to download packages using Gutsy debmirror

[marcolino]

Binary package hint: debmirror

Hi, using the following script

debmirror -m --passive --host=archive.ubuntulinux.org --root=ubuntu/ --method=ftp --progress --dist=gutsy,gutsy-updates,gutsy-security,gutsy-backports,gutsy-proposed --section=main,multiverse,universe,restricted --arch=i386 ubuntu/ --ignore-release-gpg

which works fine in Dapper and Feisty i'm unable to download packages from repositories.
This is the typescript of the session

Script iniziato su sab 01 set 2007 20:46:19 CEST
root@ubuntu710:/var/www# sh ubuagg

Mirroring to ubuntu/ from ftp://<email address hidden>/ubuntu//

Arches: i386

Dists: gutsy,gutsy-updates,gutsy-security,gutsy-backports,gutsy-proposed

Sections: main,multiverse,universe,restricted

Including source.

Passive mode on.

Checking md5sums.

Will clean up AFTER mirroring.

Pdiff mode: use.

Attempting to get lock, this might take 2 minutes before it fails.

Get Release files.

[0%] Getting: dists/gutsy/Release #

[0%] Getting: dists/gutsy/Release.gpg #

[0%] Getting: dists/gutsy-updates/Release #

[0%] Getting: dists/gutsy-updates/Release.gpg #

[0%] Getting: dists/gutsy-security/Release #

[0%] Getting: dists/gutsy-security/Release.gpg #

[0%] Getting: dists/gutsy-backports/Release #

[0%] Getting: dists/gutsy-backports/Release.gpg #

[0%] Getting: dists/gutsy-proposed/Release #

[0%] Getting: dists/gutsy-proposed/Release.gpg #

Get Packages and Sources files and other miscellany.

dists/gutsy/main/binary-i386/Packages.gz needs fetch

[ 0%] Getting: dists/gutsy/main/binary-i386/Packages.gz ##############

dists/gutsy/main/binary-i386/Release needs fetch

[ 19%] Getting: dists/gutsy/main/binary-i386/Release #

dists/gutsy/main/source/Sources.gz needs fetch

[ 19%] Getting: dists/gutsy/main/source/Sources.gz ####

dists/gutsy/main/source/Release needs fetch

[ 24%] Getting: dists/gutsy/main/source/Release #

dists/gutsy/multiverse/binary-i386/Packages.gz needs fetch

[ 24%] Getting: dists/gutsy/multiverse/binary-i386/Packages.gz ###

dists/gutsy/multiverse/binary-i386/Release needs fetch

[ 26%] Getting: dists/gutsy/multiverse/binary-i386/Release #

dists/gutsy/multiverse/source/Sources.gz needs fetch

[ 26%] Getting: dists/gutsy/multiverse/source/Sources.gz #

dists/gutsy/multiverse/source/Release needs fetch

[ 27%] Getting: dists/gutsy/multiverse/source/Release #

dists/gutsy/universe/binary-i386/Packages.gz needs fetch

[ 27%] Getting: dists/gutsy/universe/binary-i386/Packages.gz ####################################################

dists/gutsy/universe/binary-i386/Release needs fetch

[ 82%] Getting: dists/gutsy/universe/binary-i386/Release #

dists/gutsy/universe/source/Sources.gz needs fetch

[ 82%] Getting: dists/gutsy/universe/source/Sources.gz #################

dists/gutsy/universe/source/Release needs fetch

[100%] Getting: dists/gutsy/universe/source/Release #

dists/gutsy/restricted/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy/restricted/binary-i386/Packages.gz #

dists/gutsy/restricted/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy/restricted/binary-i386/Release #

dists/gutsy/restricted/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy/restricted/source/Sources.gz #

dists/gutsy/restricted/source/Release needs fetch

[100%] Getting: dists/gutsy/restricted/source/Release #

dists/gutsy-updates/main/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-updates/main/binary-i386/Packages.gz #

dists/gutsy-updates/main/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-updates/main/binary-i386/Release #

dists/gutsy-updates/main/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-updates/main/source/Sources.gz #

dists/gutsy-updates/main/source/Release needs fetch

[100%] Getting: dists/gutsy-updates/main/source/Release #

dists/gutsy-updates/multiverse/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-updates/multiverse/binary-i386/Packages.gz #

dists/gutsy-updates/multiverse/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-updates/multiverse/binary-i386/Release #

dists/gutsy-updates/multiverse/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-updates/multiverse/source/Sources.gz #

dists/gutsy-updates/multiverse/source/Release needs fetch

[100%] Getting: dists/gutsy-updates/multiverse/source/Release #

dists/gutsy-updates/universe/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-updates/universe/binary-i386/Packages.gz #

dists/gutsy-updates/universe/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-updates/universe/binary-i386/Release #

dists/gutsy-updates/universe/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-updates/universe/source/Sources.gz #

dists/gutsy-updates/universe/source/Release needs fetch

[100%] Getting: dists/gutsy-updates/universe/source/Release #

dists/gutsy-updates/restricted/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-updates/restricted/binary-i386/Packages.gz #

dists/gutsy-updates/restricted/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-updates/restricted/binary-i386/Release #

dists/gutsy-updates/restricted/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-updates/restricted/source/Sources.gz #

dists/gutsy-updates/restricted/source/Release needs fetch

[100%] Getting: dists/gutsy-updates/restricted/source/Release #

dists/gutsy-security/main/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-security/main/binary-i386/Packages.gz #

dists/gutsy-security/main/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-security/main/binary-i386/Release #

dists/gutsy-security/main/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-security/main/source/Sources.gz #

dists/gutsy-security/main/source/Release needs fetch

[100%] Getting: dists/gutsy-security/main/source/Release #

dists/gutsy-security/multiverse/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-security/multiverse/binary-i386/Packages.gz #

dists/gutsy-security/multiverse/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-security/multiverse/binary-i386/Release #

dists/gutsy-security/multiverse/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-security/multiverse/source/Sources.gz #

dists/gutsy-security/multiverse/source/Release needs fetch

[100%] Getting: dists/gutsy-security/multiverse/source/Release #

dists/gutsy-security/universe/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-security/universe/binary-i386/Packages.gz #

dists/gutsy-security/universe/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-security/universe/binary-i386/Release #

dists/gutsy-security/universe/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-security/universe/source/Sources.gz #

dists/gutsy-security/universe/source/Release needs fetch

[100%] Getting: dists/gutsy-security/universe/source/Release #

dists/gutsy-security/restricted/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-security/restricted/binary-i386/Packages.gz #

dists/gutsy-security/restricted/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-security/restricted/binary-i386/Release #

dists/gutsy-security/restricted/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-security/restricted/source/Sources.gz #

dists/gutsy-security/restricted/source/Release needs fetch

[100%] Getting: dists/gutsy-security/restricted/source/Release #

dists/gutsy-backports/main/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-backports/main/binary-i386/Packages.gz #

dists/gutsy-backports/main/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-backports/main/binary-i386/Release #

dists/gutsy-backports/main/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-backports/main/source/Sources.gz #

dists/gutsy-backports/main/source/Release needs fetch

[100%] Getting: dists/gutsy-backports/main/source/Release #

dists/gutsy-backports/multiverse/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-backports/multiverse/binary-i386/Packages.gz #

dists/gutsy-backports/multiverse/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-backports/multiverse/binary-i386/Release #

dists/gutsy-backports/multiverse/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-backports/multiverse/source/Sources.gz #

dists/gutsy-backports/multiverse/source/Release needs fetch

[100%] Getting: dists/gutsy-backports/multiverse/source/Release #

dists/gutsy-backports/universe/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-backports/universe/binary-i386/Packages.gz #

dists/gutsy-backports/universe/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-backports/universe/binary-i386/Release #

dists/gutsy-backports/universe/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-backports/universe/source/Sources.gz #

dists/gutsy-backports/universe/source/Release needs fetch

[100%] Getting: dists/gutsy-backports/universe/source/Release #

dists/gutsy-backports/restricted/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-backports/restricted/binary-i386/Packages.gz #

dists/gutsy-backports/restricted/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-backports/restricted/binary-i386/Release #

dists/gutsy-backports/restricted/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-backports/restricted/source/Sources.gz #

dists/gutsy-backports/restricted/source/Release needs fetch

[100%] Getting: dists/gutsy-backports/restricted/source/Release #

dists/gutsy-proposed/main/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-proposed/main/binary-i386/Packages.gz #

dists/gutsy-proposed/main/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-proposed/main/binary-i386/Release #

dists/gutsy-proposed/main/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-proposed/main/source/Sources.gz #

dists/gutsy-proposed/main/source/Release needs fetch

[100%] Getting: dists/gutsy-proposed/main/source/Release #

dists/gutsy-proposed/multiverse/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-proposed/multiverse/binary-i386/Packages.gz #

dists/gutsy-proposed/multiverse/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-proposed/multiverse/binary-i386/Release #

dists/gutsy-proposed/multiverse/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-proposed/multiverse/source/Sources.gz #

dists/gutsy-proposed/multiverse/source/Release needs fetch

[100%] Getting: dists/gutsy-proposed/multiverse/source/Release #

dists/gutsy-proposed/universe/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-proposed/universe/binary-i386/Packages.gz #

dists/gutsy-proposed/universe/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-proposed/universe/binary-i386/Release #

dists/gutsy-proposed/universe/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-proposed/universe/source/Sources.gz #

dists/gutsy-proposed/universe/source/Release needs fetch

[100%] Getting: dists/gutsy-proposed/universe/source/Release #

dists/gutsy-proposed/restricted/binary-i386/Packages.gz needs fetch

[100%] Getting: dists/gutsy-proposed/restricted/binary-i386/Packages.gz #

dists/gutsy-proposed/restricted/binary-i386/Release needs fetch

[100%] Getting: dists/gutsy-proposed/restricted/binary-i386/Release #

dists/gutsy-proposed/restricted/source/Sources.gz needs fetch

[100%] Getting: dists/gutsy-proposed/restricted/source/Sources.gz #

dists/gutsy-proposed/restricted/source/Release needs fetch

[100%] Getting: dists/gutsy-proposed/restricted/source/Release #

Parse Packages and Sources files and add to the file list everything therein.

Download all files that we need to get (49 MiB).

Downloaded 49 MiB in 106s at 467.1 kiB/s

Everything OK. Moving meta files.

Cleanup mirror.

All done.

root@ubuntu710:/var/www# exit

Script effettuato su sab 01 set 2007 20:48:12 CEST

Thank You

[Jason Straight (jason-jeetkunedomaster)]

I'm guessing this is why when I upgraded to gutsy last night and my mirror script ran from cron it wiped out my entire mirror of gutsy and feisty?

dists and project are there, but no pool.

[marcolino (mas-marco)]

I think I've found a solution.

I have changed line 793 of /usr/bin/debmirror from :
local $/="\n\n";
to :
local $/="\n";
Bye

[Pär Lindfors (paran)]

Confirmed. debmirror is broken in gutsy and won't download packages.

marcolinos solution worked for me as well.

[Vadim Gusev (zarbis)]

Does this change in /usr/bin/debmirror affects on feisty repository, won't it do something bad with it? I have to do feisty and gutsy repositiry at the same time.
Thanks for solution.

[marcolino (mas-marco)]

I think the matter is with Perl's libraries that handles gzipped files. And yes, whitout this change your pool directory will wiped out (feisty, gutsy, dapper and so on).
Bye

[gutocarvalho (gutocarvalho)]

some problem here... debmirror just get release files and clean any package on the existing mirror.... i lost my feisty repository :(

debmirror -v -p --arch=i386 --host=archive.ubuntu.com --root=ubuntu --section=main,multiverse,restricted,universe --dist=gutsy,gutsy-updates,gutsy-security,gutsy-backports,gutsy-proposed -e http --ignore-release-gpg --nosource --progress /storage/mirror/distros/ubuntu/gutsy

i'll try the marcolino solution...

[]'s
Guto

[gutocarvalho (gutocarvalho)]

marcolinos solution worked, tks...

[Jason Straight (jason-jeetkunedomaster)]

Sorry, but isn't it pathetic this hasn't been fixed yet?

[Malcolm Scott (malcscott)]

This is in Debian as http://bugs.debian.org/435656 -- their solution involved fixing libcompress-zlib-perl.

[Carlos Parra Camargo (carlospc)]

This is very important for Ubuntu derivatives. I'm working on Guadalinex and we have done a workaround (similar to marcolinos).

As long as this problem could erase a lot of GBs the Importance should be Medium (i recommend High).

[Cesare Tirabassi (norsetto)]

1. A statement explaining the impact

Setting up a mirror with debmirror doesn't work.
Updating a mirror with debmirror causes the whole mirror to be deleted.

This causes major disruptions to users, especially those working on Ubuntu derivatives.
The fix has been identified in Debian (see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=435656) and implemented in the package libcompress-zlib-perl 2.005-2, corrected further in version 2.005-3.

2. How the bug has been addressed

The bug is address in hardy by version 2.007-1 which contains the Debian fix applied upstream (since version 2.006-1).
This bug can be fixed by including the patch 02_restore-gzreadline-record-separator-behaviour.patch from the Debian package version 2.005-3
The same patch has been applied upstream since version 2.006-1, see http://search.cpan.org/src/PMQS/Compress-Zlib-2.007/lib/Compress/Zlib.pm lines 183-188.

3. Patch

A patch for gutsy-proposed is attached

4. Instructions to reproduce

The following command:

debmirror -m --passive --host=archive.ubuntulinux.org --root=ubuntu/ --method=ftp --progress --dist=gutsy --section=multiverse --arch=i386 ubuntu/ --ignore-release-gpg

Produces this output:

Mirroring to ubuntu/ from ftp://<email address hidden>/ubuntu//
Arches: i386
Dists: gutsy
Sections: multiverse
Including source.
Passive mode on.
Checking md5sums.
Will clean up AFTER mirroring.
Pdiff mode: use.
Attempting to get lock, this might take 2 minutes before it fails.
Get Release files.
[0%] Getting: dists/gutsy/Release #
[0%] Getting: dists/gutsy/Release.gpg #
Get Packages and Sources files and other miscellany.
dists/gutsy/multiverse/binary-i386/Packages.gz needs fetch
[ 4%] Getting: dists/gutsy/multiverse/binary-i386/Packages.gz ###
dists/gutsy/multiverse/binary-i386/Release needs fetch
[ 75%] Getting: dists/gutsy/multiverse/binary-i386/Release #
dists/gutsy/multiverse/source/Sources.gz needs fetch
[ 75%] Getting: dists/gutsy/multiverse/source/Sources.gz #
dists/gutsy/multiverse/source/Release needs fetch
[100%] Getting: dists/gutsy/multiverse/source/Release #
Parse Packages and Sources files and add to the file list everything therein.
Download all files that we need to get (2 MiB).
Downloaded 2 MiB in 11s at 135.97 kiB/s
Everything OK. Moving meta files.
Cleanup mirror.
All done.

And no packages are installed:

cesare@norsetto:~/libcompress-zlib-perl$ ll ubuntu/dists/gutsy/multiverse/binary-i386
total 1080
-rw-r--r-- 2 cesare cesare 717447 2007-10-31 16:19 Packages
-rw-r--r-- 2 cesare cesare 158201 2007-10-31 16:19 Packages.bz2
-rw-r--r-- 2 cesare cesare 205699 2007-10-18 13:20 Packages.gz
-rw-r--r-- 2 cesare cesare 99 2007-10-18 13:27 Release

After installing the corrected version of libcompress-zlib-perl:

cesare@norsetto:~/libcompress-zlib-perl$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages have been kept back:
  xserver-xorg-video-ati
The following packages will be upgraded:
  libcompress-zlib-perl
1 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 0B/34.6kB of archives.
After unpacking 0B of a...

[Cesare Tirabassi (norsetto)]

Under 4 above I should have reported in particular that the pool directory is missing.
This is solved by the above patch (see attached log).

[Martin Pitt (pitti)]

Fixed in Hardy.

[Martin Pitt (pitti)]

Patch approved for gutsy-proposed.

[Martin Pitt (pitti)]

Sponsored and accepted into gutsy-proposed.

[KeenEars (fl-blade)]

Will there be a binary file ? Alas I`m (and maybe some other too) don`t morroring the sources...

[Martin Pitt (pitti)]

It is built now.

[KeenEars (fl-blade)]

Thank you, that`s what was needed....

[aliet (alietss)]

Maybe I'm turning crazy but I remember to have updated my debmirror from proposed but now it is not in proposed or updates, what is going on with this???

[Cesare Tirabassi (norsetto)]

Its not debmirror which is in -proposed (at least, not because of this bug). It is libcompress-zlib-perl

[William Grant (wgrant)]

The fix works for me. The broken version ate most of my feisty/gutsy/hardy mirror before I noticed what it was doing. It'd be really nice to get this fixed, and it seems to now have the requisite two acks.

[Cesare Tirabassi (norsetto)]

Since this is in main I'm afraid we still need to wait for ~sru-verification (its almost a month that this is in -proposed!)

[Martin Pitt (pitti)]

I tested it successfully, too. Considering verification-done, will move it later today.

[Martin Pitt (pitti)]

Copied to gutsy-updates.

[Del Griffith (delgriffith-deactivatedaccount)]

When is this going to be available in Gutsy updates for everyone to download, please?

[Martin Pitt (pitti)]

Eww, thanks for the ping. Something went wrong with copying the packages, gutsy-updates only has the source.

[Martin Pitt (pitti)]

 libcompress-zlib-perl (2.005-1ubuntu0.2) gutsy-updates; urgency=low
 .
   * No-change upload of the -proposed version to -updates. For some reason
     gutsy-updates only got the source package copied. (LP: #136634)

[Martin Pitt (pitti)]

libcompress-zlib-perl (2.005-1ubuntu0.2) gutsy-updates; urgency=low

  * No-change upload of the -proposed version to -updates. For some reason
    gutsy-updates only got the source package copied. (LP: #136634)

libcompress-zlib-perl (2.005-1ubuntu0.1) gutsy-proposed; urgency=low

  * Make debmirror work again instead of dropping the entire contents of pool/
    and not downloading anything:
    Add patch 02_restore-gzreadline-record-separator-behaviour.patch:
    Hard-codes "\n" as record separator (do not mangle with the *global* $/,
    but set it to "\n" locally) before calling $gz->getline for the
    actual read. (LP: #136634)

 -- Martin Pitt <email address hidden> Wed, 05 Dec 2007 09:21:52 +0100