.temp/dists/.../Packages goes out of date

[Jamie Strandboge]

I haven't had time to really go through this, but I rebuilt xenial's debmirror on trusty and when I run it against the ubuntu archive I notice that Packages files become out of date but the Packages.gz file is up to date. debmirror appears to consider the Packages file for computing what to download but puts the Packages.gz in place resulting in the mirror advertising that updates are available but giving 404s when apt tries to retrieve the debs. For example, with the recent openjdk-7 update:

$ sudo apt-get update
Ign http://debmirror trusty InRelease
Get:1 http://debmirror trusty-security InRelease [65.9 kB]
Get:2 http://debmirror trusty-updates InRelease [65.9 kB]
Hit http://debmirror trusty Release.gpg
Hit http://debmirror trusty Release
Hit http://debmirror trusty-security/main Translation-en
Hit http://debmirror trusty-security/multiverse Translation-en
Hit http://debmirror trusty-security/restricted Translation-en
Hit http://debmirror trusty-security/universe Translation-en
Hit http://debmirror trusty-security/main amd64 Packages
Hit http://debmirror trusty-security/restricted amd64 Packages
Hit http://debmirror trusty-security/universe amd64 Packages
Hit http://debmirror trusty-security/multiverse amd64 Packages
Hit http://debmirror trusty-security/main i386 Packages
Hit http://debmirror trusty-security/restricted i386 Packages
Hit http://debmirror trusty-security/universe i386 Packages
Hit http://debmirror trusty-updates/main Translation-en
Hit http://debmirror trusty-updates/multiverse Translation-en
Hit http://debmirror trusty-updates/restricted Translation-en
Hit http://debmirror trusty-updates/universe Translation-en
Hit http://debmirror trusty-security/multiverse i386 Packages
Get:3 http://debmirror trusty-updates/main amd64 Packages [931 kB]
Hit http://debmirror trusty-updates/restricted amd64 Packages
Get:4 http://debmirror trusty-updates/universe amd64 Packages [460 kB]
Hit http://debmirror trusty-updates/multiverse amd64 Packages
Get:5 http://debmirror trusty-updates/main i386 Packages [895 kB]
Hit http://debmirror trusty-updates/restricted i386 Packages
Get:6 http://debmirror trusty-updates/universe i386 Packages [462 kB]
Hit http://debmirror trusty-updates/multiverse i386 Packages
Hit http://debmirror trusty/main amd64 Packages
Hit http://debmirror trusty/restricted amd64 Packages
Hit http://debmirror trusty/universe amd64 Packages
Hit http://debmirror trusty/multiverse amd64 Packages
Hit http://debmirror trusty/main i386 Packages
Hit http://debmirror trusty/restricted i386 Packages
Hit http://debmirror trusty/universe i386 Packages
Hit http://debmirror trusty/multiverse i386 Packages
Ign http://debmirror trusty/main Translation-en_US
Ign http://debmirror trusty/main Translation-en
Ign http://debmirror trusty/multiverse Translation-en_US
Ign http://debmirror trusty/multiverse Translation-en
Ign http://debmirror trusty/restricted Translation-en_US
Ign http://debmirror trusty/restricted Translation-en
Ign http://debmirror trusty/universe Translation-en_US
Ign http://debmirror trusty/universe Translation-en
Fetched 2,879 kB in 2s (984 kB/s)
Reading package lists... Done

$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  icedtea-7-jre-jamvm openjdk-7-jre-headless
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 39.6 MB of archives.
After this operation, 3,072 B disk space will be freed.
Do you want to continue? [Y/n] y
Err http://debmirror/ubuntu/ trusty-updates/main icedtea-7-jre-jamvm amd64 7u95-2.6.4-0ubuntu0.14.04.2
  404 Not Found
Err http://debmirror/ubuntu/ trusty-updates/main openjdk-7-jre-headless amd64 7u95-2.6.4-0ubuntu0.14.04.2
  404 Not Found
E: Failed to fetch http://debmirror/ubuntu/pool/main/o/openjdk-7/icedtea-7-jre-jamvm_7u95-2.6.4-0ubuntu0.14.04.2_amd64.deb 404 Not Found

E: Failed to fetch http://debmirror/ubuntu/pool/main/o/openjdk-7/openjdk-7-jre-headless_7u95-2.6.4-0ubuntu0.14.04.2_amd64.deb 404 Not Found

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Removing the Packages* and Sources* files from the .temp directory allows debmirror to pull these missing debs.

My debmirror invocation:
/usr/bin/debmirror -a amd64,i386 --source -s main,restricted,universe,multiverse -h 91.189.91.15 -d precise,precise-updates,precise-security,precise-proposed,trusty,trusty-security,trusty-updates,trusty-proposed,vivid,vivid-security,vivid-updates,vivid-proposed,wily,wily-security,wily-updates,wily-proposed,xenial,xenial-security,xenial-updates,xenial-proposed -r :ubuntu -e rsync --skip-installer=precise-updates,precise-security,precise-proposed,trusty-security,trusty-updates,trusty-proposed,vivid-security,vivid-updates,vivid-proposed,wily-security,wily-updates,wily-proposed,xenial-security,xenial-updates,xenial-proposed --verbose --rsync-options=-aL --partial --ipv4 /srv/data/debmirror/ubuntu

I noticed that after removing all the Sources* and Packages* in .temp and running the debmirror command, at the end it reports deleting some, but not all Sources.xz and Packages.xz files and doesn't delete any uncompressed or .gz files. Ie, after a successful run, there are some Sources.xz and Packages.xz, a bunch of Sources.gz and Packages.gz and a bunch of Sources and Packages files remaining.

[Colin Watson (cjwatson)]

I believe I've fixed this in git. Feel free to try it out:

  https://anonscm.debian.org/cgit/collab-maint/debmirror.git/commit/?id=c0805298f041e087cdf8db5adcecba992770a423

[Launchpad Janitor (janitor)]

This bug was fixed in the package debmirror - 1:2.23ubuntu1

---------------
debmirror (1:2.23ubuntu1) xenial; urgency=medium

  * Resynchronise with Debian. Remaining changes:
    - debian/{control,rules}: Add quilt for patch management.
    - Debian bug 455082:
      + default-settings.patch: change rsync defaults.
      + silence-errors.patch: throw away find errors.
      + drop-redundant-rsync.patch: clean up logic in rsync batching.
      + check_file-return.patch: report why a file needs to sync.
    - Debian bug 576576:
      + skip-installer.patch: allow specific releases to be skipped.
    - ubuntu-settings.patch: Changed defaults to download Ubuntu, replaced
      most instances of Debian with Ubuntu in the documentation.

debmirror (1:2.23) unstable; urgency=medium

  * Fix incorrect handling of existing uncompressed index files under .temp
    (LP: #1562118).

 -- Colin Watson <email address hidden> Sun, 27 Mar 2016 12:19:36 +0100