indices/md5sums.gz doesn't match repository

[Carlos Carvalho]

indices/md5sums.gz has many entries that don't exist in the repository. In fact it has almost twice entries. Further, it doesn't list many files that do exist in the repository. This poses two problems. The more important one is that we cannot use it to verify the integrity of files in our mirror. We normally verify the checksums twice a year; without md5sums.gz we're forced to use rsync -c, which is very expensive for the server as well.

The second problem is that it cannot be used as a reliable list of files in the archive when updating, which forces us to use a normal rsync recursive scan. If md5sums.gz was correct, as in Debian, we could avoid the expensive disk scan at the server, reducing the load on it.

[Colin Watson (cjwatson)]

indices/md5sums.gz hasn't been generated at all since the middle of last year, since the new publisher code dropped the section that runs dsync-flist. Digging around in #launchpad-dev logs for 2011-08-17:

11:43 <jtv> (Also note that dsync isn't currently used.)
11:48 <cjwatson> jtv: dsync> that's a shame
11:49 <cjwatson> jtv: what generates the md5sums index then?
11:49 <cjwatson> lp_archive@cocoplum:~$ ls -l /srv/launchpad.net/ubuntu-archive/ubuntu/indices/md5sums.gz
11:49 <cjwatson> -rw-r--r-- 1 lp_publish lp_publish 29926782 Aug 8 10:23 /srv/launchpad.net/ubuntu-archive/ubuntu/indices/md5sums.gz
11:49 <cjwatson> that looks unfortunate
11:49 <wgrant> cjwatson: It was generated by dsync.
11:49 <cjwatson> (I'm fairly sure that there exist mirroring tools which look at indices/md5sums.gz)
11:49 <wgrant> Which we should really running again.
11:49 <cjwatson> wgrant: yes, that's my point :)

Nowadays, the various Ubuntu-specific publishing hooks have been moved out to a separate ubuntu-archive-publishing project, so I'm reassigning this bug there since it seems likely to me that any attempt to resurrect this would want to be done in those hooks rather than in the generic Launchpad publisher code. It's contingent on it being possible to run this without an excessively painful performance hit.