RFD: better handling of the Packages files

[Rolf Leggewie]

Just found the following in the log:

Checking/Updating mirrordirector.raspbian.org/raspbian/dists/jessie/main/binary-armhf/Packages.gz... (12162KiB)
Checking/Updating mirrordirector.raspbian.org/raspbian/dists/jessie/main/binary-armhf/Packages.xz... (8756KiB)

I guess that once two versions of Packages{,.gz,.bz2,.xz} have made it to acng they will stay there basically forever and their information being downloaded several times. That's a waste of bandwidth.

I believe these days compressors rank from best to worst (purely in terms of compressed size) as xz < bz2 < gz < uncompressed. Furthermore, I assume that serving the uncompressed file to local clients as a fallback should always work.

Would it be possible that as a matter of policy acng tries to download the best compressed file first and if that is 404 goes through xz, bz2, gz and then uncompressed in that order and stores only that Packages file? If a client in the future requests Packages.bz2 and that is indeed available, then this is served, otherwise, fallback to Packages (which acng can get from the compressed file without any issues, recompressing with a different algorithm would possibly result in a different hashsum so I do not propose this). Any reason this would not work?

PS: if this is workable, acng might want to cache for a few weeks the information that xz is not but bz2 is available for a certain Packages file. This is unlikely to change frequently and going straight to downloading bz2 next time around will lessen the number of connections to open.

[Eduard Bloch (edi-gmx)]

Ehm... this is basically what the smart update code has been doing for the most part of the last five years.

It has been broken for Debian Sid/Experimental archives since about March of this year, and the version 0.9.3 (available since a about a week) is supposed to solve this. There will be more usability improvements in 0.9.4, coming soon.

The main reason for "breakage" was that its main information source is content of InRelease/Release files and archive started boxing out the SHA1 versions of some data which acng used exclusively for data model analysis.

[Rolf Leggewie (r0lf)]

I'm not exactly sure how the smart update code works. But if it's been present for 5 years already I'd like to ask you how I was able to reproduce this on a jessie Raspbian box the other day while doing an import. Both mirrordirector.raspbian.org/raspbian/dists/jessie/main/binary-armhf/Packages.[gx]z were fetched. This is not a sid/experimental repo so your comment about breakage there should not apply.

[Rolf Leggewie (r0lf)]

confirming this works as expected on Ubuntu Jammy, at least