support apt by-hash mirrors

[Scott Moser]

as described in bug 972077, apt's filesystem layout format is inherently racey. apt version 1.1 in debian/experimental now has support for downloading by hash rather than by filename, which reduces/removes the race conditions.

Ideally we'd have this in place for 15.10 and 16.04 in ubuntu.

relevant changelog in debian:
   * Implement simple by-hash for apt update to improve reliability of
     the update. Apt will try to fetch the Packages file via
     /by-hash/$hash_type/$hash_value if the repo supports that.
     - add APT::Acquire::$(host)::By-Hash=1 knob
     - add Acquire-By-Hash=1 to Release file

Also in this completely, we do need to have InRelease file support also.

Related bugs:
 * bug 804252: Please support InRelease files
 * bug 972077: apt repository disk format has race conditions

[Launchpad QA Bot (lpqabot)]

r13370 in db-stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-stable/revision/13370) is part of this bug's fix.

[Colin Watson (cjwatson)]

2016-03-18 17:41:04,832 INFO 2209-74-0 applied just now in 0.3 seconds

[Launchpad QA Bot (lpqabot)]

r17959 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/17959) is part of this bug's fix.

[Launchpad QA Bot (lpqabot)]

r17960 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/17960) is part of this bug's fix.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r17975 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/17975>.

[Colin Watson (cjwatson)]

The necessary code is all rolled out now and we've started generating the by-hash index files for xenial, but intentionally without the Acquire-By-Hash flag in Release for now so it requires client-side forcing to make use of it. We'll make sure this is behaving itself and then add the flag to Release.

[Dean Henrichsmeyer (dean)]

Thank you!

[Colin Watson (cjwatson)]

It is done!

  $ curl -s http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease | grep Acquire-By-Hash
  Acquire-By-Hash: yes

  http://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html

[Samuel Leslie (sdl)]

Colin Watson: Really great and super informative blog post, thank-you! Excellent to hear that this has been fixed in the latest release.

Is there any likelihood of the relevant changes being backported to older releases? In particular, the 14.04 LTS release? We see the "Hash Sum Mismatch" issue *very* frequently in AWS environments. I strongly suspect AWS EC2 mirrors aren't doing the extra work to ensure they're not distributing mismatched repository metadata when they themselves sync with upstream mirrors, and so are distributing mismatched data to clients potentially for extended periods of time. For those of us on LTS releases, it'd be fantastic to see this backported so we're not forced to upgrade to 16.04 to benefit from this fix. 12.04 LTS is probably not really worthwhile.

[Colin Watson (cjwatson)]

I'd have to defer to people who know apt better, but I'm afraid it's
very unlikely for this to be backported; the changes to apt are AIUI
quite extensive and it's a critical part of the system that mustn't
fail.

I've made a note to chase up the mirroring scripts used for the AWS EC2
mirrors to see if there's anything easy that can be done there to at
least improve the situation a bit.

[Olorunlona Juwon Enoch (juwonlona)]

Hello everyone. I am a newbie to the Linux community and I've been trying to update/upgrade my Xenial (16.04) distro all to no avail. I've read several posts on this issue and landing on this page has finally made me decided that my 'Unable to fetch archives' error is certainly a bug. I'm from Nigeria and every time I try the apt-get update, apt-get dist-upgrade etc features, I change to the best server and again to main server but it just seem to be wasting data giving me the same kind of error even after purge and clean commands. Please, if a baackport is all that will work please I need one and a full guide to getting this error eradicated using this backport. Thanks

[Robie Basak (racb)]

On Thu, Feb 02, 2017 at 12:10:06PM -0000, Olorunlona Juwon Enoch wrote:
> Hello everyone. I am a newbie to the Linux community and I've been
> trying to update/upgrade my Xenial (16.04) distro all to no avail. I've
> read several posts on this issue and landing on this page has finally
> made me decided that my 'Unable to fetch archives' error is certainly a
> bug. I'm from Nigeria and every time I try the apt-get update, apt-get
> dist-upgrade etc features, I change to the best server and again to main
> server but it just seem to be wasting data giving me the same kind of
> error even after purge and clean commands. Please, if a baackport is all
> that will work please I need one and a full guide to getting this error
> eradicated using this backport. Thanks

I believe this bug is fixed in Xenial. So if you're having an issue, it
is likely something else. Perhaps your Internet connection is altering
or corrupting downloads?

[Olorunlona Juwon Enoch (juwonlona)]

No Robie, I meant to write; after the command:

curl -s http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease | grep Acquire-By-Hash

should i just go through the normal processes and I get a solution??? Sorry for the redundant question... Thanks in advance Robie...

[Olorunlona Juwon Enoch (juwonlona)]

This is what I did: curl -s http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease | grep Acquire-By-Hash

After running this command: apt -o Debug::Acquire::http=true update

after a long list of Gets, I got:

Fetched 3,252 kB in 5min 57s (9,097 B/s)
Reading package lists... Done
E: Failed to fetch http://de2.archive.ubuntu.com/ubuntu/dists/xenial-security/universe/dep11/icons-64x64.tar.gz Hash Sum mismatch
E: Failed to fetch http://de2.archive.ubuntu.com/ubuntu/dists/xenial-updates/main/binary-i386/Packages.gz
E: Failed to fetch http://de2.archive.ubuntu.com/ubuntu/dists/xenial-updates/main/dep11/icons-64x64.tar.gz Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

[Olorunlona Juwon Enoch (juwonlona)]

FULL TERMINAL DISPLAY:

sudo apt-get -oDebug::Acquire::http=true update
0% [Connecting to de2.archive.ubuntu.com (91.189.88.152)] [Connecting to ppa.laGET /ubuntu/dists/xenial/InRelease HTTP/1.1
Host: de2.archive.ubuntu.com
Cache-Control: max-age=0
Accept: text/*
If-Modified-Since: Thu, 21 Apr 2016 23:24:48 GMT
User-Agent: Debian APT-HTTP/1.3 (1.2.12)

GET /graphics-drivers/ppa/ubuntu/dists/xenial/InRelease HTTP/1.1
Host: ppa.launchpad.net
Cache-Control: max-age=0
Accept: text/*
If-Modified-Since: Mon, 30 Jan 2017 13:07:37 GMT
User-Agent: Debian APT-HTTP/1.3 (1.2.12)

0% [Waiting for headers] [Waiting for headers]Answer for: http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu/dists/xenial/InRelease
HTTP/1.1 200 OK
Date: Thu, 02 Feb 2017 14:11:31 GMT
Server: Apache
Last-Modified: Mon, 30 Jan 2017 13:07:37 GMT
ETag: "5870f26-4693-5474f7fe8a284"
Accept-Ranges: bytes
Content-Length: 18067
Cache-Control: max-age=0, s-maxage=270, proxy-revalidate
Expires: Thu, 02 Feb 2017 14:11:31 GMT
Connection: keep-alive

Get:1 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu xenial InRelease [18.1 kB]
0% [Waiting for headers] [1 InRelease 10.7 kB/18.1 kB 59%]Answer for: http://de2.archive.ubuntu.com/ubuntu/dists/xenial/InRelease
HTTP/1.1 200 OK
Date: Thu, 02 Feb 2017 14:11:31 GMT
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Thu, 21 Apr 2016 23:24:48 GMT
ETag: "3c43e-53107042ad000"
Accept-Ranges: bytes
Content-Length: 246846
Cache-Control: max-age=0, proxy-revalidate
Expires: Thu, 02 Feb 2017 14:11:31 GMT
Connection: keep-alive

Get:2 http://de2.archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
0% [2 InRelease 1,043 B/247 kB 0%] [1 InRelease 12.1 kB/18.1 kB 67%]GET /umang/indicator-stickynotes/ubuntu/dists/xenial/InRelease HTTP/1.1
Host: ppa.launchpad.net
Cache-Control: max-age=0
Accept: text/*
If-Modified-Since: Sat, 23 Jul 2016 17:44:34 GMT
User-Agent: Debian APT-HTTP/1.3 (1.2.12)

0% [2 InRelease 20.9 kB/247 kB 8%] [Waiting for headers]Answer for: http://ppa.launchpad.net/umang/indicator-stickynotes/ubuntu/dists/xenial/InRelease
HTTP/1.1 200 OK
Date: Thu, 02 Feb 2017 14:11:33 GMT
Server: Apache
Last-Modified: Sat, 23 Jul 2016 17:44:34 GMT
ETag: "3600557-4494-538511ab0a43b"
Accept-Ranges: bytes
Content-Length: 17556
Cache-Control: max-age=0, s-maxage=270, proxy-revalidate
Expires: Thu, 02 Feb 2017 14:11:33 GMT
Connection: keep-alive

Get:3 http://ppa.launchpad.net/umang/indicator-stickynotes/ubuntu xenial InRelease [17.6 kB]
0% [2 InRelease 225 kB/247 kB 91%] 42.5 kB/s 0sGET /ubuntu/dists/xenial-security/InRelease HTTP/1.1
Host: de2.archive.ubuntu.com
Cache-Control: max-age=0
Accept: text/*
Range: bytes=102147-
If-Range: Thu, 02 Feb 2017 13:29:00 GMT
User-Agent: Debian APT-HTTP/1.3 (1.2.12)

0% [Waiting for headers] 42.5 kB/s 0sAnswer for: http://de2.archive.ubuntu.com/ubuntu/dists/xenial-security/InRelease
HTTP/1.1 200 OK
Date: Thu, 02 Feb 2017 14:11:35 GMT
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Thu, 02 Feb 2017 13:49:00 GMT
ETag: "18f03-5478c6d66cb00"
Accept-Ranges: bytes
Content-Length: 102147
Cache-Control: max-age=444, proxy-revalidate
Expires: Thu, 02 Feb 2017 14...