apt repository disk format has race conditions

[Robert Collins]

Apt archives are accessed over HTTP; this has resulted in a cluster of bugs (reported here, and upstream) about problems behind intercepting caches, problems with squid etc.

There are 3 interlocking issues:
A - mirror networks may be out of sync with each other (e.g. a file named on one mirror may no longer exist, or may not yet exist, on another mirror)
B - updating files on a single mirror is not atomic - and even small windows of inconsistency will, given enough clients, cause headaches.
C - caches exacerbate race conditions - when one happens, until the cached data expires, all clients of the cache will suffer from the race

Solving this requires one of several things:
 - file system transactions
 - an archive format that requires only weakly ordered updates to the files at particular urls with the assumption that only one file may be observed to change at a time (because a lookup of file A, then B, may get a cache miss on A and a cache hit on B, so even if all clients strictly go A, then B, updates may still see old files when paths are reused).
 - super robust clients that repeatedly retry with progressively less cache friendly headers until they have a consistent view. (This is very tricky to do).

It may be possible to do a tweak to the apt repository format though, which would allow publishing a race-free format in parallel with the existing layout, while clients migrate. To be safe against issue (A) the mirror network would need some care around handling of dns round-robin mirrors [to minimise the situation where referenced data is not available], but this should be doable - or alternatively clients doing 'apt-get update' may need to be willing to retry to accommodate round-robin skew.

What would such an archive format look like?
It would have only one well known file name (InRelease), which would be internally signed. Rather than signing e.g. Packages.gz, it would sign a uniquely named packages and sources file - e.g. Packages-$HASH.gz or Packages-$serialno.gz.

Backwards compatibility is achieved by using the same filenames for deb's and the like. We need to keep writing Packages.gz though, and Releases, until we no longer worry about old apt clients. We can optimise disk space a little by making Packages.gz a symlink to a Packages-$HASH.gz (and so on for Sources..), but it may be simpler and less prone to unexpected behaviour to keep using regular files.

tl;dr
 * Unique file names for all unique file content with one exception
 * InRelease, a self-signed file that provides hashes and names the index files (Packages, Sources, Translations etc)
 * Coexists with existing archive layout

Related bugs:
 * bug 804252: Please support InRelease files
 * bug 1430011: support apt by-hash mirrors

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robie Basak (racb)]

A related bug is bug 214612 (pdiff support in the repository).

[Scott Moser (smoser)]

This is being discussed at: http://lists.debian.org/debian-dak/2012/05/threads.html#00006

[David Kalnischkies (donkult)]

In the meantime - as it is fairly obvious that this will not be done instantly and unlikely to be in time for q-freezes - you might want to explore solutions similar to bittorrent which are available today with apt-p2p or apt-transport-debtorrent (as these are by definition hash based) or setup a service like http.debian.net which in your case would only redirect to fully updated mirrors -- which might even speeds up your updates on the machine if it redirects the same client to different mirrors (to improve this further you might like to backport deb#668111 for apt).

Either doesn't solve all problems, but should be able to fix at least a few for now.

[Johannes Martin (jmartin-notamusica)]

The following does not solve the problem, but limits the frequency of the problems when using squid as a proxy. I added the following two lines to the squid configuration:
---
acl DEBIAN_NOCACHE urlpath_regex Release Packages.bz2 Translation.*.bz2
cache deny DEBIAN_NOCACHE
---

This prevents the files that change frequently from being cached by squid. As they are relatively small, the impact on the bandwidth use is not that bad.

It does not solve the problem with mirrors being temporarily out of sync or the race condition in file access.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Reis (kiko)]

We had a report of a MAAS user yesterday with this issue affecting some of his deployments. He was not behind a proxy. He retried the deployment and it worked, but we'd like to avoid sporadic failures as much as possible so this would be nice to see addressed.

[Michael Vogt (mvo)]

Fwiw, we have support for the by-hash scheme in apt in experimetnal:
https://github.com/Debian/apt/blob/debian/experimental/apt-pkg/acquire-item.cc#L1222

[Scott Moser (smoser)]

Per mvo, there is apparently by-hash support in debian experimental with some references seen at
 https://github.com/Debian/apt/blob/debian/experimental/apt-pkg/acquire-item.cc#L1222

[Scott Moser (smoser)]

relevant changelog in debian:
   * Implement simple by-hash for apt update to improve reliability of
     the update. Apt will try to fetch the Packages file via
     /by-hash/$hash_type/$hash_value if the repo supports that.
     - add APT::Acquire::$(host)::By-Hash=1 knob
     - add Acquire-By-Hash=1 to Release file

[Scott Moser (smoser)]

I opened bug 1430011 as a request for launchpad to gain the ability to create /populate by-hash mirrors.

[Christian Brandt (brandtc)]

Just some rough observations, I am using a visible Proxy configures through /etc/apt/ and also HTTP_PROXY running on "http://proxy:3128/" since the early 1990ths. The Hash mismatch rarelly happened to me until I upgraded to Ubuntu 14.04 and since then I have it a lot on de.archive.ubuntu.com but still rarelly on eu/gb/nl mirrors. On most mirrors a quick fix is to clean /var/lib/apt/, /var/cache/squid and so on but on de mirror this seems to do nothing until the bug simply disappears after some days even without any interception from me.

I have the odd feeling that the de mirror is just exceptional erratic.

Why am I using a caching proxy? Because it speeds up updating and installing while using multiple clients (once had more than 100 after one 2MBit line, right now still five). There is little need to cache lists, indexes, hashes, keys and so on, I am only interested in caching the deb Files so maybe there is some workaround to ease the strain on the proxy?

[Dapxitlo (nhducit)]

affect to me too

[Dapxitlo (nhducit)]

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/source/Sources Hash Sum mismatch

W: Failed to fetch http://ppa.launchpad.net/ubuntu-vn/ppa/ubuntu/dists/trusty/main/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://ppa.launchpad.net/ubuntu-vn/ppa/ubuntu/dists/trusty/main/binary-i386/Packages 404 Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.

[quequotion (quequotion)]

Is there any aspect of deb package management that isn't broken?

[sunil (suneel)]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-updates/main/source/Sources 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-updates/restricted/source/Sources 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-updates/universe/source/Sources 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-updates/multiverse/source/Sources 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-updates/main/binary-amd64/Packages 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-updates/restricted/binary-amd64/Packages 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-updates/universe/binary-amd64/Packages 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-updates/multiverse/binary-amd64/Packages 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-backports/main/source/Sources 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-backports/restricted/source/Sources 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-backports/universe/source/Sources 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-backports/multiverse/source/Sources 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-backports/main/binary-amd64/Packages 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-backports/restricted/binary-amd64/Packages 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-backports/universe/binary-amd64/Packages 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://mirrors.linode.com/ubuntu/dists/utopic-backports/multiverse/binary-amd64/Packages 404 Not Found [IP: 2600:3c01:1::607e:6379 80]

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/utopic-security/main/source/Sources 404 Not Found [IP: 2001:67c:1562::17 80]

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/utopic-security/restricted/source/Sources 404 Not Found [IP: 2001:67c:1562::17 80]

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/utopic-security/universe/source/Sources 404 Not Found [IP: 2001:67c:1562::17 80]

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/utopic-security/multiverse/source/Sources 404 Not Found [IP: 2001:67c:1562::17 80]

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/utopic-security/main/binary-amd64/Packages 404 Not Found [IP: 2001:67c:1562::17 80]

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/utopic-security...

[Francois Leurent (fleurent)]

Instead of whipping /var/lib/apt/list, i found out that touch -t 1501010000 /var/lib/apt/lists/* (anytime in the past) is also sufficient

[michael jones (bcmalloy)]

mike@mike-desktop ~ $ sudo apt-get update
[sudo] password for mike:
Ign http://mirror.internode.on.net rosa InRelease
Ign http://mirror.optus.net trusty InRelease
Hit http://mirror.internode.on.net rosa Release.gpg
Hit http://mirror.optus.net trusty-updates InRelease
Hit http://mirror.internode.on.net rosa Release
Hit http://mirror.optus.net trusty Release.gpg
Hit http://mirror.internode.on.net rosa/main amd64 Packages
Hit http://mirror.internode.on.net rosa/upstream amd64 Packages
Hit http://mirror.internode.on.net rosa/import amd64 Packages
Hit http://mirror.optus.net trusty-updates/restricted amd64 Packages
Ign http://extra.linuxmint.com rosa InRelease
Hit http://mirror.internode.on.net rosa/main i386 Packages
Hit http://mirror.optus.net trusty-updates/universe amd64 Packages
Get:1 http://mirror.optus.net trusty-updates/main amd64 Packages [683 kB]
Hit http://mirror.internode.on.net rosa/upstream i386 Packages
Hit http://mirror.optus.net trusty-updates/multiverse amd64 Packages
Ign http://archive.canonical.com trusty InRelease
Hit http://mirror.internode.on.net rosa/import i386 Packages
Hit http://extra.linuxmint.com rosa Release.gpg
Hit http://mirror.optus.net trusty-updates/restricted i386 Packages
Hit http://mirror.optus.net trusty-updates/universe i386 Packages
Hit http://mirror.optus.net trusty-updates/multiverse i386 Packages
Hit http://mirror.optus.net trusty-updates/main Translation-en
Hit http://mirror.optus.net trusty-updates/multiverse Translation-en
Hit http://mirror.optus.net trusty-updates/restricted Translation-en
Hit http://extra.linuxmint.com rosa Release
Hit http://mirror.optus.net trusty-updates/universe Translation-en
Hit http://mirror.optus.net trusty Release
Get:2 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Hit http://mirror.optus.net trusty/restricted amd64 Packages
Get:3 http://mirror.optus.net trusty-updates/main i386 Packages [659 kB]
Hit http://mirror.optus.net trusty/multiverse amd64 Packages
Hit http://extra.linuxmint.com rosa/main amd64 Packages
Hit http://archive.canonical.com trusty Release.gpg
Hit http://mirror.optus.net trusty/restricted i386 Packages
Hit http://archive.canonical.com trusty Release
Hit http://mirror.optus.net trusty/multiverse i386 Packages
Hit http://mirror.optus.net trusty/main amd64 Packages
Hit http://mirror.optus.net trusty/main Translation-en_AU
Get:4 http://security.ubuntu.com trusty-security/main amd64 Packages [428 kB...

[Scott Moser (smoser)]

Given the availability of by-hash, this is not really 'apt's problem any more.
The changes left to be forever rid of 'hash sum mismatch' is to fix bug 1430011

[Abhijith Gururaj (abhiithgururaj)]

Hi I am still facing this problem . here is my console output:

sudo apt-get upgrade --fix-missing
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
gir1.2-javascriptcoregtk-4.0 gir1.2-webkit2-4.0 libjavascriptcoregtk-
4.0-18 libwebkit2gtk-4.0-37
libwebkit2gtk-4.0-37-gtk2
5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 8,650 kB/23.3 MB of archives.
After this operation, 3,997 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ubuntu.ipserverone.com/ubuntu xenial-updates/main amd64
libwebkit2gtk-4.0-37-gtk2 amd64 2.16.3-0ubuntu0.16.04.1 [8,650 kB]
Err:1 http://ubuntu.ipserverone.com/ubuntu xenial-security/main amd64
libwebkit2gtk-4.0-37-gtk2 amd64 2.16.3-0ubuntu0.16.04.1
Hash Sum mismatch
Get:1 http://ubuntu.ipserverone.com/ubuntu xenial-security/main amd64
libwebkit2gtk-4.0-37-gtk2 amd64 2.16.3-0ubuntu0.16.04.1 [8,650 kB]
Err:1 http://ubuntu.ipserverone.com/ubuntu xenial-security/main amd64
libwebkit2gtk-4.0-37-gtk2 amd64 2.16.3-0ubuntu0.16.04.1
Hash Sum mismatch
Fetched 17.3 MB in 13s (1,266 kB/s)

E: Failed to fetch http://ubuntu.ipserverone.com/ubuntu/pool/main/w/webkit2gtk/libwebkit2gtk-4.0-37-gtk2_2.16.3-0ubuntu0.16.04.1_amd64.deb Hash Sum mismatch

How to fix this?

[Julian Andres Klode (juliank)]

Switch to a different mirror. Hash sum mismatches on .deb file should not happen on any well-built mirror.

[Abhijith Gururaj (abhiithgururaj)]

@juliank I have configured the updater to use the main server. Downloading updates from the main server shouldn't cause such errors, right?

[Robie Basak (racb)]

On Thu, Jun 01, 2017 at 07:20:18AM -0000, Abhijith Gururaj wrote:
> @juliank I have configured the updater to use the main server.
> Downloading updates from the main server shouldn't cause such errors,
> right?

Some ISPs intercept their clients' downloads. The error is telling you
that the download appears to have been modified in transit, so isn't
safe to install. It cannot tell you why or where that happened,
unfortunately.

This bug was to track a known case of an index file appearing to have
been modified in transit due to a known race condition. That issue was
fixed, and it seems unlikely to me that you are affected by that
particular issue as your mismatch is on deb and not an index file.

If you need help, you're more likely to get it from a place dedicated to
that, since few people follow this particular bug. See
http://www.ubuntu.com/support/community for community support options.

[the835t (the835t)]

affects me too