apt repository disk format has race conditions

Bug #972077 reported by Robert Collins on 2012-04-03
312
This bug affects 69 people
Affects Status Importance Assigned to Milestone
MAAS
Undecided
Unassigned
apt (Ubuntu)
Medium
Unassigned

Bug Description

Apt archives are accessed over HTTP; this has resulted in a cluster of bugs (reported here, and upstream) about problems behind intercepting caches, problems with squid etc.

There are 3 interlocking issues:
A - mirror networks may be out of sync with each other (e.g. a file named on one mirror may no longer exist, or may not yet exist, on another mirror)
B - updating files on a single mirror is not atomic - and even small windows of inconsistency will, given enough clients, cause headaches.
C - caches exacerbate race conditions - when one happens, until the cached data expires, all clients of the cache will suffer from the race

Solving this requires one of several things:
 - file system transactions
 - an archive format that requires only weakly ordered updates to the files at particular urls with the assumption that only one file may be observed to change at a time (because a lookup of file A, then B, may get a cache miss on A and a cache hit on B, so even if all clients strictly go A, then B, updates may still see old files when paths are reused).
 - super robust clients that repeatedly retry with progressively less cache friendly headers until they have a consistent view. (This is very tricky to do).

It may be possible to do a tweak to the apt repository format though, which would allow publishing a race-free format in parallel with the existing layout, while clients migrate. To be safe against issue (A) the mirror network would need some care around handling of dns round-robin mirrors [to minimise the situation where referenced data is not available], but this should be doable - or alternatively clients doing 'apt-get update' may need to be willing to retry to accommodate round-robin skew.

What would such an archive format look like?
It would have only one well known file name (e.g. Releases-2), which would be internally signed. Rather than signing e.g. Packages.gz, it would sign a uniquely named packages and sources file - e.g. Packages-$HASH.gz or Packages-$serialno.gz.

Backwards compatibility is achieved by using the same filenames for deb's and the like. We need to keep writing Packages.gz though, and Releases, until we no longer worry about old apt clients. We can optimise disk space a little by making Packages.gz a symlink to a Packages-$HASH.gz (and so on for Sources..), but it may be simpler and less prone to unexpected behaviour to keep using regular files.

tl;dr
 * Unique file names for all unique file content with one exception
 * Releases-2, a self-signed file that provides hashes and names the index files (Packages, Sources, Translations etc)
 * Coexists with existing archive layout

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu):
status: New → Confirmed
Changed in apt (Ubuntu):
importance: Undecided → Medium
Robie Basak (racb) wrote :

A related bug is bug 214612 (pdiff support in the repository).

David Kalnischkies (donkult) wrote :

In the meantime - as it is fairly obvious that this will not be done instantly and unlikely to be in time for q-freezes - you might want to explore solutions similar to bittorrent which are available today with apt-p2p or apt-transport-debtorrent (as these are by definition hash based) or setup a service like http.debian.net which in your case would only redirect to fully updated mirrors -- which might even speeds up your updates on the machine if it redirects the same client to different mirrors (to improve this further you might like to backport deb#668111 for apt).

Either doesn't solve all problems, but should be able to fix at least a few for now.

information type: Public → Public Security
information type: Public Security → Public

The following does not solve the problem, but limits the frequency of the problems when using squid as a proxy. I added the following two lines to the squid configuration:
---
acl DEBIAN_NOCACHE urlpath_regex Release Packages.bz2 Translation.*.bz2
cache deny DEBIAN_NOCACHE
---

This prevents the files that change frequently from being cached by squid. As they are relatively small, the impact on the bandwidth use is not that bad.

It does not solve the problem with mirrors being temporarily out of sync or the race condition in file access.

yohan (yohan23) on 2015-01-17
Changed in apt (Ubuntu):
status: Confirmed → New
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu):
status: New → Confirmed
Christian Reis (kiko) wrote :

We had a report of a MAAS user yesterday with this issue affecting some of his deployments. He was not behind a proxy. He retried the deployment and it worked, but we'd like to avoid sporadic failures as much as possible so this would be nice to see addressed.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related blueprints