| 2012-04-03 03:42:07 |
Robert Collins |
bug |
|
|
added bug |
| 2012-04-03 17:35:57 |
Launchpad Janitor |
apt (Ubuntu): status |
New |
Confirmed |
|
| 2012-04-03 17:36:00 |
Robie Basak |
bug |
|
|
added subscriber Robie Basak |
| 2012-04-03 17:36:00 |
Scott Moser |
bug |
|
|
added subscriber Scott Moser |
| 2012-04-04 22:19:51 |
Brian Murray |
apt (Ubuntu): importance |
Undecided |
Medium |
|
| 2012-05-07 17:18:10 |
Mark Russell |
bug |
|
|
added subscriber Mark Russell |
| 2012-07-09 13:58:37 |
renbag |
bug |
|
|
added subscriber Renzo Bagnati |
| 2013-01-22 14:22:39 |
RedShift |
bug |
|
|
added subscriber RedShift |
| 2013-06-19 20:22:02 |
Hans Deragon |
bug |
|
|
added subscriber Hans Deragon |
| 2014-04-21 14:55:34 |
fiztlen |
bug |
|
|
added subscriber fiztlen |
| 2014-05-06 18:55:21 |
Paul Sokolovsky |
bug |
|
|
added subscriber Paul Sokolovsky |
| 2014-08-26 18:09:22 |
Alan Grosskurth |
information type |
Public |
Public Security |
|
| 2014-08-26 18:10:02 |
Alan Grosskurth |
information type |
Public Security |
Public |
|
| 2014-09-23 19:39:09 |
Ganton |
bug |
|
|
added subscriber Ganton |
| 2014-11-27 14:01:30 |
Claus Reibenstein |
bug |
|
|
added subscriber Claus Reibenstein |
| 2014-12-22 15:19:27 |
Alex Muntada |
bug |
|
|
added subscriber Alex Muntada |
| 2015-01-17 09:51:54 |
yohan |
apt (Ubuntu): status |
Confirmed |
New |
|
| 2015-01-17 13:16:27 |
Launchpad Janitor |
apt (Ubuntu): status |
New |
Confirmed |
|
| 2015-01-17 13:16:37 |
James Pharaoh |
bug |
|
|
added subscriber James Pharaoh |
| 2015-02-05 17:18:57 |
Mike Ray |
bug |
|
|
added subscriber Mike Ray |
| 2015-02-12 11:26:20 |
Christian Reis |
bug task added |
|
maas |
|
| 2015-02-18 10:51:25 |
sye bond |
bug |
|
|
added subscriber sye bond |
| 2015-02-19 05:05:20 |
Praveen |
bug |
|
|
added subscriber Praveen |
| 2015-03-09 22:21:45 |
William Grant |
bug |
|
|
added subscriber William Grant |
| 2015-03-19 21:39:10 |
Paolo Dina |
bug |
|
|
added subscriber Paolo Dina |
| 2015-06-06 02:58:35 |
Andres Rodriguez |
bug task deleted |
maas |
|
|
| 2015-08-24 17:58:36 |
cdoebbler |
bug |
|
|
added subscriber cdoebbler |
| 2015-08-30 04:49:41 |
Eliah Kagan |
bug |
|
|
added subscriber Eliah Kagan |
| 2015-09-03 16:54:12 |
Scott Moser |
description |
Apt archives are accessed over HTTP; this has resulted in a cluster of bugs (reported here, and upstream) about problems behind intercepting caches, problems with squid etc.
There are 3 interlocking issues:
A - mirror networks may be out of sync with each other (e.g. a file named on one mirror may no longer exist, or may not yet exist, on another mirror)
B - updating files on a single mirror is not atomic - and even small windows of inconsistency will, given enough clients, cause headaches.
C - caches exacerbate race conditions - when one happens, until the cached data expires, all clients of the cache will suffer from the race
Solving this requires one of several things:
- file system transactions
- an archive format that requires only weakly ordered updates to the files at particular urls with the assumption that only one file may be observed to change at a time (because a lookup of file A, then B, may get a cache miss on A and a cache hit on B, so even if all clients strictly go A, then B, updates may still see old files when paths are reused).
- super robust clients that repeatedly retry with progressively less cache friendly headers until they have a consistent view. (This is very tricky to do).
It may be possible to do a tweak to the apt repository format though, which would allow publishing a race-free format in parallel with the existing layout, while clients migrate. To be safe against issue (A) the mirror network would need some care around handling of dns round-robin mirrors [to minimise the situation where referenced data is not available], but this should be doable - or alternatively clients doing 'apt-get update' may need to be willing to retry to accommodate round-robin skew.
What would such an archive format look like?
It would have only one well known file name (e.g. Releases-2), which would be internally signed. Rather than signing e.g. Packages.gz, it would sign a uniquely named packages and sources file - e.g. Packages-$HASH.gz or Packages-$serialno.gz.
Backwards compatibility is achieved by using the same filenames for deb's and the like. We need to keep writing Packages.gz though, and Releases, until we no longer worry about old apt clients. We can optimise disk space a little by making Packages.gz a symlink to a Packages-$HASH.gz (and so on for Sources..), but it may be simpler and less prone to unexpected behaviour to keep using regular files.
tl;dr
* Unique file names for all unique file content with one exception
* Releases-2, a self-signed file that provides hashes and names the index files (Packages, Sources, Translations etc)
* Coexists with existing archive layout |
Apt archives are accessed over HTTP; this has resulted in a cluster of bugs (reported here, and upstream) about problems behind intercepting caches, problems with squid etc.
There are 3 interlocking issues:
A - mirror networks may be out of sync with each other (e.g. a file named on one mirror may no longer exist, or may not yet exist, on another mirror)
B - updating files on a single mirror is not atomic - and even small windows of inconsistency will, given enough clients, cause headaches.
C - caches exacerbate race conditions - when one happens, until the cached data expires, all clients of the cache will suffer from the race
Solving this requires one of several things:
- file system transactions
- an archive format that requires only weakly ordered updates to the files at particular urls with the assumption that only one file may be observed to change at a time (because a lookup of file A, then B, may get a cache miss on A and a cache hit on B, so even if all clients strictly go A, then B, updates may still see old files when paths are reused).
- super robust clients that repeatedly retry with progressively less cache friendly headers until they have a consistent view. (This is very tricky to do).
It may be possible to do a tweak to the apt repository format though, which would allow publishing a race-free format in parallel with the existing layout, while clients migrate. To be safe against issue (A) the mirror network would need some care around handling of dns round-robin mirrors [to minimise the situation where referenced data is not available], but this should be doable - or alternatively clients doing 'apt-get update' may need to be willing to retry to accommodate round-robin skew.
What would such an archive format look like?
It would have only one well known file name (e.g. Releases-2), which would be internally signed. Rather than signing e.g. Packages.gz, it would sign a uniquely named packages and sources file - e.g. Packages-$HASH.gz or Packages-$serialno.gz.
Backwards compatibility is achieved by using the same filenames for deb's and the like. We need to keep writing Packages.gz though, and Releases, until we no longer worry about old apt clients. We can optimise disk space a little by making Packages.gz a symlink to a Packages-$HASH.gz (and so on for Sources..), but it may be simpler and less prone to unexpected behaviour to keep using regular files.
tl;dr
* Unique file names for all unique file content with one exception
* Releases-2, a self-signed file that provides hashes and names the index files (Packages, Sources, Translations etc)
* Coexists with existing archive layout
Related bugs:
* bug 804252: Please support InRelease files
* bug 1430011: support apt by-hash mirrors |
|
| 2015-09-14 08:08:26 |
Colin Watson |
description |
Apt archives are accessed over HTTP; this has resulted in a cluster of bugs (reported here, and upstream) about problems behind intercepting caches, problems with squid etc.
There are 3 interlocking issues:
A - mirror networks may be out of sync with each other (e.g. a file named on one mirror may no longer exist, or may not yet exist, on another mirror)
B - updating files on a single mirror is not atomic - and even small windows of inconsistency will, given enough clients, cause headaches.
C - caches exacerbate race conditions - when one happens, until the cached data expires, all clients of the cache will suffer from the race
Solving this requires one of several things:
- file system transactions
- an archive format that requires only weakly ordered updates to the files at particular urls with the assumption that only one file may be observed to change at a time (because a lookup of file A, then B, may get a cache miss on A and a cache hit on B, so even if all clients strictly go A, then B, updates may still see old files when paths are reused).
- super robust clients that repeatedly retry with progressively less cache friendly headers until they have a consistent view. (This is very tricky to do).
It may be possible to do a tweak to the apt repository format though, which would allow publishing a race-free format in parallel with the existing layout, while clients migrate. To be safe against issue (A) the mirror network would need some care around handling of dns round-robin mirrors [to minimise the situation where referenced data is not available], but this should be doable - or alternatively clients doing 'apt-get update' may need to be willing to retry to accommodate round-robin skew.
What would such an archive format look like?
It would have only one well known file name (e.g. Releases-2), which would be internally signed. Rather than signing e.g. Packages.gz, it would sign a uniquely named packages and sources file - e.g. Packages-$HASH.gz or Packages-$serialno.gz.
Backwards compatibility is achieved by using the same filenames for deb's and the like. We need to keep writing Packages.gz though, and Releases, until we no longer worry about old apt clients. We can optimise disk space a little by making Packages.gz a symlink to a Packages-$HASH.gz (and so on for Sources..), but it may be simpler and less prone to unexpected behaviour to keep using regular files.
tl;dr
* Unique file names for all unique file content with one exception
* Releases-2, a self-signed file that provides hashes and names the index files (Packages, Sources, Translations etc)
* Coexists with existing archive layout
Related bugs:
* bug 804252: Please support InRelease files
* bug 1430011: support apt by-hash mirrors |
Apt archives are accessed over HTTP; this has resulted in a cluster of bugs (reported here, and upstream) about problems behind intercepting caches, problems with squid etc.
There are 3 interlocking issues:
A - mirror networks may be out of sync with each other (e.g. a file named on one mirror may no longer exist, or may not yet exist, on another mirror)
B - updating files on a single mirror is not atomic - and even small windows of inconsistency will, given enough clients, cause headaches.
C - caches exacerbate race conditions - when one happens, until the cached data expires, all clients of the cache will suffer from the race
Solving this requires one of several things:
- file system transactions
- an archive format that requires only weakly ordered updates to the files at particular urls with the assumption that only one file may be observed to change at a time (because a lookup of file A, then B, may get a cache miss on A and a cache hit on B, so even if all clients strictly go A, then B, updates may still see old files when paths are reused).
- super robust clients that repeatedly retry with progressively less cache friendly headers until they have a consistent view. (This is very tricky to do).
It may be possible to do a tweak to the apt repository format though, which would allow publishing a race-free format in parallel with the existing layout, while clients migrate. To be safe against issue (A) the mirror network would need some care around handling of dns round-robin mirrors [to minimise the situation where referenced data is not available], but this should be doable - or alternatively clients doing 'apt-get update' may need to be willing to retry to accommodate round-robin skew.
What would such an archive format look like?
It would have only one well known file name (InRelease), which would be internally signed. Rather than signing e.g. Packages.gz, it would sign a uniquely named packages and sources file - e.g. Packages-$HASH.gz or Packages-$serialno.gz.
Backwards compatibility is achieved by using the same filenames for deb's and the like. We need to keep writing Packages.gz though, and Releases, until we no longer worry about old apt clients. We can optimise disk space a little by making Packages.gz a symlink to a Packages-$HASH.gz (and so on for Sources..), but it may be simpler and less prone to unexpected behaviour to keep using regular files.
tl;dr
* Unique file names for all unique file content with one exception
* InRelease, a self-signed file that provides hashes and names the index files (Packages, Sources, Translations etc)
* Coexists with existing archive layout
Related bugs:
* bug 804252: Please support InRelease files
* bug 1430011: support apt by-hash mirrors |
|
| 2015-10-20 19:07:51 |
Andrew |
bug |
|
|
added subscriber Andrew |
| 2015-10-29 12:50:10 |
jcv |
bug |
|
|
added subscriber jcv |
| 2015-12-03 08:33:19 |
Oskar Liljeblad |
bug |
|
|
added subscriber Oskar Liljeblad |
| 2015-12-03 12:08:21 |
Massmo |
bug task added |
|
apt |
|
| 2015-12-07 02:15:13 |
Craig Patten |
bug |
|
|
added subscriber Craig Patten |
| 2015-12-09 06:32:45 |
Arthur Blair |
bug |
|
|
added subscriber Arthur Blair |
| 2015-12-16 20:12:23 |
Antony Bailey |
bug |
|
|
added subscriber Antony Bailey |
| 2016-01-20 16:40:15 |
Brent |
bug |
|
|
added subscriber Brent |
| 2016-02-17 10:04:58 |
varacanero |
bug |
|
|
added subscriber varacanero |
| 2016-03-11 01:17:22 |
Scott Moser |
apt: status |
New |
Fix Released |
|
| 2016-03-11 01:18:45 |
Scott Moser |
apt (Ubuntu): status |
Confirmed |
Fix Released |
|
| 2016-03-13 12:22:22 |
sye bond |
removed subscriber sye bond |
|
|
|
| 2016-04-01 09:10:57 |
Comrade-Sergei |
bug |
|
|
added subscriber Comrade-Sergei |
| 2016-06-02 11:48:02 |
Maykel Moya |
bug |
|
|
added subscriber Maykel Moya |
| 2016-08-08 15:02:59 |
jellenelis |
bug |
|
|
added subscriber jellenelis |
| 2017-06-01 04:49:38 |
Abhijith Gururaj |
bug |
|
|
added subscriber Abhijith Gururaj |
| 2019-09-06 00:24:07 |
Robert "DocSalvager" Watson |
bug |
|
|
added subscriber Robert "DocSalvager" Watson |
| 2019-09-06 00:27:25 |
Robert "DocSalvager" Watson |
removed subscriber Robert "DocSalvager" Watson |
|
|
|
| 2019-10-30 22:47:14 |
Arkadiusz Piekarz |
bug |
|
|
added subscriber Arkadiusz Piekarz |
