apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
apt (Ubuntu) |
Fix Released
|
Low
|
Unassigned | ||
Hardy |
Won't Fix
|
Undecided
|
Unassigned | ||
Lucid |
Won't Fix
|
Undecided
|
Unassigned | ||
Maverick |
Won't Fix
|
Undecided
|
Unassigned | ||
Natty |
Won't Fix
|
Undecided
|
Unassigned | ||
Oneiric |
Won't Fix
|
Undecided
|
Unassigned | ||
Precise |
Won't Fix
|
High
|
Unassigned | ||
cloud-init (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Hardy |
Won't Fix
|
High
|
Unassigned | ||
Lucid |
Fix Released
|
High
|
Unassigned | ||
Maverick |
Fix Released
|
High
|
Unassigned | ||
Natty |
Fix Released
|
High
|
Unassigned | ||
Oneiric |
Fix Released
|
High
|
Unassigned | ||
Precise |
Fix Released
|
High
|
Unassigned |
Bug Description
[Problem]
When using S3 mirrors, the apt client requests files A,B,C,D in the same request, but S3 responds with files B,C,D,A or some other arbitrary ordering. Apt then saves the files under the wrong file names.
This is due to a bug in Amazon AWS S3, where http pipelining does not work in violation of RFC 2068. apt uses http pipelining currently, so this prevents apt from using mirrors hosted on S3 cloud instances.
[Impact]
Blocks use of S3 mirrors for providing ubuntu updates. Affects Hardy, Lucid, Maverick, Oneiric, Natty, and Precise.
Use of S3 mirrors is considered as a way of enhancing our mirror services. Currently mirrors have moments of instability, so having the option of switching to S3 mirrors is very important for update reliability.
[Development Fix]
Fixed in Precise's cloud-init as of these comments, which adds an apt_pipelining option, turned off by default but configurable.
http://
http://
[Stable Fix]
TODO
[Text Case]
1) Launch an instance in EC2
2) Run: sed -i "s,ec2.
3.) Set your apt debug levels to whatever you want
4) Run "apt-get -y update"
5) Run "apt-get -y install firefox xorg libreoffice > /tmp/install.log 2>&1"
Broken Behavior: Cached files are being misnamed
Fixed Behavior: Cached files are properly named
The S3 EC2 mirrors seem to reproduce it most reliably. The downloaded files are whole and match the proper checksums, but they have the wrong content.
[Regression Potential]
http pipelining is an optimization technique in apt which allows multiple requests to be made at the same time. When it works, it provides a small performance boost, but when it doesn't the failures are arbitrary and not adequately explained.
Thus, the expected regression is slightly slower apt behavior for non-S3 cloud users. However, testing shows the impact is less than a few seconds so will likely be unnoticed. In the off chance that it does inadvertently cause non-trivial performance impact, the change is configurable on a per-host instance.
[Original Report]
[SRU: Cloud-Init]
r536 and r537 contains a fix that:
1.) Disables APT pipelining on first boot for new Cloud Images instances
2.) Disables APT pipelining on installation for existing cloud-images.
The rational for this SRU is due to the Mirror situation for EC2 Cloud images. Currently, the mirrors are proving to have moments of instability that is causing both end-user and development pain. In order to address this, we built out Amazon AWS S3 mirrors. Unfortantly, due to a bug in S3, apt http pipeling does not work with S3. As a result, in order to provide enhanced availability for the Cloud Image repositories in EC2, apt http pipelining needs to be disabled ahead of flipping the new S3 mirrors as the active mirrors.
The performance impact of this change is unnoticable and in testing, has proven to provide no ill effect. http pipeplining allows for multiple requests to be made at the same time. The apt change log states that http pipelining is a "micro-enhancement" and within the EC2 environment shaves a few seconds of many requests. When apt http pipeling works, at best, it saves a few seconds, when it fails against a buggy HTTP1.1 server or behind a proxy that does not support it, the failures are arbitrary and not adiquetly explained. This change will enhance the Cloud Images by making apt request packages sequentially instead of in batches and prevent arbitrary failures caused by non-compliant web servers. Cloud image users, both on EC2 and outside, will benefit from this fix.
http://
http://
_______
apt-get is appears to be mangling the local caching file names, which is appearing as a hashsum or size mismatch.
Evidence below:
_______
Output of "apt-get -y install firefox":
Failed to fetch http://
Failed to fetch http://
Failed to fetch http://
Failed to fetch http://
_______
apt-get -y install firefox with debug for HTTP turned on:
Get:63 http://
HTTP/1.1 200 OK^M
x-amz-id-2: EXccd6GVMfE6ly4
x-amz-request-id: 7572BAB5E6FFFAC0^M
Date: Tue, 06 Mar 2012 20:06:13 GMT^M
Last-Modified: Fri, 03 Feb 2012 08:54:27 GMT^M
ETag: "6c07f8db615cc3
Accept-Ranges: bytes^M
Content-Type: application/
Content-Length: 15142^M
Server: AmazonS3^M
^M
Get:64 http://
HTTP/1.1 200 OK^M
x-amz-id-2: IcNECY1OtzHOXnT
x-amz-request-id: 5DEEEE629E3BA012^M
Date: Tue, 06 Mar 2012 20:06:13 GMT^M
Last-Modified: Fri, 02 Mar 2012 20:55:40 GMT^M
ETag: "501373631a2418
Accept-Ranges: bytes^M
Content-Type: application/
Content-Length: 56824^M
Server: AmazonS3^M
^M
Get:65 http://
HTTP/1.1 200 OK^M
x-amz-id-2: HHNSOZKH74D4I1X
x-amz-request-id: A6FD3143E6A0BA1A^M
Date: Tue, 06 Mar 2012 20:06:34 GMT^M
Last-Modified: Sat, 21 Jan 2012 00:48:55 GMT^M
ETag: "3653165af1f20a
Accept-Ranges: bytes^M
Content-Type: application/
Content-Length: 10902^M
Server: AmazonS3^M
_______
Evaluation of the MD5sums of downloaded failed items:
root@ip-
501373631a24184
6c07f8db615cc32
566f7c92a9fcd0c
3653165af1f20a4
_______
Snipets from Packages.bz2
From Meta-data:
Package: aptdaemon
MD5sum: 6c07f8db615cc32
Package: xul-ext-ubufox
MD5sum: 501373631a24184
Package: python-aptdaemon
MD5sum: 566f7c92a9fcd0c
Package: python-defer
MD5sum: 3653165af1f20a4
_______
"dpkg -i" on pacakges downloaded
root@ip-
new debian package, version 2.0.
size 56824 bytes: control archive= 1520 bytes.
23 bytes, 1 lines conffiles
1023 bytes, 29 lines control
1804 bytes, 21 lines md5sums
Package: xul-ext-ubufox
Source: ubufox
Version: 2.0-0ubuntu1
Architecture: all
Maintainer: Ubuntu Mozilla Team <email address hidden>
Installed-Size: 383
Depends: aptdaemon, libglib2.0-0 (>= 2.26)
Recommends: firefox (>= 4.0~b6)
Enhances: firefox
Breaks: ubufox (<< 0.9~rc2-0ubuntu3)a
Replaces: ubufox (<< 0.9~rc2-0ubuntu3)
Provides: firefox-ubufox, ubufox
Section: web
Priority: optional
Homepage: https:/
Description: Ubuntu-specific configuration defaults and apt support for Firefox
Adds Ubuntu-specific modifications to Firefox.
.
Integrates the browser with Ubuntu to:
* Enable searching for missing plugins from Ubuntu software catalog
* Add the following options to the Help menu
- Get help on-line
- Help translating Firefox
- Ubuntu Release Notes
* Set homepage to Ubuntu Start Page
* Display a restart notification after upgrading Firefox
* Add ask.com to the search engines.
.
You can uninstall this if you prefer to use a pristine Firefox install.
root@ip-
new debian package, version 2.0.
size 15142 bytes: control archive= 1508 bytes.
68 bytes, 2 lines conffiles
1390 bytes, 30 lines control
1088 bytes, 15 lines md5sums
Package: aptdaemon
Version: 0.43+bzr769-
Architecture: all
Maintainer: Ubuntu Developers <email address hidden>
Installed-Size: 188
Depends: python, python-aptdaemon (= 0.43+bzr769-
Breaks: software-center (<< 1.1.21)
Section: admin
Priority: extra
Homepage: https:/
Description: transaction based package management service
Aptdaemon allows normal users to perform package management tasks, e.g.
refreshing the cache, upgrading the system, installing or removing software
packages.
.
Currently it comes with the following main features:
.
- Programming language independent D-Bus interface, which allows one to
write clients in several languages
- Runs only if required (D-Bus activation)
- Fine grained privilege management using PolicyKit, e.g. allowing all
desktop user to query for updates without entering a password
- Support for media changes during installation from DVD/CDROM
- Support for debconf (Debian's package configuration system)
- Support for attaching a terminal to the underlying dpkg call
.
This package contains the aptd script and all the data files required to run
the daemon. Moreover it contains the aptdcon script, which is a command
line client for aptdaemon. The API is not stable yet.
Original-
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: apt 0.8.16~exp12ubuntu4
ProcVersionSign
Uname: Linux 3.2.0-18-virtual x86_64
ApportVersion: 1.94-0ubuntu1
Architecture: amd64
Date: Tue Mar 6 21:21:17 2012
Ec2AMI: ami-0400dd6d
Ec2AMIManifest: (unknown)
Ec2Availability
Ec2InstanceType: m1.large
Ec2Kernel: aki-825ea7eb
Ec2Ramdisk: unavailable
SourcePackage: apt
UpgradeStatus: No upgrade log present (probably fresh install)
Related branches
- Ubuntu Development Team: Pending requested
-
Diff: 88 lines (+52/-0)3 files modifiedcloudinit/CloudConfig.py (+29/-0)
debian/changelog (+9/-0)
debian/cloud-init.postinst (+14/-0)
- Ubuntu Development Team: Pending requested
-
Diff: 150 lines (+104/-0)6 files modified.pc/applied-patches (+1/-0)
cloudinit/__init__.py (+1/-0)
debian/changelog (+9/-0)
debian/cloud-init.postinst (+14/-0)
debian/patches/lp948461-disable-add-configure-apt-pipelining.patch (+78/-0)
debian/patches/series (+1/-0)
tags: | added: server |
Changed in apt (Ubuntu Precise): | |
importance: | Undecided → High |
Changed in cloud-init (Ubuntu Precise): | |
importance: | Undecided → High |
status: | New → Fix Released |
Changed in cloud-init (Ubuntu Oneiric): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in cloud-init (Ubuntu Natty): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in cloud-init (Ubuntu Maverick): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in cloud-init (Ubuntu Lucid): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in cloud-init (Ubuntu Hardy): | |
importance: | Undecided → High |
status: | New → Triaged |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in cloud-init (Ubuntu Lucid): | |
status: | Triaged → Fix Committed |
Changed in apt (Ubuntu Hardy): | |
status: | New → Won't Fix |
Changed in apt (Ubuntu Lucid): | |
status: | New → Won't Fix |
Changed in apt (Ubuntu Maverick): | |
status: | New → Won't Fix |
tags: | added: rls-mgr-p-tracking |
Changed in apt (Ubuntu Natty): | |
status: | New → Won't Fix |
Changed in apt (Ubuntu Oneiric): | |
status: | New → Won't Fix |
tags: |
added: verification-done removed: verification-needed |
Apt log of import