Escape plus character in apt HTTP requests to work around Amazon S3 bug

Bug #1003633 reported by Doug Mitchell
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Undecided
Unassigned

Bug Description

Amazon S3 performs incorrect query-string un-escaping on URLs and improperly converts the plus ('+') character to a space (' ') when processing incoming GET requests for files. This causes apt-get to get HTTP 404 errors when downloading packages from S3-hosted mirrors which include a + character in the name of the package.

For more background info, see:
https://forums.aws.amazon.com/thread.jspa?threadID=55746

One possibly solution is to change the call to QuoteString in http.cc to include '+' in the character set, which would escape the plus character to the %2B form, which will be properly processed by Amazon S3.

Thanks,
Doug

Related branches

CVE References

Revision history for this message
Daniel Hartwig (wigs) wrote : Re: [Bug 1003633] [NEW] Escape plus character in apt HTTP requests to work around Amazon S3 bug

Also, this 2007 thread:

[1] https://forums.aws.amazon.com/thread.jspa?threadID=16361

where AWS representatives expound on their "justification" some more.
Instead of being well-behaved netizens, they just document their
non-conformance – though there is no indication in these threads where
such documentation is, other than provided in examples.

That thread also indicates that the path is expected to be
URL-encoded, not just some characters. This probably means including
':' and others to avoid more unanticipated transforms.

Revision history for this message
Daniel Hartwig (wigs) wrote :

Effectively S3 does not implement HTTP according to the spec. Rather
than working around the behaviour of S3 mirrors in methods/http we
could define methods/s3. S3 mirrors would then be properly identified
as non-http in sources.list.

This method could be a wrapper which invokes the http method with a
transformed URI.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu):
status: New → Confirmed
Revision history for this message
Julian Andres Klode (juliank) wrote :

Can't they just encode the plus character as a space in their S3 instances? Then a file like package_1.0+really0.9_all.deb would be stored as "package_1.0 really0.9_all.deb" and the URI decoding done by S3 is circumvented server-side, instead of requiring hacks on the client side.

Revision history for this message
Daniel Hartwig (wigs) wrote :

> Can't they just encode the plus character as a space in their S3 instances?

That does seem preferable.

Is there a package that is commonly used to maintain a mirror on S3 where this change can be made?

Revision history for this message
Michael Vogt (mvo) wrote :
tags: added: patch
Revision history for this message
Daniel Hartwig (wigs) wrote : Re: [Bug 1003633] Re: Escape plus character in apt HTTP requests to work around Amazon S3 bug

On 11 January 2013 16:58, Michael Vogt <email address hidden> wrote:
> ** Patch added: "Trivial extension of Daniels fix for #1086997"
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1003633/+attachment/3479965/+files/fix.diff

Please include also a comment above that line indicating why “+” will
be encoded, lest a future optimizer decides to remove it.

Michael Vogt (mvo)
Changed in apt (Ubuntu):
status: Confirmed → Fix Committed
Revision history for this message
Darik Horn (dajhorn) wrote :

This non-conformance bug works in both directions, so an easy kludge is to hard link all file names like this:

$ ln foobar-1.2+baz3_all.deb 'foobar-1.2 baz3_all.deb'

And sync to the s3 bucket normally. Afterwards, the key can be called through http with a literal "+" character or the "%2B" escape. Most clients like `apt-get` and `wget` will write out a file with the correct name using either form in the URL.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (13.3 KiB)

This bug was fixed in the package apt - 0.9.9.1~ubuntu1

---------------
apt (0.9.9.1~ubuntu1) saucy; urgency=low

  * merged from the debian/sid branch:
    - debian/gbp.conf: change build branch to ubuntu/master
    - use ubuntu keyring and ubuntu archive keyring in apt-key
    - run update-apt-xapian-index in apt.cron
    - run apt-key net-update in cron.daily
    - different example sources.list
    - APT::pkgPackageManager::MaxLoopCount set to 5000
    - apport pkgfailure handling
    - ubuntu changelog download handling
    - patch for apt cross-building, see http://bugs.debian.org/666772
    - debian/apt.auto-removal.sh
      + make kernels auto-removable

apt (0.9.9.1) UNRELEASED; urgency=low

  * debian/rules:
    - call dh_clean in clean (closes: #714980)

apt (0.9.9) unstable; urgency=low

  [ Michael Vogt ]
  * improve debug output for the Debug::pkgProblemResolver and
    Debug::pkgDepCache::AutoInstall
  * improve apt-cdrom output when no CD-ROM can be auto-detected
  * document --no-auto-detect in apt-cdrom

  [ David Kalnischkies ]
  * build the en manpages in subdirectory doc/en
  * remove -ldl from cdrom and -lutil from apt-get linkage
  * rewrite pkgOrderList::DepRemove to stop incorrect immediate setting
    (Closes: 645713)
  * prefer Essentials over Removals in ordering score
  * fix priority sorting by prefering higher in MarkInstall
  * try all providers in order if uninstallable in MarkInstall
  * do unpacks before configures in SmartConfigure (Closes: #707578)
  * fix support for multiple patterns in apt-cache search (Closes: #691453)
  * set Fail flag in FileFd on all errors consistently
  * don't explicitly init ExtractTar InFd with invalid fd
  * OpenDescriptor should autoclose fd always on error (Closes: #704608)
  * fail in CopyFile if the FileFds have error flag set
  * ensure state-dir exists before coyping cdrom files
  * fix file location for configure-index.gz in apt.conf(5) (Closes: #711921)
  * handle missing "Description" in apt-cache show (Closes: #712435)
  * try defaults if auto-detection failed in apt-cdrom (Closes: #712433)
  * support \n and \r\n line endings in ReadMessages
  * do not redownload unchanged InRelease files
  * trigger NODATA error for invalid InRelease files (Closes: #712486)

apt (0.9.8.2) unstable; urgency=low

  [ Programs translations ]
  * French translation : typo fix. Closes: #677272

  [ Guillem Jover ]
  * Update Vcs fields (Closes: #708562)

  [ Michael Vogt ]
  * buildlib/apti18n.h.in:
    - fix build failure when building without NLS (closes: #671587)

  [ Gregoire Menuel ]
  * Fix double free (closes: #711045)

  [ Raphael Geissert ]
  * Fix crash when the "mirror" method does not find any entry
    (closes: #699303)

  [ Johan Kiviniemi ]
  * cmdline/apt-key:
    - Create new keyrings with mode 0644 instead of 0600.
    - Accept a nonexistent --keyring file with the adv subcommand as well.

apt (0.9.8.1) unstable; urgency=low

  [ David Kalnischkies ]
  * apt-pkg/indexcopy.cc:
    - non-inline RunGPGV methods to restore ABI compatibility with previous
      versions to fix partial upgrades (Closes: #707771)

  [ Michael Vogt ]
  * moved source to http://git.debian.org/...

Changed in apt (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Quanah Gibson-Mount (mishikal) wrote :

I would note this is broken for https based repositories still.

Revision history for this message
Quanah Gibson-Mount (mishikal) wrote :
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.