apt-get does not properly reuse connections with https

Bug #1087543 reported by Thomas Bushnell, BSG on 2012-12-07
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Medium
Michael Vogt
Precise
Low
Unassigned

Bug Description

[Impact]

 * Apt does not re-use https connections. This results in a significant performance hit.

 * Additionally the hosts of mirrors are experiencing unnecessarily high load due to this

[Test Case]

 * Run apt-get -update using https connections. It will be noticably slower in comparison to after this change

[Regression Potential]

 * This fixes apt, by changing the way it uses curl, such that only problematic error codes cause failure. But this may only happen if the user is using https, http is not affected by this patch.

 * Might result in zero length files, or updates not being downloaded over https connectiosn.

 * https connections hanging in the event that the above logic does not correctly handle error cases.

[Other Info]
 * There's only a slight chance to hit any regressions, as the above is mostly just thinking well outside of the box.

 * This is already in Saucy.

 * This is not imperative to be pulled into precise, but it would be nice.

 * This is really more of an enhancement than a bug fix.

 * I have been using this patch for over 5 months now with no negative side affects.

 * I have been told that this patch has been deployed in a large (multi-thousand install base) organization with no negative side affects as well.

--------------------------------------------------------------------------------

When using an https: connection, apt-get does not properly reuse connections. Because https setup costs can be quite high, this can cause much slowdown when, for example, many different distributions on the same server need to be checked.

I believe the attached patch addresses the difficulty.

The attachment "enable apt to keep https connections alive for reuse" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Michael Vogt (mvo) wrote :

Thanks, I applied the fix into bzr

Changed in apt (Ubuntu):
status: New → In Progress
importance: Undecided → Medium
Dave Chiluk (chiluk) wrote :

@mvo, any update on when we can get this SRU'ed back into precise?

I have created a PPA that includes this fix. It is available here.
https://launchpad.net/~chiluk/+archive/lp1087543

Brian Murray (brian-murray) wrote :

This fix doesn't seem to have made it into raring. Where did you apply it?

Changed in apt (Ubuntu):
assignee: nobody → Michael Vogt (mvo)

On 26 April 2013 08:05, Brian Murray <email address hidden> wrote:
> This fix doesn't seem to have made it into raring. Where did you apply
> it?
>

apt (0.9.7.8~exp1) experimental; urgency=low

Launchpad Janitor (janitor) wrote :
Download full text (13.3 KiB)

This bug was fixed in the package apt - 0.9.9.1~ubuntu1

---------------
apt (0.9.9.1~ubuntu1) saucy; urgency=low

  * merged from the debian/sid branch:
    - debian/gbp.conf: change build branch to ubuntu/master
    - use ubuntu keyring and ubuntu archive keyring in apt-key
    - run update-apt-xapian-index in apt.cron
    - run apt-key net-update in cron.daily
    - different example sources.list
    - APT::pkgPackageManager::MaxLoopCount set to 5000
    - apport pkgfailure handling
    - ubuntu changelog download handling
    - patch for apt cross-building, see http://bugs.debian.org/666772
    - debian/apt.auto-removal.sh
      + make kernels auto-removable

apt (0.9.9.1) UNRELEASED; urgency=low

  * debian/rules:
    - call dh_clean in clean (closes: #714980)

apt (0.9.9) unstable; urgency=low

  [ Michael Vogt ]
  * improve debug output for the Debug::pkgProblemResolver and
    Debug::pkgDepCache::AutoInstall
  * improve apt-cdrom output when no CD-ROM can be auto-detected
  * document --no-auto-detect in apt-cdrom

  [ David Kalnischkies ]
  * build the en manpages in subdirectory doc/en
  * remove -ldl from cdrom and -lutil from apt-get linkage
  * rewrite pkgOrderList::DepRemove to stop incorrect immediate setting
    (Closes: 645713)
  * prefer Essentials over Removals in ordering score
  * fix priority sorting by prefering higher in MarkInstall
  * try all providers in order if uninstallable in MarkInstall
  * do unpacks before configures in SmartConfigure (Closes: #707578)
  * fix support for multiple patterns in apt-cache search (Closes: #691453)
  * set Fail flag in FileFd on all errors consistently
  * don't explicitly init ExtractTar InFd with invalid fd
  * OpenDescriptor should autoclose fd always on error (Closes: #704608)
  * fail in CopyFile if the FileFds have error flag set
  * ensure state-dir exists before coyping cdrom files
  * fix file location for configure-index.gz in apt.conf(5) (Closes: #711921)
  * handle missing "Description" in apt-cache show (Closes: #712435)
  * try defaults if auto-detection failed in apt-cdrom (Closes: #712433)
  * support \n and \r\n line endings in ReadMessages
  * do not redownload unchanged InRelease files
  * trigger NODATA error for invalid InRelease files (Closes: #712486)

apt (0.9.8.2) unstable; urgency=low

  [ Programs translations ]
  * French translation : typo fix. Closes: #677272

  [ Guillem Jover ]
  * Update Vcs fields (Closes: #708562)

  [ Michael Vogt ]
  * buildlib/apti18n.h.in:
    - fix build failure when building without NLS (closes: #671587)

  [ Gregoire Menuel ]
  * Fix double free (closes: #711045)

  [ Raphael Geissert ]
  * Fix crash when the "mirror" method does not find any entry
    (closes: #699303)

  [ Johan Kiviniemi ]
  * cmdline/apt-key:
    - Create new keyrings with mode 0644 instead of 0600.
    - Accept a nonexistent --keyring file with the adv subcommand as well.

apt (0.9.8.1) unstable; urgency=low

  [ David Kalnischkies ]
  * apt-pkg/indexcopy.cc:
    - non-inline RunGPGV methods to restore ABI compatibility with previous
      versions to fix partial upgrades (Closes: #707771)

  [ Michael Vogt ]
  * moved source to http://git.debian.org/...

Changed in apt (Ubuntu):
status: In Progress → Fix Released
Dave Chiluk (chiluk) wrote :

Here's a debdiff of tbushnell's patch.

It's based off of the bzr branch availalbe here. lp:~ubuntu-core-dev/apt/precise

Dave Chiluk (chiluk) wrote :

I have tested this and have linked my related bzr branch as well In case that makes life easier.

Dave Chiluk (chiluk) on 2013-07-12
description: updated
Dave Chiluk (chiluk) on 2013-07-12
tags: added: precise
Dave Chiluk (chiluk) on 2013-07-12
description: updated
Changed in apt (Ubuntu Precise):
status: New → Triaged
importance: Undecided → Low
Brian Murray (brian-murray) wrote :

I've uploaded the debdiff to the proposed queue for precise.

Hello Thomas, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.8.16~exp12ubuntu10.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apt (Ubuntu Precise):
status: Triaged → Fix Committed
tags: added: verification-needed
Dave Chiluk (chiluk) on 2013-07-29
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 0.8.16~exp12ubuntu10.12

---------------
apt (0.8.16~exp12ubuntu10.12) precise; urgency=low

  * Add patch from Thomas Bushnell to make apt-get reuse https connections
    (LP: #1087543)
 -- Dave Chiluk <email address hidden> Fri, 12 Jul 2013 12:16:26 -0500

Changed in apt (Ubuntu Precise):
status: Fix Committed → Fix Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.