Failing to fetch https sources with apt-get update

[Timothy R. Chavez]

Raring packages fail to build in PPAs that use https:// sources as external dependencies when 'apt-get update' is called early on in the build process. This is not happening when building packages for Precise (which is the Ubuntu series we primarily work off of). Here is a pastebin of an actual log:

https://pastebin.canonical.com/91731/

Here's what we know:

 - The credentials used to access the external dependency are valid (confirmed by timrc and lamont)

 - The external dependency sources are reachable from a buildd where the "apt-get update" failed via wget (confirmed by timrc, lamont, and mthaddon)

 - The problem does not appear architecture specific - the build failure is confirmed for i386, amd64, and armhf (confirmed by timrc, lamont, and infinity)

 - The problem does not appear buildd specific - the build failure will happen on which ever builder is used to build the package (confirmed by timrc)

 - The problem can be reproduced by downloading the raring chroot from LP (e.g. chroot-ubuntu-raring-amd64.tar.bz2), adding the sources to chroot-autobuild/etc/apt/sources.list.d/bootstrap.list (as they appear in the external_dependency PPA field), and running "chroot chroot-autobuild apt-get update" (confirmed by timrc, lamont, infinity)

 - The problem occurs on Quantal (apt 0.9.7.5ubuntu5.4), Raring (apt 0.9.7.7ubuntu4), and Saucy (apt 0.9.7.7ubuntu4). Precise ( 0.8.16~exp12ubuntu10.10) is fine (confirmed by infinity)

Other info:

 - Quantal fails with a slightly different log message, e.g. https://pastebin.canonical.com/91733/

 - At least for the armhf failures on Raring, the build log shows apt-get update printing garbage to the log when referencing one of these external dependencies. Could it be that there is some sort of memory corruption, here?

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: apt-transport-https 0.9.7.7ubuntu4
Uname: Linux 3.8.0-19-generic x86_64
ApportVersion: 2.9.2-0ubuntu8
Architecture: amd64
Date: Tue May 28 19:34:07 2013
MarkForUpload: True
ProcEnviron: Error: [Errno 2] No such file or directory: '/proc/22833/environ'
SourcePackage: apt
UpgradeStatus: No upgrade log present (probably fresh install)

[David Kalnischkies (donkult)]

Thankfully the bureport doesn't mention the error message(s?) and the used pastebin is at least behind a login-wall (and provided that I seem to be unable to get more than an empty page even if I log in with a launchpad account it might even be limited to employees…).

So, a general remark: wget is not of interest here as curl library is used. And the " libcurl3-gnutls" variant of it, so make sure you are trying with that. As we have frequent problems with this library, check which versions are used and try if it is really a problem of apt-transport-https and not of the underlying library.

There also shouldn't be too many changes to the https transport inbetween known good/bad, so feel free to bisect. ;)

[Timothy R. Chavez (timrchavez)]

David, using Canonical's pastebin was intentional since it contained sensitive information, but I can see how that would be a problem for someone looking at the problem who's not an employee of Canonical. So, I'm really sorry about that, and here's a version of the log with redact information. This attachment corresponds to the first pastebin, https://pastebin.canonical.com/91731/

[Adam Conrad (adconrad)]

So, a confirmed workaround to this is to install ca-certificates in the chroots, which I may do as a stop-gap. I'll set up a reproduction environment that proves this issue on sid as well, but it's interesting that precise works fine without ca-certificates installed.

[Timothy R. Chavez (timrchavez)]

Ans here is the log information corresponding to https://pastebin.canonical.com/91733

[David Kalnischkies (donkult)]

Mhh, the "Problem with the SSL CA cert (path? access rights?)" is not from us, it comes from libcurl and I see no change between 0.8.16~exp12 and 0.9.7.5 which could be responsible for this (as the two changes in this timeframe is a user-agent string change and adding of the Accept: request-header), so check libcurl please.

The char* buffer used for the error message is provided to libcurl via CURLOPT_ERRORBUFFER; we never check if the string we get is valid/set/whatever, but we init it as empty and only use it if libcurl tells use there was some kind of error, so that shouldn't be a problem.

Maybe setting some debug variables will give you some insights:
-o Debug::Acquire::https=1 # enables CURLOPT_VERBOSE
-o Debug::pkgAcquire::Worker=1 # shows the messages passed to the methods and back

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

ca-certificates and apt-transport-https packages should be available in the chroot, which they are in stock chroots available on launchpad.

I've validated that certificate chain is correct, valid, and trusted to hosts in question. And thus apt update should operate correctly.

If anybody else has problems accessing https based repositories please open new bug reports.