pull-lp-source fails due to SSL error - pull-lp-source: Error: Failed to download: [SSL] unknown error (_ssl.c:726)

[Trent Lloyd]

pull-lp-source fails to download packages on xenial and bionic due to a non-descript SSL error

lathiat@optane ~$ pull-lp-source percona-toolkit xenial
pull-lp-source: Downloading percona-toolkit version 2.2.16-1
pull-lp-source: Error: Failed to download: [SSL] unknown error (_ssl.c:726)

I have determined the cause for this is python-httplib2 (for python2, not python3) does not support SNI - which is where the HTTP hostname is sent to the server during SSL negotiation before the certificate is sent, so that the remote host can send the correct certificate.

Change request to support SNI on Python 2.7.9+ (xenial has 2.7.11):
https://github.com/httplib2/httplib2/pull/13

Support for this is implemented in newer httplib2 since mid-2016/early 2016 - for some reason this new package version is not in Debian (and by proxy Ubuntu) - even Bionic. https://github.com/httplib2/httplib2/blob/master/CHANGELOG

You can reproduce the failure with this test case:
python -c "import httplib2; httplib2.Http().request('https://launchpad.net/ubuntu/+archive/primary/+files/percona-toolkit_2.2.16-1.dsc')"

While this starts on launchpad.net it's actually redirected to https://launchpadlibrarian.net/225717435/percona-toolkit_2.2.16-1.dsc

You can verify that SNI is the issue in two parts

(1) These two commands, the first sends no SNI and gets no certificate and fails; the second gets the correct certificate
openssl s_client -connect launchpadlibrarian.net:443
openssl s_client -servername launchpadlibrarian.net -connect launchpadlibrarian.net:443

(2) It works with the python3 version of httplib2, which has SNI support
python3 -c "import httplib2; httplib2.Http().request('https://launchpad.net/ubuntu/+archive/primary/+files/percona-toolkit_2.2.16-1.dsc')"

It's not entirely clear to me whether backporting the httplib2 fix to Xenial+Bionic will be reasonable or not.

In theory it could change the behavior of some programs, e.g. it suddenly starts sending SNI and gets a different certificate/vhost, and could in theory break some applications even though arguably they were broken to begin with.

So we may need to consider whether a backported httplib2 patch is reasonable, whether to somehow patch ubuntu-dev-tools to use some other library that supports SNI or whether we can get launchpadlibrarian.net changed not to require SNI

[Trent Lloyd (lathiat)]

Workaround: pip install httplib2

[William Grant (wgrant)]

Turns out monopolistic Australian ISPs are bad and hijack DNS.

[Trent Lloyd (lathiat)]

Telstra DNS serversr are hijacking the DNS for launchpadlibrarian.net and then transparently proxying it. They aren't intercepting the SSL, but without SNI it fails to pass the connection through.

Seems suspiciously similar to the URL filtering used for thepiratebay and various other piracy sites, although, the IP address returned is different to the one I get for those sites.

Weird.. will chase up Telstra about it.

[Mattia Rizzolo (mapreri)]

"pip install httplib2" is no workaround for a bug in a debian package anyway.

And if you already know it's something that has to do with httplib2, why did you assign this bug to ubuntu-dev-tools instead of python-httplib2 (which will be updated at some point, a contributor contacted me to update the library in Debian just a few days ago :))