python3.4.3 SRU break requests

[Didier Roche]

SRU reproducer / acceptance criteria: see comment #6

Sicne the upgade to python 3.4.3 on trusty, I'm getting this error when using a squid proxy:
https://jenkins.qa.ubuntu.com/view/All/job/udtc-trusty-tests/1946/label=ps-trusty-desktop-amd64-1,type=large/testReport/tests.large.test_android/AndroidSDKTests/test_default_android_sdk_install/

The code is using python-requests, with verify=True for ssl connection (default). Some tests are testing that invalid certificates are rejected: https://github.com/ubuntu/ubuntu-make/blob/master/umake/network/download_center.py#L129

Rerunning the same code with previous trusty package (3.4.0~trusty1) doesn't show up this issue. It seems that SNI is broken for the trusty version of python3-requests with 3.4.3. (See the FAQ http://www.python-requests.org/en/latest/community/faq/ with "What are “hostname doesn’t match” errors?" and the stackoverflow question.

I did run a test, grabbing requests 2.7 and backporting it to trusty (I needed to as well to take python3-urllib3 willy version).

So, 3.4.3 has an incompatible change for existing projects and people with proxys are starting to see some breakage like in https://bugs.launchpad.net/ubuntu/+source/ubuntu-make/+bug/1499890.

Can we get it fix somehow, reverting the incompatible change breaking SNI (I wonder if this is "Changed in version 3.4.3: This class now performs all the necessary certificate and hostname checks by default. To revert to the previous, unverified, behavior ssl._create_unverified_context() can be passed to the context parameter." in https://docs.python.org/3/library/http.client.html or something else) so that existing code can either get a new compatible python-requests or avoid incompatible changes in python 3.4.3?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Didier Roche (didrocks)]

Barry told that he's looking at it

[Barry Warsaw (barry)]

PEP 476 <https://www.python.org/dev/peps/pep-0476/> is the impetus for this, and issue 22417 is the implementation <https://bugs.python.org/issue22417>

The PEP says this <https://www.python.org/dev/peps/pep-0476/#id28>:

"This change will have the appearance of causing some HTTPS connections to "break", because they will now raise an Exception during handshake.

This is misleading however, in fact these connections are presently failing silently, an HTTPS URL indicates an expectation of confidentiality and authentication. The fact that Python does not actually verify that the user's request has been made is a bug, further: "Errors should never pass silently."

Nevertheless, users who have a need to access servers with self-signed or incorrect certificates would be able to do so by providing a context with custom trust roots or which disables validation (documentation should strongly recommend the former where possible). Users will also be able to add necessary certificates to system trust stores in order to trust them globally.

Twisted's 14.0 release made this same change, and it has been met with almost no opposition."

So the immediate question I have is, you say you're running the tests with verify=True and you're getting new failures. If that's the case, then since you've apparently always been verifying certificates, the change to make verification by default shouldn't affect you. But you're saying it has affected you and other packages too, is that right?

You say you ran a test with the backport of requests 2.7, but you don't mention what the outcome of that test was.

If you install the monkey patch to restore the old behavior according to the PEP, does that fix the problem for you?

I'm also curious because I ran the system-image tests in a trusty chroot and haven't seen any problems. It does a lot of self-signed certificate checks, including ensuring that bad certificates are rejected. It doesn't use requests though.

[Barry Warsaw (barry)]

How can I reproduce this on my own machine?

[Didier Roche (didrocks)]

Ok, let me clarify some points here.

First, I pointed that change as it's the closest thing that changed in python 3.4.3 which can have an impact on SNI, it may be something different which broke requests.
The invalid certificate tests is a test that is passing (this was to underline that I already had verfiy=true and checked that the SSL certificate check was done, as I was aware about the non check of it in python)

So, what is broken? I think the target is the SNI support with older request
1. Setup
- Install ubuntu-make from ubuntu-desktop/ubuntu-make ppa
- Have a proxy (squid.internal) for instance, filtering your network
- Export HTTPS_PROXY to point to your squid internal proxy

2. With python 3.4.0:
- Run "umake android" -> android-studio resolve, show the license page and install

3. With python 3.4.3 (current 14.04):
- Run "umake android" -> you will get the hostname 'squid.internal' doesn't match either of '*.google.com', '*.android.com', '*.appengine.google.com', '*.cloud.google.com', '*.google-analytics.com', '*.google.ca', <…>' message (content depending on the allowed site from your proxy).

4. With python 3.4.3 + requests 2.7 backported in my ppa (didrocks/ppa):
- Run "umake android" -> android-studio resolve, show the license page and install

As you can see, Ubuntu Make code didn't change at all in any of those tests, I think requests post 2.2.1 has something that made SNI work with python 3.4.3.

As long as you have the right proxy config, I think this should simply fail:
import requests
requests.get('https://api.github.com/events')

[Barry Warsaw (barry)]

A simple reproducer is:

$ sudo apt-get install squid3
$ export HTTP_PROXY=http://localhost:3128 HTTPS_PROXY=https://localhost:3128
$ python3 -c "import requests; requests.get('https://api.github.com/events')"

This is a tough one to assign blame to but I actually think that Python 3.4.3 *fixed* a problem in this space, but that urllib3 --not requests!-- was buggy. What isn't obvious from the above is that upstream requests vendors urllib3, but in Ubuntu, we unbundle it. Further, your PPA includes not just a new requests but also a new urllib3.

From what I've read in the requests tracker, SNI support is actually the responsibility of urllib3. I've looked through the changelogs of both requests and urllib3 and haven't been able to match a bug fix with the relevant upstream releases in Ubuntu. Trusty has requests 2.2.1 and urllib3 1.7.1. But I still think ultimately it was a bug in urllib3 that got fixed in your PPA backport of 1.11 which fixes the above reproducer.

So, I would propose that we backport both requests 2.7.0-3 and urllib3 1.11-1 (Wily versions both) to trusty-backports as a resolution of this problem. Would that work for you?

[Didier Roche (didrocks)]

Do you mean -backports or -updates? The issue is that people not enabling backports have now a broken request/urllib3 with - updates only, so as long as people not enabling backports and only using -updates are fixed, I'm fine.

If the solution to fix it is to requires people to enable -backports, that's not a good solution IMHO as the breakage happened in -updates with python 3.4.3.

[Sebastien Bacher (seb128)]

you can't introduce regressions or incompatible changes in a LTS and then trying to deal with those through -backport, some users could have production code relying on us that needs to keep working, behaviour changes don't have their place in stable updates even if that's to fix bugs. If there are cases which can't be fixed we should probably revert and suggest users who want those fixes to update to more recent Ubuntu versions

[Barry Warsaw (barry)]

This is the requests bug that lead me to suspect urllib3: https://github.com/kennethreitz/requests/issues/2435

[Didier Roche (didrocks)]

For now, the python 3.4 package has been revert in -updates.

[Axel Beckert (xtaran)]

So you leave all users of the trusty-updates with packages installed that are newer than what is available in the archives and hence take the chance to get proper security updates in the future?

# apt-cache policy python3.4
python3.4:
  Installed: 3.4.3-1ubuntu1~14.04.1
  Candidate: 3.4.3-1ubuntu1~14.04.1
  Version table:
 *** 3.4.3-1ubuntu1~14.04.1 0
        100 /var/lib/dpkg/status
     3.4.0-2ubuntu1.1 0
        500 http://ubuntu.ethz.ch/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     3.4.0-2ubuntu1 0
        500 http://ubuntu.ethz.ch/ubuntu/ trusty/main amd64 Packages

Please do reverting in the future _only_ by uploading a package with properly increased version number, e.g. 3.4.3+really3.4.0-0ubuntu1~14.04.1.

[Lenin (gagarin)]

+1 @#8

[Didier Roche (didrocks)]

Steve Langasek did this revert, I guess Barry and Matthias (the uploader of this python version) are still working on getting it fixed properly.

[Oli (oli)]

The update and reversion has left people beached on 3.4.3. Apt doesn't automatically downgrade people back down. They're just orphaned on an obsolete version.

I understand that this is a WiP and this has been done to stop more people hitting this issue, but is there a good way to downgrade to the last version temporarily?

[Oli (oli)]

Additionally, 3.4.3 seems to be missing the pyvenv script and documentation that routes through to python -m venv. It's not the end of the world, but certainly a regression.

[Axel Beckert (xtaran)]

Hi Oli,

Oli wrote:
> I understand that this is a WiP and this has been done to stop more
> people hitting this issue, but is there a good way to downgrade to the
> last version temporarily?

We used "apt-get install -y --force-yes python3.4{,-minimal}=3.4.0-2ubuntu1.1 libpython3.4{,-minimal,-stdlib}=3.4.0-2ubuntu1.1".

You might want to adjust the list of packages depending on which
packages you have installed. (Usage of "{…,…}" requires bash, zsh or
similar as shell.)

  Regards, Axel
--
 ,''`. | Axel Beckert <email address hidden>, http://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
  `- | 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE

[graingert (tagrain)]

If I grab the latest cloud image (https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box) I'm completely broken. Is there a cloud image that works?

[graingert (tagrain)]

Oh that link 404s now and I get an empty directory https://cloud-images.ubuntu.com/vagrant/trusty/current/

What's going on?

[Bill Miller (wbmilleriii)]

I tried the fix in #16 and it broke update-manager and the software center. I had to enable trusty-proposed and upgrade python3.4 to get it to work again. So I am "beached" again.

[Chris Barnes (4-chris-2)]

Beached here as well.
Ubuntu 14.04.3 LTS, brand new install today.
This bug affects the installation of LibreOffice from ppa:libreoffice/libreoffice-5-0 which has been working fine for several months.
Depends: python3-uno (>= 4.0~)

[Steve Langasek (vorlon)]

On Thu, Oct 08, 2015 at 12:11:06PM -0000, Chris Barnes wrote:
> Beached here as well.
> Ubuntu 14.04.3 LTS, brand new install today.

No install from today would include the withdrawn SRU. You are seeing an
unrelated issue.

[Chris Barnes (4-chris-2)]

Just installed from scratch with latest ubuntu-14.04.3-desktop-amd64.iso
After installation:
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get purge -y libreoffice*
sudo add-apt-repository ppa:libreoffice/libreoffice-5-0
sudo apt-get install libreoffice
-- Fails due to Depends: python3-uno (>= 4.0~)
sudo apt-get install python3-uno
-- Fails due to: python3-uno : Depends: libpython3.4 (>= 3.4.2) but 3.4.0-2ubuntu1.1 is to be installed

Looks like a symptom of the same issue to me ?
Are you sure they are unrelated ?

[Steve Langasek (vorlon)]

On Thu, Oct 08, 2015 at 08:06:56PM -0000, Chris Barnes wrote:
> Just installed from scratch with latest ubuntu-14.04.3-desktop-amd64.iso
> After installation:
> sudo apt-get update
> sudo apt-get upgrade -y
> sudo apt-get purge -y libreoffice*
> sudo add-apt-repository ppa:libreoffice/libreoffice-5-0
> sudo apt-get install libreoffice
> -- Fails due to Depends: python3-uno (>= 4.0~)
> sudo apt-get install python3-uno
> -- Fails due to: python3-uno : Depends: libpython3.4 (>= 3.4.2) but 3.4.0-2ubuntu1.1 is to be installed

Well, it's related in the sense that the python3-uno in this ppa is built
against a newer libpython3.4 than is currently available in trusty-updates.
So yes, it's related after all.

You could ask the ppa maintainer to do a rebuild of libreoffice against
current trusty to fix this.

[Bill Miller (wbmilleriii)]

On Thu, Oct 08, 2015 at 08:06:56PM -0000, Chris Barnes wrote:
> Just installed from scratch with latest ubuntu-14.04.3-desktop-amd64.iso
> After installation:
> sudo apt-get update
> sudo apt-get upgrade -y
> sudo apt-get purge -y libreoffice*
> sudo add-apt-repository ppa:libreoffice/libreoffice-5-0
> sudo apt-get install libreoffice
> -- Fails due to Depends: python3-uno (>= 4.0~)
> sudo apt-get install python3-uno
> -- Fails due to: python3-uno : Depends: libpython3.4 (>= 3.4.2) but 3.4.0-2ubuntu1.1 is to be installed

You could enable trusty-proposed and upgrade to the python3.4 from their, then install your libreoffice. But you would then be stuck with an obsolete version like the rest of us.

[Sebastian Unger (sebunger44)]

Please note that not all people are "beached". I manage some 56 machines and they regularly run apt-get update dist-upgrade and are pinned to versions from a local mirror of the archive.

Those machines all downgraded python after the revert which broke a large number of apps amongst them apport, unattended-upgrades and others (see attached reports).

I believe this may be due to pyc files not being cleaned out?

What can I do to get these machines working properly again?

[Sebastian Unger (sebunger44)]

Looks like I can attach only one crash report per comment?!

[Ang Bongsong (notinsideofhereiamnotinside)]

Like Bill Miller (#19) I downgraded via the suggestion in #16, and it broke [amongst other things] Software Updater, Software & Updates (which also affects Synaptic, in that its interface for managing repositories becomes inaccessible). I had to temporarily enable trusty proposed via adding it to /etc/apt/sources.list, and upgrade the packages again. This is a terribly annoying situation, and really should not be happening on an LTS release.

[Ben Howard (utlemming)]

See Bug #1505164:

# apt-get install libapache2-mod-wsgi-py3
The following packages have unmet dependencies:
 libapache2-mod-wsgi-py3 : Depends: libpython3.4 (>= 3.4~b1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

[Cory Benfield (lukasaoz)]

Hi folks, I'm a requests and urllib3 maintainer.

Looks like you're running into urllib3 issue #385. This was fixed in urllib3 1.8.3 by the changelog. However, it should be a simple matter to backport the relevant patch from #385, which should get your systems working again.

[0]: https://github.com/shazow/urllib3/pull/385

[Brian Murray (brian-murray)]

Hello Didier, or anyone else affected,

Accepted python-urllib3 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-urllib3/1.7.1-1ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Matthias Klose (doko)]

this lets the reproducer in comment #6 work with the python3.4 3.4.3 in trusty-proposed.

[Sebastian Unger (sebunger44)]

Am I understanding this right that with the above proposed fix to urllib3, python itself will go back up to version 3.4.3? I.e. the archive revert is reverted?

[Matthias Klose (doko)]

to test the python3 behaviour for certificate verification, you have to use urllib.request.urlopen. requests does it's own certificate verification.

[Matthias Klose (doko)]

See LP: #1348954 for the 3.4.3-1ubuntu1~14.04.2 update, currenty in trusty-proposed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-urllib3 - 1.7.1-1ubuntu4

---------------
python-urllib3 (1.7.1-1ubuntu4) trusty-proposed; urgency=medium

  * SRU: LP: #1500768.
  * Backport pull request #385, fixing TLS verification when using a proxy.

 -- Matthias Klose <email address hidden> Mon, 12 Oct 2015 18:03:36 +0200

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for python-urllib3 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Dan Watkins (daniel-thewatkins)]

This problem still exists in the version of urllib3 that is vendored by the version of requests that is vendored by the pip wheel that is included in the python-virtualenv package.

[Barry Warsaw (barry)]

We won't be fixing virtualenv in Trusty, there's just too much to backport. The solution is to use the built-in virtual environment support in Python 3 for Trusty. Install the `python3.4-venv` package and use `python3 -m venv <dir>`. Then proceed as you normally would with a virtual environment.

[Matthias Klose (doko)]

people claiming too much work should really look at the package in the first place, before making such claims. Now waiting in trusty-proposed.

[Matthias Klose (doko)]

test upload of python-virtualenv in the ubuntu-toolchain-r/ppa PPA

[Chris J Arges (arges)]

Hello Didier, or anyone else affected,

Accepted python-virtualenv into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-virtualenv/1.11.4-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Matthias Klose (doko)]

checked that virtualenv still works with the proxy setting from comment #6 and the python 2.7.10 from the ubuntu-toolchain-r/ppa archive. works as well with the python2.7 from trusty.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-virtualenv - 1.11.4-1ubuntu1

---------------
python-virtualenv (1.11.4-1ubuntu1) trusty-proposed; urgency=medium

  * SRU: LP: #1500768.
  * Backport pull request #385, fixing TLS verification when using a proxy.

 -- Matthias Klose <email address hidden> Sat, 12 Dec 2015 15:00:33 +0100