proxy isn't used after a dropped connection

Bug #1412545 reported by Stratos Moros
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-urllib3 (Ubuntu)
Fix Released
Undecided
Unassigned
Trusty
Fix Released
Undecided
Diogo Matsubara
python-virtualenv (Ubuntu)
Invalid
Undecided
Unassigned
Trusty
Invalid
Undecided
Unassigned

Bug Description

Ubuntu version: Ubuntu 14.04.1 LTS
python-urllib3 version: 1.7.1-1build1

Steps to reproduce:

1. set up an http proxy
2. configure a ProxyManager to use said http proxy
3. make successive GET requests to https://pypi.python.org/

example script: https://gist.github.com/stratoukos/7545c5c909fa9b5d1cfb

What happens:

urllib3 stops using the http proxy after the connection is dropped (after 24 requests in my testing with pypi. other people have seen different numbers)

What I expected to happen:

urllib3 should always use the proxy

Other Info:

This has been fixed in commit 1c30a1f3 of urllib3 and included in its 1.8.3 release. This bug also affects pip and requests as reported here: https://github.com/pypa/pip/issues/1805

I really hope the bugfix will be backported, since pip is currently unusable behind an outgoing firewall on 14.04

[Impact]
urllib3 stops using the proxy after a connection is dropped making users of
python-urllib3 (such as pip) that are behind a firewall unable to connect to external sites.

[Test Case]
1. Start a trusty VM
2. Get the test script ($ wget https://gist.githubusercontent.com/stratoukos/7545c5c909fa9b5d1cfb/raw/456381dff95d503818d35c393e71ec0272ab08d3/gistfile1.py -O test.py)
3. Install Squid ($ apt-get install squid)
4. Install python-urllib3 if it's not installed ($ apt-get install python-urllib3)
5. Block outgoing connections (
    $ sudo iptables -A OUTPUT -m owner --uid-owner root -j ACCEPT
    $ sudo iptables -A OUTPUT -m owner --uid-owner proxy -j ACCEPT
    $ sudo iptables -A OUTPUT -p tcp --dport 80 -j DROP
    $ sudo iptables -A OUTPUT -p tcp --dport 443 -j DROP
)
6. Run the test script ($ python test.py)
7. In another terminal, tail the squid log ($ sudo tailf /var/log/squid3/access.log)

With python-urllib3 1.7.1-1build1, one would see a connection timeout after 24
or so requests, while with the backported package, one sees hits in the proxy
log meaning the requests are going through the proxy after a reconnect.

[Regression Potential]

The fix was released on 1.8.3 and the current upstream code is on 1.9.1 so the
potential for the upstream fix code to be wrong is minimal. That said, the
current upstream code moved some modules around and the backport is an attempt
to recreate the intent of the patch in the old codebase available on Trusty
making the mininmal amount of changes necessary to get things working.

Related branches

Revision history for this message
Diogo Matsubara (matsubara) wrote :

This bug has been fixed on Vivid and I nominated it to be backported on Trusty.

Changed in python-urllib3 (Ubuntu):
status: New → Fix Released
Changed in python-urllib3 (Ubuntu Trusty):
status: New → Triaged
Robie Basak (racb)
Changed in python-urllib3 (Ubuntu Trusty):
assignee: nobody → Diogo Matsubara (matsubara)
description: updated
Revision history for this message
Stratos Moros (stmoros) wrote :

After doing some more digging, it seems that while the above is enough to fix pip, requests and urllib3 it will not fix virtualenv.

Virtualenv comes with its own version of pip which it installs in every virtualenv, regardless of the OS version of pip. The included version of pip is in /usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl. If you look inside the wheel (it's a zip archive), you can see that it has not been patched to use the apt installed packages. Instead it uses its vendored requests, which in turns uses its vendored urllib3 which still exhibits this bug.

Should I raise this in virtualenv's tracker?

Revision history for this message
Diogo Matsubara (matsubara) wrote :

Hi Stratos, thanks for digging further. I think you're correct and should raise the bug within virtualenv's tracker so they can patch the version of urllib3 shipped with pip. Let's use this bug to track the backport to trusty.
Thank you!

Revision history for this message
Diogo Matsubara (matsubara) wrote :

Added a new task for python-virtualenv on trusty as this bug affects the embedded urllib shipped with it. It s a bit complicated to fix that part of the bug report and it's being tracked on: https://bugs.launchpad.net/ubuntu/+source/python-virtualenv/+bug/1415028

As a workaround for virtualenv one can go with Barry's suggestion:
<barry> rbasak: i think that's reasonable. probably as a workaround, you can create the virtualenv, and then `pip install --upgrade urllib3`

Changed in python-virtualenv (Ubuntu Trusty):
status: New → Triaged
Revision history for this message
Robie Basak (racb) wrote :

Uploaded Diogo's merge proposal. Thanks! Now awaiting SRU team approval.

Changed in python-urllib3 (Ubuntu Trusty):
status: Triaged → In Progress
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Stratos, or anyone else affected,

Accepted python-urllib3 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-urllib3/1.7.1-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-urllib3 (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Stratos Moros (stmoros) wrote :

The bug is fixed using the test script with version 1.7.1-1ubuntu0.1 0

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-urllib3 - 1.7.1-1ubuntu0.1

---------------
python-urllib3 (1.7.1-1ubuntu0.1) trusty; urgency=medium

  * debian/patches/05_backport_proxy_fix.patch: Backport upstream fix 1c30a1f3
    from 1.8.3 release. Fixes the issue when a dropped connection makes
    further connections ignore the proxy. (LP: #1412545)
 -- Diogo Matsubara <email address hidden> Mon, 26 Jan 2015 12:02:13 -0200

Changed in python-urllib3 (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of the Stable Release Update for python-urllib3 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Stefano Rivera (stefanor) wrote :

I assume this is all done. Closing the bug.

Changed in python-virtualenv (Ubuntu):
status: New → Invalid
Changed in python-virtualenv (Ubuntu Trusty):
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.