pip 1.5.4 import an invalid dependencies

[Claustry Ginne]

With a Ubuntu 14.04 constantly updated:

$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 11, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar # noqa
  File "/usr/lib/python2.7/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 25, in <module>
    from requests.compat import IncompleteRead
ImportError: cannot import name IncompleteRead

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Barry Warsaw (barry)]

I've try several ways to reproduce this but can't seem to manage it. It works for me on both 14.04 and 14.10 (e.g. python-pip 1.5.4-1ubuntu1 on 14.04.2) in chroots and VMs, inside a virtualenv and outside a virtualenv (with both --user and sudo into /usr/local). `pip --help` works, as does just `pip`.

I'd be happy to help debug this, but I can't trigger the problem locally, so I'll need more information.

Let's start with this: is python-requests installed on your system? What does `dpkg-query -W python-requests` say?

[Barry Warsaw (barry)]

For anyone else hitting this bug, please get in contact via IRC (nick: barry on #ubuntu-devel) or email me, or follow up here with more information on how to reproduce it. If it's happening to you we can do some live debugging over IRC.

[Brian Murray (brian-murray)]

I've been able to recreate this via the following steps:

schroot -u root -c trusty-amd64
apt-get install python-pip debsums
pip install httpie
pip --help
(ImportError traceback)
------------------------------------------------
Here's the output:

(trusty-amd64)root@impulse:/home/bdmurray# pip install httpie
Downloading/unpacking httpie
  Downloading httpie-0.9.2-py2.py3-none-any.whl (66kB): 66kB downloaded
Downloading/unpacking requests>=2.3.0 (from httpie)
  Downloading requests-2.7.0-py2.py3-none-any.whl (470kB): 470kB downloaded
Downloading/unpacking Pygments>=1.5 (from httpie)
  Downloading Pygments-2.0.2-py2-none-any.whl (672kB): 672kB downloaded
Installing collected packages: httpie, requests, Pygments
  Found existing installation: requests 2.2.1
    Uninstalling requests:
      Successfully uninstalled requests
Successfully installed httpie requests Pygments
Cleaning up...
(trusty-amd64)root@impulse:/home/bdmurray# pip --help
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 11, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar # noqa
  File "/usr/lib/python2.7/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 25, in <module>
    from requests.compat import IncompleteRead
ImportError: cannot import name IncompleteRead
(trusty-amd64)root@impulse:/home/bdmurray# debsums python-requests
debsums: missing file /usr/lib/python2.7/dist-packages/requests-2.2.1.egg-info (from python-requests package)
/usr/lib/python2.7/dist-packages/requests/__init__.py OK

[Brian Murray (brian-murray)]

In particular notice the following:

debsums: missing file /usr/lib/python2.7/dist-packages/requests-2.2.1.egg-info (from python-requests package)

This happens with either the python-pip package from -updates or the release pocket.

[Brian Murray (brian-murray)]

This also seems to be happening in 14.10:

https://errors.ubuntu.com/problem/6c29a66a2f199b9a3a1fef92d78b74834ddd3c2f
https://errors.ubuntu.com/problem/456029ce0b329deb4b4dadaf27af55fd515c1136

[Brian Murray (brian-murray)]

I've overridden the specific errors that were preventing the python-pip SRU from continuing to phase.

=== modified file 'phased-updates-overrides.txt'
--- phased-updates-overrides.txt 2015-05-04 15:11:57 +0000
+++ phased-updates-overrides.txt 2015-05-14 16:21:50 +0000
@@ -654,3 +654,8 @@
 virtualbox, 4.3.18-dfsg-2ubuntu2, https://errors.ubuntu.com/problem/7f00260fc0706bef5a73f0de8b5a91f3e9f5abd3
 # skipped version - previous version had no errors but others did
 apt, 1.0.1ubuntu2.7, https://errors.ubuntu.com/problem/b4050ecf674d4872b4257787c5ab7818f44699c8
+# not a regression - happens with the package version from release pocket
+python-pip, 1.5.4-1ubuntu1, https://errors.ubuntu.com/problem/6c29a66a2f199b9a3a1fef92d78b74834ddd3c2f
+python-pip, 1.5.4-1ubuntu1, https://errors.ubuntu.com/problem/3b7dcc9e045e1cc176cc883ee14a42240a2c7cdc
+python-pip, 1.5.4-1ubuntu1, https://errors.ubuntu.com/problem/456029ce0b329deb4b4dadaf27af55fd515c1136
+python-pip, 1.5.4-1ubuntu1, https://errors.ubuntu.com/problem/38cc7eadb438bc637121a8725369ff192ece75de

[Barry Warsaw (barry)]

FWIW, I fixed this in Debian's 1.5.6-5. I'll work on an SRU to trusty-updates.

[Barry Warsaw (barry)]

Well, I fixed the import error but haven't figured out the debsums (removal of .egg-info file) problem yet.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pip - 1.5.6-6ubuntu1

---------------
python-pip (1.5.6-6ubuntu1) wily; urgency=medium

  * Merge with Debian. Remaining changes:
    - debian/patches/set_user_default.patch: Default to --user on non
      virtualenv.
    - d/control: update-maintainer
  * Merge closes LP: #1324391

python-pip (1.5.6-6) unstable; urgency=medium

  [ Felix C. Stegerman ]
  * d/patches/0006-bug-785787.patch: Allow pip 1.5.6 to handle the PEP 440
    "epoch" field. (Closes: #785787)

  [ Barry Warsaw ]
  * d/control:
    - Update Vcs-* header for conversion to git-dpm.
    - Add myself to Uploaders.
  * d/watch: Update to pypi.debian.net redirector.
  * wrap-and-sort

 -- Barry Warsaw <email address hidden> Thu, 28 May 2015 11:22:01 -0400

[Barry Warsaw (barry)]

Now that 1.5.6-6ubuntu1 has finally been promoted to wily, I've verified that the repro recipe bdmurray describes in #4 is fixed, both the import error and the debsums error.

I'll now be working on backporting the patch to SRU a fix into Trusty.

[Barry Warsaw (barry)]

SRU information.

[Impact]
pip installing a package as root into /usr/local is a supported use case. This bug describes two problems with that in Trusty. First, installing some packages, such as httpie, can upgrade requests to an incompatible version, causing a subsequent ImportError when running pip. It can also mess with the system's requests egg.info causing an error in debsums

[Test Case]
See comment #4:

$ sudo pip install httpie
$ pip --help
$ debsums -s python-requests

[Regression Potential]
While no regressions have been detected during testing, watch for system packages getting messed up or replaced after a sudo pip install, or the user's packages, or /usr/local admin packages getting messed up.

[Barry Warsaw (barry)]

1.5.4-1ubuntu2 has been uploaded to trusty-proposed and awaits SRU approval.

[Chris J Arges (arges)]

Hello Claustry, or anyone else affected,

Accepted python-pip into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/1.5.4-1ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I've tried seeing if this SRU fixes the issue I encountered in comment #4 and it does not seem to.

Setting up python-pip (1.5.4-1ubuntu2) ...
Setting up python-wheel (0.24.0-1~ubuntu1) ...
(trusty-amd64)root@impulse:~# pip install httpie
Downloading/unpacking httpie
  Downloading httpie-0.9.2-py2.py3-none-any.whl (66kB): 66kB downloaded
Downloading/unpacking requests>=2.3.0 (from httpie)
  Downloading requests-2.7.0-py2.py3-none-any.whl (470kB): 470kB downloaded
Downloading/unpacking Pygments>=1.5 (from httpie)
  Downloading Pygments-2.0.2-py2-none-any.whl (672kB): 672kB downloaded
Installing collected packages: httpie, requests, Pygments
  Found existing installation: requests 2.2.1
    Not uninstalling requests at /usr/lib/python2.7/dist-packages, owned by OS
Successfully installed httpie requests Pygments
Cleaning up...
(trusty-amd64)root@impulse:~# pip --help
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 61, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar # noqa
  File "/usr/lib/python2.7/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 25, in <module>
    from requests.compat import IncompleteRead
ImportError: cannot import name IncompleteRead

The debsums all check out for python-requests.

[Barry Warsaw (barry)]

On Jun 16, 2015, at 06:53 PM, Brian Murray wrote:

>I've tried seeing if this SRU fixes the issue I encountered in comment
>#4 and it does not seem to.

Looks like a versioned binary package dependency got lost. Testing a fix now.

[Barry Warsaw (barry)]

1.5.4-1ubuntu3

[Brian Murray (brian-murray)]

Hello Claustry, or anyone else affected,

Accepted python-pip into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/1.5.4-1ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I've used the same test case as before and with python-pip version 1.5.4-1ubuntu3 this bug is now fixed. Thanks!

Setting up python-pip (1.5.4-1ubuntu3) ...
Setting up python-wheel (0.24.0-1~ubuntu1) ...
(trusty-amd64)root@impulse:/home/bdmurray# pip install httpie
Downloading/unpacking httpie
  Downloading httpie-0.9.2-py2.py3-none-any.whl (66kB): 66kB downloaded
Downloading/unpacking requests>=2.3.0 (from httpie)
  Downloading requests-2.7.0-py2.py3-none-any.whl (470kB): 470kB downloaded
Downloading/unpacking Pygments>=1.5 (from httpie)
  Downloading Pygments-2.0.2-py2-none-any.whl (672kB): 672kB downloaded
Installing collected packages: httpie, requests, Pygments
  Found existing installation: requests 2.2.1
    Not uninstalling requests at /usr/lib/python2.7/dist-packages, owned by OS
Successfully installed httpie requests Pygments
Cleaning up...
(trusty-amd64)root@impulse:/home/bdmurray# pip --help

Usage:
  pip <command> [options]

Commands:
  install Install packages.
  uninstall Uninstall packages.
  freeze Output installed packages in requirements format.
  list List installed packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pip - 1.5.4-1ubuntu3

---------------
python-pip (1.5.4-1ubuntu3) trusty-proposed; urgency=medium

  * d/control: Add missing versioned dependencies.
  * d/control, d/python-pip-whl.install: wrap-and-sort.

python-pip (1.5.4-1ubuntu2) trusty-proposed; urgency=medium

  * SRU, d/patches:
    - use-venv-wheels.patch: Backport from 1.5.6-5: Use the .whl files for
      de-vendorized dependencies both inside and outside the virtual
      environments. (Closes: #744145) (LP: #1324391)
    - debian-771794.patch: Backport from 1.5.6-4: Prevent pip from
      removing system Python packages (Closes: #771794)
  * d/control: update-maintainer.

 -- Barry Warsaw <email address hidden> Tue, 16 Jun 2015 15:26:04 -0400

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for python-pip has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[William Grant (wgrant)]

python-pip from trusty-updates is now uninstallable. python-pip depends on python-pip-whl, which depends on three binaries that exist only in -proposed: python-requests-whl, python-six-whl and python-setuptools-whl.

[William Grant (wgrant)]

Reverted trusty-updates from 1.5.4-1ubuntu3 to 1.5.4-1ubuntu1.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pip - 1.5.4-1ubuntu3

---------------
python-pip (1.5.4-1ubuntu3) trusty-proposed; urgency=medium

  * d/control: Add missing versioned dependencies.
  * d/control, d/python-pip-whl.install: wrap-and-sort.

python-pip (1.5.4-1ubuntu2) trusty-proposed; urgency=medium

  * SRU, d/patches:
    - use-venv-wheels.patch: Backport from 1.5.6-5: Use the .whl files for
      de-vendorized dependencies both inside and outside the virtual
      environments. (Closes: #744145) (LP: #1324391)
    - debian-771794.patch: Backport from 1.5.6-4: Prevent pip from
      removing system Python packages (Closes: #771794)
  * d/control: update-maintainer.

 -- Barry Warsaw <email address hidden> Tue, 16 Jun 2015 15:26:04 -0400

[Steve Langasek (vorlon)]

The dependencies are now landed in trusty-updates, so re-promoting.