virtualenv's included pip does not use system libraries

[Stratos Moros]

Ubuntu version: Ubuntu 14.04.1 LTS
python-virtualenv version: 1.11.4-1

Normally, pip vendors a few python libraries under a _vendor directory in its source tree. When packaged for Ubuntu, pip is patched to use system installed of these libraries.

python-virtualenv installs its own version of pip (found in /usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl) in every virtualenv it creates. Unfortunately, this version of pip is not patched and does not use the system wide python libraries.

There is an open issue affecting urllib3 (https://bugs.launchpad.net/ubuntu/+source/python-urllib3/+bug/1412545) that makes pip fail under certain circumstances. Patching virtualenv's pip would make it get the fix when urllib3 is updated, instead of requiring a separate fix.

[Barry Warsaw (barry)]

On Jan 27, 2015, at 01:02 PM, Stratos Moros wrote:

>python-virtualenv installs its own version of pip (found in /usr/share
>/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl) in every
>virtualenv it creates. Unfortunately, this version of pip is not patched
>and does not use the system wide python libraries.

I don't think that's actually true. We build the .whl files when we update
the dependencies. They all come from the same source package. In fact,
Debian Python policy only allows for wheel packages in a limited set of cases,
explicitly to support virtualenv and pyvenv.

It's definitely true that when a new dependency is released, any existing
virtualenvs would have to be recreated or explicitly updated in order to get
the fix.

>There is an open issue affecting urllib3
>(https://bugs.launchpad.net/ubuntu/+source/python-urllib3/+bug/1412545)
>that makes pip fail under certain circumstances. Patching virtualenv's
>pip would make it get the fix when urllib3 is updated, instead of
>requiring a separate fix.

Ah, this is for Trusty. I think things were done differently in LTS, which
does make things more complicated. So it probably requires an SRU.

[Stratos Moros (stmoros)]

I haven't looked into newer Ubuntu versions, so everything I'm posting is about Trusty.

As far as I can tell, virtualenv's bundled pip wheel is not built by Debian maintainers, but by virtualenv's developers themselves. In fact, if you compare the wheel python-virtualenv installs in /usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl with the one in virtualenv's tree (https://github.com/pypa/virtualenv/raw/6dc9b1d376b041d83472c03c1ef334919a3b7c12/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl) their contents are identical.

You can also tell that the bundled pip is not patched by comparing its contents with the contents of the python-pip package. The bundled pip has a _vendor directory and all imports reference it (eg "from pip._vendor.requests import ***"). The pip that is installed through apt is missing that directory and all imports have been patched to import system wide libraries (eg "from pip import ***).

The larger issue is whether the above behavior is correct. The pypa developers (maintainers of pip, virtualenv etc) prefer vendoring libraries, while Debian's python packagers prefer splitting packages and patch them accordingly. How this applies to virtualenv's bundled wheel which is installed as part of virtualenv's normal operation is not clear to me.

I am not very familiar with Debian Python Policy, but I would argue that the contents of the bundled wheel should be patched as well, since:

1. The bundled pip's version is the same as the apt pip's version, so bugfixes on dependencies would equally apply to both.
2. Currently, the bundled pip is not the same as the pip apt installs, even though the are both the same version.

Esesntialy, I think that the pip virtualenv installs should be the same as the one apt installs.

If this can't be done at this point of the release, bugfixes would need to be applied to pip's vendored libraries (and their vendored libraries).

[Barry Warsaw (barry)]

On Jan 28, 2015, at 12:46 PM, Stratos Moros wrote:

>I haven't looked into newer Ubuntu versions, so everything I'm posting
>is about Trusty.
>
>As far as I can tell, virtualenv's bundled pip wheel is not built by
>Debian maintainers, but by virtualenv's developers themselves.

You're right, Trusty does it differently. This is in fact the reason why
virtualenv in Trusty works, but pyvenv doesn't.

The bundling of the wheel files violates Debian policy. This was discovered
for pyvenv and disabled before it was fixed properly. The proper fix (made in
the Debian first) didn't make it into Ubuntu until after Trusty. virtualenv
*also* violated Debian policy but that wasn't discovered until later, so the
policy-violating-but-working version of virtualenv is present in Trusty.

Later, in Debian (and you'll find all these tools fixed in Jessie) we fixed
both virtualenv and pyvenv to be compliant with Debian policy *and* actually
work. ;) Unfortunately Trusty didn't get any of these fixes.

I've talked to many many people about this and I think it's a real problem.
Ideally, and especially because Trusty is an LTS, we would fix both virtualenv
and pyvenv to be both policy-compliant and working. That would certainly be
in the best interest of our users. The difficulty is because there's a stack
of packages that would need SRUing to do it properly; not just virtualenv and
Python 3.4, but also the 6 or so dependencies which all build the .whl files
that virtualenv and pyvenv vendor (in upstream). We'd also need to pull the
python-wheel package into Trusty since it didn't appear in Debian until after
Trusty was released.

Needless to say, this is a complex problem with a proper solution requiring a
significant amount of resources to accomplish. That's assuming that such SRUs
would be approved in the first place.

I'll try to at least have some conversations with the appropriate people to
figure out whether and how we can fix this for Trusty.

[Stratos Moros (stmoros)]

Maybe I'm missing something, but I don't see way any other package would need an SRU. The dependencies of the bundled pip are the same as the dependencies of apt's pip (since they're both the same version), so they are already available in apt.

[Barry Warsaw (barry)]

On Jan 28, 2015, at 05:14 PM, Stratos Moros wrote:

>Maybe I'm missing something, but I don't see way any other package would
>need an SRU. The dependencies of the bundled pip are the same as the
>dependencies of apt's pip (since they're both the same version), so they
>are already available in apt.

Except that in Trusty, we don't build the .whl files so if we fix it the way
it's fixed in e.g. Jessie and Vivid, we need the toolchains to build that
stack.

[Stratos Moros (stmoros)]

Ah ok, my bad. I thought you were talking about pip's dependencies on your previous comment. Thanks for looking into this.

[Stefano Rivera (stefanor)]

This was fixed years ago.