pyvenv fails due to mising ensurepip module

[wizwiz50]

Hello,

I noticed the following

# fails
python3.4 -m venv --clear python-venv
Error: Command '['.../external/python-venv/bin/python3.4', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

# works, but no pip
python3.4 -m venv --clear --without-pip python-venv

Thank you

[raylu (raylu)]

venv actually fails before creating the activate{,.csh.fish} scripts, so it's a little tricky to activate the venv.

Interestingly, the docs for ensurepip are there: http://packages.ubuntu.com/trusty/all/python3.4-doc/filelist

[Matthias Klose (doko)]

we do not want to include all the pre-packaged python modules in the wheel files, python-pip, setuptools, distlib, python-six, python-requests, etc ...

so what we want to do:

 - if pyvenv3 is called with --system-site-packages, then don't require ensurepip at all,
   but try to import the system python3-pip and fail with an import error if it fails,
   pointing the user to install python3-pip

- when called without --system-site-packages:
  provide a wheel package/something which is built from the system supplied
  python-pip, setuptools, distlib, ... packages and then used in the venv. Maybe
  use rewheel for this, but in any case build this in a separate source.

[Matthias Klose (doko)]

and then provide the pyvenv3 symlink in the python3-defaults package

[Barry Warsaw (barry)]

On Apr 11, 2014, at 09:27 PM, Matthias Klose wrote:

> - if pyvenv3 is called with --system-site-packages, then don't require
> ensurepip at all, but try to import the system python3-pip and fail with
> an import error if it fails, pointing the user to install python3-pip

This doesn't work currently (at least in my limited testing). python3-pip
doesn't install into the right location in a venv.

>- when called without --system-site-packages:
> provide a wheel package/something which is built from the system supplied
> python-pip, setuptools, distlib, ... packages and then used in the
> venv. Maybe use rewheel for this, but in any case build this in a separate
> source.

We'll have to do some work on rewheel to make it work recursively (e.g. all
the bundles that the bundles) and then package it up in PyPI and then the
archive. But I think this is the right long term solution.

[raylu (raylu)]

Remind me why we don't want to include everything in the wheel files?

[Beau (w-beau)]

Python 3.4's bundled ensurepip module is missing, and as a result its bundled virtualenv module is broken. Crucial Python 3.4 functionality is missing.

Is there a chance this will be fixed in the near future?

[Barry Warsaw (barry)]

On Apr 17, 2014, at 04:44 PM, Beau wrote:

>Python 3.4's bundled ensurepip module is missing, and as a result its
>bundled virtualenv module is broken. Crucial Python 3.4 functionality
>is missing.
>
>Is there a chance this will be fixed in the near future?

The plan is to fix it in Debian and SRU it into Trusty. I will be working on
that next week.

[Beau (w-beau)]

Cool. Thanks Barry.

[Donald Stufft (dstufft)]

As Barry said you can't gate installing pip on system packages or not. Further more I would suggest that installing the OS modified pip into a virtualenv will lead to surprising behavior for people. People should be free to update their pip inside of a virtual environment as they wish, however doing that will trigger people to go from a OS modified pip to an upstream pip and can subtly change the behavior.

One thing to ensure that if you use rewheel that it should also rebundle the bundled libraries. Pip bundles those things because otherwise we assert what valid versions of things you can have installed alongside pip. If you just simple install say requests into the virtual environment alongside pip then when someone comes along and wants to install their app which uses a different version of requests that isn't compatible then either pip or their app breaks.

[Matthias Klose (doko)]

Am 17.04.2014 22:51, schrieb Donald Stufft:
> As Barry said you can't gate installing pip on system packages or not.

Could you clarify what you mean by "gate installing"? I'd rather hint people to
install a system package if it is available instead of an arbitrary wheel (if it
is installed for system python, and not into venv).

I would like to see this as the default behaviour, unless an option
--go-get-the-cheese is given to pip.

[Donald Stufft (dstufft)]

I mean you can't decide not to install pip into the virtual environment if --system-site-packages is passed to the virtual environment.

[Barry Warsaw (barry)]

Let's look at use cases.

1) User wants to pip install something into their ~/.local directory. Having python3-* packages installed does not give them pip3. They have to apt-get install python3-pip. If they try to `pip3 install foo` it will try to install it into the system dist-packages and this will fail with a permission denied. They must `pip3 install --user foo` and it will end up in their ~/.local/lib/pythonX.Y/site-packages directory, and it will be importable by the system Python. All of this works right out of the box today:

% pip3 install --user nose2
[...]
% python3.4 -c "import nose2; print(nose2)"
<module 'nose2' from '/home/barry/.local/lib/python3.4/site-packages/nose2/__init__.py'>

2) User wants to pip install something into the system package path. Again they must install python3-pip, and this time they must use `sudo pip3 install foo`. foo gets installed into /usr/local/lib/python3.4/dist-packages. All of this works right out of the box today:

% sudo pip3 install nose2
[...]
% python3.4 -c "import nose2; print(nose2)"
<module 'nose2' from '/usr/local/lib/python3.4/dist-packages/nose2/__init__.py'>

3) Use creates a virtualenv and wants to install foo into the virtualenv. Regardless of whether --system-site-packages is used or not, and regardless of whether pyvenv-3.4 or virtualenv is used, the virtualenv should already contain `/my/venv/bin/pip` and the user should be able to enter the venv and run `pip install foo`. This should install foo into /my/venv/lib/pythonX.Y/site-packages and have it be importable.

This latter is the use case that's currently broken with pyvenv-3.4 due to the disabling of ensurepip, so this is the use case we need to fix. System pip package shouldn't enter into the picture. The question we have to answer, IMHO, is whether we want ensurepip to use the recursively bundled pip (and its dependencies) or to rewheel system pip and dependencies (in which case we'd need to handle the circular apt dependencies from python3.4->python3-pip).

I think the determining factor should be Debian policy w.r.t. build-from-source. Wheels are just zips so unzipping the two bundled wheels results in all the recursive source files contained within the wheels. But how are the wheels built to be included in the tarball? That recipe must be embodied in the Debian package so that the bundled wheels (with their own vendored libraries) can be completely reproduced from upstream source.

If it's not possible to do that reproducibly, then we need an approach like rewheel.

[Donald Stufft (dstufft)]

Your use cases are correct. Ideally in the future 1) won't require --user but that's a different discussion :)

The wheels in ensurepip are taken from PyPI and are built with ``python setup.py bdist_wheel --no-script`` (I may have the command parameter wrong for excluding scripts, and the latest Wheel excludes them by default). You should be able to reproduce them by downloading the tarball, unpacking it, and running that command. If you can't let me know and I'll fix it. It may not be a byte for byte reproduction, I don't know if ``setup.py bdist_wheel`` for pure python has a deterministic output, but the contents should be the exact same.

[Barry Warsaw (barry)]

If I'm reading the rewheel code correctly, I think we are going to have to do something like the following:

* Package up wheel for Debian (python-wheel, and python3-wheel). This gives us the setuptools bdist_wheel command.

* At Python build-time we need *.dist-info/RECORD files for setuptools, pip, and recursively vendored packages. We do not currently generate these, and there's a bit of a bootstrapping problem. But let's say for the moment we can generate all these dist-info directories, we need to put them some place where the python3.4 debian/rules can find it *without* downloading anything from PyPI. My thought is that we bootstrap this by building setuptools-wheel, pip-wheel, etc binary packages which only contain the dist-info/ directories. We can then build-dep python3.4 on those *-wheel packages.

* We need to package up rewheel too, both in PyPI and Debian. It'll take a little work to whip the code in shape, but it's not too bad I guess. I really don't like patching Python the way rewheel does it, so I'll look at alternatives. Hopefully bkadrda will accept pull requests.

Anyway, that's a general plan, and the devil is in the details, but I think I see how it all hangs together.

[Donald Stufft (dstufft)]

So I think that the debundling of requests, html5lib, etc that python-pip does and rewheel are mutually incompatible and are going to lead to a lot of end user pain.

Specifically the reason we bundle those things in pip is because if pip depends on requests, and someone does ``pip install requests==1.0`` then their pip will break and they'll have to manually uninstall requests and manually install the correct version of requests. Specifically this policy in pip of bundling instead of depending on has come from our experience with our current single dependency (setuptools) and the pain that people have had release after release where things would just completely break unless they were very careful to do things in a particular order. I suspect that problem would get even worse with a dependency that other people actually depend on and not something that most people just happen to have installed alongside their own dependencies.

I think supporting the ability to arbitrarily install different versions of things, even if pip itself depends on it, is important because that's one of the primary use cases for a virtual environment.

You *could* theortically make it work by making rewheel rebundle those things inside of pip, but you'd also need to undo the patch Debian does to the import statements (although likely pip would just be smarter about those instead) but the real killer there is that pip has modified the import statments in some of those bundled modules in order to have them import other bundled things instead of globally installed things.

In general I think the decision has to be between:

1) Treat the .whl files in ensurepip as data files, or alernatively bundled copies of code that are intended to be bundled (as per the Debian manual in 4.13), and continue to do your normal modifications to python-pip.
2) Don't debundle things from python-pip (thus making you have to maintain a second set of patches against anything bundled) and rewheel into the venv.
3) Leave ensurepip broken and make myself and lots of Python developers sad ;(

In general my preferences are 1, 2 and a very very very very far in the back 3. Additionally 1 matches with how python-virtualenv is treated (which has copies of pip/setuptools in exactly the same way as ensurepip) in that they are moved to /usr/share instead of being kept inside of the package directory.

One *possible* solution is to install the bundled things into their own private directory and modify sys.path inside of pip to import from there, but I think that is going to cause a lot of bugs and broken behavior because pip looks at sys.path to figure out what is installed and it will probably see requests is installed and try to uninstall it to install an upgraded version, or not install requests into the virtual environment where others can access it because it sees requests installed on it's sys.path.

There's also a relatively minor UX concern if 2) is the solution that issuing a ``pip install --upgrade pip`` in a virtual environment will get you an unmodified copy of pip so you'll end up with a slight set of subtle differences, the most obvious being what CAs are trusted. However some of that will h...

[Matthias Klose (doko)]

> 1) Treat the .whl files in ensurepip as data files, or alternatively bundled copies
> of code that are intended to be bundled (as per the Debian manual in 4.13),
> and continue to do your normal modifications to python-pip.

chapter 4 talks about source packages, not binary packages. And these have to be rebuilt from source. At least the windows installer binaries don't have any source code.

> 2) Don't debundle things from python-pip (thus making you have to maintain
> a second set of patches against anything bundled) and rewheel into the venv.

That would mean, using another copy of the pip & friends sources, and having different pip versions on the system.

3) Leave ensurepip broken and make myself and lots of Python developers sad ;(

Sure.

So why not build the wheel files from the Debian python-pip source package, build-depending on python-six, python-requests, etc, then revert the de-vendorize patch. The binary package should have Built-Using information about these build dependencies. Then provide exactly this wheel in the python-pip and the python3-pip packages.

[Yatharth Agarwal (yatharth999)]

As a stop-gap solution for n00bs like me, download the lib/ensurepip directory from the Python 3.4 source code, and place it in /usr/lib/python3.4 with the proper permissions.

Hoping this gets resolved soon...

[Barry Warsaw (barry)]

On Apr 24, 2014, at 07:38 PM, Matthias Klose wrote:

>So why not build the wheel files from the Debian python-pip source
>package, build-depending on python-six, python-requests, etc, then
>revert the de-vendorize patch. The binary package should have Built-
>Using information about these build dependencies. Then provide exactly
>this wheel in the python-pip and the python3-pip packages.

I think we'll have to basically build the recursive vendorized pip from
scratch, using the build-depend packages. We'll probably have to write some
scripts to put all that together, which will include reversing the
de-vendorizing patches, recursively.

[Ivo (teh-ivo)]

What makes the situation with these wheels so much more drastic than what happens with python-virtualenv already?

https://packages.debian.org/sid/all/python-virtualenv/filelist

I don't understand why there needs to be an entire effort de-bundling and de-(re-?)-vendoring files and why this hasn't been applied to virtualenv as well?

With the `--without-ensurepip` flag on, the code is only used in isolated venvs, so it can't effect the system.

[Barry Warsaw (barry)]

On May 22, 2014, at 09:32 PM, Ivo wrote:

>I don't understand why there needs to be an entire effort de-bundling
>and de-(re-?)-vendoring files and why this hasn't been applied to
>virtualenv as well?

python-virtualenv violates policy too. It just wasn't noticed before.
Ideally the same set of patches for pyvenv should be applied to virtualenv.

[sumguy (sumguy)]

Can we get a status update? This is still broken and it's sad that such a critical and good-for-the-comunity feature is needlessly broken by default.

Forturnately this doesn't completely break docker as I can just do everything system wide and isolate the image. However it would be lovely to use the features of the language as intened by upstream rather than have them broken by so called "maintainers."

[Barry Warsaw (barry)]

On Jun 03, 2014, at 08:33 PM, Stephen Tanner wrote:

>Can we get a status update?

Matthias and I are actively working on this stack in Debian. I've had to wait
until several packages clear Debian NEW, including one NMU. I've also had to
update several other packages, and provide an update to the ensurepip patch to
Matthias. All of this will hopefully land within the next week is my best
estimate.

Once we have it working in Debian and Utopic, we'll SRU it into 14.04.

[Moppers (moppers)]

Stephen Tanner: There is a second way to easily deploy a virtual environment: You can also use `virtualenv`. In Ubuntu I belive the package is named python-virtualenv (might need a 3!). Installing that with apt-get will give you (as far as I can tell) the same tools for python 3.4 as venv-3.4.

[Donald Stufft (dstufft)]

virtualenv relies on some dirty hacks (right now at least, in the future it'll reuse venv where available) so using venv is preferable. However the python-virtualenv package was allowed to (at least for now, I believe barry is planning to update it to the venv solution once it's done) violate the Debian policy whereas the venv/ensurepip solution wasn't. So python-virtualenv will work correctly and venv won't until barry's work is done.

[Matthias Klose (doko)]

Am 03.06.2014 22:33, schrieb Stephen Tanner:
> Can we get a status update? This is still broken and it's sad that such
> a critical and good-for-the-comunity feature is needlessly broken by
                                                  ^^^^^^^^^^
> default.

If you don't understand Ubuntu's guidelines for packaging and distribution,
that's fine. However this is no reason to troll about it.

[sumguy (sumguy)]

Matthias Klose, coming from a distro who does a lot less mucking about with upstream packages, having deafult language behavior broken infavor of idealistic guidlines is just plain bad for UX.

Spin it how you want to but shiping broken packages means some one just means someone spent the time trying to make the package conform (and I agree with Ivo, why is this just now an issue when virtualenv was shipping "incorrectly" before.) but failed to take the time to make it actually work as expected. It happens, but it should not happen when it does not need to. This was clearly an idealistic need rather than technical need. Thats why I say "needlessly."

Barry Warsaw, thank you for the bump. Too often some one says they will fix it only to let it languish.

[Moppers (moppers)]

Thank you for releasing fix. I for one am genuinely grateful.

[Matthias Klose (doko)]

Am 04.06.2014 15:51, schrieb btreecat:
> Matthias Klose, coming from a distro who does a lot less mucking about
> with upstream packages, having deafult language behavior broken infavor
> of idealistic guidlines is just plain bad for UX.

there is nothing broken in "default language behaviour". There is a binary
missing which enables you to create a virtual environment.

> Spin it how you want to but shiping broken packages means some one just
> means someone spent the time trying to make the package conform (and I
> agree with Ivo, why is this just now an issue when virtualenv was
> shipping "incorrectly" before.) but failed to take the time to make it
> actually work as expected. It happens, but it should not happen when it
> does not need to. This was clearly an idealistic need rather than
> technical need. Thats why I say "needlessly."

shipping virtualenv in such a way is a mistake, which only was discovered after
the release. Knowingly shipping something which we cannot ship makes things worse.

The issue was known way before the release, and it was promised to get fixed
before the release (which didn't happen). So the next time if you are
interested in a fix included in a release, please help understanding and fixing
these issues.

[Barry Warsaw (barry)]

On Jun 05, 2014, at 04:10 PM, Matthias Klose wrote:

>shipping virtualenv in such a way is a mistake, which only was discovered
>after the release.

I'm working on an update to virtualenv which will use the same technique as
pyvenv, so now that we know the problem, we'll be able to solve it there too.

Cheers,
-Barry

[Tin Tvrtkovic (tinchester)]

I just had a brush with this issue. Thank you kind folks for working on fixing it.

[Jonathan Hartley (tartley)]

Apologies for the noise but also: Thanks for working on a thoughtfully-derived correct fix.

[jose (jose-cybergalvez)]

Does anyone know when the fix will be available in the ubuntu repos?

[Barry Warsaw (barry)]

On Jun 10, 2014, at 01:18 PM, jose wrote:

>Does anyone know when the fix will be available in the ubuntu repos?

It's a bit of a long, complicated process, but rest assured we are on it.

[jose (jose-cybergalvez)]

Thanks I appreciate the update and the the effort.

[Adrien Beau (adrienbeau)]

On 2014-06-10, Barry Warsaw wrote:
>
> It's a bit of a long, complicated process, but rest assured we are on it.

It's been three weeks since the status of this bug was changed to "Fix Released", but it is apparently still not available in Ubuntu 14.04. In what time frame can we expect to see it land in the current stable Ubuntu repos? Weeks? Months? Should we hack the installed package as described in comment #17 in the mean time?

[Pierre Equoy (pieq)]

Hi everyone,

I came across this issue this morning when trying to enable a virtualenv with Python 3.4.

First of all, thanks to everyone who worked on that to make a fix available in Debian!

Same question as Adrien Beau: Any update on when a fix will be available? In the meantime, what is the best way to create a virtualenv (featuring pip) using Python 3.4?

[Eddie Dunn (eddie-dunn)]

Pierre, this seemed to work for me:

$ sudo apt-get install python-virtualenv
$ virtualenv /desired/path/to/env
$ which pip
/desired/path/to/env/bin/pip

[orbisvicis (orbisvicis)]

How can I keep track of the update process or download pre-release versions? Fixes have yet to hit the testing (pre-release) repository.

[machrider (machrider)]

Anyone know if and when this fix is planned to be backported to trusty (i.e. the current Ubuntu release)? There doesn't seem to be any information in this bug report to indicate one way or the other. I actually don't even know which version of the python3.4 package fixes it. (Isn't that something bug trackers normally track?)

[uranusjr (uranusjr)]

This is already released to trusty; just run apt-get upgrade and you will get it.

[Rob Speer (rspeer)]

> This should work after you apt-get install python3-venv
> (python3.4-venv).

venv is a module in the Python 3 standard library. I don't understand why it would need a separate package on Ubuntu.

Someone who's installing additional packages could just as well install the better-understood python3-virtualenv. But the idea of venv was that it'd be built into Python, so that someone could finally write example code about setting up a Python environment without the "...and here's what you need to do differently on Ubuntu" section.

[Rob Speer (rspeer)]

As an example of why this matters, you do not typically need root access to use the Python standard library.

[JanMalte (janmalte)]

I have to agree Rob. Having python3-venv as a new and separate package makes it only confusing for developers. I could live with this, if the package would be a dependency of python3, so it gets installed automatically.

[David McKenney-Barschall (david-mckenney-barschall)]

If this is fixed, why is pyvenv-3.4 not working for me without "--without-pip"?

[ellie (et1234567)]

Is there any chance this will be made available without a separate python3-venv package? Since python 3.3 or 3.4 venv is considered as a core package, so the requirement to install python3-venv is somewhat non-intuitive.

Python developers in any sort of support chat will also always tell you that there's no need to install venv for a recent python3 since it's now part of python, so this is unusual packaging in terms of common user expectations.

Combined with the somewhat helpful error when python3-venv is not installed, it seems like this really should work without python3-venv to start with.

[Barry Warsaw (barry)]

On Jun 16, 2015, at 11:37 AM, Jonas Thiem wrote:

>Is there any chance this will be made available without a separate
>python3-venv package? Since python 3.3 or 3.4 venv is considered as a
>core package, so the requirement to install python3-venv is somewhat
>non-intuitive.

Would it be good enough if the Suggests for python3{,.4}-venv were turned into
a Recommends? Then most people would get it but you could use apt-get's
--no-install-recommends if you really didn't want it. (Of course, today you
can use --install-suggests but as that's not the default, most people won't
get it.)

[Matthias Klose (doko)]

I don't think the installation by default would be a good idea. This is distro land, and I think we should promote using the distro packages. People wanting to install venv and pip know how to do this.

[Donald Stufft (dstufft)]

virtual environments and distro packages solve entirely different problems
though. They aren’t two solutions to the same problem.

On June 17, 2015 at 12:11:28 PM, Matthias Klose (<email address hidden>) wrote:
> I don't think the installation by default would be a good idea. This is
> distro land, and I think we should promote using the distro packages.
> People wanting to install venv and pip know how to do this.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1290847
>
> Title:
> pyvenv fails due to mising ensurepip module
>
> Status in python3-defaults package in Ubuntu:
> Fix Released
> Status in python3.4 package in Ubuntu:
> Fix Released
> Status in python3.4 package in Debian:
> Fix Released
>
> Bug description:
> Hello,
>
> I noticed the following
>
> # fails
> python3.4 -m venv --clear python-venv
> Error: Command '['.../external/python-venv/bin/python3.4', '-Im', 'ensurepip',
> '--upgrade', '--default-pip']' returned non-zero exit status 1
>
> # works, but no pip
> python3.4 -m venv --clear --without-pip python-venv
>
> Thank you
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/python3-defaults/+bug/1290847/+subscriptions
>

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Pietro Albini (pietroalbini)]

> I don't think the installation by default would be a good idea.
> This is distro land, and I think we should promote using the distro packages.
> People wanting to install venv and pip know how to do this.
But pyvenv is, since release 3.4, part of the Python standard library. It's not a separated tool or something like that. It's like if the Python os module is shipped in a separated package, and users must install it before importing it in a program.
And I think that, for a module which is meant to be in the standard library, "but on Ubuntu you should install the python3-venv package before" isn't a great thing, is it?

[Matthias Klose (doko)]

On 06/17/2015 06:12 PM, Donald Stufft wrote:
> virtual environments and distro packages solve entirely different problems
> though. They aren’t two solutions to the same problem.

if they address a different issue, the venv package doesn't need to be installed
by default.

[Harry P (hjwp2)]

I do a fair bit of work with beginner Python tutorials. It's not often that we have to say "on linux, this is harder", but this is one area where ubuntu/debian are failing pretty hard compared to the mac + windows experience.

[Donald Stufft (dstufft)]

I think it should be installed by default because it is a documented part
of the Python language. I think the Debian/Ubuntu Python is fundamentally
broken by default as it is currently.

On June 17, 2015 at 12:30:58 PM, Matthias Klose (<email address hidden>) wrote:
> On 06/17/2015 06:12 PM, Donald Stufft wrote:
> > virtual environments and distro packages solve entirely different problems
> > though. They aren’t two solutions to the same problem.
>
> if they address a different issue, the venv package doesn't need to be installed
> by default.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1290847
>
> Title:
> pyvenv fails due to mising ensurepip module
>
> Status in python3-defaults package in Ubuntu:
> Fix Released
> Status in python3.4 package in Ubuntu:
> Fix Released
> Status in python3.4 package in Debian:
> Fix Released
>
> Bug description:
> Hello,
>
> I noticed the following
>
> # fails
> python3.4 -m venv --clear python-venv
> Error: Command '['.../external/python-venv/bin/python3.4', '-Im', 'ensurepip',
> '--upgrade', '--default-pip']' returned non-zero exit status 1
>
> # works, but no pip
> python3.4 -m venv --clear --without-pip python-venv
>
> Thank you
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/python3-defaults/+bug/1290847/+subscriptions
>

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Pietro Albini (pietroalbini)]

> if they address a different issue, the venv package doesn't need to be installed by default.
But again, pyvenv isn't a separated thing from Python. Everywhere except in Debian/Ubuntu, It's shipped with Python itself, since it's part of the standard library.
If you need to install extra packages for the standard library, why we don't have python3-json, python3-hashlib, python3-os and so on? All not installed by default?

[Matthias Klose (doko)]

On 06/17/2015 06:33 PM, Donald Stufft wrote:
> I think it should be installed by default because it is a documented part
> of the Python language. I think the Debian/Ubuntu Python is fundamentally
> broken by default as it is currently.

No, a distribution has it's own conventions and policies. We don't install
development packages and X libraries by default. We shouldn't install the gdbm
module by default, which makes it easy to combine incompatible software
licenses. I can see your point, but unless you change these distro conventions,
there is no point insisting on that. Sure Python comes with batteries, but it
doesn't include the whole power plant.

[Matthias Klose (doko)]

On 06/17/2015 06:27 PM, Harry P wrote:
> I do a fair bit of work with beginner Python tutorials. It's not often
> that we have to say "on linux, this is harder", but this is one area
> where ubuntu/debian are failing pretty hard compared to the mac +
> windows experience.

$ pyvenv
The program 'pyvenv' is currently not installed. You can install it by typing:
sudo apt-get install python3-venv

so you even see the exact command to use on your system to install this.

[Rob Speer (rspeer)]

Matthias, that command clearly only works if you have sudo access. This makes it a bad idea for *most* of the scripts that would need to use the pyvenv command.

Like I said, you should not need elevated privileges to use the Python standard library. Especially something whose purpose is to make it easier to install packages as a user.

[ellie (et1234567)]

Who calls pyvenv directly? People just use python3 -m venv which, as people pointed out above, is expected by everyone to be fully working and available with any python3 install _without_ separately installed packages. Sure distributions have different conventions, but python was always packaged with all core stuff contained on pretty much all distributions includung Ubuntu. Therefore if you change that arbitrarly for the core venv component, you should expect people to get caught by surprise and wonder why Ubuntu's python is so broken until this is changed.

[ellie (et1234567)]

Regarding a different matter: above python3-venv was mentioned for Ubuntu 14.04 (Trusty), but I can't find such a package (neither python3-virtualenv). Maybe a listing for which ubuntu releases and which python versions this is supposed to be fixed would be helpful to stop people from asking.

[Barry Warsaw (barry)]

On Jun 25, 2015, at 11:13 AM, Jonas Thiem wrote:

>Who calls pyvenv directly?

I do.

[Jim Fulton (jim-zope)]

I call it directly also.

[Jim Fulton (jim-zope)]

FTR,

    python3 -m venv env

Doesn't work either for me.

[Donald Stufft (dstufft)]

FWIW I think that ``python -m venv`` is the easiest (only?) way to reliably say that you want to create a virtual environment for *this* particular Python. This is especially important when multiple versions of Python that have venv start to be able to be installed (either via the system repos, or via PPAs like deadsnakes). In particular, there is an inprogress rewrite of virtualenv to use the venv module for isolation that will essentially be doing ``python -m venv``.

[Barry Warsaw (barry)]

On Jun 25, 2015, at 04:31 PM, Donald Stufft wrote:

>FWIW I think that ``python -m venv`` is the easiest (only?) way to
>reliably say that you want to create a virtual environment for *this*
>particular Python.

Even though I don't generally use it, I agree with the above.

[Rob Speer (rspeer)]

Would it be too much to ask for this bug to not be marked as "fix released", given that it's not fixed?

[Wolfgang Richter (wolfgang-richter)]

I just encountered this bug. I do not see the aforementioned pyvenv packages anymore, and this renders programmatically creating virtual envs (a la EnvBuilder style) completely broken.

Is there any roadmap to fixing this in 14.04 ?

[pixelfairy (pixelfairy)]

i dont know what happens in distro land, but the rest of the world expects to be able to follow the docs at python.org without any more fuss than "install python3"

in the future, if you really need a broken up python package for whatever reason, could you make a python3-tiny like vim-tiny, that the full version can be installed on top of or replaces? maybe call that full version "python3" which depends on python3.6 or whatever is current by then. just a thought from an end user.

[Dario Bertini (berdario)]

@rspeer @wolfgang-richter

The bug is "sort of fixed"

You have to install python3-venv, this will make the ensurepip module available, and the creation of a virtualenv will succeed

Imho this is an horrible naming choice: the error you'll get when trying to run `python3 -m venv` mentions ensurepip, and most ubuntu/debian users might have heard about the fact that ensurepip is disabled in the distro

then, `apt list ensurepip` or `apt search ensurepip` won't yield any useful results (just like I tried to do, before being puzzled about the "fix released" status of this bug)

moreover, `--without-pip` will just work even without the new package, making the name "python3-venv" even more misleading

Still, I'd caution you AGAINST using venv... I just found another trivial bug. You're better off using virtualenv, given how it's battle tested

[Dario Bertini (berdario)]

PS: the bug I found is not truly a bug, just a difference in behaviour between venv and virtualenv:

the former won't allow you to locally install packages when created with --system-site-packages (making that flag effectively useless, imho), while virtualenv will allow you to layer local packages upon the globally installed ones

[Dario Bertini (berdario)]

This is it: https://bugs.python.org/issue24875

[Barry Warsaw (barry)]

There have been some discussions about this via other channels (e.g. IRC). We're probably at least going to patch Python so that if you try to do `python3 -m venv` and ensurepip (i.e. via python3-venv binary package) isn't installed, you'll get a more helpful message that instructs you to `sudo apt-get install python3-venv`. I think we all agree this would be good to add.

The second part of our discussions, which is still unsettled, is whether to promote `python3.X-venv` to a Recommends. Some viewpoints see this as a (mild) abuse of Debian Policy, but OTOH it may be an acceptable one to improve the user-friendliness of the tool. apt-get by default installs Recommends (with an option to not do so) where as Suggests are not installed by default (with an option to do so).

Stay tuned.

[jose (jose-cybergalvez)]

There is no python3-venv package in 14.04 how do I get it? or where do I
find it?
Jose

On 09/14/2015 10:18 AM, Dario Bertini wrote:
> @rspeer @wolfgang-richter
>
> The bug is "sort of fixed"
>
> You have to install python3-venv, this will make the ensurepip module
> available, and the creation of a virtualenv will succeed
>
> Imho this is an horrible naming choice: the error you'll get when trying
> to run `python3 -m venv` mentions ensurepip, and most ubuntu/debian
> users might have heard about the fact that ensurepip is disabled in the
> distro
>
> then, `apt list ensurepip` or `apt search ensurepip` won't yield any
> useful results (just like I tried to do, before being puzzled about the
> "fix released" status of this bug)
>
> moreover, `--without-pip` will just work even without the new package,
> making the name "python3-venv" even more misleading
>
> Still, I'd caution you AGAINST using venv... I just found another
> trivial bug. You're better off using virtualenv, given how it's battle
> tested
>

[Matthias Klose (doko)]

On 25.09.2015 02:40, jose wrote:
> There is no python3-venv package in 14.04 how do I get it? or where do I
> find it?

this is work in progress, for now please install the python3.4-venv package.

[jose (jose-cybergalvez)]

Thanks that seems to work
Jose

On 09/24/2015 08:57 PM, Matthias Klose wrote:
> On 25.09.2015 02:40, jose wrote:
>> There is no python3-venv package in 14.04 how do I get it? or where do I
>> find it?
> this is work in progress, for now please install the python3.4-venv
> package.
>

[Mali Akmanalp (mali-t)]

Hi Matthias et al,

Did this used to be on trusty? We added it to a deployment script about a week and a half ago and it worked on an 14.04 ec2 image, but then as of today doing a deploy with a fresh image it doesn't seem to be finding python3.4-venv or python3-venv, and we're back to square one. :/

Cheers,
~mali

[Jeroen Ruigrok van der Werven (asmodai)]

I can confirm what Mali says. No python3[*]-venv in the repositories. And the python3 -m venv gives the error that kickstarted this lengthy fix process.

[Harry P (hjwp2)]

I'm now testing Ubuntu 15.10, which seems *even more* broken, since it seems to have a mixed mode of Python3.4 and 3.5. there's no python3.4-pip package, which means there is essentially no way of getting a working pip for python 3.4.

I know I'm not a contributor, so I don't *deserve* answers, but I still don't see why Python has to be broken under debian/ubuntu? Why can't it just ship with all of its standard library, like it does on all other platforms?

With apologies for the moaning. I do appreciate all the hard work, and I know there are probably good reasons.

[Barry Warsaw (barry)]

On Oct 16, 2015, at 03:17 PM, Harry P wrote:

>I'm now testing Ubuntu 15.10, which seems *even more* broken, since it
>seems to have a mixed mode of Python3.4 and 3.5. there's no
>python3.4-pip package, which means there is essentially no way of
>getting a working pip for python 3.4.

Tools like pip, which depend on a specific Python version are fundamentally
flawed on *nix. There's healthy debate in many upstream forums about this
issue. Basically, nobody likes distributing things like pip3.4 and pip3.5,
etc.

If you install python3-pip you will get a pip3 but even that isn't great.
While having a command like `pip3` or `pip3.4` might seem like a useful
convenience, really the best way to invoke pip for a specific Python version
is to use `python3.4 -m pip` or `python3.5 -m pip`. This goes for a small
handful of other packages like nose (with scripts like `nose2-3.4` being the
worst offenders! better to use `pythonX.Y -m nose2`).

We're looking at ways of making this better and more consistent across all
Linux distributions.

[garyo (garyo)]

I'm sorry to post to this ancient bug, but I just did a fresh install of Ubuntu 18.04, and I just want to create a virtual env with python 3.7, including pip (or some way to install modules). Doing this does not work:

% sudo apt install python3.7 python3.7-venv
% python3.7 -m venv py37-venv

It gives the error "The virtual environment was not created successfully because ensurepip is not available".

Doing it with --without-pip works, but then there is no way to install modules in the virtualenv. (python -mpip in the virtualenv correctly says "No module named pip").

So what is the recommended way to install python 3.7 on Ubuntu 18.04 and create a "normal" virtual env?

[Jack J. Woehr (jackwoehr)]

Yes, 5 years later this is still broken. What can we do to install a venv, please?

[Daniel Murray (d-daniel-t)]

Not a fix but a workaround: I use the virtualenv package instead, rather than -m venv.