Should default to --user to not fail default pip install usage

[Didier Roche-Tolomelli]

As part of vivid cycles, we want all largely known developer-oriented package manager to default to local installation to not mess with the system installed packages (default of most of those system nowdays).

We will keep coherent behavior between all those package managers.

We won't change the default in a subdirectory technology like virtualenv, rubygem and so on…
More information on https://blueprints.launchpad.net/ubuntu/+spec/udtc-prevent-library-clash
vUDS video: http://summit.ubuntu.com/uos-1411/meeting/22331/ubuntu-developer-tools-community-input/

[Didier Roche-Tolomelli (didrocks)]

FYI: pip upstream bug (couldn't link github bug in launchpad): https://github.com/pypa/pip/issues/1668

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pip - 1.5.6-4ubuntu1

---------------
python-pip (1.5.6-4ubuntu1) vivid; urgency=medium

  * add debian/patches/set_user_default.patch,
    debian/pip-manpage.rst: (LP: #1419695)
    Default to --user on non virtualenv setup to ensure that
    default pip install works. Ignore as well system installed package
    detection on pip install in that case. We keep the existing (working)
    behavior in virtualenv environment.
 -- Didier Roche <email address hidden> Mon, 09 Feb 2015 11:13:19 +0100

[Donald Stufft (dstufft)]

Speaking as a pip core developer, please revert this and wait for us to fix it.

As I mentioned in a similar bug for Debian (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=725848) jumping ahead of upstream on this is going to make things WAY worse for a lot of people. You're using a custom flag that's only going to exist in versions of pip that comes from Ubuntu's archive. This means that behavior is going to change drastically if end users switch between systems or upgrade (or downgrade!) or even just reinstall their pip through pip itself. This is going to confuse people, it's going to break lots of tools like salt, puppet, chef, etc.

Changing this is a massively incompatible change that needs to be done carefully and with a lot of thought about how best to do it without causing a lot of negative impact for pip's users. Going off on your own is only going to make a bad situation (Python packaging) that we're slowly trying to make better a lot more confusing and a lot worse.

[Barry Warsaw (barry)]

I'm also concerned about getting ahead of Debian here. While I understand release cycle skew (e.g. Jessie is still in freeze), I think this will make it more difficult to merge later and remove the delta. If we don't do that, then there will be even more confusing user experience between upstream, Debian, and Ubuntu. Let's be careful here.

[Didier Roche-Tolomelli (didrocks)]

We are trying to get the same behavior between multiple package system installation here (ruby, npm, pip…). All the others have the sane use default.

When we started to write about it, the Debian bug was in discussion and opened and I saw that you discussed that it was in progress upstream. However, multiple months after the initial discussion, I don't see any progress and Ubuntu is a week away of its feature freeze. Are you close to ship an upstream solution that will fit in our Ubuntu feature freeze date? How can we help there?

I would of course prefer an upstream-compatible solution, but you need to understand that we can't always wait for releases (the upstream bug is opened about a year ago) without going forward. I'm opened to any idea to help you getting there in that timeframe.

As one of the goal of vivid is to get a good develoepr story, which pip doesn't provide at the moment, we need at least a solution for this cycle.

Note that I wrote in the man on purpose that this is a default specific to ubuntu and the flag as well, I can make it even more clear if needed.

I'm opened to any re

[Didier Roche-Tolomelli (didrocks)]

Sorry, shift+Enter

"I'm opened to any real proposal, compatible with the vivid feature freeze timeframe."
Other way to deal with it:
- pip install without the right perm to write to /usr/local, instead of getting the not-so-great traceback it's writing at the moment, would redirect to a wiki page writing why installing system-wide is wrong.
- get this specific behavior under an environment variable which is installed when ubuntu-make is (that way, scripts should be safe).
- getting the proper upstream behavior now in trunk, and backporting the patch

[Donald Stufft (dstufft)]

We're unlikely to ship a solution within a week, however we can likely hammer out what solution we're going to go with and probably land something in trunk so that Ubuntu isn't different/broken it's just ahead of the curve.

I wish y'all had reached out to us so we could have prioritized the issue, it looks like this was decided to happen ~4 months ago which would have been a great time to come to us and say that Ubuntu needs a solution to this issue by X date. We haven't made a lot of progress on the issue because it's a thorny issue with a lot of far reaching complications and while it should be fixed it's at least not flat out broken and the current behavior as well understood. At the same time there were other, lower hanging fruit which were places where there were actual broken pieces of behavior.

A side comment by someone who wasn't involved in the change is not a great way to find out that Ubuntu broke a fairly major use case by default, especially given that there's only a week or so to figure out a good solution before the feature freeze. It's obviously too late to do anything about it in this case besides us (pip) change gears and try to quickly hammer out a solution in time. However I would suggest that things will be better for everyone involved if engaging upstream is treated as the first step to these things and is done early in the process instead of being done after the fact, if at all. Doing it the other way makes upstreams like myself feel like we have to try and monitor all of the downstream distributors for breaking changes to try and protect our users.

[Didier Roche-Tolomelli (didrocks)]

Well, the debian bug was about this, and you commented there telling to hold on as upstream/you was working on it and going to ship something. I didn't want at the time putting pressure with a firm date for ubuntu, thinking that would be shipped way in advance upstream.

I understand that the issue is complex and backward compatibility is important, I'm happy that now we went ahead of the curve and pushed something, you are more eager to tackle this issue. You probably understand well the current issues, I can tell you that developers don't and just end up breaking their system installation mixing package system installation and pip sources (because they got the permission traceback and just bindly typed "sudo").

I did introduce the change in purpose on a non LTS release, so that we can swap this solution with a better upstream one even after Feature Freeze, but before beta I guess (as it can be seen as a bug fix, supporting better and wider use-case). I'm willing to help you, as said in my previous comment, to get that issue fixed and happy to bring any help. How can we move then forward from here and work together to getting that done so that we can get something in pip trunk soonish that I'll backport to the ubuntu package then?