virtualenv not working under opensuse

Bug #248558 reported by Malte Marquarding on 2008-07-15
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Virtualenv
In Progress
Undecided
Unassigned
python-virtualenv (Fedora)
Invalid
Medium

Bug Description

opensuse 10.1 or greater, sets up a system 'distutils.cfg' with default install location being '/usr/local'.

The attached patch fixes this.

Stephen Pascoe (stephen-pascoe) wrote :

The patch almost completely fixes the problem for me. A reference to global site-packages doesn't get added to sys.path.

Delving into SuSE's python installation I'm not surprised ... site.py has been patched to add '/usr/local' to the list of prefixes. I hope to report this to SuSE and will reference it here when done.

Two invocations of python follow. One via virtualenv, the other not. Note the former doesn't include /usr/local/lib64/python2.5/site-packages.

.....

(myenv2)spascoe@bond:~/host/bond/expt> python
Python 2.5.1 (r251:54863, Sep 21 2007, 22:46:31)
[GCC 4.2.1 (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, pprint
>>> pprint.pprint(sys.path)
['',
 '/disks/bond1/spascoe/expt/myenv2/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg',
 '/disks/bond1/spascoe/expt',
 '/disks/bond1/spascoe/expt/myenv2/lib/python25.zip',
 '/disks/bond1/spascoe/expt/myenv2/lib64/python2.5',
 '/disks/bond1/spascoe/expt/myenv2/lib64/python2.5/plat-linux2',
 '/disks/bond1/spascoe/expt/myenv2/lib64/python2.5/lib-tk',
 '/disks/bond1/spascoe/expt/myenv2/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/disks/bond1/spascoe/expt/myenv2/lib/python2.5/site-packages']
>>>
(myenv2)spascoe@bond:~/host/bond/expt> /usr/bin/python
Python 2.5.1 (r251:54863, Sep 21 2007, 22:46:31)
[GCC 4.2.1 (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, pprint
>>> pprint.pprint(sys.path)
['',

 ... eggs-snipped...

 '/usr/local/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/disks/bond1/spascoe/expt',
 '/usr/lib/python25.zip',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/plat-linux2',
 '/usr/lib64/python2.5/lib-tk',
 '/usr/lib64/python2.5/lib-dynload',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/local/lib64/python2.5/site-packages']
>>>

Ian Bicking (ianb) wrote :

Applied in r3602

Changed in virtualenv:
status: New → Fix Committed

Description of problem:
Inside a virtual environment created with the 'virtualenv' command, the /usr/lib64/python2.5/site-packages path is missing.

Version-Release number of selected component (if applicable):
python-virtualenv-1.3-1.fc9.noarch

Steps to Reproduce:
1. Create a virtual environment like with 'virtualenv /tmp/env'
2. Compare paths outside and inside using
   "python -c 'import sys, pprint; pprint.pprint(sys.path);'" and
   "/tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'"
   respectively.

Actual results:
% python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/usr/lib64/python25.zip',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/plat-linux2',
 '/usr/lib64/python2.5/lib-tk',
 '/usr/lib64/python2.5/lib-dynload',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib/python2.5/site-packages']

Expected results:
See how /usr/lib64/python2.5/site-packages (and thus all paths below, like Numeric, PIL, etc.) are missing in the second output.

Additional info:
A similar problem (on OpenSuSE) was described here https://bugs.launchpad.net/virtualenv/+bug/248558/comments/2 (note that that is different from the original bug report) and propagated to OpenSuSE's bugtracker here: https://bugzilla.novell.com/show_bug.cgi?id=430761. That might be related if Fedora is also patching site.py.

Thanks for the report. I'll look into pushing up an upstream patch.

ashcrow (stevem-gnulinux) wrote :

Bug reported to Fedora as well ....

https://bugzilla.redhat.com/show_bug.cgi?id=471617

I'll give the patch listed at https://bugs.launchpad.net/fedora/+source/python-virtualenv/+bug/2485 a run and see if that fixes the issue on a 64bit host. It was applied on 2008-09-29.

Changed in python-virtualenv:
status: Unknown → In Progress

python-virtualenv-1.3.1-1.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/python-virtualenv-1.3.1-1.fc10

(In reply to comment #3)
> python-virtualenv-1.3.1-1.fc10 has been submitted as an update for Fedora 10.

This doesn't fix the issue -- and wasn't supposed to, because upstream's 1.3.1 fixes the original problem reported in launchpad's bug 248558, not the problem described in comment 2 thereof.

Here's the current output on fc10, with
python-2.5.2-1.fc10.x86_64
python-setuptools-0.6c8-1.fc10.noarch
python-virtualenv-1.3.1-1.fc10.noarch

% python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/usr/lib64/python25.zip',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/plat-linux2',
 '/usr/lib64/python2.5/lib-tk',
 '/usr/lib64/python2.5/lib-dynload',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

vs.

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib/python2.5/site-packages']

Still, in the second output, paths /usr/lib64/python2.5/site-packages and below are missing.

The problem is that fedora patches site.py, which virtualenv doesn't (can't?) detect.

Ok, I see what you mean. virtualenv is writing out it's site.py via a zipped version it holds in virtualenv.py as base64 encoded zipped data. I'll take a look at the patches being applied in the python fedora package that might make sense to pull into virtualenv's copy.

These all end up mapping to pth files and are owned by different packages ... here is an example from my F9 laptop ...

[steve@tachikoman site-packages]$ for x in `find . -name "*.pth"`; do
    echo -n "$x: ";
    echo `rpm -q --whatprovides /usr/lib/python2.5/site-packages/$x`;
done
./PasteDeploy-1.3.1-py2.5-nspkg.pth: python-paste-deploy-1.3.1-2.fc9.noarch
./Numeric.pth: python-numeric-24.2-11.fc9.i386
./Paste-1.6-py2.5-nspkg.pth: python-paste-1.6-1.fc9.noarch
./PIL.pth: python-imaging-1.1.6-9.fc9.i386
./pygtk.pth: pygobject2-2.14.2-1.fc9.i386
./easy-install.pth: file /usr/lib/python2.5/site-packages/easy-install.pth is not owned by any package
./pygst.pth: gstreamer-python-0.10.11-2.fc9.i386
./DecoratorTools-1.7-py2.5-nspkg.pth: python-decoratortools-1.7-1.fc9.noarch
./PasteScript-1.6.2-py2.5-nspkg.pth: python-paste-script-1.6.2-2.fc9.noarch
./wx.pth: wxPython-2.8.9.1-1.fc9.i386

or on my F10 desktop

[steve@powerhouse site-packages]$ for x in `find . -name "*.pth"`; do
> echo -n "$x: ";
> echo `rpm -q --whatprovides /usr/lib64/python2.5/site-packages/$x`;
> done
./pygtk.pth: pygobject2-2.15.4-3.fc10.x86_64
./Numeric.pth: python-numeric-24.2-11.fc9.x86_64
./PIL.pth: python-imaging-1.1.6-12.fc10.x86_64
./pygst.pth: gstreamer-python-0.10.12-1.fc10.x86_64
./scim.pth: scim-python-0.1.13rc1-1.fc10.x86_64

I looked quickly at the patching that is done on site.py in the python package and it's pretty minimal and it didn't look to be related.

I guess I could make a patch that generates full path pth's off of whats in the site-packages dir(s). I don't know if that really makes sense though since it's per package.

I could still be missing something though ... let me know if I am Thomas.

(In reply to comment #6)

> I looked quickly at the patching that is done on site.py in the python package
> and it's pretty minimal and it didn't look to be related.

Well, it is surely related, because it (at least) changes one of the sitedirs and adds another one on a 64 bit system. Here's one of the most relevant chunks from python/F-10/python-2.5-lib64.patch (applied as patch No. 102):

--- Python-2.5b1/Lib/site.py.lib64 2006-06-12 04:23:02.000000000 -0400
+++ Python-2.5b1/Lib/site.py 2006-06-22 12:20:35.000000000 -0400
@@ -182,9 +182,14 @@
                 sitedirs = [os.path.join(prefix, "Lib", "site-packages")]
             elif os.sep == '/':
                 sitedirs = [os.path.join(prefix,
- "lib",
+ "lib64",
                                          "python" + sys.version[:3],
                                          "site-packages"),
+ os.path.join(prefix,
+ "lib",
+ "python" + sys.version[:3],
+ "site-packages"),
+ os.path.join(prefix, "lib64", "site-python"),
                             os.path.join(prefix, "lib", "site-python")]
             else:
                 sitedirs = [prefix, os.path.join(prefix, "lib", "site-packages")]

Imho that's exactly the sort of patch that should either be removed from the python package, or added to the virtualenv package. And there might be some other relevant chunks in that patch.

Maybe we should get one of the main python package's maintainers on board here?

> I guess I could make a patch that generates full path pth's off of whats
> in the site-packages dir(s). I don't know if that really makes sense though
> since it's per package.

That doesn't sound like the right (and neither a feasible) solution to me.

When building, grab site.py from the python package and patch it as necessary.

The problem with grabbing site.py from the python package is that it is vastly different than the site.py zipped and base encoded in virtualenv. I can modify what virtualenv holds though using the python package's site.py file.

In the actual unzipped virtualenv source they talk about extra paths using pth's and give an example of debain's lib locations which is what made me look that the pth's being placed by things like Numeric, PIL, gst and gtk.

If you guys really think using the python packages patched version will fix the issue I'm willing to give it a shot ... but at first run virtualenv failed for me when trying to use it like that.

(In reply to comment #9)
> The problem with grabbing site.py from the python package is that it is vastly
> different than the site.py zipped and base encoded in virtualenv.

Most of the differences come from the fact though that virtualenv's site.py is from python 2.3. The differences can be seen using (zsh syntax):

diff -u /usr/lib64/python2.5/site.py <(python -c 'import virtualenv; print virtualenv.SITE_PY')

I have a confession.

This is not the first time I've seen this bug.

I thought about it LONG AND HARD for the better part of a month when I initially hit it, and this was the only sane method I could find for solving it. Unfortunately I never got around to actually implementing it.

Here's a naive patch:

Index: python-virtualenv.spec
===================================================================
RCS file: /cvs/pkgs/rpms/python-virtualenv/devel/python-virtualenv.spec,v
retrieving revision 1.5
diff -r1.5 python-virtualenv.spec
12a13
> Patch0: virtualenv-lib64.patch
28c29,33
< %setup -q -n virtualenv-%{version}
---
> %setup0 -q -n virtualenv-%{version}
> %if "%{_lib}" == "lib64"
> %patch0 -p1 -b .lib64
> %endif
> %{__python} rebuild-script.py

diff -up virtualenv-1.3.1/support-files/site.py.orig virtualenv-1.3.1/support-files/site.py
--- virtualenv-1.3.1/support-files/site.py.orig 2008-11-25 03:27:26.000000000 +0100
+++ virtualenv-1.3.1/support-files/site.py 2008-12-03 00:38:40.000000000 +0100
@@ -213,6 +213,10 @@ def addsitepackages(known_paths, sys_pre

             elif os.sep == '/':
                 sitedirs = [os.path.join(prefix,
+ "lib64",
+ "python" + sys.version[:3],
+ "site-packages"),
+ os.path.join(prefix,
                                          "lib",
                                          "python" + sys.version[:3],
                                          "site-packages"),

This seems to work, but now there are too many dirs in the path:

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/tmp/env/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/site-packages',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

Note that simply copying the system site.py did not work for me neither. According to a message found on their mailinglist, the version in the virtualenv package is slightly modified. So maybe the right way is to use system's site.py, but apply virtualenv's modfications.

Well, if that works and the extra paths are not harming anything, I'm fine with using that patch ... at least for now. I'm going to run some tests with a modified version of the patch (- original virtualenv site.py, + my changes to the original virtualenv setup.py)....

--- /tmp/site.py.virtualenv 2008-12-11 19:40:58.000000000 -0500
+++ lib/python2.5/site.py 2008-12-11 20:50:55.000000000 -0500
@@ -201,11 +201,27 @@
                                              "python" + sys.version[:3], "site-packages")]

             elif os.sep == '/':
- sitedirs = [os.path.join(prefix,
- "lib",
- "python" + sys.version[:3],
- "site-packages"),
- os.path.join(prefix, "lib", "site-python")]
+ # PATCH:BZ471617
+ # https://bugzilla.redhat.com/show_bug.cgi?id=471617
+ import platform
+ # Maps the processor type with the library path name
+ lib_map = {'lib': 'lib',
+ 'x86_64': 'lib64'}
+
+ def __easy_path(cpu):
+ """
+ Simply returns a python path based on cpu and lib_map.
+ """
+ if cpu not in lib_map.keys():
+ return ''
+ return os.path.join(prefix,
+ lib_map[cpu],
+ "python" + sys.version[:3],
+ "site-packages")
+
+ sitedirs = [__easy_path("lib")]
+ sitedirs.append(__easy_path(platform.processor()))
+ # END PATCH:BZ471617
                 try:
                     # sys.getobjects only available in --with-pydebug build
                     sys.getobjects

(In reply to comment #13)
> Well, if that works and the extra paths are not harming anything, I'm fine with
> using that patch ... at least for now.

Did you talk to upstream about this problem already? Maybe they have some idea how to cleanly solve it.

> modified version of the patch (- original virtualenv site.py, + my changes to
> the original virtualenv setup.py)....

That code looks far more complicated to me than it needs to be, but if it works *shrug*.

Maybe Ignacio can comment?

Ian Bicking (ianb) wrote :

I've had to revert this because setting prefix leads to some weird problems in other cases, and the problems are very hard to track down at that (you can tell it has to do with distutils.cfg because removing this setting from that file fixes it).

The system distutils.cfg shouldn't really be used in this context, so I'll look into why that might be happening.

Ian Bicking (ianb) on 2008-12-17
Changed in virtualenv:
status: Fix Committed → In Progress

From 1.3.2 release

    * Added platform-specific paths, like /usr/lib/pythonX.Y/plat-linux2

I now get the paths added in this version:

<snip>
'/usr/lib/python2.5/site-packages',
'/usr/lib64/python2.5/site-packages',
'/usr/lib64/python2.5/site-packages/Numeric',
'/usr/lib64/python2.5/site-packages/PIL',
'/usr/lib64/python2.5/site-packages/gst-0.10',
'/usr/lib64/python2.5/site-packages/gtk-2.0',
'/usr/lib64/python2.5/site-packages/scim-0.1']

Just an FYI as I'll be pushing the new version up with a spec fix with this new version.

Changed in python-virtualenv:
status: In Progress → Fix Committed

The package has been pushed to stable.

Changed in python-virtualenv:
status: Fix Committed → Fix Released
Carl Meyer (carljm) wrote :

Not migrating to new tracker at https://github.com/pypa/virtualenv/issues - appears to have been fixed, no recent updates.

Seeing this again on F17:

# virtualenv /tmp/env
New python executable in /tmp/env/bin/python
Installing setuptools............done.
Installing pip...............done.

# python -c 'import sys; print "\n".join(sys.path)' | grep -v '\.local'

/usr/lib64/python27.zip
/usr/lib64/python2.7
/usr/lib64/python2.7/plat-linux2
/usr/lib64/python2.7/lib-tk
/usr/lib64/python2.7/lib-old
/usr/lib64/python2.7/lib-dynload
/usr/lib64/python2.7/site-packages
/usr/lib64/python2.7/site-packages/PIL
/usr/lib64/python2.7/site-packages/gst-0.10
/usr/lib64/python2.7/site-packages/gtk-2.0
/usr/lib64/python2.7/site-packages/wx-2.8-gtk2-unicode
/usr/lib/python2.7/site-packages
/usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info

# /tmp/env/bin/python -c 'import sys; print "\n".join(sys.path)'

/tmp/env/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/tmp/env/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/tmp/env/lib64/python27.zip
/tmp/env/lib64/python2.7
/tmp/env/lib64/python2.7/plat-linux2
/tmp/env/lib64/python2.7/lib-tk
/tmp/env/lib64/python2.7/lib-old
/tmp/env/lib64/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib64/python2.7
/usr/lib64/python2.7/lib-tk
/tmp/env/lib/python2.7/site-packages

This yields to import failures, e.g.

# python -c 'import yaml'
# /tmp/env/bin/python -c 'import yaml'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named yaml

Works on F21 with virtualenv --system-site-packages.

Changed in python-virtualenv (Fedora):
importance: Unknown → Medium
status: Fix Released → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.