python applications fail due to library problem

[BryanLawrence]

I have a version of python in /usr/local.

When /usr/local is mounted during login, a number of applications do not appear in my taskbar, and fail with bizarre messages when requested from system settings.

Two that really matter are guidance and the power manager (battery monitor) (Actually, I think these are the same, but I'm not quite sure).

It appears that while the (some?) kde python applications correctly use /usr/bin/python, they are nonetheless picking up the python site-library from /usr/local

Investigating with guidance shows the following error when run from the command-line

./guidance-power-manager
bnl@sstdlbnl:/usr/bin$ Error in sys.excepthook:
Traceback (most recent call last):
  File "/var/lib/python-support/python2.5/apport_python_hook.py", line 30, in apport_excepthook
    import apport.report, apport.fileutils
  File "/var/lib/python-support/python2.5/apport/__init__.py", line 1, in <module>
    from apport.report import Report
  File "/var/lib/python-support/python2.5/apport/report.py", line 18, in <module>
    from xml.parsers.expat import ExpatError
  File "/usr/local/lib/python2.5/site-packages/_xmlplus/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ImportError: /usr/local/lib/python2.5/site-packages/_xmlplus/parsers/pyexpat.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

Original exception was:
Traceback (most recent call last):
  File "/usr/share/python-support/kde-guidance/guidance-power-manager.py", line 38, in <module>
    import dbus
  File "/var/lib/python-support/python2.5/dbus/__init__.py", line 198, in <module>
    from dbus._dbus import Bus, SystemBus, SessionBus, StarterBus, Interface
  File "/var/lib/python-support/python2.5/dbus/_dbus.py", line 38, in <module>
    from dbus.proxies import ProxyObject, BUS_DAEMON_NAME, BUS_DAEMON_PATH, \
  File "/var/lib/python-support/python2.5/dbus/proxies.py", line 31, in <module>
    from dbus._expat_introspect_parser import process_introspection_data
  File "/var/lib/python-support/python2.5/dbus/_expat_introspect_parser.py", line 22, in <module>
    from xml.parsers.expat import ExpatError, ParserCreate
  File "/usr/local/lib/python2.5/site-packages/_xmlplus/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ImportError: /usr/local/lib/python2.5/site-packages/_xmlplus/parsers/pyexpat.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

[BryanLawrence (b-n-lawrence)]

Sorry, I should have said version: kubuntu feisty release.
(And the title should have had python in it, but I can't change that ... sorry)

[Jeremy Teale (jteale)]

How is your Python path set up?
This seems to be a problem with your system's confguration and not Kubuntu.
Thanks.

[Jeremy Teale (jteale)]

Apologies, ignore me.

[BryanLawrence (b-n-lawrence)]

Just to clarify: provided I don't mount /usr/local, things work fine.

However, even with a vanilla kde user (i.e. not "me"), things are broken with
the /usr/local/ code in place - so it's not my personal configuration.

Actually, I now know that all /usr/bin/python applications are broken with /usr/local
mounted ...

(NB: this configuration worked fine under dapper with ubuntu, not kubuntu)

[BryanLawrence (b-n-lawrence)]

- sorry about the bug spam. But to clarify yet again: until last week I was running dapper ubuntu, with this configuration, and it was fine (my own python 2.5 in /usr/local).
- on Friday, I installed feisty kubuntu, and mounted my old /usr/local, and the system
python is now broken ...

[Scott Kitterman (kitterman)]

First, upgrades from Dapper direct to Feisty are not supported. Second, if you installed your own Python 2.5 outside the packaging system there are now two installed, also not a configuration we can support. So this is not an Ubuntu bug (thus marked rejected).

I believe your analysis is correct about some programs using the local version of Python. What I would suggest you do is remove that version of Python and see if that solves your problem. Since things work with /usr/local is not mounted, then I think removing the Python version there is likely to succeed. If it doesn't, then I'd suggest opening a support request in the Answers section of LP and see if someone there has suggestions.

[BryanLawrence (b-n-lawrence)]

Scott:

1) I didn't do an *upgrade*, I did an *install*. However, after the install, I mounted my old /usr/local, and things started going wrong.

2) I think it is a *bug* in ubuntu if the system python is looking into the site-packages of another python which lives in /usr/local. Their library paths should not intermingle. They have never done so before, in ubuntu, or any other linux with which I am familiar (and I have been doing this a long time).

3) Having multiple pythons on a system is how you work out how to upgrade systems. It's ludicrous to suggest that it shouldn't be done. However, I'm obviously willing to admit I may have done it in a way that wasn't optimal, but see point 2) above ... this is not dissimilar to what would happen if you have a script which users /usr/bin/env python rather than /usr/bin/python lurking in your system ... as I say, I suspect something similar ...

How about not *rejecting* it quite yet ...

[Scott Kitterman (kitterman)]

OK. Fair enough. I thought you had upgraded direct to Feisty. There were a lot of Python changes Dapper --> Edgy --> Feisty to support the new Python policy.

If you fire up Python with usr/local mounted and do:

import sys
print sys.path

What do you get?

I'm not sure about the right package for this, but python defaults seems like a step in the right direction. This is unlikely to be something KDE/Kubuntu specific.

[BryanLawrence (b-n-lawrence)]

Agreed, I don't think this is kde/kubuntu specific, I thought it was, but I've got a bit further ...

The bottom line is that an import statement
 from xml.parsers.expat import ExpatError, ParserCreate
is giving an
ImportError: /usr/local/lib/python2.5/site-packages/_xmlplus/parsers/pyexpat.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

Which comes down to a problem which is well understood:
http://www.python.org/doc/faq/extending.html#when-importing-module-x-why-do-i-get-undefined-symbol-pyunicodeucs2

i.e: two pythons compiled with different numbers of bytes for unicode characters.

But, why it's not finding the right thing within the system python (first) is the question (and what other things this might cause that we've not yet identified).

[BryanLawrence (b-n-lawrence)]

ok, and quite clearly my path has all the /usr/local stuff in it first:
>>> import sys
>>> print sys.path
['', '/usr/local/lib/python2.5/site-packages/simplejson-1.4-py2.5.egg', '/usr/local/lib/python2.5/site-packages/Amara-1.1.9-py2.5.egg', '/usr/local/lib/python2.5/site-packages/flup-0.5-py2.5.egg', '/usr/local/lib/python2.5/site-packages/scgi-1.12-py2.5-linux-i686.egg', '/usr/local/lib/python2.5/site-packages/Paste-1.0-py2.5.egg', '/usr/local/lib/python2.5/site-packages/ZSI-2.0.dev_r1293-py2.5.egg', '/usr/local/lib/python2.5/site-packages/dap-2.2.5.1-py2.5.egg', '/usr/local/lib/python2.5/site-packages/httplib2-0.2.0-py2.5.egg', '/usr/local/lib/python2.5/site-packages/Cheetah-2.0rc7-py2.5-linux-i686.egg', '/usr/local/lib/python2.5/site-packages/dap.plugins.grads-0.1.1-py2.5.egg', '/usr/local/lib/python2.5/site-packages/arrayterator-0.2.5-py2.5.egg', '/usr/local/lib/python2.5/site-packages/PasteDeploy-1.0-py2.5.egg', '/usr/local/lib/python2.5/site-packages/PasteScript-1.0-py2.5.egg', '/usr/local/lib/python2.5/site-packages/dap.plugins.netcdf-0.3.2-py2.5.egg', '/usr/local/lib/python2.5/site-packages/pupynere-0.2.1-py2.5.egg', '/usr/local/lib/python2.5/site-packages/wsgistate-0.4-py2.5.egg', '/usr/local/lib/python2.5/site-packages/SQLAlchemy-0.3.3-py2.5.egg', '/usr/local/lib/python2.5/site-packages/WSGIUtils-0.7-py2.5.egg', '/usr/local/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg', '/usr/local/lib/python2.5/site-packages/kid-0.9.5-py2.5.egg', '/usr/local/lib/python2.5/site-packages/PyOpenGL-3.0.0a5-py2.5.egg', '/usr/local/lib/python2.5/site-packages/Genshi-0.4-py2.5.egg', '/usr/local/lib/python2.5/site-packages/wsgilog-0.1-py2.5.egg', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/var/lib/python-support/python2.5']

So that's where the problem lies. Now the question is why ...

[BryanLawrence (b-n-lawrence)]

OK, and my reading is that the /usr/local stuff should come after the system
stuff, and only be searched if the system doesn't come up with the goods.

But see:
http://www.rexx.com/~dkuhlman/python_comments.html

It does look like it's all my easy_install packages that are the problem, and maybe
I've never had this before because I've only been using eggs for the last
few months.

OK, this doesn't look like an ubuntu problem per se ... it's a python problem.

Damn. Feel free to reject this ... now :-)

[Scott Kitterman (kitterman)]

Done (rejected) and good luck.

[BryanLawrence (b-n-lawrence)]

Well, actually, this is a DEBIAN problem, not a python problem per se.

/usr/lib/python2.5/Site.py has been patched as follows:

FOR DEBIAN, this sys.path is augmented with directories in /usr/local.
Local addons go into /usr/local/lib/python<version>/site-packages
(resp. /usr/local/lib/site-python), Debian addons install into
/usr/{lib,share}/python<version>/site-packages.

This is not the default python install behaviour (and nor should it be). Obviously I now need to get round this, and will, but actually I reckon this is an ubuntu problem that needs to push back to DEBIAN ...

[Scott Kitterman (kitterman)]

OK. I've marked this back to unconfirmed then. I've got no say in Python policy in Ubuntu, but I'd guess we are going to tend to follow Debian on this. Please file a bug in Debian BTS and then link this bug to the Debian one (or comment with the Bug number and I'll do it if you don't know how).

[Matthias Klose (doko)]

> This is not the default python install behaviour (and nor should it be).
> Obviously I now need to get round this, and will, but actually I reckon
> this is an ubuntu problem that needs to push back to DEBIAN ...

No, it's Debian policy to include this path.

[David Geller (dgeller)]

I was bitten by the same bug a while ago! I had thought that my developmental and experimental python on /usr/local was totally independent of the installed python on /usr - and then certain applications, like the update manager started failing. It is then I saw that /usr/local was being appended to the path. To me, as well, this should not be! Otherwise, you really have to go through hoops to have your own "private" python on /usr/local not pollute the installed version, used by various system utilities, etc.