#!/usr/bin/env python breaks Python-based Ubuntu packages in the presence of virtualenvs, local installations

Bug #912625 reported by David Warde-Farley on 2012-01-06
24
This bug affects 1 person
Affects Status Importance Assigned to Milestone
HPLIP
Undecided
Unassigned
c2esp (Ubuntu)
Undecided
Unassigned
foo2zjs (Ubuntu)
Undecided
Unassigned
gconf (Ubuntu)
Undecided
Unassigned
gnome-applets (Ubuntu)
Undecided
Unassigned
hplip (Ubuntu)
Medium
Unassigned
mercurial (Debian)
Fix Released
Unknown
mercurial (Ubuntu)
Undecided
Unassigned
pidgin (Ubuntu)
Undecided
Unassigned
pitivi (Ubuntu)
Undecided
Unassigned
pyppd (Ubuntu)
Undecided
Unassigned

Bug Description

Currently (as of 11.04, and I suspect in 11.10), several packages I've discovered will potentially break if you have a non-system Python executable on your PATH, e.g. using virtualenv or a custom-built Python. As per the Debian Python Policy (I can't find a similarly thorough document for Ubuntu),

> The preferred specification for the Python interpreter is /usr/bin/python or /usr/bin/pythonX.Y. This ensures that a Debian installation of python is used and all dependencies on additional python modules are met.

> Maintainers should not override the Debian Python interpreter using /usr/bin/env python or /usr/bin/env pythonX.Y. This is not advisable as it bypasses Debian's dependency checking and makes the package vulnerable to incomplete local installations of python.

I think this is reasonable, and also supported by the majority of the Python scripts in my /usr/bin directory.

This also has potential security implications, i.e. someone with only user-level access could override the system Python in a user's ~/.bash_profile and install a malicious version of certain package dependencies.

dwf@barricade:~$ lsb_release -rd
Description: Ubuntu 11.04
Release: 11.04
dwf@barricade:~$ grep '#!/usr/bin/env python' /usr/bin/* /usr/sbin/* |cut -d ':' -f 1|xargs dpkg -S
gconf2: /usr/bin/gsettings-schema-convert
mercurial-common: /usr/bin/hg-ssh
hplip: /usr/bin/hp-align
hplip: /usr/bin/hp-check
hplip: /usr/bin/hp-clean
hplip: /usr/bin/hp-colorcal
hplip: /usr/bin/hp-firmware
hplip: /usr/bin/hp-hpdio
hplip: /usr/bin/hp-info
hplip: /usr/bin/hp-levels
hplip: /usr/bin/hp-makeuri
hplip: /usr/bin/hp-pkservice
hplip: /usr/bin/hp-plugin
hplip: /usr/bin/hp-probe
hplip: /usr/bin/hp-query
hplip: /usr/bin/hp-scan
hplip: /usr/bin/hp-setup
hplip: /usr/bin/hp-testpage
hplip: /usr/bin/hp-timedate
hplip: /usr/bin/hp-unload
gnome-applets: /usr/bin/invest-chart
pitivi: /usr/bin/pitivi
libpurple-bin: /usr/bin/purple-remote
libpurple-bin: /usr/bin/purple-url-handler
hplip: /usr/sbin/hpssd

dwf@barricade:~$ grep '#!/usr/bin/env python' /usr/bin/* /usr/sbin/* |cut -d ':' -f 1 |xargs dpkg -S |cut -d':' -f 1|xargs apt-cache policy
gconf2:
  Installed: 2.32.2-0ubuntu2
  Candidate: 2.32.2-0ubuntu2
  Version table:
 *** 2.32.2-0ubuntu2 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main i386 Packages
        100 /var/lib/dpkg/status
hplip:
  Installed: 3.11.1-2ubuntu2
  Candidate: 3.11.1-2ubuntu2
  Version table:
 *** 3.11.1-2ubuntu2 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main i386 Packages
        100 /var/lib/dpkg/status
gnome-applets:
  Installed: 2.32.1.1-0ubuntu5
  Candidate: 2.32.1.1-0ubuntu5
  Version table:
 *** 2.32.1.1-0ubuntu5 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main i386 Packages
        100 /var/lib/dpkg/status
libpurple-bin:
  Installed: 1:2.7.11-1ubuntu2.1
  Candidate: 1:2.7.11-1ubuntu2.1
  Version table:
 *** 1:2.7.11-1ubuntu2.1 0
        500 http://security.ubuntu.com/ubuntu/ natty-security/main i386 Packages
        100 /var/lib/dpkg/status
     1:2.7.11-1ubuntu2 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main i386 Packages
pitivi:
  Installed: 0.13.5-1ubuntu4
  Candidate: 0.13.5-1ubuntu4
  Version table:
 *** 0.13.5-1ubuntu4 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main i386 Packages
        100 /var/lib/dpkg/status
mercurial-common:
  Installed: 1.7.5-1ubuntu1
  Candidate: 1.7.5-1ubuntu1
  Version table:
 *** 1.7.5-1ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/universe i386 Packages
        100 /var/lib/dpkg/status

Related branches

affects: hplip (Ubuntu) → ubuntu
visibility: private → public
Till Kamppeter (till-kamppeter) wrote :

David, thank you for the bug report. Such a report needs to be assigned to the individual packages to make the appropriate maintainers aware of the problem. I have done so now.

affects: ubuntu → hplip (Ubuntu)
Changed in hplip (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Till Kamppeter (till-kamppeter) wrote :

To the HP developers at HP: Please use "#!/usr/bin/python" in your Python executables, the distro's Python interpreter is always at that path, or use "@PYTHONPATH@" and let the "configure" script determine the correct Python path and replace the placeholder with it.

Changed in hplip (Ubuntu):
status: Triaged → In Progress
Jiri Popelka (jpopelka) wrote :

I would like to second this request as in Fedora we also change the shebang
'#!/usr/bin/env python' to '#!/usr/bin/python'
in python executables when building hplip packages.
For more details see
http://fedoraproject.org/wiki/Features/SystemPythonExecutablesUseSystemPython

Thanks.

--
Jiri

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package hplip - 3.11.10-1ubuntu4

---------------
hplip (3.11.10-1ubuntu4) precise; urgency=low

  * debian/rules: Use "#!/usr/bin/python" instead of "#!/usr/bin/env python"
    to specify the path to the Python interpreter, to assure that always
    the system's interpreter is used and never a possibly incompatible
    alternative Python interpreter (LP: #912625).
 -- Till Kamppeter <email address hidden> Fri, 6 Jan 2012 09:37:00 +0100

Changed in hplip (Ubuntu):
status: In Progress → Fix Released
Till Kamppeter (till-kamppeter) wrote :

OdyX, can you have a look at the Debian packages of c2esp, foo2zjs, and pyppd? Thanks.

Didier Raboud (odyx) wrote :

I might take a look at c2esp, foo2zjs and pyppd in due time.

So far I'm not sure this bug is as uncontroversial as it might seem and would really prefer a discussion about it on <email address hidden> before rushing out any actions. The Debian Python Policy currently says "Maintainers _should_ not use env ..": this doesn't mean it's (currently) forbidden; this even means it discouraged, but allowed.

I don't think creating a Ubuntu-specific diff for this type of bug is the way to go: either there is a valid reason to do it and it should be done in Debian first (with appropriate Debian Python Policy change, lintian warning, etc) or there is none and, well, the cost of the diff will have to be carried by Ubuntu hands. (Note that given bugs on the respective packages, such a change could be done in very few days in Debian.) Not doing it in "Debian first" now means that it might eventually be done there later and that a merge will be needed; that sounds like unneccessary work on both sides (and I like to avoid unneccessary work).

In particular, this topic was previously discussed there: http://lists.debian.org/debian-python/2009/09/msg00132.html and there seemed to be reasonable agreement around a lintian warning and/or python policy amendment (although without actions); this should really come before blindly fixing all affected packages in Ubuntu only. (IMHO, eh).

Cheers, OdyX

Marc Deslauriers (mdeslaur) wrote :

This isn't a security issue. A regular user can't modify which python gets used by other users.

security vulnerability: yes → no
security vulnerability: yes → no
Barry Warsaw (barry) wrote :

/usr/bin/env is a perfectly good solution for *development* branches of packages, but very definitely not for deployed production versions of applications, for exactly the reasons described in this bug report. Meaning: if you are developing a Python application, by all means use /usr/bin/env in your own code, since this will make it easier to test against a variety of Python versions. But packaging should always install the application using the explicit path to the appropriate Python executable. I'm pretty sure distribute and setuptools do this munging automatically.

Scott Kitterman (kitterman) wrote :

I'm not convinced this is a bug at all. Debian Python policy (which Ubuntu uses) gives a preference to /usr/bin/python*, but says /usr/bin/env python is allowable. You shouldn't put a non-system python in your system's python path.

Marc: Fair enough. I guess the same kind of hijacking I mentioned could be accomplished in a lot of ways, including setting PYTHONPATH, so it's probably alright.

Scott: The Debian policy reads:

> Maintainers should not override the Debian Python interpreter using /usr/bin/env

The word "should" is somewhat ambiguous in its level of severity, but I would read that as a "strongly discouraged" even if not a hard and fast "you must not do this". I would say there ought to be a really good reason if a system-installed executable is figuring out which interpreter to use at runtime.

> You shouldn't put a non-system python in your system's python path.

I assume you mean on your shell's PATH as the PYTHONPATH is something different -- at any rate, this is an unworkable demand for just about anyone who does anything resembling serious Python development. In addition to virtualenv being a ubiquitous tool for deployment management and environment isolation, several specialized Python distributions exist (both commercial and FLOSS) such as Enthought Python Distribution, ActivePython, FEMhub, Sage, etc. and isolate themselves from the system Python (as they should).

Placing the bin directory of one of these distributions, or of a virtualenv, on your shell's PATH (i.e. adding it to your PATH in ~/.bash_profile) should not cause random system-installed executable scripts to start breaking, and I would very much consider it a bug in the package that installed the executable if this does happen.

Furthermore, most of the Python scripts in /usr/bin on my machine follow the "hard code which interpreter you want" convention, and as Barry pointed out above, even the native Python packaging system, broken as it is in many ways, performs this kind of munging. I would consider this a strong case for not using #!/usr/bin/env python.

Changed in mercurial (Debian):
status: Unknown → Fix Released

The mercurial task is fixed in precise:

$ head -1 /usr/bin/hg-ssh
#!/usr/bin/python

Changed in mercurial (Ubuntu):
status: New → Fix Released
Roman Yepishev (rye) on 2012-06-06
no longer affects: ubuntu-sso-client
papukaija (papukaija) on 2012-06-06
tags: added: natty
tags: added: oneiric precise
Roman Yepishev (rye) on 2012-06-06
no longer affects: ubuntu-sso-client (Ubuntu)
Logan Rosen (logan) on 2012-09-14
affects: gconf2 (Ubuntu) → gconf (Ubuntu)
Jackson Doak (noskcaj) wrote :

Can someone please check this is still an issue? I though we are meant to use /env shebangs now

Changed in gnome-applets (Ubuntu):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.