Python build from source lack some modules due to non-standard libraries placement

Bug #738213 reported by Dmitry Tantsur
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Python
Fix Released
Unknown
db4.8 (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Unassigned
gdbm (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Unassigned
python2.6 (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Unassigned
python2.7 (Ubuntu)
Critical
Matthias Klose
Natty
Critical
Matthias Klose
python3.1 (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Unassigned
python3.2 (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Unassigned
sqlite3 (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Unassigned
zlib (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Unassigned

Bug Description

Modules: bsddb (libdb4.8 libraries are not in /usr/lib), zlib, dl and maybe some other.
Python build system expects libraries to be in the same branch and e.g. libdb so are somewhere in /usr/lib/x86_64-linux-gnu.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: libdb4.8-dev 4.8.30-5ubuntu2
ProcVersionSignature: Ubuntu 2.6.38-6.34-generic 2.6.38-rc7
Uname: Linux 2.6.38-6-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Sat Mar 19 16:36:32 2011
InstallationMedia: Ubuntu 10.04 "Lucid Lynx" - Beta amd64 (20100318)
ProcEnviron:
 LANGUAGE=ru_RU:en
 PATH=(custom, no user)
 LANG=ru_RU.UTF-8
 SHELL=/bin/bash
SourcePackage: db4.8
UpgradeStatus: Upgraded to natty on 2011-03-07 (11 days ago)

Revision history for this message
Dmitry Tantsur (divius) wrote :
Revision history for this message
Dmitry Tantsur (divius) wrote :

Reproduce: try to install python from sources (after doing `apt-get build-dep python 2.7` of course).
You'll see warnings about missing modules:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb _sqlite3 bsddb185
dbm dl gdbm
imageop sunaudiodev zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:
crypt nis

Please importance critical, as you'll lose half of your archive on next rebuild.
BTW, there's build failure with setuptools:

Traceback (most recent call last):
  File "setup.py", line 94, in <module>
    scripts = scripts,
  File "/tmp/ibd3-1/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/tmp/ibd3-1/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/tmp/ibd3-1/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/ibd3-setup-bJCwNZ/setuptools-0.6c11/setuptools/command/install.py", line 76, in run
    self.do_egg_install()
  File "/tmp/ibd3-setup-bJCwNZ/setuptools-0.6c11/setuptools/command/install.py", line 96, in do_egg_install
    self.run_command('bdist_egg')
  File "/tmp/ibd3-1/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/tmp/ibd3-1/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/ibd3-setup-bJCwNZ/setuptools-0.6c11/setuptools/command/bdist_egg.py", line 236, in run
    dry_run=self.dry_run, mode=self.gen_header())
  File "/tmp/ibd3-setup-bJCwNZ/setuptools-0.6c11/setuptools/command/bdist_egg.py", line 527, in make_zipfile
    z = zipfile.ZipFile(zip_filename, mode, compression=compression)
  File "/tmp/ibd3-1/lib/python2.7/zipfile.py", line 679, in __init__
    "Compression requires the (missing) zlib module"
RuntimeError: Compression requires the (missing) zlib module

Revision history for this message
Steve Langasek (vorlon) wrote :

The change in location of the libraries is an intended transition; marking as invalid for those tasks. The focus is on getting python to build against the new location.

Changed in python2.7 (Ubuntu):
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Steve Langasek (vorlon)
milestone: none → ubuntu-11.04-beta-1
Changed in db4.8 (Ubuntu):
status: New → Invalid
Changed in gdbm (Ubuntu):
status: New → Invalid
Changed in sqlite3 (Ubuntu):
status: New → Invalid
Changed in zlib (Ubuntu):
status: New → Invalid
tags: added: multiarch
Revision history for this message
Dmitry Tantsur (divius) wrote :

And what the hell should I do with the official Python distribution???

For my job I have to run bootstrap script that has to install custom Python. It just cannot be done with Natty. So I have to either manually setup symlinks or downgrade.

Revision history for this message
Dmitry Tantsur (divius) wrote :

Sorry for being rude, but the possibility for working under Natty is at risk for me, and I'm quite used to Ubuntu=)

Revision history for this message
Steve Langasek (vorlon) wrote :

The patch in the linked branch corrects the previous deb-setup patch and gives a correct build in natty again with the same set of modules.

For custom python builds that don't use the Debian/Ubuntu packaging, I believe it is possible to configure additional library paths using configure options such as --libdir, but I'm afraid I'm not familiar enough with the python build system to speak to this authoritatively.

Changed in python2.7 (Ubuntu Natty):
status: Triaged → Fix Committed
assignee: Steve Langasek (vorlon) → Matthias Klose (doko)
Revision history for this message
Dmitry Tantsur (divius) wrote :

I guest I can adopt your deb-setup.diff for my purposes... Can it be used with patch utility?

Revision history for this message
Dmitry Tantsur (divius) wrote :

I ended up with the following (leaving here for someone with the same trouble):

--- setup.py.orig 2011-03-23 11:30:58.000000000 +0300
+++ setup.py 2011-03-23 11:34:46.000000000 +0300
@@ -349,6 +349,14 @@
         # Ensure that /usr/local is always used
         add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib')
         add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
+ try:
+ triplet = os.popen('dpkg-architecture -qDEB_HOST_MULTIARCH').readline()[:-1]
+ except EnvironmentError:
+ pass
+ else:
+ add_dir_to_list(self.compiler.library_dirs, '/usr/lib/%s' % triplet)
+ add_dir_to_list(self.compiler.include_dirs, '/usr/include/%s' % triplet)
+

         # Add paths specified in the environment variables LDFLAGS and
         # CPPFLAGS for header and library files.

Apply:
cd Python-2.7.1
patch ./setup.py /path/to/patches/python-setup.py-natty.diff

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python2.7 - 2.7.1-5ubuntu1

---------------
python2.7 (2.7.1-5ubuntu1) natty; urgency=low

  * debian/patches/deb-setup.diff: DEB_GNU_TYPE was never being set, so
    the expected multiarch path support wasn't working. Consolidate to
    a single call to dpkg-architecture -qDEB_HOST_MULTIARCH in setup.py.
    LP: #738213.
 -- Steve Langasek <email address hidden> Wed, 23 Mar 2011 15:02:47 -0700

Changed in python2.7 (Ubuntu Natty):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python2.6 - 2.6.6-6ubuntu7

---------------
python2.6 (2.6.6-6ubuntu7) natty; urgency=low

  * Search libraries in multiarch directories too. LP: #738213.
 -- Matthias Klose <doko@timbuktu> Fri, 25 Mar 2011 17:45:24 +0100

Changed in python2.6 (Ubuntu Natty):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python3.2 - 3.2-1ubuntu1

---------------
python3.2 (3.2-1ubuntu1) natty; urgency=low

  * Search libraries in multiarch directories too. LP: #738213.
 -- Matthias Klose <email address hidden> Fri, 25 Mar 2011 18:00:03 +0100

Changed in python3.2 (Ubuntu Natty):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python3.1 - 3.1.3-1ubuntu1

---------------
python3.1 (3.1.3-1ubuntu1) natty; urgency=low

  * Search libraries in multiarch directories too. LP: #738213.
 -- Matthias Klose <email address hidden> Fri, 25 Mar 2011 18:05:31 +0100

Changed in python3.1 (Ubuntu Natty):
status: New → Fix Released
Revision history for this message
Barry Warsaw (barry) wrote :

The question is: what should we do about this upstream?

From the mercurial repo, the following modules fail to build in 2.7:

_bsddb
_sqlite3
bsddb185 (don't care)
dbm
dl
gdbm
imageop (don't care)
sunaudiodev (don't care)
zlib

I don't know if it would be acceptable to include the dpkg-architecture call in upstream, though definitely not as given in deb-setup.diff (since that's too platform specific). I also don't know how far back we can or should fix upstream.

Revision history for this message
Barry Warsaw (barry) wrote :
Revision history for this message
Barry Warsaw (barry) wrote :

Scratch that. Use http://bugs.python.org/issue11715 instead.

Changed in python:
status: Unknown → New
Changed in python:
status: New → Fix Released
Changed in python:
status: Fix Released → New
Changed in python:
status: New → Fix Released
Changed in python:
status: Fix Released → Fix Committed
Changed in python:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.