python modules need to work during dist-upgrades

[jfenlason]

Binary package hint: apport

I was trying to upgrade from Edgy to Fiesty and I got the following error.
Could not install '/var/cache/apt/archives/samba_3.0.24-2ubuntu1.2_i386.deb'
The upgrade aborts now. Please report this ...
subprocess new post-removal script returned error exit status 134.

/var/log/dist-upgrade ends with the following.
QGVector::insert: Index 3952 out of range
QGVector::insert: Index 3953 out of range
Traceback (most recent call last):
  File "/usr/share/apport/package_hook", line 15, in <module>
    import apport, apport.fileutils
ImportError: No module named apport

[Martin Pitt (pitti)]

This is not easy to solve properly. During a dist-upgrade, many packages are unconfigured for a while, and thus Python modules do not work. This needs to be coordinated with apt's apport integration.

[Martin Pitt (pitti)]

We get so many crashes which are due to Python modules not being functional at all during dist-upgrades (or after a partially failed one), that we really need to fix this properly once and for all. We should investigate the following possibilities:

 * Drop all the symlink magic and instead add the non-version-dependent module directories to the default module search path of all python versions. (My prefered solution)

 * Do all the symlink removal and updates in postinst only, so that it happens atomically.

[Matthias Klose (doko)]

Readded apport:

 * Disable apport during upgrades

Your preferred solution doesn't work. extension modules are not found for the case of python-support, and adding two location for the same package leads to errors in the import order of subpackages.

[Martin Pitt (pitti)]

It does not just affect apport, but also restricted-manager, update-manager, update-notifier, and all other Python packages.

I do not quite understand your reply. "Extension modules" are compiled ones which need to exist per-version, right? But those need separate paths per version anyway? I was just talking about the Python modules (.py files) which are version independent. Right now they are linked to two different places (once for each supported Python version), and those symlinks (or rather, removing them during upgrades) are the bit that cause the pain.

What do you mean with "adding two locations for the same package"?

[Matthias Klose (doko)]

* update-manager, update-notifier and restricted-manager either already run during an upgrade, or they should be started only after the package is configured; apport is started *during* the update. so there is a difference.

* python-support has extensions located at two different locatations, which are both not part of the directory which is found on sys.path; splitting a package in such a way causes differences in the import order. changing the upstream behaviour is no option.

[Ralph Corderoy (ralph-inputplus)]

AMD64 7.04 install, fully up to date, and then chose to upgrade to the new 7.10 installation that was available with the GUI. It worked quite happily for a while and got to the stage of downloading lots of files. Left it to it. Now it's hung. The GUI is still present and responds to X expose events but nothing's happening. The end of dist-upgrade/apt.log says

 Traceback (most recent call last):
   File "/usr/share/apport/package_hook", line 15, in <module>
     import apport, apport.fileutils
 ImportError: No module named apport

Some of the running processes include

 10757 pts/0 Ss+ 0:00 /usr/bin/python /tmp/tmpurZIGY/gutsy --have-prerequists --with-network
 20408 pts/0 S+ 0:00 /bin/sh /var/lib/dpkg/info/ttf-indic-fonts-core.postinst configure
 20420 pts/0 D+ 0:02 [fc-cache]
 32245 pts/0 S+ 0:01 /tmp/tmpurZIGY/backports/usr/bin/dpkg --force-overwrite --status-fd 4 --configure libc6-i386 linux-libc-dev libc6

32245 is sitting in waitpid().

If this bug is the cause of a 7.04 to 7.10 failure I can't believe that it's sitting here as High importance yet users are still allowed to topple into its abyss by upgrading to 7.10. What do I do to recover?

[Ralph Corderoy (ralph-inputplus)]

There's a group of crash files, all around the time things started going wrong.

$ ls -ltra /var/crash
total 56
drwxr-xr-x 15 root root 4096 2007-11-18 17:53 ..
-rw-r--r-- 1 root root 182 2007-11-18 21:19 update-manager.0.crash
-rw-r--r-- 1 root root 184 2007-11-18 21:19 update-notifier.0.crash
-rw-r--r-- 1 root root 196 2007-11-18 21:19 software-properties-gtk.0.crash
-rw-r--r-- 1 root root 188 2007-11-18 21:19 gnome-app-install.0.crash
-rw-r--r-- 1 root root 188 2007-11-18 21:19 language-selector.0.crash
-rw-r--r-- 1 root root 179 2007-11-18 21:19 ubuntu-desktop.0.crash
-rw-r--r-- 1 root root 190 2007-11-18 21:20 restricted-manager.0.crash
-rw-r--r-- 1 root root 166 2007-11-18 21:20 apturl.0.crash
-rwxr-xr-x 1 root root 0 2007-11-19 10:56 .lock
drwxrwxrwt 2 root root 4096 2007-11-19 10:56 .

They're all similar. Here's one.

ProblemType: Package
Date: Sun Nov 18 21:19:41 2007
ErrorMessage:
 ErrorMessage: dependency problems - leaving unconfigured

Package: software-properties-gtk
SourcePackage: software-properties

[Martin Pitt (pitti)]

Fixed in bzr head (use 'nomove').

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 0.102

---------------
apport (0.102) hardy; urgency=low

  [ Martin Pitt ]
  * problem_report.py: Support reading reports with legacy zlib
    compression in 'retain compressed values' mode (as used nowadays by
    apport when reporting a crash). Add a test case, too. (LP: #129616)
  * debian/control, debian/rules: Switch from python-support to
    python-central, and use 'nomove' option so that apport works during
    upgrades, too. (LP: #121341)
  * debian/rules: Use dh_icons instead of dh_iconcache.
  * debian/apport.init: Do not stop apport in any runlevel (LSB header).
  * apport/ui.py, run_crash(): Catch zlib.error on invalidly compressed core
    dumps. (LP: #176977)
  * apport/ui.py: Give a meaningful error message instead of crashing if the
    package for a crash report is not installed any more. (LP: #149739)
  * apport/ui.py: Do not include ProcCmdline in bug reports, since these are
    not ack'ed by the user and might contain sensitive data. (LP: #132800)
  * apport/ui.py: Add various test cases for crash reports whose packages have
    been uninstalled between the crash and the report. This reproduces
    LP #186684.
  * apport/ui.py, load_report(): Produce proper error message if
    executable/interpreter path do not exist any more. (LP: #186684)
  * cli/apport-cli: Intercept SIGPIPE when calling sensible-pager, to avoid
    crash when quitting it prematurely. (LP: #153872)
  * bin/apport-checkreports: Print out a list of program names/packages which
    have a pending crash report. (LP: #145117)
  * apport/ui.py, run_argv(): Add return code which indicates whether any
    report has been processed.
  * cli/apport-cli: If no pending crash reports are present, say so and refer
    to --help. (LP: #182985)
  * apport/ui.py: Waive check for obsolete packages if environment defines
    $APPORT_IGNORE_OBSOLETE_PACKAGES. Document this in the apport-cli manpage.
    (LP: #148064)

  [ Daniel Hahler ]
  * .crash file integration for KDE3 (LP: #177055)
    - debian/apport-qt.install: install added files qt4/apport-qt-mime.desktop
      and qt4/apport-qt-mimelnk.desktop
  * Fixed minor warnings/errors from desktop-file-validate in
    gtk/apport-gtk-mime.desktop.in and qt4/apport-qt.desktop.in (LP: #146957)

 -- Martin Pitt <email address hidden> Wed, 06 Feb 2008 12:55:53 +0100

[Matthias Klose (doko)]

fixed in python-central 0.5.5x

[Robert D. Martin (ets3rodamar)]

Please notice the mention of "nevow" (and of "twitsted") within the following quote of status/error messages sent during a recent re-try at installing python2.6 undertaken after a dist-upgrade for ubuntu9.04 jaunty jackolope failed, which I at first (I know assume, mistakenly) thought to have been caused by faulty coding within the python2.6 package.

[quote]
Richte python2.6-minimal ein (2.6.1-0ubuntu8) ...
Linking and byte-compiling packages for runtime python2.6...
/usr/lib/python2.6/dist-packages/twisted/web/microdom.py:181: SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert (oldChild.parentNode is self,
/usr/lib/python2.6/dist-packages/numpy/lib/tests/test_io.py:68: SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(c.readlines(),
Compiling /usr/lib/python2.6/dist-packages/nevow/context.py ...
SyntaxError: ('invalid syntax', ('/usr/lib/python2.6/dist-packages/nevow/context.py', 37, 12, ' def with(self, tag):\n'))
[/quote]

Having in the meanwhile noticed several mentions upstream of "nevow" causing trouble for upgrades due to a self-reference within its installation procedures, whereby building the "nevow" software required access to its already built self in order to get itself built.

(see for example
http://allmydata.org/trac/tahoe/ticket/455
http://bugs.python.org/setuptools/issue20
http://divmod.org/trac/ticket/2699
http://allmydata.org/trac/tahoe/ticket/530
http://bugs.python.org/setuptools/issue57)

The "nevow" build process includes a self-reference; the code being carried out to build python2.6 contains a tautology. Might not that be the actual cause of the self-reference and thus of the build problems not only of "nevow" itself but also of the packages such as python2.6 dependent upon it for their own build?
_____________________
I see no way to attach text files to this report at the moment, so I will include here inline the following four textual records of the messages displayed pertinent to my attempts to finally complete the dist-upgrade without leaving out any but the python2.6 packages. That effort eventually seemed to me to succeed, which initially got me believing that the python2.6 package had caused the problems. But that was before I caught onto the "nevow" problems upstream.

it now seems obvious to me that possibly not python2.6 itself be at fault but, of course, "twisted" and / or "nevow", be the troublemakers. At any rate, neither the python2.6 package itself nor that software, as such, need be at fault, were the installation software, itself, to be the culprit here.

[Begin of the four inline textual inclusions]
___________________
Di Feb 24 11:15:43 CET 2009

/var/cache/apt/archives/rhythmbox_0.11.6svn20090209-0ubuntu1_i386.deb konnte nicht installiert werden

Die Aktualisierung wird fortgesetzt, aber das '/var/cache/apt/archives/rhythmbox_0.11.6svn20090209-0ubuntu1_i386.deb' Paket ist möglicherweise nicht funktionsfähig. Bitte erwägen Sie, einen Fehlerbericht darüber einzusenden.

Unterprozess neues pre-removal-Skript gab den Fehlerwert 1 zurück
___________________________
Di Feb 24 11:23:52 CET 20...

[Matthias Klose (doko)]

nevow and numpy are updated in jaunty.

[Steve Langasek (vorlon)]

python-support is now deprecated. Packages should switch to dh_python2 to avoid this issue.

[Thomas Hotz (thotz-deactivatedaccount)]

Is this still an issue for you? What Ubuntu version do you use? Thank you for telling us!

[Matthias Klose (doko)]

this is still an issue as long as there are third party modules which create symlinks on configuration time

[dino99 (9d9)]

Support for this version has ended

[dino99 (9d9)]

dh-python2 is now used with python2.7