landscape-sysinfo crashed with ImportError in <module>()

[Jim Garten]

The Landscape Team has requested an SRU for this bug. The required information follows below, and the original bug description is available in the link below.

=== Statement explaining the impact ===

This bug is typically triggered by release upgrades from jaunty to karmic. During the upgrade some Python modules are not importable and cause a crash in the landscape-sysinfo script, which is in turned run by pam-motd upon user login. For this reason the bug potentially affects everyone upgrading from jaunty to karmic. The bug in itself has no other effect than firing an apport bug, however this is likely to make the user think that something went wrong, so fixing it is really important from a user-experience point of view.

=== How the bug has been addressed ===

The landscape-sysinfo script has been modified to catch possible module import errors and exit in that case. The fix is already included in version 1.4.0-0ubuntu0.9.10.0 of the landscape-client package, which the current one in lucid and karmic.

=== Patch ===

The comment #27 of this bug sports a patch created against the landscape-client 1.3.2.3-0ubuntu0.9.04.0 package currently in jaunty-updates.

=== How to reproduce the bug ===

Issue a release-upgrade from jaunty to karmic and switch user or login as a new user.

=== Regression potential ===

The change is very isolated and there's no possibility of regression.

[Andreas Hasenack (ahasenack)]

Hello Jim,

can you please run this command in a terminal, as your user?

landscape-sysinfo

Please paste the whole output here.

[Kees Cook (kees)]

/usr/lib/python2.6/dist-packages/twisted/internet/_sslverify.py:5: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import itertools, md5
/usr/lib/python2.6/dist-packages/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
  System load: 0.39 Swap usage: 2% Users logged in: 1
  Memory usage: 34% Processes: 249

  => There is 1 zombie process.

[Matt Zimmerman (mdz)]

Try:

python -c 'import zope.interface'

[Kees Cook (kees)]

This works fine. I commented in my dup'd bug that this happened during upgrade, so I suspect that zope gets into a funny state while update-motd tries to run landscape-sysinfo while the upgrade is happening.

[Andreas Hasenack (ahasenack)]

Matt, Kees, is there something going on with python packaging that we should fix? Or is it perhaps just jaunty's transient nature (alpha, beta, etc)? At some point during the upgrade, landscape-sysinfo just wasn't able to find zope.interface, but the package is installed. At least at the end of the upgrade it is.

[Kees Cook (kees)]

I think the issue is that crontab is running landscape-sysinfo (by way of update-motd) in the middle of an upgrade. It would be best if landscape-sysinfo handled this in a more graceful way (instead of causing a traceback). This is especially true since it is installed by default.

[Matt Zimmerman (mdz)]

I expect Kees is right.

Probably the cron job should not run if the package is unconfigured.

[Andreas Hasenack (ahasenack)]

But in this case it was a dependency that was being installed, not landscape-common itself (which contains landscape-sysinfo). It was python-zopeinterface that was not available. How can we address this?

[Matt Zimmerman (mdz)]

On Mon, Mar 30, 2009 at 09:39:14PM -0000, Andreas Hasenack wrote:
> But in this case it was a dependency that was being installed, not
> landscape-common itself (which contains landscape-sysinfo). It was
> python-zopeinterface that was not available. How can we address this?

The packaging system takes care of this for you if your dependencies are
correct. It won't allow a package to be configured until its dependencies
have been configured.

--
 - mdz

[Andreas Hasenack (ahasenack)]

I'm sorry, I don't mean to transform this into a Debian Packaging 101 :) We can continue elsewhere, or just point me to some docs or other packages that have a similar problem.

So what if only python-zopeinterface is being upgraded? landscape-sysinfo is installed and is not being upgraded in this example. Then suddenly when python-zopeinterface is being configured (the new version), the cron job hits. How can the cron job detect this situation and decide to not run?

[Matt Zimmerman (mdz)]

On Mon, Mar 30, 2009 at 10:03:28PM -0000, Andreas Hasenack wrote:
> I'm sorry, I don't mean to transform this into a Debian Packaging 101 :)
> We can continue elsewhere, or just point me to some docs or other
> packages that have a similar problem.
>
> So what if only python-zopeinterface is being upgraded? landscape-
> sysinfo is installed and is not being upgraded in this example. Then
> suddenly when python-zopeinterface is being configured (the new
> version), the cron job hits. How can the cron job detect this situation
> and decide to not run?

Oh, that wasn't clear from this report. If only python-zopeinterface was
being upgraded, I would expect the window for this race to be very small
(but still present). The window is very large, though, when upgrading from
8.10 to 9.04. Because so many packages are being upgraded at once, the
packages will stay unconfigured (and potentially non-functional) for a long
time.

I'm surprised this hasn't come up before (which means it probably has and
I'm not aware of it). There may be a standard pattern for dealing properly
with this. I suggest taking the question to <email address hidden>.

--
 - mdz

[Thomas Herve (therve)]

So, the situation has changed a bit with Karmic as update-motd is gone, replaced by pam-motd. landscape-sysinfo is not run by cron anymore, apparently on login directly. But the problem is basically the same: if you login during an upgrade, an apport bug will be fired. Note that the motd won't be broken, probably because it detects the error code and thus don't copy the new motd.

Anyway, the way to fix it is to catch the error in landscape-sysinfo, so that the exception doesn't bubble up, but still exit with an error status, so that the motd is not updated. This is what I've done in the branch.

[Kevin McDermott (bigkevmcd)]

Nice simple fix, +1

[Free Ekanayaka (free.ekanayaka)]

Untested, but effective, +1.

I'll need to decide if we want this fix to be included in karmic before it gets out.

[Thomas Herve (therve)]

The fix has been merged in landscape-client trunk in r148.

[Andreas Hasenack (ahasenack)]

qa + 1, apport stays quiet now regarding import errors in landscape-common, while with the previous landscape-common an import error would trigger it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 1.3.2.4-0ubuntu0.9.10.0

---------------
landscape-client (1.3.2.4-0ubuntu0.9.10.0) karmic; urgency=low

  * New upstream release:
    - Catch import errors in the landscape-sysinfo script to prevent
      errors when landscape-sysinfo is run to update the motd during
      upgrade (LP: #349996)
    - Fix a long-standing bug in the client which causes resynchronisations
      on the server (LP: #144475)
    - When downloading hash-id stores, pass possible custom SSL certificates
      to the fetch fuction (LP: #435887)
    - Handle unicode username in custom graphs, and also report missing user
      to the server properly (LP: #406388)
    - Handle a SQlite bug when creating package store database, by making an
      extra query to flush the table cache (LP: #416629)

 -- Free Ekanayaka <email address hidden> Fri, 09 Oct 2009 18:21:24 +0200

[Thomas Herve (therve)]

The fix is released in Karmic. The problem happens with the jaunty version still running during the upgrade.

[Walldorf2000 (walldorf2000)]

I have updated to Karmic with all available updates. After a reboot there are still jaunty packages active?

Synaptic does not report any obsolete packages whatsoever.

[Andreas Hasenack (ahasenack)]

Walldorf2000, the crash happened *during* the upgrade, but it is being *reported* to you via apport after the reboot when you login.

Unless I completely misunderstood how apport crash reports work...

[David Grant (davidgrant)]

This is happening to me near the end of the upgrade to Karmic.

[Kevin McDermott (bigkevmcd)]

Hi,

The Landscape team would like to apologise for the confusion that this bug has caused, and the bad experience users have had during their upgrades.

This bug was originally identified during the early releases of Karmic, but we didn't get a fix in place until the 10th of October 2009 - after the freeze for Ubuntu had occurred, essentially the bug was caused by the automated cron job that updates the landscape-sysinfo information continuing to run during the upgrade process, which failed because some of the libraries that it relied on were removed briefly during the actual upgrade routine.

The error that is reported is caused by the import of these libraries, and is trivial in nature, it just means that the cron job didn't complete once during the upgrade, unfortunately apport catches these errors and reports them as bugs (which we accept it is).

A fix has now been put in place which will hopefully avoid this problem in future, and this will make it into Ubuntu at the earliest possible opportunity.

Thanks,
The Landscape Team

[The Founder (president-yooter)]

Dear Landscape Team,

Trust us, we are grateful that this is fixed!

If this was Windows 7 or Vista.. it wouldn't be fixed until the next service pack update... interaction like this is why we all adopted Ubuntu as our OS of choice.

Best Regards,
Roger

[Martin Pitt (pitti)]

Since this appears to be a regression in jaunty-updates, I'm bumping this to critical.

[Thomas Herve (therve)]

Martin: it's not a regression, this bug has been present for a while. It's fixed in Karmic, but the fix was done after the last SRU was submitted.

[Free Ekanayaka (free.ekanayaka)]

This bug was actually present in the jaunty package, so I'm removing the regression-update tag. As mentioned in comment #19, this is already fixed in Karmic.

The attached diff created against lp:ubuntu/jaunty-proposed/landscape-client solves the situation. I'm going to ask Mathias Gug to sponsor an SRU upload to jaunty-proposed for it.

[Scott Kitterman (kitterman)]

Was it present in Jaunty at release?

[Free Ekanayaka (free.ekanayaka)]

@Scott: yes and no.

Yes, in the sense that if you try to upgrade a jaunty system with the original landscape-client package version from jaunty at release, you will hit the bug, in the same way you would it with the landscape-client package version now in jaunty-updates. The code of landscape-sysinfo hasn't changed at all between these two versions.

No, in the sense that the landscape-sysinfo script in the original jaunty landscape-client package is working fine in jaunty per se. The bug is triggered by the upgrade, and in particular because of the transition from update-motd (in jaunty) to pam-motd (in karmic), which makes the script run at every login.

[Mathias Gug (mathiaz)]

Uploaded to jaunty-proposed.

[Mathias Gug (mathiaz)]

Marking Fixed Released in Karmic and Lucid

[Martin Pitt (pitti)]

Accepted landscape-client into jaunty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Free Ekanayaka (free.ekanayaka)]

The updated landscape-sysinfo script works as expected, and doesn't tracebacks if a Python module is not importable. Please move the landscape-client package on to jaunty-updates.

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 1.3.2.3-0ubuntu0.9.04.1

---------------
landscape-client (1.3.2.3-0ubuntu0.9.04.1) jaunty-proposed; urgency=low

  * Fix crash in landscape-sysinfo due to an import error during system wide
    upgrades (LP: #349996)

 -- Free Ekanayaka <email address hidden> Tue, 03 Nov 2009 11:44:39 +0100