crash in indexer.py, AttributeError: 'NoneType' object has no attribute 'unlink'

Bug #1652641 reported by Carlo Vanini on 2016-12-26
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
apt-xapian-index (Ubuntu)
Medium
Unassigned

Bug Description

[Impact]

When the apt-xapian-index is already up to date, and under certain conditions, update-apt-xapian-index throws an exception.

The index /var/lib/apt-xapian-index is up to date
Exception ignored in: <bound method ServerProgress.__del__ of
<axi.indexer.ServerProgress object at 0x7f04a5ae4c18>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
AttributeError: 'NoneType' object has no attribute 'unlink'

This happens more frequently when it is run with nice and ionice, which is the case in the cron.weekly/apt-xapian-index cron job.

This is not severe, since the exception is thrown on exit, and depends on the way python3 cleans up the environment. Anyway, this leaves a socket file behind, and creates noise in cron logs (see comment #2).

The patch keeps a reference to the os.unlink function in the ServerProgress instance, such that it has the same lifespan.

[Test Case]

* `sudo nice -n 19 ionice -c 3 update-apt-xapian-index --quiet`
* may need to run 4 times or more
* no exception is thrown

[Regression Potential]

* If the os module is unloaded anyway and self.os_unlink becomes unreachable, we get the same exception again.

[Other Info]

* There is probably a more "pythonic" way to fix this. For instance, avoiding __del__. That requires a larger change.

André Verwijs (verwijs) wrote :

(k)ubuntu version 17.04
kernel version: 4.9.0-11.12
apt-xapian-index version: 0.47ubuntu11

Andreas Roth (aroth) wrote :

This morning i received this mail from the cron.weekly run:

/etc/cron.weekly/apt-xapian-index:
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f373e677710>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
AttributeError: 'NoneType' object has no attribute 'unlink'

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt-xapian-index (Ubuntu):
status: New → Confirmed
Andreas Roth (aroth) wrote :

Sorry, but forgot to mention that i'm running Ubuntu yakkety on my machine.

tags: added: yakkety
Brian Murray (brian-murray) wrote :

I didn't see any instances of this Traceback at errors.ubuntu.com. It's worth noting that apt-xapian-index is only included in Task: kubuntu-desktop, kubuntu-full, lubuntu-qt-desktop, ubuntukylin-desktop.

Carlo Vanini (silhusk) wrote :

It looks like the same bug reappearing https://bugs.launchpad.net/ubuntu/+source/apt-xapian-index/+bug/1530518/comments/2

Running a-x-i directly doesn't trigger this bug, but the combination of nice and ionice makes it show up much more often. (When the index is already up to date)

Steps to reproduce:
 * `sudo update-apt-xapian-index`
 * Now that the index is updated we trigger the bug by running the following command (which is the same used in the cron job)
 * `sudo nice -n 19 ionice -c 3 update-apt-xapian-index --quiet`
 * may need to run 4 times or more

The intention of the proposed patch is to work around the problem by keeping a reference to the os.unlink function in the ServerProgress instance.
See http://stackoverflow.com/questions/17084260/imported-modules-become-none-when-running-a-function for an explanation of the issue in python.
Also see http://pydev.blogspot.ch/2015/01/creating-safe-cyclic-reference.html for a possible implementation using weakrefs.

The attachment "keep a reference to os.unlink" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Changed in apt-xapian-index (Ubuntu):
importance: Undecided → Medium
tags: added: zesty
Carlo Vanini (silhusk) on 2017-05-05
description: updated
Simon Quigley (tsimonq2) wrote :

Hello Carlo,

Could you please clarify which release this should go to? Yakkety is EOL now.

Also, irt to your patch, DEP-3 headers don't need "##" in front of them, please remove that.

Thank you for your interest in fixing this bug!

(unsubscribing ~ubuntu-sponsors as there's nothing to sponsor)

Carlo Vanini (silhusk) wrote :

Hello Simon,

version 0.47ubuntu13 in zesty-updates is still affected.
version 0.47ubuntu8.3 in xenial-updates is affected too.

Artful beta1 has 0.47ubuntu13 like Zesty does, but I could not reproduce the bug. The reason could be that Python changed from 3.5.3 (in Zesty) to 3.6.2 (in Artful). The way Python unloads modules might have changed, but I couldn't tell for sure from the docs https://docs.python.org/3/whatsnew/3.6.html

Patch updated, as you suggested.

Carlo Vanini (silhusk) wrote :

Sorry for the noise, patch updated again. This time it uses a weakref.finalizer instead of __del__. This is a little a cleaner solution. Moreover the finalizer is guaranteed to be called (at the latest at exit), whereas __del__ is not.

Amr Ibrahim (amribrahim1987) wrote :

Carlo, I subscribed ~ubuntu-sponsors for you to review the debdiff.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers