/etc/cron.weekly/apt-xapian-index reports TypeError

Bug #1530518 reported by glitsj16 on 2016-01-02
48
This bug affects 9 people
Affects Status Importance Assigned to Milestone
apt-xapian-index (Ubuntu)
High
Barry Warsaw
Xenial
High
Barry Warsaw

Bug Description

Received the following email from cron:

To: <email address hidden>
Subject: Anacron job 'cron.weekly' on lab16
Date: Sat, 2 Jan 2016 05:53:34 +0100 (CET)

/etc/cron.weekly/apt-xapian-index:
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f226fad1710>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
TypeError: 'NoneType' object is not callable

$ lsb_release -rd
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04

$ apt-cache policy apt-xapian-index
apt-xapian-index:
  Installed: 0.47ubuntu6
  Candidate: 0.47ubuntu6
  Version table:
 *** 0.47ubuntu6 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial/main i386 Packages
        100 /var/lib/dpkg/status

$ sudo /usr/sbin/update-apt-xapian-index
The index /var/lib/apt-xapian-index is up to date
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f5b2a3736d8>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
TypeError: 'NoneType' object is not callable

tags: added: xenial
Changed in apt-xapian-index (Ubuntu):
status: New → Triaged
importance: Undecided → High
dino99 (9d9) wrote :

Also got it on cold boot; join the crash file

Barry Warsaw (barry) wrote :

Almost positive what's happening is that Python shutdown process is interacting badly with ServerProgress's __del__().

__del__() methods are not the best way to reclaim resources, especially when you leave it to Python's shutdown process to implicitly free objects, like what happens in update-apt-xapian-index. Here's what happens:

When Python shutdown, after it's run the atexit handlers, it then tries to free up all objects that are still implicitly alive because of reference counts. It does this first by emptying the __dict__s of all modules, which will decref any still referenced globals. It actually does this first by setting all keys in the module's globals to None. Because Python can't know what's a safe order to do this, it will do it randomly.

Notice a few things. In update-a-x-i, indexer is a module global. When its .lock() method is called, it sets self.progress to a ServerProgress instance and neither instance is explicitly freed, so they live until Python shutdown. Now, when Python gets around to clearing the module dicts as described above, it will set u-a-x-i's indexer instance to None, which will decref it to zero, and so freeing the indexer instance.

Because the ServerProgress instance is kept alive by the reference from the indexer's self.progress attribute, once the indexer is freed, the ServerProgress instance probably decrefs to zero and then *it* gets freed. This will call ServerProgress.__del__(). Line 306 is the os.unlink() call and you'll notice that you get an exception that None is not callable. Why is that?

Well, because if the os module's dictionary got cleared *before* the ServerProgress instance's __del__() got called, the os module is still valid, and it still has a name bound in it called 'unlink' but that name has been bound to None! If you're lucky, the ServerProgress instance gets decrefed away before os gets cleared, but as you've observed in IRC, about every third time, you're unlucky, and os gets cleared first (remember, the order is random). In those cases... boom!

I don't like the design here, since relying on __del__ methods to free resources isn't good Python style, especially when those resources can be freed explicitly, but fixing that is probably more work than we want to do for a package that is abandoned in Debian. A bandaid might be easy though: before the sys.exit() call in u-a-x-i, let's try to explicitly `del indexer`. That will cause all the cascading decrefs before Python's shutdown machinery clears out the os module. I'll attach a diff to try.

Barry Warsaw (barry) wrote :

diff --git a/update-apt-xapian-index b/update-apt-xapian-index
index 5b38c5f..f6e434a 100755
--- a/update-apt-xapian-index
+++ b/update-apt-xapian-index
@@ -103,4 +103,6 @@ if opts.update:
 else:
     indexer.rebuild(opts.pkgfile)

+# Free the resources explicitly. See LP: #1530518
+del indexer
 sys.exit(0)

Brian Murray (brian-murray) wrote :

I've tested Barry's change multiple times and it works for me.

Barry Warsaw (barry) on 2016-04-13
Changed in apt-xapian-index (Ubuntu Xenial):
assignee: nobody → Barry Warsaw (barry)
status: Triaged → In Progress

Hello glitsj16, or anyone else affected,

Accepted apt-xapian-index into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt-xapian-index/0.47ubuntu8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apt-xapian-index (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
dino99 (9d9) wrote :

Note that this issue has not happened since early Feb on my system (was a transitional problem)

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt-xapian-index - 0.47ubuntu8

---------------
apt-xapian-index (0.47ubuntu8) xenial; urgency=medium

  * debian/patches/05_python3.patch: del the indexer instance explicitly
    before sys.exit(), so that it gets decrefed and freed before Python's
    implicit shutdown machinery can potentially leave an empty os module for
    ServerProgress.__del__() to find. (LP: #1530518)

 -- Barry Warsaw <email address hidden> Wed, 13 Apr 2016 17:36:22 -0400

Changed in apt-xapian-index (Ubuntu Xenial):
status: Fix Committed → Fix Released
glitsj16 (glitsj16) wrote :

Confirming apt-xapian-index 0.47ubuntu8 fixes the issue. Thanks to all involved.

Ian Proudler (ikp1) wrote :

I've been getting identical emails to the above from my system recently.

I have checked /usr/sbin/update-apt-xapian-index and it has the bug fix suggested above.

$ uname -a
Linux 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -rd
Description: Ubuntu 16.04.2 LTS
Release: 16.04

$ apt-cache policy apt-xapian-index
apt-xapian-index:
  Installed: 0.47ubuntu8.3
  Candidate: 0.47ubuntu8.3
  Version table:
 *** 0.47ubuntu8.3 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        500 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
        100 /var/lib/dpkg/status
     0.47ubuntu8 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://gb.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

$ sudo /usr/sbin/update-apt-xapian-index
The index /var/lib/apt-xapian-index is up to date
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7fd7aeb14748>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
TypeError: 'NoneType' object is not callable

ruedihofer (ruedihofer) wrote :

I confirm the same behaviour and error output as ikp1 reports! This with Ubuntu 16.10:

:~$ uname -a
Linux rudix 4.8.0-41-generic #44-Ubuntu SMP Fri Mar 3 15:27:17 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
:~$ lsb_release -rd
Description: Ubuntu 16.10
Release: 16.10
:~$ sudo /usr/sbin/update-apt-xapian-index
The index /var/lib/apt-xapian-index is up to date
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f3f25b46c88>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
AttributeError: 'NoneType' object has no attribute 'unlink'
hofer@rudix:~$

David Grayston (davidtg) wrote :

I'm also still getting this error.

:~$ uname -a
Linux request 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
:~$ lsb_release -d
Description: Ubuntu 16.04.2 LTS
:~$ apt-cache policy apt-xapian-index
apt-xapian-index:
  Installed: 0.47ubuntu8.3
  Candidate: 0.47ubuntu8.3
  Version table:
 *** 0.47ubuntu8.3 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
        100 /var/lib/dpkg/status
     0.47ubuntu8 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

Not sure if its meaningful but apt-xapian-index script is old and the server has been updated through several Ubuntu LTS versions

/etc/cron.weekly:
drwxr-xr-x 2 root root 4096 Apr 21 09:15 .
drwxr-xr-x 122 root root 12288 Jun 9 15:29 ..
-rwxr-xr-x 1 root root 730 Sep 24 2011 apt-xapian-index

demon (d3m0nkingx) wrote :

I confirm here as well the Bug still occurs on Ubuntu 16.04 64-bit

System Package Version:

apt-xapian-index:
  Installed: 0.47ubuntu8.3
  Candidate: 0.47ubuntu8.3
  Version table:
 *** 0.47ubuntu8.3 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
        100 /var/lib/dpkg/status
     0.47ubuntu8 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

Running Command:

$ sudo /usr/sbin/update-apt-xapian-index
The index /var/lib/apt-xapian-index is up to date
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f82bd7c5898>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
AttributeError: 'NoneType' object has no attribute 'unlink'

AlainKnaff (kubuntu-misc) wrote :

There are 2 more sys.exit(0) calls in the code, and they need their del indexer as well. Indeed, we keep getting the error message despite the presence of del indexer before the sys.exit(0) at the end of the file. Adding del indexer to the other 2 fixes it.

... and possibly the two sys.exit's with non-zero exit-status need it as well.

Alex (normadize) wrote :

I can confirm that I have 0.47ubuntu8.3 installed and I'm *STILL* getting this ... definitely not fixed

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers