Comment 2 for bug 1530518

Almost positive what's happening is that Python shutdown process is interacting badly with ServerProgress's __del__().

__del__() methods are not the best way to reclaim resources, especially when you leave it to Python's shutdown process to implicitly free objects, like what happens in update-apt-xapian-index. Here's what happens:

When Python shutdown, after it's run the atexit handlers, it then tries to free up all objects that are still implicitly alive because of reference counts. It does this first by emptying the __dict__s of all modules, which will decref any still referenced globals. It actually does this first by setting all keys in the module's globals to None. Because Python can't know what's a safe order to do this, it will do it randomly.

Notice a few things. In update-a-x-i, indexer is a module global. When its .lock() method is called, it sets self.progress to a ServerProgress instance and neither instance is explicitly freed, so they live until Python shutdown. Now, when Python gets around to clearing the module dicts as described above, it will set u-a-x-i's indexer instance to None, which will decref it to zero, and so freeing the indexer instance.

Because the ServerProgress instance is kept alive by the reference from the indexer's self.progress attribute, once the indexer is freed, the ServerProgress instance probably decrefs to zero and then *it* gets freed. This will call ServerProgress.__del__(). Line 306 is the os.unlink() call and you'll notice that you get an exception that None is not callable. Why is that?

Well, because if the os module's dictionary got cleared *before* the ServerProgress instance's __del__() got called, the os module is still valid, and it still has a name bound in it called 'unlink' but that name has been bound to None! If you're lucky, the ServerProgress instance gets decrefed away before os gets cleared, but as you've observed in IRC, about every third time, you're unlucky, and os gets cleared first (remember, the order is random). In those cases... boom!

I don't like the design here, since relying on __del__ methods to free resources isn't good Python style, especially when those resources can be freed explicitly, but fixing that is probably more work than we want to do for a package that is abandoned in Debian. A bandaid might be easy though: before the sys.exit() call in u-a-x-i, let's try to explicitly `del indexer`. That will cause all the cascading decrefs before Python's shutdown machinery clears out the os module. I'll attach a diff to try.