python-apt can SIGSEGV when encountering Packages stanzas with no Description field (was: update-apt-xapian-index crashed with SIGSEGV in File())

[aborland]

Errors Bucket:
http://errors.ubuntu.com/bug/1220013

There are 600+ reports of this on errors.ubuntu.com.

Something wrong

ProblemType: CrashDistroRelease: Ubuntu 13.10
Package: apt-xapian-index 0.45ubuntu2
ProcVersionSignature: Ubuntu 3.11.0-4.9-generic 3.11.0-rc7
Uname: Linux 3.11.0-4-generic i686
ApportVersion: 2.12.1-0ubuntu3
Architecture: i386
CrashCounter: 1
Date: Tue Sep 3 01:23:23 2013
ExecutablePath: /usr/sbin/update-apt-xapian-index
InterpreterPath: /usr/bin/python2.7
MarkForUpload: True
PackageArchitecture: all
ProcCmdline: /usr/bin/python /usr/sbin/update-apt-xapian-index --quiet
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 SHELL=/bin/sh
SegvAnalysis:
 Segfault happened at: 0xb6c9e020: imul $0x3c,(%eax),%ebp
 PC (0xb6c9e020) ok
 source "$0x3c" ok
 destination "(%eax)" (0xd7ac1000) not located in a known VMA region (needed writable region)!
SegvReason: writing unknown VMA
Signal: 11SourcePackage: apt-xapian-index
StacktraceTop:
 ?? () from /usr/lib/python2.7/dist-packages/apt_pkg.so
 ?? ()
 PyObject_GetAttr ()
 PyEval_EvalFrameEx ()
 ?? ()
Title: update-apt-xapian-index crashed with SIGSEGV in PyObject_GetAttr()
UpgradeStatus: Upgraded to saucy on 2013-08-16 (17 days ago)
UserGroups:

[Apport retracing service (apport)]

StacktraceTop:
 File (this=0xbff71af4, this=0xbff71af4) at /usr/include/apt-pkg/cacheiterators.h:415
 DescriptionGetFileList (Self=0x1834dd24) at python/cache.cc:851
 getset_get.11201 (descr=0x9eaf86c, obj=0x1834dd24, type=0xb6cc92e0 <PyDescription_Type>) at ../Objects/descrobject.c:146
 _PyObject_GenericGetAttrWithDict (dict=<optimized out>, name=0x9eaf3c0, obj=<optimized out>) at ../Objects/object.c:1439
 PyObject_GenericGetAttr (name=<optimized out>, obj=<optimized out>) at ../Objects/object.c:1461

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nazar Mokrynskyi (nazar-pc)]

Occures after log in.
Both on Ubuntu 13.10 and 14.04.

Now I have Intel Core i7-4900MQ CPU, but as I remember it was even on AMD FX-8120

[Iain Lane (laney)]

I'm not sure if this is a bug in apt, python-apt or software-center. Here's my brain dump

ddebs.ubuntu.com currently has a package with no Description field

Package: php5-gearman-dbgsym
Priority: extra
Section: php
Installed-Size: 243
Maintainer: Debian PHP PECL Maintainers <email address hidden>
Architecture: amd64
Source: php-gearman
Version: 1.1.2-1build1
Depends: php5-gearman (= 1.1.2-1build1)
Filename: pool/universe/p/php-gearman/php5-gearman-dbgsym_1.1.2-1build1_amd64.ddeb
Size: 50610
MD5sum: 0c9b4932b6a1de927c2f4fb7e388e5d5
SHA1: 60697f6457e71e5197f2a5109d62b7e100616684
SHA256: 8f7f500dd44fa58bb0af6ac573bb22b09b55d536baf28bb20a8aabd373cbdd65
SHA512: 3809e68065c7b8e28c35143411aec449585d443a8a3101839e27671ad1129a33fe69679c9278d6ecea927e6b2b03db216a3812467306fac08826173c81ec3d32
Homepage: http://pecl.php.net/package/gearman

I used python-faulthandler to print out a python stack trace when the segfault happens:

Current thread 0x00007fdc45a3c740:
  File "/usr/lib/python2.7/dist-packages/apt/package.py", line 273 in _translated_records
  File "/usr/lib/python2.7/dist-packages/apt/package.py", line 331 in description
  File "<stdin>", line 1 in <module>
  File "/usr/lib/python2.7/pdb.py", line 234 in default
  File "/usr/lib/python2.7/cmd.py", line 220 in onecmd
  File "/usr/lib/python2.7/pdb.py", line 279 in onecmd
  File "/usr/lib/python2.7/cmd.py", line 142 in cmdloop
  File "/usr/lib/python2.7/pdb.py", line 210 in interaction
  File "/usr/lib/python2.7/pdb.py", line 158 in user_line
  File "/usr/lib/python2.7/bdb.py", line 67 in dispatch_line
  File "/usr/lib/python2.7/bdb.py", line 49 in trace_dispatch
  File "/usr/share/apt-xapian-index/plugins/display_name.py", line 81 in index
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 540 in get_document_from_apt
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 577 in gen_documents_apt
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 719 in buildIndex
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 744 in rebuild
  File "/usr/sbin/update-apt-xapian-index", line 111 in <module>
Segmentation fault (core dumped)

display_name.py is a plugin from software-center which is doing this:

  def index(self, document, pkg):
        …
        ver = pkg.candidate

        if ver is None:
            return
        if self.val_display_name != -1:
            name = ver.summary

The `ver.summary' line is the one that triggers the segfault by eventually calling into the crashing code in python-apt.

python/cache.cc crashes in this line in DescriptionGetFileList() at the call to File():

        DescFile = CppPyObject_NEW<pkgCache::PkgFileIterator>(Owner,&PyPackageFile_Type,I.File());

So I'm not sure if the problem is in python-apt here or if there's a bad iterator coming from libapt.

[Iain Lane (laney)]

In a python REPL, I didn't manage to get a Package object which has a candidate() method, for some reason.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Barry Warsaw (barry)]

And of course now the Packages file has been fixed to include a Description, at least for that package. But this is still worth further investigation. Trying to get a reproducible test case eliminating axi. I suspect it's a problem in python-apt.

[Julian Andres Klode (juliank)]

The APT command-line code does the same loop python-apt does. If it fails in python-apt, APT fails as well. So in that case, the best idea might be to change APT by overriding end() to do the checks isGood() does, so all clients work correctly. This would require a rebuild of all reverse dependencies though, and is formally an ABI break.

[Nazar Mokrynskyi (nazar-pc)]

Can' t use synaptic at all because of this bug. Please, fix it ASAP.
Someone did broke it. I'm on 14.10 already, still suffering from it.

[Julian Andres Klode (juliank)]

This is fixed.

[Thomas Ward (teward)]

Hello.

It is still unclear whether this is still an issue in apt-xapian-index or the underlying python apt bindings. I am marking this as Incomplete for apt-xapian-index as this is still marked as affecting it and has been since 2015, and the Trusty that was affected is End of Life. Please verify if this remains an issue or not.