apt-get crashed with SIGSEGV in pkgCacheGenerator::ListParser::NewProvides()

[Timo Jyrinki]

Starting today, most apt operations started crashing to the extent that I can't even run ubuntu-bug apt. rm -f /var/lib/apt/* does not help either, or switching from a mirror to main archive. The crash happens in apt-get update when reading package lists after they have been downloaded.

TEST CASE:
1. Press shift on boot to display the boot menu
2. Press F6 and select the option 'Free software only'
3. Continue with a default installation
4. At the end of the installation reboot
5. Login and run 'apt-get update'

ALTERNATIVE TEST CASE:
1. Copy the file sources.list attached to this report (https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1066445/+attachment/3398341/+files/sources.list) to /etc/apt/ on an existing system (backup the exiting file first!)
2. Run 'apt-get update'

ACTUAL RESULT
apt will crash with the trace below

StacktraceTop:
 pkgCacheGenerator::ListParser::NewProvides (this=0x7ffffeba7910, Ver=..., PkgName=..., PkgArch=..., Version=...) at pkgcachegen.cc:920
 debListParser::NewProvidesAllArch (this=this@entry=0x7ffffeba7910, Ver=..., Package=..., Version=...) at deb/deblistparser.cc:722
 debListParser::ParseProvides (this=this@entry=0x7ffffeba7910, Ver=...) at deb/deblistparser.cc:711
 debListParser::NewVersion (this=0x7ffffeba7910, Ver=...) at deb/deblistparser.cc:171
 pkgCacheGenerator::MergeListVersion (this=this@entry=0x7ffffeba84a0, List=..., Pkg=..., Version=..., OutVer=@0x7ffffeba75e8: 0x0) at pkgcachegen.cc:388

===

gdb apt-get upgrade gives (and update crash at package list reading looks identical):

#0 0x00007ffff7b4dd20 in pkgCacheGenerator::ListParser::NewProvides(pkgCache::VerIterator&, std::string const&, std::string const&, std::string const&) ()
   from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#1 0x00007ffff7b864d1 in debListParser::NewProvidesAllArch(pkgCache::VerIterator&, std::string const&, std::string const&) ()
   from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#2 0x00007ffff7b883e3 in debListParser::ParseProvides(pkgCache::VerIterator&)
    () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#3 0x00007ffff7b8902c in debListParser::NewVersion(pkgCache::VerIterator&) ()
   from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#4 0x00007ffff7b4c458 in pkgCacheGenerator::MergeListVersion(pkgCacheGenerator::ListParser&, pkgCache::PkgIterator&, std::string const&, pkgCache::VerIterator*&) () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#5 0x00007ffff7b4d6a3 in pkgCacheGenerator::MergeList(pkgCacheGenerator::ListParser&, pkgCache::VerIterator*) ()
   from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#6 0x00007ffff7ba173a in debPackagesIndex::Merge(pkgCacheGenerator&, OpProgress*) const () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#7 0x00007ffff7b46632 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#8 0x00007ffff7b49409 in pkgCacheGenerator::MakeStatusCache(pkgSourceList&, OpProgress*, MMap**, bool) () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#9 0x00007ffff7b400c1 in pkgCacheFile::BuildCaches(OpProgress*, bool) ()
   from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#10 0x00007ffff7b4045c in pkgCacheFile::Open(OpProgress*, bool) ()
   from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#11 0x000000000042113c in ?? ()
#12 0x00000000004160a3 in ?? ()
#13 0x00007ffff7afa562 in CommandLine::DispatchArg(CommandLine::Dispatch*, bool) () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
#14 0x000000000040a2ac in ?? ()
#15 0x00007ffff720276d in __libc_start_main (main=0x40a070, argc=2,
    ubp_av=0x7fffffffe328, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffe318) at libc-start.c:226
#16 0x000000000040a591 in ?? ()
#17 0x00007fffffffe318 in ?? ()
#18 0x000000000000001c in ?? ()
#19 0x0000000000000002 in ?? ()
#20 0x00007fffffffe60c in ?? ()
#21 0x00007fffffffe61d in ?? ()
#22 0x0000000000000000 in ?? ()

[Harry (harry33)]

I might be seeing this too.
Synaptic will crash and won't even start any longer.
In terminal when I run
   "sudo apt-get update"
It stops reading data when 45 % is done and will not go further.

Interesting part is that if I remove the repository universe from /etc/apt/sources.list, all is well again and also synaptic runs OK.

Is there some temporary issue with universe repo now?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Timo Jyrinki (timo-jyrinki)]

I can confirm that commenting out universe repo (binaries) workarounds the problem, and adding it back causes the crash again.

[Gergely Nagy (algernon)]

I'm seeing the same too, but this only seems happens on amd64, my i386 chroot is not affected.

[Jean-Baptiste Lallement (jibel)]

I got the same trace after a fresh installation of Quantal Desktop amd64 20121014

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1066445

[Jean-Baptiste Lallement (jibel)]

Stack trace

[Fabio Marconi (fabiomarconi)]

Hallo
Just to notice that I cannot reproduce this in a fully updated Quantal (upgraded from Precise yesterday).
Everythings work in software updarter or apt-get in terminal (nothing to upgrade)

[LaMont Jones (lamont)]

With _either_ universe or main as the only component in sources.list* (others commented out), apt behaves fine.

having both main and universe there (quantal Release file with Date: Sun, 14 Oct 2012 15:20:26 UTC) leads apt-get update to die with:
0x00007ffff7b4dd20 in pkgCacheGenerator::ListParser::NewProvides(pkgCache::VerIterator&, std::string const&, std::string const&, std::string const&) () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12

[Kate Stewart (kate.stewart)]

@lamont what was the environment you were starting from?

[LaMont Jones (lamont)]

< skaet> lamont, jibel saw the failure on hardware and VM, fresh installation, all amd64. hggdh hasn't been able to see it though. what are the specifics of the system you saw 1066445 on?
< lamont> skaet: I'd have to dig a bit, but I'm pretty sure this was an install from this year, call it precise, amd64, real hw. do-release-upgraded to quantal at beta(1?), and dist-upgraded pretty much daily since then. has quantal{,-{security,updates}} {main,universe,multiverse} (no restricted), as well as a ppa or 2 and a local archive
< lamont> the mirror I hit is synced from us.archive at :50 each hour
< skaet> thanks lamont, can you add that to the bug, so we see if we can get the pattern figured out?

In my debugging, I commented out all the deb lines in all of sources.list and sources.list.d/* and got it down to failure with quantal main and universe, but not either one separately.

here are more package versions than I think you want to know.

# dpkg -l | grep -e apt -e dpkg
ii apt 0.9.7.5ubuntu3 amd64 commandline package manager
ii apt-doc 0.9.7.5ubuntu3 all documentation for APT
ii apt-listchanges 2.85.8ubuntu2 all package change history notification tool
ii apt-transport-https 0.9.7.5ubuntu3 amd64 https download transport for APT
ii apt-utils 0.9.7.5ubuntu3 amd64 package managment related utility programs
ii apt-xapian-index 0.44ubuntu7 all maintenance and search tools for a Xapian index of Debian packages
ii aptdaemon 0.45+bzr861-0ubuntu9 all transaction based package management service
ii aptdaemon-data 0.45+bzr861-0ubuntu9 all data files for clients
ii aptitude 0.6.8.1-2ubuntu1 amd64 terminal-based package manager
ii aptitude-common 0.6.8.1-2ubuntu1 all architecture indepedent files for the aptitude package manager
ii apturl-common 0.5.1ubuntu6 amd64 install packages using the apt protocol - common data
ii dpkg 1.16.7ubuntu6 amd64 Debian package management system
ii dpkg-dev 1.16.7ubuntu6 all Debian package development tools
ii dpkg-repack 1.37 all puts an unpacked .deb file back together
...
ii libapt-inst1.3 0.8.16~exp5ubuntu14.2 amd64 APT's deb package format runtime library
rc libapt-inst1.4:amd64 ...

[Harry (harry33)]

I can confirm what LaMont Jones just wrote here.
When both quantal main and universe are in the same deb line, the crash follows.

[C de-Avillez (hggdh2)]

(1) on my quantal (working laptop) I do not get a failure. Main, Universe, Multiverse, Restricted, Security, Partners (-updates and -proposed) enabled.
(2) on a fresh install of Quantal I do get it Main, Universe (-updates enabled).

[C de-Avillez (hggdh2)]

Oh, I forgot to add this: on the new install, I *only* got it after I disabled the default enabled repositories to just main and universe.

[Steve Langasek (vorlon)]

Can someone who sees this bug please attach /var/cache/apt/pkgcache.bin to this bug?

[Steve Langasek (vorlon)]

Also, if you move /var/cache/apt/pkgcache.bin aside and run 'apt-get update', does the problem persist?

[Jean-Baptiste Lallement (jibel)]

/etc/apt/sources.list from a system affected.

[Jean-Baptiste Lallement (jibel)]

Content of /var/cache/apt/. Note that there is no pkgcache.bin but 2 files named pkgcache.bin.XXXXXX

[Ryan Finnie (fo0bar)]

Also confirmed with smoser's beta2 (amd64) AMI on Canonistack.

smoser-cloud-images/ubuntu-quantal-12.10-beta2-amd64-server-20120925
ba3de67e-dfeb-4a18-b4b3-e2b08decc87d (ami-00000155)

[Kate Stewart (kate.stewart)]

TO DO: add to release notes, instructions for manually installing apt as a work around if this bug is encountered.

<cjwatson> Ah - well, it'll take me a bit longer to actually fix it, but I've tracked it down to excruciatingly unlucky timing of cache remapping in a function unequipped to handle it
<cjwatson> Unfortunately there may be a hard-to-resolve upgrade problem with old versions of apt; don't know what we can do about that short of (a) randomly permute size of cache by continuing normal development (b) SRU whatever fix we come up with
<cjwatson> And possibly (c) release-note instructions for manually installing new apt

[Colin Watson (cjwatson)]

A temporary workaround is:

  echo 'APT::Cache-Start 50331648;' >/etc/apt/apt.conf.d/99cache-start

This doubles the initial size of APT's cache in memory, thereby happening to avoid this remapping problem. This will probably cause APT to use more memory, so you should remove /etc/apt/apt.conf.d/99cache-start after upgrading to version 0.9.7.5ubuntu4 or newer.

[Colin Watson (cjwatson)]

Seems to have been introduced in, er, I think somewhere between lucid and natty but I would need to be rather more awake to trace the full history. It's probably not worth SRUing this back very far, but it would be helpful to at least put the fix in precise.

[Harry (harry33)]

The new apt (0.9.7.5ubuntu4) did fix this issue for me immediately.

I did not use the workaround suggested.
Instead, I removed the "universe" from /etc/apt/sources.list first,
then updated apt and finally reintroduced the "universe" back to the sources.list.

[Timo Jyrinki (timo-jyrinki)]

Also confirming that 0.9.7.5ubuntu4 from -proposed without workarounds fixed the issue for me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 0.9.7.5ubuntu4

---------------
apt (0.9.7.5ubuntu4) quantal-proposed; urgency=low

  [ Colin Watson ]
  * apt-pkg/pkgcachegen.cc:
    - Fix crash if the cache is remapped while writing a Provides version
      (LP: #1066445).

  Cherry-pick from http://bzr.debian.org/bzr/apt/apt/debian-sid:

  [ David Kalnischkies ]
  * apt-pkg/pkgcachegen.cc:
    - add a missing remap registration causing a segfault in case
      we use the not remapped iterators after a move of the mmap again
 -- Colin Watson <email address hidden> Sun, 14 Oct 2012 23:54:27 +0100

[Adam Conrad (adconrad)]

Hello Timo, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.8.16~exp12ubuntu10.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

I've dropped the release note for this from the wiki page; the fix was uploaded to quantal, so this doesn't seem to be a noteworthy issue now. (It's still possible that users in precise will hit this bug, but I don't see any reason to think they're more likely to hit it when upgrading to quantal than at any other time.)

[Brian Murray (brian-murray)]

I've tried multiple different cache sizes to recreate this bug and have been unsuccessful. Instead I performed a regression test by installing apt version 0.8.16~exp12ubuntu10.5 from precise-proposed and then installing 143 packages from -updates. This upgrade completed successfully so I think this bug can be considered verification-done.

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 0.8.16~exp12ubuntu10.5

---------------
apt (0.8.16~exp12ubuntu10.5) precise-proposed; urgency=low

  * apt-pkg/pkgcachegen.cc:
    - Fix crash if the cache is remapped while writing a Provides version
      (LP: #1066445).

apt (0.8.16~exp12ubuntu10.4) precise-proposed; urgency=low

  [ David Kalnischkies ]
  * apt-pkg/packagemanager.cc:
    - do not run into loop on new-pre-depends-breaks (Closes: #673536)
      LP: #1050791
  * apt-pkg/cachefile.cc:
    - clean up lost atomic cachefiles with 'clean' (Closes: #650513)
      LP: #1050779

  [ TJ ]
  * apt-pkg/contrib/netrc.cc:
    - increase LOGINSIZE/PASSWORDSIZE limits and add proper error
      if the limits are reached (LP: #1065429)

  [ Michael Vogt ]
  * lp:~mvo/apt/lp346386-precise:
    - fail gracefully when a InRelease file is not valid, e.g. behind
      paywalls (LP: #346386)
 -- Colin Watson <email address hidden> Mon, 15 Oct 2012 05:42:45 +0100