update-apt-xapian-index crashed with SystemError in open(): E:Read error - read (5: Input/output error), E:The package lists or status file could not be parsed or opened.

[alexander_iax]

the software crashed during the update

ProblemType: Crash
DistroRelease: Ubuntu 12.04
Package: apt-xapian-index 0.44ubuntu5
Uname: Linux 3.3.0-030300-generic x86_64
ApportVersion: 1.94.1-0ubuntu2
Architecture: amd64
Date: Tue Mar 20 14:21:10 2012
ExecutablePath: /usr/sbin/update-apt-xapian-index
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Alpha amd64 (20120312)
InterpreterPath: /usr/bin/python2.7
PackageArchitecture: all
ProcCmdline: /usr/bin/python /usr/sbin/update-apt-xapian-index --quiet
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 SHELL=/bin/sh
PythonArgs: ['/usr/sbin/update-apt-xapian-index', '--quiet']
SourcePackage: apt-xapian-index
Title: update-apt-xapian-index crashed with SystemError in open(): E:Read error - read (5: Input/output error), E:The package lists or status file could not be parsed or opened.
UpgradeStatus: Upgraded to precise on 2012-03-12 (7 days ago)
UserGroups:

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paul F (boxjunk)]

Appears to be caused by corrupted files in /var/lib/apt/lists (see ls -l of affected files attached).

My theory is that these are corrupted when user switches between two different internet connections (2 routers) when the software updater is (automagically) using the connection to check for updates. In my case the corrupted files were filled with html source from (one of) the routers web interface pages. Weird.

Paul

[Paul F (boxjunk)]

Had a closer look at this. Scratch my theory (comment #3) about switching routers.

The corrupted package list files in /var/lib/apt/lists are caused by the router redirecting to an internal help page when it realises that its internet connection is down. So, when a fetch is attempted from, say gb.archive.ubuntu.com/ubuntu/dists/precise-updates/universe/binary-i386/Packages when checking for updates what comes back is the html source from the router's help page (example attached -- line 52 contains the requested url).

It would appear that no sanity check is done on the returned data leaving subsequent parse attempts to choke. The corrupted files remain and may propagate (???) causing other update failures.

On a security note, it occurs to me that an attacker in control of the router could return crafted files in place of apt's package lists to introduce malware as part of the normal automated update process. I trust checks are in place to prevent this???

NB My experience may or may not be the cause of the original bug poster's crash. As apt-xapian-index is not the originating culprit I will file a new bug report about corrupted apt package lists.

[Tomas (tomasvandenooijevaer)]

Paul F's second post (#4) makes perfect sense to me. Ran into this issue after I had some internet issues and I have a modem known to send html pages stating the internet connection is unavailable.

After this crash Ubuntu Software Centre refuses to start: pops up, appears to load for a second, then closes again. Workaround for this is to refresh the packages as decribed in the accepted answer here:
http://askubuntu.com/questions/30072/how-do-i-fix-a-problem-with-mergelist-error-when-trying-to-do-an-update

Remove the merge list:
sudo rm /var/lib/apt/lists/* -vf
Then update the package list:
sudo apt-get update

[Paul F (boxjunk)]

This problem dates back to 2009. Marking this as a duplicate of BUg #346386 which appears to be the original report.

[Paul (i41bktob-launchpad-net)]

At least in my case this doesn't look like bug #346386, just a race condition caused by running an aptitude update at the same time update-apt-xapian-index was running.