"Reading Database" takes too long

[polvoazul]

Binary package hint: apt

I know that the step Reading Database depends on the number of packages i have installed. But now it is taking so long that it is unbereable. For me to install a small program, it takes more time reading database than anything else.

I have two sugestions to speed up that process, i dont know if they can help, but here they are:

* Cache the database information in some kind of file. Just like the 'updatedb' and 'locate'.

* multithread the aplication to read the database from the start of the process, so that it downloads and reads at the same time. clearly these two things are independent and this would probably save some time.

And please can someone inform me why is it so slow? does it check your hardrive on every single package?
How many packages a clean ubuntu has? mine has 229452 files and directories currently installed. (i have ubuntu-studio-audio)

thank you very much.

Using apt 0.7.20.2ubuntu6 for i386 compiled on Apr 17 2009 04:25:29
Ubuntu 9.04 Jaunty

[Atanas Atanasov (thenasko)]

I recently notices a significant slow down in the "reading database" operation after updating to Jaunty. This is most probably a similar issue, though not necessarily identical.

[Steven (stebalien)]

This problem is also discused in bug report #69192 on bugs.debian.org (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=69192).

[polvoazul (fredpublico)]

This bug report http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=69192 is 9 years old, is it still not taken care of?

[Arnaud Jeansen (ajeans)]

I remembered seeing a blog entry about it some time ago...

http://antti-juhani.kaijanaho.fi/newblog/archives/521

I did the two steps mentioned there and it really made a difference for me:
$ sudo dpkg --clear-avail
$ sudo dpkg --forget-old-unavail

Hope this helps.

[Eddie Dunn (eddie-dunn)]

Thanks, Arnaud, that really helped me too!

Every time I use apt I get annoyed when I remember how much faster pacman for Arch Linux is. I've used Arch, but don't want to go back there -- can't something be done to speed up apt?

[Sam Freilich (l33tminion)]

Is this just a problem for upgrades? If so, maybe it would be sufficient to just make the commands above part of the upgrade process.

[Daniel Kończyk (drmartens)]

You mean OS upgrades or package upgrades?
Still, it feels like a regression in Karmic, because I cannot recall anything like this since I've been running Feisty on this machine.

[Michał Gołębiowski-Owczarek (mgol)]

On my 4-year-old notebook on Hardy "Reading database" used to go fast. Now I have Core2 Duo P8600 (2,4 GHz) and on Karmic it goes slowly with steps 5%-sized, maybe 3 such steps per second. What takes so much time? It was faster in the past...

On the other side, I don't get why apt in Ubuntu still doesn't download deltas only - it's also killing Canonical servers so I think it should be a priority...

[Michał Gołębiowski-Owczarek (mgol)]

Moreover, I checked and this "reading database" goes fast even on virtual machine run on VirtualBox if only I run older Ubuntu! What has happened to apt?

[Michał Gołębiowski-Owczarek (mgol)]

This dpkg-related bug isn't actually what we're talking about. Sure, it causes apt to be slower than it could be but it still doesn't explain Karmic regression here.

[Anakin Starkiller (sunrider)]

I can confirm this issue on Karmic too....

[Philipp Weissenbacher (philipp-weissenbacher)]

On the Debian mailing-list this has been dicussed here http://<email address hidden>/msg245661.html.

One solution that helped me a bit (just one or two seconds faster) is the following (taken from ):
1. $ sudo su
1. % for x in /var/lib/apt/lists/*Packages;do dpkg --merge-avail "$x";done
2. Rebuild /info (to work against fragmentaton; for me this took +3 minutes!)
  % cd /var/lib/dpkg/
  % cp -a info info.tmp
  % mv info info.orig
  % mv info.tmp info
3. Finally: % dpkg --clear-avail

_and_ reboot (for whatever reason ...)

On the mailinglist above there was also a pointer to http://people.debian.org/~seanius/dpkg-sqlite/ for a proof-of-concept sqlite powered cache, which would fix the slowness problems.

[Anakin Starkiller (sunrider)]

Philipp Weissenbacher >> unfortunately, this workaround doesn't change anything for me...
Maybe it's because I was using aptitude full-upgrade ? I'll try with apt-get next time ;)

[Graham Poulter (grahampo)]

#12 (Phillip Weissenbacher's) solution worked for me! -

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/398870/comments/12

I don't know how much good the dpkg --merge-avail "$x" did, but making a copy of "/var/lib/dpkg/info" to defragment it did take a few minutes to complete, and afterwards the "reading database" was MUCH faster!

[Graham Poulter (grahampo)]

A moment ago, I did another copy of "/var/lib/dpkg/info"

This time it took 1 second to copy, whereas the first time took about 3 minutes.

So for me, I conclude that the cause of reading database taking 5+ seconds was fragmentation of /var/lib/dpkg/info

[Hans van den Bogert (hbogert)]

we should get a button to mark a comment as a viable solution.

Just like graham, the first copy took ages, but the reading now takes place within 2 seconds, where it used to be 30s.
Thanks! Philipp Weissenbacher, this is some HUGE performance gain, I always suspected fragmentation was an issue on my small 40GB partition.

[Eddie Dunn (eddie-dunn)]

It is a workaround, not a solution.

A solution would be if this were solved automatically so users wouldn't have to track down this bug report in order to have sane package database reading times.

[Graham Poulter (grahampo)]

Cron work-around: a weekly "cd /var/lib/dpkg && mv info info.bak && cp -a info.bak info && rm -R info.bak" to keep the directory contiguous.

ReiserFS work-around: create a ReiserFS partition especially to hold /var/lib/dpkg/info. ReiserFS was specifically designed to handle zillions of tiny files in one massive directory (under the philosophy that databases mostly because filesystems aren't up to the task). As /var/lib/dpkg/info demonstrates, ext3 is having having fragmentation trouble on huge database-like directories.

Short-term solution: Have the default Ubuntu installation defragment /var/lib/dpkg/info regularly

Long-term solution: Have dpkg use a database instead of /var/lib/dpkg/info. I deduce that the person who specified /var/lib/dpkg/info as a massive directory of tiny files was expecting ReiserFS to become the standard filesystem. Since that's not going to happen, it's about time to use a database.

[Graham Poulter (grahampo)]

Sorry, the "long-term solution" does not explain why "Reading database" used to be fast under older versions of Ubuntu. Can anyone explain why fragmentation came to be a problem for /var/lib/dpkg/info in recent versions of Ubuntu?

[Michał Gołębiowski-Owczarek (mgol)]

I don't know what exactly this "workaround" does, but it does NOT work for me at all. "Reading database" in Karmic still lasts ages as opposed to what used to be in Hardy.

[frt975 (frt975-deactivatedaccount)]

This helps: http://ubuntuforums.org/showthread.php?t=1004376

[Akshay (akshaykkulkarni)]

For me, this solved the problem:

apt-get clean

This basically deletes all the downloaded ".deb" files (which are no longer needed after they have been installed), which has the added advantage of freeing up some disk space.

[Akshay (akshaykkulkarni)]

edit to my comment above - apt-get clean doesn't actually solve this problem. I was seeing a speed-up simply due to the database being in the hard drive cache :(

[Philipp Weissenbacher (philipp-weissenbacher)]

In fact a database has already been implemented. See the following Brainstorm Idea: http://brainstorm.ubuntu.com/idea/24027/

If you want that feature, please vote for this idea!

[Luca Bruno (lucabru)]

I'd like to remember you all dpkg/experimental is a lot faster now... from about 14sec to 3sec here. Well using the database would be about 1sec but I wouldn't recommend to a distribution to use a wrapper (just like I don't recommend installing the library in your system, not because it's unstable but because a bad user usage could lead to unexpected results for the system). Patch dpkg directly instead.

[Anakin Starkiller (sunrider)]

This bug has been fixed in Lucid.

[Mike.lifeguard (mikelifeguard)]

Could we get a pointer to the commit which fixed this in apt? I assume this will go in lucid-proposed and such, yes?

[torzsmokus (torzsmokus)]

will there be a Jaunty backport?