reading database slow since upgrading to karmic

[Micah Gersten]

Binary package hint: dpkg

Ever since I upgraded to karmic, reading database when installing takes 10-20 secs where it was almost instantaneous before. I have over 300k files under apt package control.

ProblemType: Bug
Architecture: amd64
Date: Mon Oct 19 20:48:57 2009
DistroRelease: Ubuntu 9.10
Package: dpkg 1.15.4ubuntu2
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-14.48-generic
SourcePackage: dpkg
Uname: Linux 2.6.31-14-generic x86_64

[Daniel Kończyk (drmartens)]

Same with a clean install. I installed beta when it came out and "Reading database..." was painfully slow since the very beginning. I've never seen something like this in any previous ubuntu incarnation.

[Ian Corne (icorne)]

I have the same problem.

(Reading database ... 754388 files and directories currently installed.)

[Eloy Paris (peloy-chapus)]

Opening a large mailbox (in Maildir format) using mutt (mutt -f =mailboxname) seems a lot slower in Karmic than what it was in Jaunty. The versions of dpkg in Debian and in Karmic are not the same, but are very close (1.15.4.1 versus 1.15.4ubuntu1). I do not see a slowness problem with dpkg in Debian.

I am using disk encryption. The kernel went from 2.6.28 in Jaunty to 2.6.31, so my first thought is that the problem could be there, and not in dpkg.

Are you guys that are experiencing this problem using disk encryption?

I've never hard a problem with disk encryption in Ubuntu but since the slowness when doing package updates is so noticeable I am trying to think about all possibilities...

Cheers,

Eloy Paris.-

[Stepan Roucka (rouckas)]

The solution mentioned in bug #398870 worked for me:

Arnaud Jeansen wrote on 2009-08-17:
I remembered seeing a blog entry about it some time ago...

http://antti-juhani.kaijanaho.fi/newblog/archives/521

I did the two steps mentioned there and it really made a difference for me:
$ sudo dpkg --clear-avail
$ sudo dpkg --forget-old-unavail

this is probably duplicate

[bJXjLjEHIaWT0tFd (bjxjljehiawt0tfd-deactivatedaccount)]

This is *not* a duplicate of #398870.

I have never experienced this issue with *any* *previous* *version* of Ubuntu yet it occurred immediately after updating to karmic. Thus I must assume that it is caused by something that changed between jaunty and karmic. Please note that #398870 has been reported for jaunty which is why don't agree with the duplicate status. Please remove it, Stepan.

dpkg --clear-avail had no noticable effect. --forget-old-unavail is obsolete since karmic.

Eloy might be on to something though. I myself am using disk encryption as well. I've got a logical LVM volume that is LUKS encrypted. This is unencrypted on boot and fed back into LVM as a PV that hosts my ext4 partitions. The issue we are seeing might very well just be a symptom of encryption related disk IO trouble with karmic.

Could you find any issues better describing that possible root cause, Eloy?

[Daniel Kończyk (drmartens)]

I for one do not use any encryption on my root partition, so I don't think it's encryption related issue...

[Zillode (zillode)]

I can confirm the 'reading database' is much slower after upgrading to Karmic. I've been using the same install since Feisty but it's the first time it's going slow.

The --clear-avail had no effect as described by Niels Ganser.
I am not using an encrypted filesystem (except for my /media/backup/).

* 'Not a duplicate' +1

[Ergün KOÇAK (ergun)]

i have the same problem since upgraded to karmic from jaunty

[Anonym25712 (anonym25712)]

On my system also, reading database takes much more time than before (it was actually almost instantaneous previously). This happens since I did a *clean install* of Karmic, it's not an upgrade.

[Barış (penguen)]

I started having this problem after upgrading to Karmic on all of my three systems. (one is 64-bit server)

"sudo dpkg --clear-avail" seems to do some trick, but after some time the duration of "reading database" increases back to the problematic level. I tried this on all systems and experienced the same outcome.

[Jonne (jonathan-vanherpe)]

I've been experiencing this too, on 3 different machines (it's especially painfully slow on my netbook). No encryption, and filesystem is ext3 on 2 boxes and ext4 on the netbook.

[Kreuger Burns (car-crazy33)]

Add me to the list of people with this problem. I have no encryption and the clear does nothing to fix it either. Fresh install using ext4

[arnuschky (abrutschy)]

same here on karmic x86

[arnuschky (abrutschy)]

aw, sorry, wasn't finished yet.

Upgraded from Jaunty. Single ext4 root fs. "dpkg --clear-avail" didn't show any effect either.

[Kenny Meyer (knny-myer)]

Same problem here, since installing Ubuntu Karmic. No necessity to detail my problem, because it fits exactly into the description of the bug thread opener.

[Donatas Olsevičius (donatas-o)]

I have the same issue with Lucid Lynx.

[Istvan Toth (tothi)]

I also have this issue on Karmic (amd64) after an upgrade. Before Karmic I hadn't. dpkg --clear-avail doesn't help.

[bJXjLjEHIaWT0tFd (bjxjljehiawt0tfd-deactivatedaccount)]

As this bug has already been widely confirmed, there's no need to further comment on it for that purpose. Simply mark it as affecting you (right under the bug title). Thanks!

[Istvan Toth (tothi)]

running update-apt-xapian-index seems to solve the problem...

[Istvan Toth (tothi)]

sorry, #20 is no solution...

[nzadLithium (cmaster2)]

I'm not sure if its the same for all of you, but on my system it takes a lot longer as everyones said on the first run of a command that causes the database to be read, after that until I restart, I can then run any other commands seemingly as fast as i could before (I don't notice any difference).

[Everett Guerny (everett)]

Painfully slow in Karmic with ext4 on /. Never had this problem, going way back to breezy, but I've also never used ext4 before this install.

[Berk Demirkır (perq87-deactivatedaccount)]

Following method resolved my problem: http://ubuntuforums.org/showthread.php?t=1004376

[David Coconut (coconut)]

The method posted by Berk Demirkir also fixes the problem for me. I followed the instructions exactly as described - no problems.

[Michał Gołębiowski-Owczarek (mgol)]

> The method posted by Berk Demirkir also fixes the problem for me. I
> followed the instructions exactly as described - no problems.
And I broke my dpkg database this way and had to reinstall Ubuntu...
I'd be careful with these solutions, they can seriously mess up one's
files.

--
Regards,
Michał Gołębiowski

[Matthew Walster (dotwaffle)]

Just as an aside, I also get this bug, but it looks like disk activity in general is slow:

 time dd if=/dev/zero of=test.zero bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 10.1394 s, 10.1 MB/s
dd if=/dev/zero of=test.zero bs=1024 count=100000 0.04s user 0.68s system 6% cpu 10.350 total

This server has two dual-core processors, 4GB ECC RAM and RAID-1 disks. On a 1.2GHz Celeron with 1GB RAM I get 22.5MB/s, and over 60MB/s at home on a dual-core RAID-5 unit.

Reading appears to be fine - just writing is the issue. Same circumstances as others - fast on intrepid, upgrade seems t slow down system. iotop show no higher than about 10MB/s, reading apt db takes 5 seconds or more.

[David Coconut (coconut)]

> And I broke my dpkg database this way and had to reinstall Ubuntu...
> I'd be careful with these solutions, they can seriously mess up one's
> files.

I agree that it is very risky, and a backup should be performed beforehand.

Sorry to hear that it didn't work for you...

[Roy Tam (roytam)]

There is a "safer" version in that thread:
http://ubuntuforums.org/showpost.php?p=8995096

[thsths (thomas-steffen+ubuntu)]

@Matthew:

Yes, I think something more general is wrong. I noticed that a number of the larger files used by aptitude, apt and dpkg have extreme fragmentation on my disk (hundreds of fragments). And I have no idea why that is - do they grow by a little bit every time, for example? Or are concurrent processing writing files the problem?

I have Dropbox installed, and use KDE, but that is the only thing that all my systems really have in common.

[Raphaël Hertzog (hertzog)]

This has been fixed in dpkg 1.15.6 and later.

  * Use FIEMAP when available (on Linux based systems) to sort the .list
    files loading order. With a cold cache it improves up to a 70%.
    Thanks to Morten Hustveit <email address hidden>.
  * When FIEMAP is not available use posix_fadvise() to start preloading the
    .list files before loading them. With a cold cache it improves up to 40%.
    Thanks to Stefan Fritsch <email address hidden>. Closes: #557560

[Angel Guzman Maeso (shakaran)]

This bug should be reopened (I don't have permissions) since upgrading to raring has the same issue. I even test with new saucy and it get a poor perfomance. The process "reading database list" is running more than 5 minutes for me in a quad-core i7 laptop with 20 GB RAM. I think that it is not a problem with low hardware specs, it is a software problem. Also It probably needs a pararell implementation version of dpkg for allow to use all the cores and optimize the perfomance.

In my opinion this should be high or critical priority since install packages is a common operation and each release we have more packages and programs installed or available.

Related:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=69192
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/398870