My computer updates are very slow since latest dpkg update

[Nicolas DERIVE]

Binary package hint: dpkg

Since latest dpkg updates with cache optimisations, dpkg seems very slow (and update of 198 packages from 2 days of lucid updates takes more that 15 minutes... and counting...)

ProblemType: Bug
Architecture: i386
CheckboxSubmission: c4273c0f4f2b8d26cd6bf31cadfd2912
CheckboxSystem: a871981cb5bdf4d6ebd55be46becf77e
Date: Thu Mar 11 11:19:12 2010
DistroRelease: Ubuntu 10.04
EcryptfsInUse: Yes
NonfreeKernelModules: nvidia
Package: dpkg 1.15.5.6ubuntu2
ProcEnviron:
 LANGUAGE=fr_FR:fr:en_GB:en
 LANG=fr_FR.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.32-16.24-generic
SourcePackage: dpkg
Uname: Linux 2.6.32-16-generic i686

[Nicolas DERIVE (kalon33)]

Take in account that these times are given on a intel core2quad @ 2,66 Ghz...

[Jean-Baptiste Lallement (jibel)]

Thanks.

That was expected because few fsync have been added to prevent data loss when the system crashes during an upgrade (bug 512096)
Are you able to compare the performances of the latest version of dpkg with the previous version?

[Nicolas DERIVE (kalon33)]

not really, unless you give me a link to the previous version package and a list of packages which could make a good benchmark. In that last case I would be happy to help testing and comparing.

[Nicolas DERIVE (kalon33)]

I suppose in that case that trying reinstalling packages could make the benchmark, and that it won't matter reinstall a package which the previous dpkg package.

[Nicolas DERIVE (kalon33)]

I got the dpkg packages (thanks archive.u.com not delete previous versions too fast !) but if you want a reliable benchmarking test I need a list of packages to reinstall (maybe a kernel, kernel headers and packages with triggers ? Seems to me they were the slowest ones). I'm waiting for your advice on this. I will have quite some time tomorrow to deal with this if you want.

[Jean-Baptiste Lallement (jibel)]

I use to play with the texlive package. They are non critical, and some of them are shipping a large number of files.
texlive-latex-extra-doc (+8k files)
texlive-generic-recommended (~ 300 files and install-info and tex-common triggers )
hello or bsdgames are rather small but good for testing purpose.

[Nicolas DERIVE (kalon33)]

For all these packages, install time (without downloading) is 16sec with old and 192sec with new dpkg. (will soon add files with start and end time, and dpkg actions with these packages (using apt))

[Nicolas DERIVE (kalon33)]

for installing the -pae kernel with its headers, 23sec with old and 420sec with new dpkg... it's a dramatic slow down.

[Nicolas DERIVE (kalon33)]

Many files, but you have the transparency !

My "script" was :

date -R > txtnewdpkg2.txt && apt-get -y -q install linux-image-generic-pae linux-headers-generic-pae &> actions_apt_dpkg_new2.txt && date -R >> txtnewdpkg2.txt

or

date -R > txtnewdpkg.txt && apt-get -y -q install texlive-latex-extra-doc texlive-generic-recommended hello bsdgames &> actions_apt_dpkg_new.txt && date -R >> txtnewdpkg.txt

for both dpkg versions. I renamed the files in order to have a more understandable title.

[Rogério Theodoro de Brito (rbrito)]

Hi.

I am also seeing the bug that Nicolas just reported. Since my computer is slower, the "slowness" is also magnified when used with the newer version of dpkg.

If any further information is needed, please let me know.

Regards, Rogério Brito.

[Jean-Baptiste Lallement (jibel)]

Thanks for testing, that's much appreciated.

Could you please provide the following information:
- Filesystem type (ext3/4, xfs, ...)
- mount options
- Any specific configuration (lvm, raid, ...)
- hard drive specs
- If possible, an estimation of the slowdown (you'd better use the 'time' command instead of 'date')

Thanks in advance.

[Nicolas DERIVE (kalon33)]

for me, ext4, /dev/sda6 / ext4 rw,relatime,errors=remount-ro 0 0 (from mtab)
no lvm, no raid (a really classical installation, but with a separate /home)

Disk : average access time : 15,2ms, average reading speed : 66,2Mo/s (tested with palimptest, no write data available because not empty)

Do you need any more info ?

[Alessandro Ghersi (alessandro-ghersi)]

Same here, dpkg is very slow. I don't know if it's related to this bug, maybe so, but I installed Lucid daily and it takes one hour to complete the installation with quad core 3.2 Ghz on a fast hard disk.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpkg - 1.15.5.6ubuntu3

---------------
dpkg (1.15.5.6ubuntu3) lucid; urgency=low

  * Revert fsync during package unpack for now; it's unacceptably slow for
    packages with lots of small files, and we can't ship beta-1 this way.
    We'll do something better once it's decided upstream (LP: #537241).
 -- Colin Watson <email address hidden> Tue, 16 Mar 2010 10:04:38 +0000

[Phillip Susi (psusi)]

Reopening this bug since -ubuntu4 has reintroduced it. It may not be as bad as before, since cjwatson said on irc that syncing is now only performed after unpacking all files in a single package, but I just tested upgrading beta 2 to current and the time to upgrade went from 6m21s to 11m15s.

To test I made a clean install of beta 2 to a new lv on my ssd, booted into the new install, ran apt-get update ; apt-get upgrade -d to download all the files needed. Then I booted back to my main install and made a backup of the test volume using dd. Booted back into test system and locked the version of dpkg to -ubuntu3 and ran a time apt-get upgrade -y. I then booted my main system and restored the dd backup, booted back into the test volume, and first upgraded dpkg to -ubuntu4, then again ran time apt-get upgrade -y.

It seems that this "fix" may be to solve a problem caused by people using data=writeback mount option on ext4, which is documented to be unsafe, and comes at a hefty cost to upgrade a number of packages at once. Because of this I feel this fix should be reverted.

[Colin Watson (cjwatson)]

To be perfectly honest, a factor of two is acceptable in my view when
the trade-off is *hard failures* elsewhere. I believe I was already
quite clear about this on IRC, but I'm not going to change this further
for Lucid. Sorry.

[Colin Watson (cjwatson)]

And BTW it is definitely not anywhere near as bad as before from all the measurements I've taken and seen.

[Phillip Susi (psusi)]

I believe that the failure only occurs when you intentionally *BREAK* your filesystem by using data=writeback, so a factor of two performance penalty for a protection that is unneeded on the default data=ordered mode is unacceptable to me. At the very least the syncing should only be done if data=writeback is detected, or the syncing and renaming should be delayed until after all packages are unpacked if possible, so that it is only done once.

Leaving this in for lucid release to be on the safe side is probably the right thing to do at this point, but this should be revisited later.

[Jean-Baptiste Lallement (jibel)]

Philip, the 247 reporters of the original report (#512096) didn't intentionally break their system. I suppose that most of them don't even know what data=writeback is.

To reproduce it on a fresh lucid setup:
- to be sure that the right mount options are set add "auto_da_alloc,data=ordered" to the ext4 line in fstab and remount.
- install a previous version of dpkg (e.g http://archive.ubuntu.com/ubuntu/pool/main/d/dpkg/dpkg_1.15.4ubuntu2_i386.deb)
- simulate a crash:
$ apt-get install bsdgames ; sleep 5; echo b > /proc/sysrq-trigger

When your system as rebooted, see all those 0 bytes files:
$ ls -l /var/lib/dpkg/info/bsdgames.*
$ ls -ld $( dpkg -L bsdgames )

auto_da_alloc doesn't even detect the replace via rename situation (in fact it does, but only in the simplest cases)

Once the files in /var/lib/dpkg/info/ and /var/lib/dpkg are corrupted there is no other to recover than running some manual operations. Which are like black magic to the common user who generally end up with a complete reinstall. Users can complain about performance but won't complain about loosing their data.

This situation is not specific to dpkg. Try :
$ ar x dpkg_1.15.4ubuntu2_i386.deb; sleep 5; echo b > /proc/sysrq-trigger
it will leave 0 bytes files too.

It's not the place to discuss about it since this report is about performance, but I don't think the solution is to add fsyncs to all the applications around and it is not realistic.
The solution is to fix ext4 itself. To draw a parallel, there is no single database engine leaving empty rows when a transaction failed. Why should a fs behave differently ?

[Phillip Susi (psusi)]

On 4/20/2010 6:31 AM, Jean-Baptiste Lallement wrote:
> auto_da_alloc doesn't even detect the replace via rename situation
> (in fact it does, but only in the simplest cases)

Ahh, so it *IS* broken, even with data=ordered.

> It's not the place to discuss about it since this report is about
> performance, but I don't think the solution is to add fsyncs to all
> the applications around and it is not realistic. The solution is to
> fix ext4 itself. To draw a parallel, there is no single database
> engine leaving empty rows when a transaction failed. Why should a fs
> behave differently ?

Yes, ext4 should be fixed then. A rename() replacing an existing file
must act as a barrier causing the new file to be flushed before the
rename is committed.

[Phillip Susi (psusi)]

The build daemons, brand new installs and certain other environments really don't care if the install is corrupted during a crash, and so would very much prefer to disable the syncing if this could be changed to a runtime option that can be defeated.

[Colin Watson (cjwatson)]

Jean-Baptiste, could you please report an upstream bug about this on bugzilla.kernel.org, since you have the reproduction recipe to hand? (ext4 upstream isn't particularly keen on Launchpad.)

[Bertrand Mathieu (bmat)]

My concern is not related to the time it takes, but the disk activity and loss of responsiveness during package updates: it's all about user perception.

When the change has been introduced, I noticed a lot of writes to the disk with a lot of time spent in syscall (quite normal for frequent fsync).
My immediate reaction was to suspect that the hard drive of upcoming failure, since it behaved like it had difficulties to read or write (and then I found this bug report). I think that regular users may be confused too by this behavior.

[Michael Kofler (michael-kofler)]

I guess this bug also causes the massive slow-down during the Ubuntu installation (compared to earlier Ubuntu versions).

I had already several reports about the standard installation hanging at 95 % (or whatever other percentage). At this point the dpkg database is built/updated/whatever. The installation does not really hang, but depending on how slow your disk is, this step can take up to 15 minutes (perhaps even longer on really old computers).

This also affects the alternate installation, but in a somewhat lesser degree (according to my own experiences).

When Ubuntu 10.04 has nearly achieved a boot time of 10 seconds (it's more like 20 seconds on my computers, but it really boots much faster than before), installation time has almost quadrupled :-( One step forwards here, one step backwards there

[jgreenso (james-green-mjog)]

Forgive me if this is unrelated but we've just installed 10.04 from a network install cd onto a Dell R210 with RAID-1 using ext4 as /. The disks report back damn fast using hdparm -tT and using dd, but when dpkg says it's unpacked it's files there seems to be a massive delay before progressing to configuring the packages. Are we seeing this bug too or something similar?

[Jakob Unterwurzacher (jakobunt)]

Am 02/06/10 17:45, schrieb jgreenso:
> Forgive me if this is unrelated but we've just installed 10.04 from a
> network install cd onto a Dell R210 with RAID-1 using ext4 as /. The
> disks report back damn fast using hdparm -tT and using dd, but when dpkg
> says it's unpacked it's files there seems to be a massive delay before
> progressing to configuring the packages. Are we seeing this bug too or
> something similar?

Looks like the same thing, but different bug#:
https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/570805

[Q (louwrentius)]

We use a PXE environment to deploy user laptops based on Ubuntu.

The installation process is a real pain due to this issue. Hardware is stressed a lot, the hard drive of our notebooks are constantly making grinding noises.

A full desktop installation takes more than an hour which is just unnecessary. Especially during installation, protection against disk crashes is not relevant.

Please, please revert whatever may have changed, to revert it. Or provide a preseed option to disable this behavior. It is slow and is mangling our hard drives.

[zzarko (zzarko-gmail)]

I have the same problem, system slowdown and a very long updates. I have the measurement for speed-dreams package 1.4.0 installed from command line (sudo dpkg -i speed*) for Ubuntu 9.10 and 10.04. Both installations are on the same machine, same HD (different partitions, ext4 for 10.04, ext3 fot 9.10), and speed-dreams files (~320MB) are on a another HD. Results for unpacking phase are:
9.10 : 9s
10.04 : 36s
which is way more than two times slowdown...
If it is acceptable by Colin Watson, could there be at least an opton of choosing should fsyncs be done new or old way? I.e. dpkg --nofysync or something?

[Philip Muškovac (yofel)]

see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=584254 for adding a --force-unsafe-io option. (Can someone please really implement that? the sync() calls are wasting a lot of time if you use pbuilder frequently)

[Phillip Susi (psusi)]

This was fixed upstream and made it into Natty, so marking development task as fixed.

[Bráulio (brauliobo)]

really slow here on a updated trusty. several minutes to update; before it was a few seconds!