dpkg gets slower as /var/lib/dpkg/info gets fragmented

[Krastanov]

Binary package hint: dpkg

dpkg gets slower as /var/lib/dpkg/info gets fragmented especially after a distro upgrade or when /var is more than 70% full.

A working solution is http://ubuntuforums.org/showthread.php?t=1004376, which should be added as a functionality to dpkg or synaptic or software center.

ProblemType: Bug
Architecture: amd64
Date: Sun Oct 4 12:44:26 2009
DistroRelease: Ubuntu 9.10
NonfreeKernelModules: nvidia
Package: dpkg 1.15.4ubuntu2
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-11.38-generic
SourcePackage: dpkg
Uname: Linux 2.6.31-11-generic x86_64

[Zack Evans (zevans23)]

It says in that post that xfs suffers dir fragmentation easily. xfs is probably quite an unusual choice for /var so there's a bit of a danger this will be considered a corner case...

Are you saying you are also seeing this on ext4?

[Krastanov (krastanov-stefan)]

Yes, I'm seeing it on ext4.

All my filesystems are ext4 (except for /boot).

After upgrade to Karmic my /var was about 85% full (+- 5%). Scanning the database took 30 to 45 seconds each time I tried to install something. After doing the defrag from that forum post it takes under 7 seconds.

I'm doing an "apt-get clean" every now and then and my /var is usually at 50%, so I believe the greatest fragmentation was during the upgrade.

[Philip Muškovac (yofel)]

This bug is an upstream one and it would be quite helpful if somebody experiencing it could send the bug the to the people writing the software. You can learn more about how to do this for various upstreams at https://wiki.ubuntu.com/Bugs/Upstream. Thanks in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpkg - 1.15.5.6ubuntu2

---------------
dpkg (1.15.5.6ubuntu2) lucid; urgency=high

  * Backport from upstream:
    - Use FIEMAP when available (on Linux based systems) to sort the .list
      files loading order. With a cold cache it improves up to a 70%.
      Thanks to Morten Hustveit <email address hidden>. LP: #442114
    - Call fsync(2) after writing files on disk, to get the atomicity
      guarantees when doing rename(2). Based on a patch by Jean-Baptiste
      Lallement <email address hidden>.
      Closes: #430958, LP: #512096
  * Security fixes by Raphaël Hertzog, also backported from upstream
    (CVE-2010-0396):
    - Modify dpkg-source to error out when it would apply patches containing
      insecure paths (with "/../") and also error out when it would apply a
      patch through a symlink. Those checks are required as patch will
      happily modify files outside of the target directory and unpacking a
      source package should not be able to have any side-effect outside of
      the target directory. LP: #532445
    - Also error out when the quilt series contains a path with "/../" as
      this can cause patch to create files outside of the source package due
      to the -B .pc/$path option that it gets.
 -- Colin Watson <email address hidden> Thu, 11 Mar 2010 00:34:28 +0000

[Ryan (ryan-farmer-personal-deactivatedaccount)]

@Zack Evans: I don't know what the deal is. XFS is actually quite hard to fragment, and even if you do manage, you have an online defragmenter (xfs_fsr in the xfsdump package) which can take care of that for you.

This would be more likely to become a problem on Ext4, and much more likely on Ext3. The only way to fix fragmentation there is to format the partition. (e4defrag was promised, but still never appeared).

[Krastanov (krastanov-stefan)]

It's not about fragmentation of files. It's about small files scattered all over the hard drive. It affects all file systems, and defragmentation is not helping as the files are not fragmented, but scattered, so sequential read is not possible. Maybe I used the word "fragmented" incorrectly while reporting the bug - if that's the case I'm sorry. But the bug (more of a wishlist) is fixed. Compliments to the devs, and mister Hustveit in particular.