DPKG is very slow when using BTRFS

[Ivy]

Binary package hint: dpkg

Installed using the Maverick daily build from 19th July (32 bit)
UPDATE: Confirmed also in release, both 64 and 32 bit

Installed using a root partition formatted as BTRFS

Noted that DPKG was noticeably slower and took longer for every operation than the same system with an ext4 partition.
UPDATE: ext3, ext2 partitions are roughly the same speed as ext4. Only btrfs is slower.

Examples of slowness include:
* apt-get update takes a very long time, particularly at the end after downloading the package lists
* Installing packages takes 2 to 3 times as long as on ext4/3/2
* Particularly unpacking the deb file, this takes the longest. Basically any IO operations are slower.

[Brian Rogers (brian-rogers)]

All the slowness can be attributed to dpkg's use of sync and fsync. I created a PPA that removes those calls, and dpkg is fast again.

https://launchpad.net/~brian-rogers/+archive/btrfs

Of course, this takes away safety in the event of a crash or power failure. It'd be a good idea to keep a snapshot around and know how to use the snapshot in case your package database gets corrupted in a crash.

[Jorge Castro (jorge)]

I see this on a clean install of Alpha 3.

[UbuntuFlo (ubuntuflo)]

Installed Maverick's Netbook Alpha 3 choosing btrfs for / and home on a Asus Eee 1000H. During updates (using apt-get) a remarkable loss in speed/performance is noticeable.

[Patrick le Roux (pleroux-gmail)]

I can confirm that the slowness is there on the 10.10 beta release as well. My lawn now grows faster than apt-get unpacks packages when useing btrfs.

[Anton Kudris (kudris)]

I've got 10.10 release version running on btrfs partition (only /boot is ext2) and can confirm that dpkg runs way slower then before (before I used ext3)

[Fabus (fabian-gebert-hh)]

I can confirm this with a fresh 32-bit install (with updates during install enabled), /boot on ext4 and / on btrfs, /home on ext3.

[Fabus (fabian-gebert-hh)]

I must add the install speed was alright, it's just when installing things using the Update Manager

[Ivy (blazemore-deactivatedaccount)]

I'd like to revisit this bug and confirm that it occurs on both 32 and 64-bit.
When I get back home I'll do some benchmarks to see exactly what kind of speed-hit is occurring. Needless to say, if it's noticeable it must be significant.

[Mikael Nordfeldth (mmn)]

I am experiencing this problem on an Asus Eee901 formatted with btrfs (/ and /home are btrfs-formatted SSDs, but everything else disk-related is fast, so it's not a hardware or btrfs bug).

[Ferdinand Hagethorn (ferdinand-hagethorn)]

Same issue here, seems btrfs doesn't honor the fsync call?

[Ferdinand Hagethorn (ferdinand-hagethorn)]

Okay, quick and dirty workaround that proves its an fsync issue:

//nosync.c:
void sync() { }
int fsync(int fildes) { return 0; }

//build it:
gcc -shared -fPIC -O2 -g nosync.c -o /usr/local/lib/libnosync.so

//use it as preload with dpkg/apt
LD_PRELOAD=/usr/local/lib/libnosync.so apt-get install $package

dpkg should work 'rather' quickly now

[Christian Kujau (christiank)]

Thanks Ferdinand, this helped a lot. Quick PoC with installing/purging a samba4 (13.5MB):

# grep btrfs /proc/mounts && uname -r
/dev/sda2 / btrfs rw,nodev,noatime 0 0
2.6.35-27-generic

# sysctl -w vm.drop_caches=3
# time (LD_PRELOAD=/usr/local/lib/libnosync.so apt-get -y install samba4 && sync)
real 0m31.115s user 0m1.620s sys 0m3.150s

# time (LD_PRELOAD=/usr/local/lib/libnosync.so apt-get -y purge samba4 && sync)
real 0m30.940s user 0m1.370s sys 0m2.690s

# sysctl -w vm.drop_caches=3
# time (apt-get install samba4 && sync)
real 0m41.611s user 0m1.530s sys 0m3.250s

# time (apt-get purge -y samba4 && sync)
real 0m40.015s user 0m1.390s sys 0m2.850s

And this is reproducible: so it's always 30 or 31 seconds with libnosync and always ~10sec more w/o libnosync.

[Raphaël Hertzog (hertzog)]

There's not much we can do to improve dpkg on btrfs. There's an option --force-unsafe-io if you want to get rid of fsync() on installed files but it will still use fsync() on the internal DB.

Those calls are required to ensure the consistency of the dpkg database and we value that over the performance.

[Vin Shankar (v-shankar)]

I don't see how very slow dpkg operation on btrfs is a duplicate of dpkg reporting that packages are already configured, especially when the bug this is marked as a duplicate of doesn't mention btrfs at all.
Additionally: It would be good if something could be done to fix this, as because of it a Maverick->Natty upgrade took nearly 24 hours post-download just to unpack and configure packages, which is unacceptably slow.

[Marius Storm-Olsen (marius-storm-olsen)]

Also confirmed here.

On a Mac Pro, 2xQuad Xeon E5620 @ 2.40 GHz, BTRFS on a ATA WDC WD6400AAKS-4 (547.53GiB partition), it's telling me 1day 8hours to upgrade from Maverick to Natty, post download of packages (~1800 of them). And that's on an idle machine.

[Raphaël Hertzog (hertzog)]

Do you guys still have very poor performance with dpkg 1.16.0 and btrfs ?

A 25% slowdown due to fsync() usage does not seem unreasonable. BTW, you can try with "eatmydata" to disable the fsync() calls.

[Isaac Aaron (e-tsik)]

I can confirm that the nosync workaround worked for me.
I tried to upgrade Ubuntu 11.04 to 11.10 on a netbook using a SD card with btrfs.
The time estimate before I killed was 1 day and 22 hours.
I couldn't even run dpkg --reconfigure -a

After the fix the dpkg --reconfigure -a took about 3 minutes and upgrade (just the package installation) took about 1.5 hour