btrfs fsync() is extremely slow

[Steve Walton]

Installation and upgrades on btrfs are *extremely* slow. This is because dpkg makes extensive fsync() calls to make sure that a power loss in the middle of an operation does not leave the system in a broken state. These calls have a significant performance penalty on other filesystems ( such as ext[234] ), but on btrfs, the penalty is multiple orders of magnitude.

As a workaround, you can use the eatmydata package/command around dpkg/apt-get to disable the fsync() calls and restore good performance during upgrades, at the risk of hosing the system if it crashes. The apt-btrfs-snapshot package will have apt make a snapshot before it begins an upgrade so that if things do go wrong, you can at least roll back to the snapshot.

[Conn O Griofa (psyke83)]

I can confirm the same issue; installation via the alpha 2 32bit alternate media takes ~2 hours when installing to a btrfs partition, compared to ~15 minutes to ext4. I'm not sure that system specs are necessary, because I don't think this is a bug (or if it is, it should be addressed upstream).

There's an old thread on the linux-btrfs list[1] which mentions a similar problem, so perhaps btrfs' performance characteristics are unfavourable towards file activity related to package unpacking/installation.

[1] http://<email address hidden>/msg03104.html

[toobuntu (toobuntu)]

the btrfs (filesystem) is in the linux kernel, btrfs-tools provides userland tools for working with the filesystem (like taking snapshots, defragmenting, etc.)

[Artem Chudinov (artemka373)]

"Setting up" is very slow, too. But tar -jxvf has same speed as of ext4.

[toobuntu (toobuntu)]

The slowness is not just with installation of the OS, sudo aptitude safe-upgrade takes a very long time if there are a decent number of pkgs to upgrade. Seems to be an issue with dpkg. Unpacking and setting up are where the slowness is most noticeable.

BTW, the problem the fedora user had with yum described in the link in comment #1 seems to have been resolved with a newer kernel version (2.6.31.4-88.fc12.x86_64). Even lucid is on 2.6.32.

[Anthony Hook (anthonyhook)]

I'm marking this bug I opened https://bugs.launchpad.net/ubuntu/+source/linux/+bug/600846 as a duplicate.

[James Lewis (james-fsck)]

I can confirm this... on a thinkpad T61p, running Alpha 3 both the install and subsequent patching is outrageously slow on BTRFS.

What I did notice is that testing in Virtualbox on Lucid on the same hardware prior to installing for real did not show any slowness what so ever!

[James Lewis (james-fsck)]

I've found reference to what I believe is this issue on the BTRFS mailing list... it seems that there may be a significant performance regression in certain configurations, but Chris Mason has identified the cause, and it should be patched... the issue for many of us of course is that Maverick is frozen on the 2.6.35 kernel and so if this is the issue, we may not see a fix before 2.6.36 unless canonical backport it, which may not happen since it's not officially supported... :(

I have posted a link to this bug to the btrfs mailing list thread to confirm that the problem is not just theoretical:-

[James Lewis (james-fsck)]

This issue seems to have been picked up by Phoronix also...

http://www.phoronix.com/scan.php?page=news_item&px=ODQ4Nw

Lets hope that Canonical will backport whatever fix is forthcoming before the release of Maverick, otherwise it seems this could set back the testing and ultimate adoption of BTRFS in Ubuntu by 6 months.

[Conn O Griofa (psyke83)]

James,

In comment #6 you mentioned that you experience no btrfs slowdowns in a virtualized environment. That's probably because btrfs is running on an underlying filesystem, with a separate caching mechanism by the physical hard disk and host OS's handling of your disk. The net effect seems to mitigate whatever performance issues exist with btrfs, as opposed to what happens when used directly on bare metal.

[Bráulio (brauliobo)]

hope to see a kernel patch available and applied to maverick soon!

[Anthony Hook (anthonyhook)]

Added tags from my first previous bug.

[Anthony Hook (anthonyhook)]

When installing, it took over 12 hours to complete the setup. I have two ASUS-PHISON SSDs in this netbook. After seeing this: https://btrfs.wiki.kernel.org/index.php/FAQ#Is_Btrfs_optimized_for_SSD.3F I tried to add 'ssd' to my options in fstab, but this still does not help.

[Rocko (rockorequin)]

On a btrfs-related note, the developers have only just realised that it might be worth advising btrfs users of the following caveat (see the wiki at https://btrfs.wiki.kernel.org/index.php/Main_Page):

"Note that Btrfs does not yet have a fsck tool that can fix errors. While Btrfs is stable on a stable machine, it is currently possible to corrupt a filesystem irrecoverably if your machine crashes or loses power on disks that don't handle flush requests correctly. This will be fixed when the fsck tool is ready."

ie it is ridiculously easy to lose an entire partition!

@James: I can confirm what Conn said in comment #9: when I tried the Maverick btrfs install in a VirtualBox environment, it ran chronically slowly until I enabled the host disk cache, and now it runs at normal speed.

[Kenny Strawn (realkstrawn93)]

Trying to 'sudo apt-get upgrade' takes upwards of ~2-3 hours on my Acer Aspire One AOA110 with Ubuntu 10.10 Alpha 3 and btrfs. In particular, when there is over 100 packages to install, I just let the updates install overnight, and in some cases it's still not done.

[Tournier Mathieu (mathieutournier)]

Same problem with my dell xps 1640 : dpkg is very slow, so updates are abnormally slow... (same problem with 2.6.36 rc1)

[Rocko (rockorequin)]

The patch posted by Chris Mason on the btrfs mailing list http://news.gmane.org/gmane.comp.file-systems.btrfs (patch attached) makes a huge difference on my Maverick VM.

My test case was to turn off the host SATA cache for the VM and then install a hefty package (netbeans in this case). atop showed avio maxing out at 2ms during the install compared to the 20-30ms I was seeing with the stock kernel.

[James Lewis (james-fsck)]

Can we request that Canonical ensure this patch is in the kernel shipped with Maverick for release as having a serious problem with BTRFS in 10.10 would be a serious setback to the filesystem in Ubuntu.

[Bráulio (brauliobo)]

@james
that would be rather nice

[James Lewis (james-fsck)]

Looks like the patch above may not be the answer, or certainly not the complete answer... keep an eye on the mailing list for updates!

[RussianNeuroMancer (russianneuromancer)]

Any news about other additional patches for slow 2.6.35 btrfs?

[Giovanni Condello (nanomad)]

Same issue here with the latest 2.6.35.
Any news?

[Dave Johnson (dave-scomple)]

I'm experiencing this on Maverick Beta. I installed the 64-bit beta of 10.10 last night with btrfs on "/" and "/home" leaving "/boot" on ext4 and now that I've gotten around to running the update manager I'm into hour 4 of the process. I first noticed the slowness when using Firefox to search for some suitable wallpaper to replace the default that looks like someone's nose bled on my screen, but this deal with the update manager takes the cake.

I have a quad-core AMD Phenom CPU with 8GB of memory and a standard 7200rpm SATA drive.

I'm normally very willing to deal with bugs in an effort to help weed them out, but I get considerably better performance on a fifteen-year-old machine running WattOS. It may have been premature to include btrfs in this release.

[Brian Rogers (brian-rogers)]

Constant calls to sync and fsync by dpkg are what slow down btrfs so much. If you want to try dpkg with syncing removed, I made a PPA for that: https://launchpad.net/~brian-rogers/+archive/btrfs

[Abram Hindle (abram-hindle)]

I don't think it is just btrfs. I am using XFS on top of software raid 1 in Maverick, and if I am doing any disk I/O (e.g. a big copy) dpkg will take a very long time to unpack. Stopping the disk I/O often won't solve the problem (maybe patience will).

I installed Brian Roger's dangerous sync free dpkg and I could install packages with heavy disk I/O without any significant lag or latency. dpkg Unpack would not wait for long periods with D - Uninterruptible sleep status.

* XFS on RAID 1
* dpkg was slow to install during high I/O
* dpkg w/o sync/fsync worked fine
* During the first installation it was also very slow.

[Robin van Leeuwen (robinvanleeuwen)]

Ok, 10.10 is just released and I downloaded it. I tried installing it with / on btrfs and after 5 hours the install was at 75%. This is on a AMD Turion II Dual Core M500 with 6 GB. My / was 550 GB.

It's a shame this issue wasn't resolved before releasing Maverick and this will hurt BTRFS adoption for sure. IMHO this should never had been released in this fashion since this bug was already reported in the Alpha release. Why bother with alpha and beta releases when isssues like this are not resolved. Sorry if this sounds like a rant, but a filesystem is a essential part of every setup and it should either work correctly or it shouldn't be a option during install (to say the least).....

[Ken Bloom (kbloom)]

I did a related benchmark on a Debian unstable system (with dpkg 1.15.8.5 and linux-image-2.6.35-trunk-amd64) and an ext3/4 file system. Here were the results:

This affects my ext3/4 filesystem (ext3 mounted with the ext4 driver, but none of the advanced ext4 features enabled) as well. Iin a cowbuilder chroot (with all of the packages pre-cached by apt-cacher-ng):

      # time eatmydata apt-get install –no-install-recommends openoffice.org
      0 upgraded, 142 newly installed, 0 to remove and 0 not upgraded.
      …
      real 0m57.682s
      user 0m37.030s
      sys 0m7.220s

      # time apt-get install –no-install-recommends openoffice.org
      0 upgraded, 142 newly installed, 0 to remove and 0 not upgraded.
      …
      real 3m17.158s
      user 0m37.186s
      sys 0m11.057s

Over three times as long. So maybe you're getting a double hit with btrfs, but the first hit that dpkg is giving you on *any* filesystem is pretty bad to begin with.

[jdobry (jdobry)]

Just one more confirmation. Unbelievably slow.

[bsfmig (bigslowfat)]

My confirmation too.Too slow to wait for the installation to complete.

[Andriy Yurchuk (ch00k)]

It won't be fixed earlier than 11.04 is out, as per this info: https://help.ubuntu.com/community/btrfs#Stability

[RussianNeuroMancer (russianneuromancer)]

Looks like not problem anymore since Natty.

[Gary T. Giesen (giesen)]

Any chance of a backport?

[Arne Bockholdt (u-launchpad-net-bockholdt-com)]

Still present in Natty, latest nightly build.

[James Lewis (james-fsck)]

If this is as a result of dpkg calling "fsync" a lot, un-necessarily... then perhaps we should consider this partly an issue with dpkg as well as with btrfs?... I might even go so far as to say that it could be described as a bug in other filesystems which are not honouring the fsync call?

I welcome comments on that, but having read this thread... I'm not sure this is 100% the fault of btrfs.

[RussianNeuroMancer (russianneuromancer)]

Probably Arne Bockholdt is right. I test Natty in VM, then maybe I mistake. I missed commentary six.

[Chris Halse Rogers (raof)]

On Thu, 2011-01-13 at 17:36 +0000, James Lewis wrote:
> If this is as a result of dpkg calling "fsync" a lot, un-necessarily...
> then perhaps we should consider this partly an issue with dpkg as well
> as with btrfs?... I might even go so far as to say that it could be
> described as a bug in other filesystems which are not honouring the
> fsync call?

Other filesystems *are* honouring the fsync call (you can check by
preloading libeatmydata); btrfs is just desperately slow when something
fsyncs a lot.

[Andy Whitcroft (apw)]

There are a number of factors in play here. btrfs is not production ready in the Maverick kernels, it is slow in general and very slow in the face of an fsync centric load. Btrfs was enabled in the installer as an experimental feature, as a technology preview, and as such it is not likely anyone is going to work to backport any performance fixes to Maverick. For Natty we are expecting some direct improvements from layout improvements, and from upstream fixes. Closing off the Maverick task Won't Fix.

[Andy Whitcroft (apw)]

We are currently moving the Natty kernel up to v2.6.38-rcN. Once we have any kind of stability there it would be good to get btrfs performance retested; before then we are likely wasting the testing time.

[James Lewis (james-fsck)]

>Other filesystems *are* honouring the fsync call (you can check by
>preloading libeatmydata); btrfs is just desperately slow when something
>fsyncs a lot.

Fair point...

BUT, does this perhaps present an opportunity?... during initial install, a power failure will probably not result in a bootable system anyway, and so perhaps there are performance gains to be had (even if smaller) with other filesystems by having a "no fsync" mode of dpkg for initial installation.

Not that this specific issue with btrfs isn't important, I'm just wondering if we can learn somthing from why this particular load is slow... and perhaps make some other things faster in the process.

[James Lewis (james-fsck)]

Verified that the issue still occurs with 2.6.38rc2 (By updating a 10.10 install to the mainline 2.6.38rc2 kernel, and then attempting to update to natty)... It currently says that it will take 11 hours to complete the update (This may be somewhat worse than the estimated time for 2.6.35, although I've not given it as long to settle, so I can't confirm that..

[James Lewis (james-fsck)]

It seems that this issue is only "worse" on BTRFS (OK, much worse)... but it is a sufficient issue even on ext4 that there is a bug report requesting a "no fsync" option for dpkg/apt in Debian's bug tracker!

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=588254

[Eugene San (eugenesan)]

Confirmed on Natty (Ubuntu 2.6.38-3.30-generic 2.6.38-rc4)

[Tommy_CZ (t-kijas)]

I can confirm it in latest Natty (Alpha 3), 2.6.38-5

[Joke de Buhr (joke)]

Still the same problem with 2.6.38-7.

By the way activating the dpkg option --force-unsafe-io doesn't really speed things up.

[canthus13 (canthus13)]

I had to give up on an install after 45 minutes and only 30% finished, using Beta 2.

[vadimo (michalgejdos-azet)]

Today i install Xubuntu 11.04 64bit (finall) to USB stick with btrfs as /. Lately remount as compress.
All worked now good.

[gregrwm (gregrwm)]

natty's nifty fast startup also bites the dust in btrfs. 5 minutes instead of 15 seconds.

[Philip Muškovac (yofel)]

@greg: I can't confirm those 5 minutes here. But I used compression and remove the fsck.btrfs link as that slows down the boot and doesn't do much yet. Can you install bootchart to check where it gets stuck for so long?

[Alexandre (ab-linuxfr)]

I can confirm the slowness with kubuntu natty 2.6.38.8.
I posted a bug #798988 before finding out that thread.

Btw, the general btrfs slow response can be "feel" even after startup, specially if you use btrfs for /home too.

[Alexandre (ab-linuxfr)]

I can confirm the slowness with kubuntu natty 2.6.38.8.
I posted a bug report 798988 before finding out that thread.

Btw, the general btrfs slow response can be "feel" even after startup, specially if you use btrfs for /home too.

[Ketil Malde (ketil-ii)]

I just did an 'apt-get upgrade' and not only did it appear to take somewhere close to forever, it also bogged down the rest of the system. In the end, I interrupted the process, did the workaround from here:

https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/607632/comments/12

and restarted the process using LD_PRELOAD, which completed very quickly.

Ubuntu Natty, 2.6.38-8-generic, x86-64, btrfs on SSD disk.

I don't think it was this slow before, did something change recently?

[Ketil Malde (ketil-ii)]

I think that in my case, this is caused by me using the 'discard' option. See e.g:

http://lists-archives.org/linux-kernel/27396958-btrfs-and-apt-package-manager-in-ubuntu-discard-stalls.html

I did the test suggested, and with discard enabled, it took about 45 seconds, without discard it took 12 seconds. Looks like there is some nasty interaction between fsync and discard? Anyway, this is likely a different issue from what others are reporting, so I'll open a new bug (as soon as I can find out &%¤##" how, the "Report a bug" link just points into a maze of long wiki pages, all alike...)

[Rocko (rockorequin)]

@Ketil: you can report a new bug with 'ubuntu-bug <name of package>'. The trick is in finding the right package name. (I find the "Report a bug" links very frustrating as well.)

[Ketil Malde (ketil-ii)]

Yes, thanks, it's now here:
  https://bugs.launchpad.net/ubuntu/+source/linux/+bug/797075

It's hard to tell how many of the subscribers to this bug are affected by the same issue, or whether there are other performance issues with fsync/btrfs. I don't think btrfs was set up with 'discard' by default, so the original bug report was probably something else, and probably fixed by now.

[Brad Figg (brad-figg)]

The attached patch was never committed to the upstream kernel by the btrfs developers. I've therefor marked it as not a patch for this bug.

[EricDHH (ericdhh)]

12.04b1 on thinkpad x41, update from 11.10 installed on btrfs.

Started at 00:00 AM with all packages for the update downloaded, the reboot and finish occurs on 04:00 PM, thats 16 hours for 4.5gb of data. The laptop is equipped with an solidata ssd that reed and write around 120mb/s. Bootup for the fresh 12.04b1 system is around 6 minutes to login!

Thats simply horrible slow, i will kick btrfs out for now and use xfs instead.

[Chris Halse Rogers (raof)]

The slow bootup problem is due to btrfs' fsck tool (which is unable to
*fix* any problems it finds) not bailing on a clean filesystem; this
causes a full filesystem check each boot.

[Brad Figg (brad-figg)]

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

You can update to the latest development kernel by simply running the following commands in a terminal window:

    sudo apt-get update
    sudo apt-get dist-upgrade

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

[FlashBuster (flashbuster)]

I have the same problem.. on an update my CPU load is at 4, but CPU usage percentage around 5%, enough memory.
The update process takes extremely long, where it took seconds before.
I'll have to reinstall with ext4 the next days, because btrfs is obviously not ready to be used.
This is after a clean Ubuntu precise installation, no ext4 upgrade.

[DiegoWoitasen (diego-woitasen)]

The bug still exists in Precise, kernel 3.2.0-24-generic

[Jérôme Poulin (jeromepoulin)]

I can confirm that dpkg is very slow when BTRFS is used as root filesystem. Using eatmydata fixes the problem.

[Manuel Gysin (manuel-gysin)]

Summary after some years where this problem exists:

The btrfs developers are aware of this problem, but they will not fix it. Why? They standpoint is clear and simply, dpkg and many other application uses and relay on fsync, but fsync is no POSIX standard and really a bad way to provide the integrity of file operations. So the btrfs develops will not fix the behavior of btrfs itself.

So on the other hand the dpkg developers pointing to btrfs and are not fixing the bug, while it's working with other file systems.
This is a dead end situation, nobody is going to change anything because both sides are telling the other side made a mistake.

Ubuntu as one of the big distributors should take steps to fix this. Write a patch which makes dpkg works without fsyncs (eatmydata is no solution!) and adopt the btrfs idea with the rename of files.

Else we hit in some near time the point that debian based distributions are not fully compatible with new file systems like btrfs.
On the other hand when Ubuntu is not fixing this, the btrfs support should be completely removed! At the current state debian based distributions are not working with btrfs in a usable way.

Links:
https://btrfs.wiki.kernel.org/index.php/Talk:FAQ
https://btrfs.wiki.kernel.org/index.php?title=FAQ
http://old.nabble.com/Bug-635993%3A-dpkg-is-very-slow-with-btrfs-filesystem-td32185922.html

[X (u78qir8a9-deactivatedaccount)]

The linked commits have been merged for Linux 3.5 and should improve dpkg on btrfs a bit (I did not test personally).

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=5bdbeb2187a99d690b374a8c5ec9911fcbcfe739
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=22ee6985de7d3e81ec0cef9c6ba01b45ad1bafeb

[Trung Ngô (ndtrung4419)]

There are some new and significant development in the lastest kernel regarding btrfs' fsync performance. Can someone check it out?

https://patchwork.kernel.org/patch/1372281/

[RussianNeuroMancer (russianneuromancer)]

What the status of this bug in Linux 3.7?

[rogerdpack (rogerdpack)]

the "turbo charge fsync" (aug. 2012) patch should be in 3.6 AFAICT. Anybody care to try dpkg with it and let us know? Apparently 3.9 includes a "further" fsync speedup patch as well.

[EricDHH (ericdhh)]

Can speak for 12.04-2 on 3 computers, btrfs is a bit faster but installing a kernel package take full 5 minutes. All machines are equipped with ssd's up to 250mb/s. Latest kernel seen here is 3.5.0-28. Hope someone will backport the speedup benefits.

[Maciek Sujkowski (maciejus)]

I have noticed a great speed improvement in the Linux kernel 3.7 and above

[Alberto Salvia Novella (es20490446e)]

If you are still experiencing this issue, please set this bug status back to "confirmed".

[Phillip Susi (psusi)]

The debian bug you link to did not actually fix this. The debian devs simply dodged the issue by adding a --force-unsafe-io flag to dpkg that slightly reduces ( but nowhere near eliminates ) the ridiculous number of syncs dpkg does. The idea was that debian-installer should use this flag to speed up installations. Not only does it not go nearly far enough during installation, but it does nothing for installing or upgrading packages on an existing system.