Better file preallocation with fallocate()

[Bryan Quigley]

Binary package hint: transmission

Torrent was about halfway done. Computer crashed (from pulseaudio, I think unrelated). Now torrent is at 4 or megs (was at 150).

This is the log for the hard drive (from dmesg) on the next bootup:
[ 1.423899] sda5 > sda3 sda4
[ 2.264091] EXT4-fs (sda4): INFO: recovery required on readonly filesystem
[ 2.264096] EXT4-fs (sda4): write access will be enabled during recovery
[ 2.301665] EXT4-fs (sda4): barriers enabled
[ 4.295714] kjournald2 starting: pid 394, dev sda4:8, commit interval 5 seconds
[ 4.295735] EXT4-fs (sda4): delayed allocation enabled
[ 4.299343] EXT4-fs (sda4): orphan cleanup on readonly fs
[ 4.299350] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 1319
[ 4.299423] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 264002
[ 4.299437] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 263983
[ 4.299448] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 263980
[ 4.299458] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 263958
[ 4.299469] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 263955
[ 4.299479] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 263945
[ 4.299490] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 263944
[ 4.299500] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 1316
[ 4.299553] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 1315
[ 4.299564] EXT4-fs (sda4): ext4_orphan_cleanup: deleting unreferenced inode 326
[ 4.299569] EXT4-fs (sda4): 11 orphan inodes deleted
[ 4.299572] EXT4-fs (sda4): recovery complete
[ 4.858060] EXT4-fs (sda4): mounted filesystem with ordered data mode
[ 12.985525] EXT4-fs (sda4): internal journal on sda4:8

Perhaps transmission needs to be take advantage of some ext4 features?

ProblemType: Bug
Architecture: amd64
Date: Wed Oct 7 21:39:48 2009
DistroRelease: Ubuntu 9.10
ExecutablePath: /usr/bin/transmission
NonfreeKernelModules: nvidia
Package: transmission-gtk 1.75-0ubuntu1
ProcEnviron:
 LANG=en_IN
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-12.40-generic
SourcePackage: transmission
Uname: Linux 2.6.31-12-generic x86_64
XsessionErrors:
 (gnome-settings-daemon:1859): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
 (gnome-settings-daemon:1859): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
 (polkit-gnome-authentication-agent-1:1910): GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed
 (nautilus:1896): Eel-CRITICAL **: eel_preferences_get_boolean: assertion `preferences_is_initialized ()' failed
 (gnome-panel:1891): Gdk-WARNING **: /build/buildd/gtk+2.0-2.18.2/gdk/x11/gdkdrawable-x11.c:952 drawable is not a pixmap or window

[John Dong (jdong)]

Not surprising due to delayed allocation, if cache writeout pressure isn't high then this will happen. The "workaround" would be for transmission to periodically flush the filesystem but that'll just cause awful IO spiking latency and defeat the purpose of delayed allocation.

If you'd rather not lose that much data, then perhaps you need a more aggressive dirty_writeback_centisecs value.

[Bryan Quigley (bryanquigley)]

Would another solution be to use Persistent pre-allocation?

From wikipedia "The ext4 filesystem allows for pre-allocation of on-disk space for a file. The current methodology for this on most file systems is to write the file full of 0s to reserve the space when the file is created. This method would no longer be required for ext4; instead, a new fallocate() system call was added to the Linux kernel for use by filesystems, including ext4 and XFS, that have this capability. The space allocated for files such as these would be guaranteed and would likely be contiguous. This has applications for media streaming and databases."

As my other understanding from the error message and the above, is that it lost data because they were stored in different places. Then ext4 filecheck cleared them all, because they weren't properly recorded in ext4s records. If all space is alloted ahead of time all data actually written to disk will always be recoverable by transmission.

[John Dong (jdong)]

No, that's likely not the case either. You lost data because it is in the writeback cache which hadn't been flushed to disk. Whether or not you fallocate(), the writeback cache is still used.

To bypass that, you either flush to disk more often as a whole on your system, or have Transmission open its output file with O_SYNC.

I wouldn't consider transmission downloads to be critical enough (unlike say saving an OpenOffice document) to deserve O_SYNC but if losing 100MB of in-transit data is a big deal for you, then you should definitely tune your VM subsystem to flush the writeback cache way more often.

[Bryan Quigley (bryanquigley)]

It was indeed not that big a deal, it's a torrent (just redownload it) , and I think it likely only lost so much because it was downloading quite quickly.

Still I think persistent pre-allocation, might help, as it didn't download all 150 megs in under a minute (which I thought was the maximum delay time) and one of the reasons to do the delayed allocation is to prevent fragmentation (which isn't really an issue if you allocate it all at once).

Maybe make it a wishlist feature request?

[John Dong (jdong)]

1 minute isn't an upper bound for the amount of in-transit data that can be lost; delayed allocation has to do with losing entire files (zero-size files), not just data.

Nonetheless, I do agree with you that offering a persistent preallocation method like KTorrent does which tries to use FS-specific preallocation whenever possible is a good idea.

[Charles Kerr (charlesk)]

Changing the default to preallocation in Transmission only requires a one-line change.
This is turned off by default because fallocate() can be extremely slow on ext3.
The current plan is to change the default after ext4 becomes more widespread.

Is there a way to test for the ext4 filesystem?

[John Dong (jdong)]

On Fri, 09 Oct 2009 15:09 +0000, "charles" <email address hidden>
wrote:
> Is there a way to test for the ext4 filesystem?
>

Hmm maybe we should peek in the KTorrent source code. They even are
smart enough to handle XFS vs ext3 vs ext4 vs vfat independently. For
some unknown definition of "smart" :)

[Charles Kerr (charlesk)]

> Hmm maybe we should peek in the KTorrent source code. They even are smart enough to handle XFS vs ext3 vs ext4 vs vfat independently. For some unknown definition of "smart" :)

KTorrent uses ftruncate64() or ftruncate() in its "quick" mode, and posix_fallocate64() or posix_fallocate() in its non-quick mode. The problem with this is described in detail (even in terms of p2p apps) in http://www.linux-mag.com/id/7272/2/ -- if you use posix_fallocate() on ext3, it will attempt to emulate fallocate() by writing all zeros to the file. This emulation step can be very slow.

John, are fallocate64() and fallocate() (note the lack of the "posix_" prefix) present in the glibc that ships with 9.10?

[Charles Kerr (charlesk)]

looks like glibc 2.10.1 is in Karmic, but http://repo.or.cz/w/glibc.git?a=commitdiff;h=1f3615a1c97a030bca59f728f998947f852679b9;hp=9e471dad8e173806cbbfb704875d5ae40e36fa34 says that the fallocate*() functions are new in glibc 2.11.

We should revisit this in 10.4... this would be an improvement.

[Charles Kerr (charlesk)]

Since the original intent of this ticket seems to have been resolved as an OS tuning issue that the user can configure, but the follow-up discussion about flie preallocation on ext4 has implications that are relevant to future versions of Transmission's code, I'm going revising this ticket's attributes to reflect this.

[Launchpad Janitor (janitor)]

This bug was fixed in the package transmission - 1.90-0ubuntu1

---------------
transmission (1.90-0ubuntu1) lucid; urgency=low

  [ Krzysztof Klimonda ]
  * New upstream release (LP: #522726)
  * Fixes bugs:
    - Support Application Indicators (LP: #497882)
    - Better file preallocation with fallocate() (LP: #445592)
    - Speed limit of 0 is not honored (LP: #473652)
    - incorrectly sent information (LP: #521140)
  * Refreshed 99_autoreconf.patch
  * debian/control:
    - Build-Depend on libappindicator-dev

  [ Chris Coulson ]
  * Add ${misc:Depends} in debian/control to make Lintian happy
  * Update Standards-Version to 3.8.4. No other changes necessary
 -- Krzysztof Klimonda <email address hidden> Tue, 16 Feb 2010 17:50:22 +0100