Copying files from Linux to NTFS Partition (using NTFS-3G) makes random-placed non-movable system blocks which causes file fragmentation

[Joshia]

Binary package hint: ntfs-3g

When I'm copying files from Ubuntu to any NTFS partition, firstly the files become heavily fragmented, so I defragged the volume using Microsoft Disk Defragmenter in Windows XP and O&O Defrag in Windows Vista.

I wonder why my volume could not defragged perfectly like the other times when i'm not copying the files using Linux.

I see a lot of system block colored identifier in the Cluster view in O&O Defrag scattered everywhere. I don't know how to eliminate these unnecessary system blocks (I wonder what data contained there) without reformatting the partition.

So, each time when i'm going to copy an abundant amount of files (in this case 120000+ files, from a few KBs to 6 GBs), the data will be fragmented because thete are not enough contiguous blocks for the system to write in.

Linux Distributions : Probably Any linux Distributions (maybe, coz i've tried fedora 8, ubuntu 8.04 LTS, mandriva 2007.1 (spring), etc and the results are the same from the fresh install)

NTFS-3G version : any version to date (any stable or either unstable versions included in the distributions listed above)

This have been happened since i were using ubuntu 7.04 (first time using ubuntu)

I hope that there is:
1. a way to eliminate these sectors by hand without reformatting the partition
2. a way to prevent this thing further occurrence

[Szabolcs Szakacsits (szaka)]

Here are some info what kind of fragmentations exist. It's important to read because fragmentation types are often misunderstood: http://en.wikipedia.org/wiki/Fragmentation_(computer)

The NTFS-3G driver tries hard not to fragment files and if you're doing a lot of concurrent writes then you will find that it's much more efficient than the Microsoft NTFS driver which fragments files usually at every 16 clusters meanwhile NTFS-3G fragments them only at around 4000 clusters. This means the elimination of 250 very costly, slow disk seeks and result much better write and read performance.

What you're seeing is external fragmentation. The NTFS-3G driver indeed places different files a bit far from each other to eliminate and optimize file fragmentation to improve the disk performance. How much does it matter? Usually technically and performance wise it's absolutely irrelevant.

If you don't believe me then please feel free to send your NTFS debug info to szaka AT ntfs-3g.org folllowing the instructions at http://www.linux-ntfs.org/doku.php?id=ntfsclone#store_only_ntfs_metadata
and I will tell you the TRUTH about your real fragmentation and what you can check yourself to be true by reading all the open source code we are using to do these REAL analyzes.

About the "system" blocks. These are always Windows deficiencies explained nicely to Windows users. There are many reasons for them. For instance files in use by Windows are unmovable. Sadly Windows caches a lot of attributes absolutely needlessly and prevents defragmenters to do their jobs. The built-in Windows defragmenter code, what all the Windows defragmenters use, apparently also has technical limitations, like not being able to relocates files having attribute list attribute. In such case they don't say "We are very sorry, this functionality is not implemented!" but instead "System block". So users won't be upset with Microsoft but try to find somebody else to blame and people always do: the defragmenter, a file system driver, etc.

What you can do is offline defragmentation, i.e. Windows should not be running during the process.

Regards, Szaka

--
NTFS-3G Lead Developer: http://ntfs-3g.org

[Joshia (joshia-setiawan)]

what i did is offline defragmentation, because i took the hard drive from my brother's computer to my computer, and defrag it by an USB connection.
The confusing thing is, when i'm copying with Windows, there are no such "system" blocks left, which described before.
I'll post the NTFS debug later there. please give me explanations later.
One question, will those "system" blocks gone after i delete the files, or they will keep standing there?

[Doru Barbu (barbu.doru-deactivatedaccount)]

I can also confirm this behaviour, the scattered "system" blocks cause fragmentation by limiting the amount of continuous free space. They don't seem to dissapear after deleting the files, but they where definitely created after copying stuff from ubuntu with ntfs-3g. I will do a metadata image as soon as possible.

[Michael Fritscher (michael-fritscher)]

Fixed, in 2009.03.08, in #349569 there is also a solution to get rid of these system blocks.

[Thomas Hotz (thotz-deactivatedaccount)]

Marking as fixed.