wipe_inodes by itself blows up NTFS MFT

[Andrew Ziem]

BleachBit 1.1 git: when running wipe_inodes on an empty disk, the MFT grows and isn't shrunk by deleting the files and exiting BleachBit.

Windows XP
NTFS

[Andrew Ziem (ahziem1)]

Temporarily resolved in 1.1/1.2 by disabling wipe_inodes

[verdy_p (verdy-p)]

This is normal behavior of NTFS: Usually the MFT is not shrunk automatically because inodes in the MFT normally never move: if a single file is added at end of the MFT on, an inode number with high value, and this file is not deleted, all other previous empty inodes will remain.

When you "wipe inodes" on NTFS, what you do is that you create tons of dummy inodes (for empty temporary files) and then other concurrent processes that need to create or update their own files (e.g. log files, or NTFS journal extents, or special directories like "$Deleted" that will need cleanup after reboot), or need to add extended attributes (e.g. adding fragmentated extents, or adding space in a bitmap or index) will cause the MFT size to be increased (as there's no more free inode in the MFT). These extended attributes will also keep their size and cannot be defragmented on disk (NTFS treat these EA as unmovable, both those that are "resident" in the MFT, and these attributes stored externally if they don't fit in the 1KB inode of the MFT itself).

There's little we can do: we should ask to Microsoft to updat its NTFS driver so that it can use a background worker to track all indexes referencing an inode, then lock that inode and virtualize it to relocate it elsewhere in the MFT, then perform a transaction that will update all indexes with the new location of the inode, then unlock the old inode to erase it. This should "compact" the MFT to its used area and then would allow freeing the unused blocks at end of the MFT.

Note that the MFT also has a minimum size on the volume, but there's nothing in NTFS that will make sure that there won't be megabytes of consecutive free inodes, and nothing to freeup the unused space inside extended attributes (notably those that are not resident in the MFT due to their size), and in commited and terminated transaction files: they keep their allocation permanently and fragmentation of the volume then grows over time and can only be solved by "offline" management (where you'll backup the volume into a new VHDX to preserve all security and attributes contents, then reformat it, then restore the content of the volume, ignoring their former cluster or inode location in the original volume).

As Windows 10 now fully supports virtualized filesystems, I'm convinced that this should even be doable online (using a virtual cache in RAM while creating a new separate NTFS structure on disk and moving extents in the background to recompact the NTFS volume).

For now what I can do: perform an system image backup (this creates a VHDX file) to another volume.
Then reboot Windows from the DVD or an USB key in RAM. In that BootPE environment, unmount the disk partition using diskpart (REMOVE C:) delete the partitions, then restore the system image: it will recreate the filesystem and its structure in a much cleaner way, and much less fragmented, and the MFT will just have the desired size. Finally perform an offline defragmentation (of folders). Then reboot...

(Note that the MFT is also defragmentable online (as opposed to what the doc says!): but it necessarily uses 2 extents (one for the "\$MFT" itself, another for its mirrorable leading are...

[Andrew Ziem (ahziem1)]

I am closing because at the moment there is not a path forward. Also, I am moving issues from Launchpad to GitHub