Files modified on NTFS partition are seen as corrupted files in Windows

[Karim Sonbol]

I have Ubuntu 13.10 and Windows 8.1 installed, both 64-bit (dual boot). Whenever I create or copy files to any of the NTFS partitions from Ubuntu, these files get deleted once I login to Windows. When I login to Ubuntu again, the files are not there. I shut down Windows properly and do not hibernate. I have disabled fast startup option in Windows, and Fastboot from BIOS. So it is not a hibernation problem. The same problem occurred when using Linux Mint.

I have been able to find the exact event log concerning this issue in Windows Event Viewer, it says: "The file system structure on volume D: has now been repaired." and in the details section:

EventData
  VolumeIdLength 2
  VolumeId D:
  RepairDetail 25008: Start repair on 02/05/2014 at 20:53:12:946 25017: Processing repair verb IndexEntry: 0x5000000000005, "$I30", "line-mozart.avi" Flags: 0x32, 0x0 26065: Deleting index entry line-mozart.avi in index 0x5000000000005 of file 0x2000000003498. 25009: End repair on 02/05/2014 at 20:53:12:946
  RepairDataLength 152
...
Note: "line-mozart.avi" is the name of the file that got deleted this time.

Since Windows sees the modified files as corrupted files that need repair, I think Ubuntu does something wrong when saving files to NTFS partitions, and it is probably a bug in NTFS-3g.

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: ntfs-3g 1:2013.1.13AR.1-2ubuntu1
ProcVersionSignature: Ubuntu 3.11.0-15.23-generic 3.11.10
Uname: Linux 3.11.0-15-generic x86_64
ApportVersion: 2.12.5-0ubuntu2.2
Architecture: amd64
Date: Sat Feb 15 14:07:58 2014
InstallationDate: Installed on 2014-01-17 (28 days ago)
InstallationMedia: Ubuntu 13.10 "Saucy Salamander" - Release amd64 (20131016.1)
MarkForUpload: True
SourcePackage: ntfs-3g
UpgradeStatus: No upgrade log present (probably fresh install)

[Jean-Pierre (jean-pierre-andre)]

> Whenever I create or copy files to any of the NTFS partitions from Ubuntu, these files
> get deleted once I login to Windows.

Does this also happen to files on external devices (such as a usb key) ?

> I have been able to find the exact event log concerning this issue in Windows

A file has also probably been created in "Win/System Volume Information/Chkdsk"
(Replace Win by your actual mount point, use quotes or escape the spaces)
which you can determine from its time stamp (02/05/2014 at 20:53:12:946).
Please post it.
Note : this is a UTF16-encoded file, you may have to post it as a binary file.

> I shut down Windows properly and do not hibernate. I have disabled fast startup
> option in Windows, and Fastboot from BIOS.

When you start Ubuntu after having logged off Windows normally, what is the
output of :
sudo head -c 30 'Win/$LogFile' | tail -c 4 | od -t x2
(Replace Win by your actual mount point, keep the quotes or escape the '$')

What does the same command return when you switch to Ubuntu from
Windows through a "restart" instead of "shut down" ?

> I think Ubuntu does something wrong when saving files to NTFS partitions,
> and it is probably a bug in NTFS-3g.

And a possible cause is Microsoft having changed the rules unilaterally.

[Karim Sonbol (karim-sonbol279)]

> Does this also happen to files on external devices (such as a usb key) ?
No, I copied a file to my USB flash drive and was able to access it in Windows and copy it to an NTFS partition.

> A file has also probably been created in "Win/System Volume Information/Chkdsk"..
File attached.

> When you start Ubuntu after having logged off Windows normally, what is the
> output of :
> sudo head -c 30 'Win/$LogFile' | tail -c 4 | od -t x2
> (Replace Win by your actual mount point, keep the quotes or escape the '$')
>
> What does the same command return when you switch to Ubuntu from
> Windows through a "restart" instead of "shut down" ?

The output in both cases is:
0000000 0001 0001
0000004

[Jean-Pierre (jean-pierre-andre)]

Thank you for your report.

The file you attached is probably not the right one. It was apparently created on Jan 18th, whereas the strange repair you reported was on Feb 5th. The one you posted might be a consequence of resizing the ntfs partition from 400GB to 126GB, it is not related to the issue you reported. The useful one is approximately named Chkdsk201402052053.... (it is named after the date and time it was created).

The data from the LogFile is supposed to mean Windows was not ready to be fast restarted (unless Microsoft changed the rules).

Also please post the output of (replacing sdxx by your partition identification) :
sudo ntfsinfo -fm /dev/sdxx

According to http://ubuntuforums.org/showthread.php?t=2199632 this problem only occurs in the root directory. Can you make a try in an inner directory ?

[Karim Sonbol (karim-sonbol279)]

Unfortunately, this is the only file available at OS/System Volume Information/Chkdsk. The problem occurs anywhere on the partition, not only in the root directory, and for any change in files. For example, I renamed an .mp4 file in an inner directory in the partition from Ubuntu, and when I logged into Windows, the file was corrupt and didn't play. When I opened Ubuntu again, it was fine !

This is the output of the command:

Forced to continue.
Volume Information
 Name of device: /dev/sda5
 Device state: 11
 Volume Name: Data
 Volume State: 91
 Volume Flags: 0x0000
 Volume Version: 3.1
 Sector Size: 512
 Cluster Size: 4096
 Index Block Size: 4096
 Volume Size in Clusters: 127999998
MFT Information
 MFT Record Size: 1024
 MFT Zone Multiplier: 0
 MFT Data Position: 24
 MFT Zone Start: 786432
 MFT Zone End: 16786431
 MFT Zone Position: 786432
 Current Position in First Data Zone: 16786431
 Current Position in Second Data Zone: 0
 Allocated clusters 21632 (0.0%)
 LCN of Data Attribute for FILE_MFT: 786432
 FILE_MFTMirr Size: 4
 LCN of Data Attribute for File_MFTMirr: 2
 Size of Attribute Definition Table: 2560
 Number of Attached Extent Inodes: 0
FILE_Bitmap Information
 FILE_Bitmap MFT Record Number: 6
 State of FILE_Bitmap Inode: 80
 Length of Attribute List: 0
 Number of Attached Extent Inodes: 0
FILE_Bitmap Data Attribute Information
 Decompressed Runlist: not done yet
 Base Inode: 6
 Attribute Types: not done yet
 Attribute Name Length: 0
 Attribute State: 3
 Attribute Allocated Size: 16003072
 Attribute Data Size: 16000000
 Attribute Initialized Size: 16000000
 Attribute Compressed Size: 0
 Compression Block Size: 0
 Compression Block Size Bits: 0
 Compression Block Clusters: 0
 Free Clusters: 29422008 (23.0%)

[Anton Altaparmakov (aia21)]

Have you got any 3rd party security related software / antivirus software installed that might be protecting the volume against modifications? If yes what happens if you turn it off?

Have you got any of the change tracking features enabled in Windows? If yes what happens if you turn them off?

Perhaps most importantly, would you be happy to create and send us metadata images of the volume in question? Probably most useful to have a "post ntfs-3g write" and "post windows boot after ntfs-3g write". We can give you instructions how to gather the metadata images - note those would expose the names of all your files/directories but none of the content (except for metadata related files).

Best regards,

Anton

[Anton Altaparmakov (aia21)]

btw. The fact that this works on usb key but not on Windows main volume suggests ntfs-3g is not doing anything wrong as such but that something in Windows is guarding against external changes to the system volume.

Do you by any chance have an SSD or hybrid drive that Windows might be using as a boot cache or some such?

[Karim Sonbol (karim-sonbol279)]

@Anton Altaparmakov

> Have you got any 3rd party security related software / antivirus software installed that might be protecting the volume against
> modifications? If yes what happens if you turn it off?
Yes, I have an antivirus and I already tried turning it off with no luck.

> Have you got any of the change tracking features enabled in Windows ? If yes what happens if you turn them off?
I don't know what you mean by "change tracking features", could you explain ?

> Perhaps most importantly, would you be happy to create and send us metadata images of the volume in question? Probably
> most useful to have a "post ntfs-3g write" and "post windows boot after ntfs-3g write". We can give you instructions how to
> gather the metadata images - note those would expose the names of all your files/directories but none of the content (except
> for metadata related files).
I have three NTFS partitions (besides the Recovery and Restore partitions) and they all have the same problem, do you want the metadata images for any one of them ? and will that be very helpful ? If yes, please tell me the instructions

> Do you by any chance have an SSD or hybrid drive that Windows might be using as a boot cache or some such?
I have a 1 TB hard drive with a 24 GB SSD cache.

[Jean-Pierre (jean-pierre-andre)]

I have a few more questions after having read your replies to Anton.

First, I would like to check whether there is a difference from Windows 8 in processing the "fast restart mode". Please turn this feature back on, then log off Windows (to be ready for a fast restart), boot into Ubuntu and show the result of :
sudo head -c 30 'Win/$LogFile' | tail -c 4 | od -t x2

Then please provide information about your storage and SSD cacheing. I have a computer on which I had to disable the SSD cacheing, because it is used differently by Windows and Linux. I applied the procedure under "Remove fake raid" on
http://askubuntu.com/questions/159645/dual-boot-installation-of-ubuntu-12-04-lts-on-hp-ultrabook-envy-4-1002tx
The "sudo lspci -v" output related to your storage controller may also be useful.

I interpret the "change tracking features" mentioned by Anton as a configuration in which your storage is backed up on some external server (such as a mirror on the cloud).

Finally, for providing a metadata image, better select a partition with the smallest number of files, otherwise the image might be too big. You may also prefer a partition hosting files whose names are not confidential.
You can built such an image by (replace sdxx by relevant partition id, it must not be mounted, and bzip2 would be better than gzip, but it is much slower) :
sudo ntfsclone -mst -O - /dev/sdxx | gzip > sdxx.image.gz
The resulting file is probably too big for being attached to this issue tracker or sent by mail. Uploading to a public server may be required.

[Karim Sonbol (karim-sonbol279)]

Enabling fast restart back on prevents me from mounting the partitions, giving this error message:
Unable to mount OS
Error mounting /dev/sda4 at /media/karim/OS: Command-line `mount -t "ntfs" -o "uhelper=udisks2,nodev,nosuid,uid=1000,gid=1000,dmask=0077,fmask=0177" "/dev/sda4" "/media/karim/OS"' exited with non-zero exit status 14: Windows is hibernated, refused to mount.
Failed to mount '/dev/sda4': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

Output of "sudo lspci -v" is attached in "lspci.txt" file.

> Then please provide information about your storage and SSD cacheing. I have a computer on which I had to disable the SSD
> cacheing, because it is used differently by Windows and Linux. I applied the procedure under "Remove fake raid"
I hope I can fix this without disabling the SSD caching, but if that's the only solution I will do it !
BTW, is it safe to run the command described "sudo dmraid -E -r /dev/sda" ?

> I interpret the "change tracking features" mentioned by Anton as a configuration in which your storage is backed up on some
> external server (such as a mirror on the cloud).
My storage isn't backed up on any server. The only thing backed up is the dropbox folder.

I will create the metadata image and get back to you.

[Jean-Pierre (jean-pierre-andre)]

> Enabling fast restart back on prevents me from mounting
> the partitions, giving this error message:

That is the expected consequence of enabling the "fast restart" mode. It is a good point which shows ntfs-3g correctly detects the mode, and Windows 8.1 appears to behave like Windows 8. So I discard a fast restarting cause.

> BTW, is it safe to run the command described "sudo dmraid -E -r /dev/sda" ?

Do not issue this command : the output of lspci shows your SATA controller is in AHCI mode (which is the normal mode, mine was in RAID mode).

You mentioned cacheing on a SSD, and I see no clue this to be done by the hardware. Where did you get the information about the SSD being used for cacheing ? It is probably configured at the Windows level (I have no idea where to look at in Windows, use your favorite search engine).

> I will create the metadata image and get back to you.

Ok.

[Karim Sonbol (karim-sonbol279)]

> You mentioned cacheing on a SSD, and I see no clue this to be done by the hardware. Where did you get the information about
> the SSD being used for cacheing ? It is probably configured at the Windows level (I have no idea where to look at in Windows, use
> your favorite search engine).

It's a 1 TB HDD with another 24 GB SSD used as a cache. The notebook specs says that, and Gparted listing two disks of these sizes confirms that. How the caching is done, however, is something I don't know. All I know is that this SSD is used to cache recently opened applications and so on.

Metadata image attached.

[Karim Sonbol (karim-sonbol279)]

I forgot to say that when I first ran the ntfsclone command it gave me the following error (although I logged into Ubuntu after a restart from Windows):
ntfsclone v2013.1.13AR.1 (libntfs-3g)
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 229999992832 bytes (230000 MB)
Current device size: 230000000000 bytes (230000 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Cluster accounting failed at 30036592 (0x1ca5270): extra cluster in $Bitmap
Cluster accounting failed at 30063507 (0x1cabb93): extra cluster in $Bitmap
Cluster accounting failed at 32118944 (0x1ea18a0): missing cluster in $Bitmap
Cluster accounting failed at 32225272 (0x1ebb7f8): missing cluster in $Bitmap
Cluster accounting failed at 32229368 (0x1ebc7f8): missing cluster in $Bitmap
Cluster accounting failed at 32233464 (0x1ebd7f8): missing cluster in $Bitmap
Cluster accounting failed at 32241656 (0x1ebf7f8): missing cluster in $Bitmap
Totally 7 cluster accounting mismatches.
ERROR: Filesystem check failed! Windows wasn't shutdown properly or inconsistent
filesystem. Please run chkdsk /f on Windows then reboot it TWICE.

and running chdsk on Windows as told I got:
C:\Windows\system32>chkdsk /f F:
The type of the file system is NTFS.
Volume label is Work.

Stage 1: Examining basic file system structure ...

  98560 file records processed.

File verification completed.

  4 large file records processed.

  0 bad file records processed.

Stage 2: Examining file name linkage ...

  132348 index entries processed.

Index verification completed.
CHKDSK is scanning unindexed files for reconnect to their original directory.

  2 unindexed files scanned.
Recovering orphaned file test (98552) into directory file 40399.
CHKDSK is recovering remaining unindexed files.

  1 unindexed files recovered.

Stage 3: Examining security descriptors ...
Security descriptor verification completed.

  16895 data files processed.
CHKDSK is verifying Usn Journal...
Usn Journal verification completed.
Correcting errors in the master file table's (MFT) BITMAP attribute.
CHKDSK discovered free space marked as allocated in the volume bitmap.

Windows has made corrections to the file system.
No further action is required.

 224609368 KB total disk space.
 128978408 KB in 81577 files.
     31476 KB in 16897 indexes.
         0 KB in bad sectors.
    171380 KB in use by the system.
     65536 KB occupied by the log file.
  95428104 KB available on disk.

      4096 bytes in each allocation unit.
  56152342 total allocation units on disk.
  23857026 allocation units available on disk.

then I restarted twice and ran the command and it gave me the image I attached earlier.

Hope this information helps !

[Jean-Pierre (jean-pierre-andre)]

First, what is the model of your hard disk ? The backup boot
sector is located at a place unusual for a 512-sector disk.

Then, you have built the image after having started chkdsk twice,
nevertheless when I start chkdsk on the image, it still finds an
error. The error is about directory Django/GSWD/projects/old_microblog
which is just named microblog in the parent directory. This appears
to be a directory created on Windows as microblog, which you renamed
as old_microblog on Linux. I am surprised this was not fixed by
chkdsk on your computer.

The missing clusters found by chkdsk on your computer are related
to some earlier fixing of files created on Ubuntu. The first one
is about found.001/dir_00000000.chk which was obviously created
by Windows. The last three ones are related to git files such as
found.001/dir_00000000.chk/.git/objects/f9/cf35626bdb4bcbbb5....
but other files apparently created by the same process were not
flagged (such as found.001/dir_00000000.chk/microblog/settings.pyc)

It looks like Windows and Ubuntu do not read the same data.

Can you create on Windows a file a bit over 4096 bytes (for
example 5000 bytes), then switch to Ubuntu and *append* a few
bytes (for example "echo more data >> testfile"). Do not use
a text editor which would relocate the file after updating.
Check the time stamp of the file while on Ubuntu. Then switch
back to Windows. Is the file visible ? Has it been moved to a
quarantine directory (found.001) ? What is its time stamp, what
contents ?
(better use a partition different from the one for which you
sent the metadata, so that it remains meaningful).

> The notebook specs says that, and Gparted listing two disks
> of these sizes confirms that.

Can these specs be accessed on-line ?
I assume Gparted does not mention the SSD is used as a cache.

[Karim Sonbol (karim-sonbol279)]

> Is the file visible ? Has it been moved to a quarantine directory (found.001) ?
> What is its time stamp, what contents ?
The file is visible. But it is the old version before changes in Ubuntu. The time stamp has NOT been changed. Opening the file in Ubuntu again gives the same results.

> The error is about directory Django/GSWD/projects/old_microblog
> which is just named microblog in the parent directory.
The directory is seen now as "microblog" in both Windows and Ubuntu.

> what is the model of your hard disk ?
WDC

[Karim Sonbol (karim-sonbol279)]

Sorry, I posted before completing the comment by mistake.

> what is the model of your hard disk ?
WDC WD10PVX-80JC3T0 (s a Western Digital drive)
and the SSD drive:
SanDisk SSD U100 24 GB

> Can these specs be accessed on-line ?
> I assume Gparted does not mention the SSD is used as a cache.
I couldn't find anything yet, but it's a 24 GB SSD that is only visible in disk management utilities, and not visible in Windows Explorer. What else can it be used for ?

[Jean-Pierre (jean-pierre-andre)]

So apparently, Windows and Ubuntu do not read the same data, which point at a cache somewhere not synced when shutting down.

Here are a few URL pointing at a tool from Intel to configure a SSD as a cache.
http://www.pugetsystems.com/labs/articles/How-it-Works-Intel-SSD-Caching-148/
http://www.pcworld.com/article/248828/how_to_setup_intel_smart_response_ssd_caching_technology.html
http://download.intel.com/support/chipsets/sb/intel_smart_response_technology_user_guide_3.pdf

Can you use this tool to determine whether a SSD cache has been configured ? I would recommend not to change the current setting, as I do not know the consequences on the installed Windows.

[Karim Sonbol (karim-sonbol279)]

OK, so after some research I found the following:
I have SSD caching enabled in Windows, but through a program called ExpressCache not Intel RST. It comes bundled with Sandisk SSDs.
https://www.condusiv.com/partners/oem/technologies/expresscache/
http://www.sandisk.com/products/software/express-cache/

It seems that ExpressCache does not require the SATA configuration mode be set to RAID.
https://forums.lenovo.com/t5/tkb/articleprintpage/tkb-id/ideaPad@tkb/article-id/424

I've attached a screenshot of the cache info for ExpressCache on my machine.

I also installed Intel RST software and there was no option to "accelerate", probably because there is no RAID mode. Strangely enough, Intel RST driver is also included in the drivers page for my laptop.

Anyway, I think what I can try now is uninstall ExpressCache or stop SSD caching somehow and see if this fixes the problem and maybe reinstall it afterwards. However, I don't know if this could break the system in any way. And if I disabled SSD caching, will I be able to use the SSD as storage device ?

[Jean-Pierre (jean-pierre-andre)]

> Anyway, I think what I can try now is uninstall ExpressCache or
> stop SSD caching somehow and see if this fixes the problem and
> maybe reinstall it afterwards.

Looks reasonable. I have no experience with this.
Please share your findings.

> And if I disabled SSD caching, will I be able to use the SSD as
> storage device ?

A SSD can be partitioned and used the same way as a hard disk. You can make a
8 or 10GB partition to hold the Linux root directory (keeping /var, /home and
swap on the hard disk). The remaining space can probably be used for Windows
utilities or some system subsets.

[Karim Sonbol (karim-sonbol279)]

That was it ! I uninstalled ExpressCache and the problem was gone, so was SSD caching. I tried re-installing it but the problem occurred again. I think I will have to give up SSD caching :(

Thanks a lot Jean and Anton for your help, I really appreciate it. One last request, If you know any way to make SSD caching work with dual booting, please let me know.

[Jean-Pierre (jean-pierre-andre)]

Glad you made it.

> any way to make SSD caching work with dual booting

IMHO that would require either :

- a cacheing software which exists in both Windows and Linux variant,
- or which can be customized to cache the Windows system partition only,
- or which can be customized to sync the SSD to HD on shutdown.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[drkwv (bl3-drkwv)]

I'd like to be able to disable ntfs caching at all. Because when I copy files on a USB drive I want to see real progress bar, not the fake and fast one. Because this is what a progress bars actually for. Is this possible?

[András Korn (kornandras)]

@drkwv, your question is off-topic (not related to the issue discussed here). However if you mount your usb stick with they 'sync' option, then writes will not be buffered and your progress bar will also show realistic progress for small writes that would otherwise be buffered.

[Aroun (aroun-olorin)]

Hello, I confirm I had the same problem using Windows 8.1 x64 and Linux Mint 17.
I also have ExpressCache installed to faster my Windows system.
I found that the eccmd command on Windows had an exclude option to exclude a partition to be cached.
So I typed "eccmd -exclude d" so that it only caches my system partition.
It does not print any confirmation message so I just guess it worked.
Then on my linux system, I mount the d drive with rw rights and the c drive with ro.
Seems to work fine so far.
Hope this can help.