Using special characters in filenames prevents Windows from opening

[Evan Huus]

Currently, the only characters that Ubuntu doesn't support in filenames and paths are slashes, while Windows has a considerable list (apostrophes, asterisks, etc).

Naming a file on a Windows drive in Ubuntu with one of these characters causes Windows to be unable to read the file and anything beneath it in the directory structure.

Ubuntu needs to automatically strip these characters (or prompt for a rename) when creating/moving/renaming files on NTFS drives.

[Richard Seguin (sectech)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please answer these questions:

* Is this reproducible?
* If so, what specific steps should we take to recreate this bug?

This will help us to find and resolve the problem.

What package is this in relation to, Samba?

[Richard Seguin (sectech)]

* Marked as incomplete pending enough information to complete triage

[Evan Huus (eapache)]

It is reproducible on XP. I do not have a Vista machine to test with.

On a NTFS partition, use Ubuntu to name a folder with one of Windows' forbidden characters.
Try to open it with Windows.

[Evan Huus (eapache)]

I'm not sure which package to link it to. Ntfs-3g? Nautilus?

What handles other forbidden characters (like slash)?

[Andrew Sayers (andrew-bugs-launchpad-net)]

Would you mind trying this with an ext2 or ext3 filesystem using one of the ext2-on-Windows drivers, such as http://www.fs-driver.org/ ? That would show whether this is a filesystem issue or an operating system issue.

[Evan Huus (eapache)]

This issue also occurs with ext2 partitions and the fs-driver.org driver.

Definitely an OS issue not a filesystem issue.

[Andrew Sayers (andrew-bugs-launchpad-net)]

A bit of googling and testing shows that the null character is invalid in ext2, and that it's silently removed from files:

$ echo > ~/foo$'\0'bar
$ ls ~/foobar
/home/andrew/foobar

If these are actually valid characters in the filesystem, it seems inappropriate to call this a "bug" anywhere in Linux. What if my digital camera only supports FAT but I want to create a file "Me & my friends.jpg"?

Reading the open(2) man page, it's not clear to me that the kernel has any concept of rejecting illegal filenames: the null character is silently discarded and '/' is treated as a directory separator. However, mount(8) shows that there's a "uni_xlate" option for vfat which translates Unicode characters that can't be represented in a FAT filesystem.

If you don't mind rousing the kernel gods, this problem could be fixed by adding the specified characters to those that uni_xlate handles, and by adding uni_xlate to NTFS-3G. This has the advantages that it's relatively transparent to the user and doesn't create weird side-effects like filename collisions. However, it needs to be explicitly turned on, and doesn't help with the (admittedly rare) case of people wanting to read funnily-named files in ext2-on-Windows FSs.

Alternatively, bugs could be filed against all of the major graphical filesystem browsers to add a "warn on Windows-illegal characters" mode. One problem with this is that it creates annoying "are you sure?" boxes that people will ignore then complain about the non-existence of. Another problem is that there will have to be a line drawn between programs that support this feature and programs that don't - it'll be a long time before `cp` gets such a feature, for example. Users crossing that line will face significant confusion as programs on the other side of the line do the wrong thing (from that user's point-of-view).

[Przemek K. (azrael)]

vfat doesn't have an uni_xlate option, ntfs does.
vfat has check=...

[Milan Bouchet-Valat (nalimilan)]

We should do with those invalid chars just like the vfat module is doing with invalid (on VFAT) Unicode chars. It's better to replace them silently than to prevent Windows from reading them. At least, GVFS could manage the UI part, but this may be a little tricky...

[Richard Seguin (sectech)]

 Thanks for reporting this bug and any supporting documentation. Since this bug has enough information provided for a developer to begin work, I'm going to mark it as confirmed and let them handle it from here. Thanks for taking the time to make Ubuntu better!

[Szabolcs Szakacsits (szaka)]

http://ntfs-3g.org/support.html#posixfilenames2

Why does the driver allow special characters in the filenames?

    NTFS supports several filename namespaces at the same time: DOS, Win32 and POSIX. While the NTFS-3G driver handles all of them, it always creates new files in the POSIX namespace for maximum portability and interoperability reasons. This means that filenames are case sensitive and all characters are allowed except '/' and '\0'. This is perfectly legal on Windows, though some application may get confused. If you find so then please report it to the developer of the relevant Windows software.

    Workaround: If case insensitivity handling and/or restriction of special character usage is desirable then you may export the NTFS volume via Samba which supports this functionality the same way as it does for other POSIX file systems.

    Status: Not ntfs-3g problem.

[Evan Huus (eapache)]

The Support page on the NTFS-3G website doesn't make a whole lot of sense to me. This isn't a case of certain applications on Windows not supporting the special characters, this is a case of the entire filesystem layer on Windows not supporting these characters.

While it is technically not the fault of the NTFS-3G driver, it can still cause problems for the uninformed user who just wants to share files between OSes.

Whatever mechanism we use to prevent invalid vfat characters, it should be added to prevent invalid win32 chars on NTFS.

[Gabe Gorelick (gabegorelick)]

@eapache ever since XP, Windows has supported the notion of "subsystems" within the operating system. This allows support of, for example, a POSIX subsystem within Windows. In theory, this is a useful feature, but really almost no one uses any other subsystem except for the standard Windows NT one. So ntfs-3g can't make the assumption that a certain character is invalid for a filename, since even if it's invalid in one subsystem, it may not be in another.

The best way to fix this bug would be to add some kind of check in nautilus (or dolphin, etc.) so that if it detects you are trying to write a file to an ntfs filesystem, it warns you. This could be expanded to support other filesystems as well. For example, if you have some filesystem for BeOS mounted that doesn't support some random character, nautilus can warn you before you write to it. This would be kind of a big fix though. Anyone want to draw up a blueprint or do a brainstorm for this or otherwise take the lead?

[Rygle (rygle)]

Some good news on this bug. I put up a comment on the NTFS-3G forums and got a very positive reply from Jean-Pierre, one of the lead programmers for NTFS-3G;

I asked...
[QUOTE]
Could you please include a switch in NTFS-3G to allow a workaround for this Windows limitation? (i.e. to turn off use of special characters in order to allow compatibility with Windows for the many users who wouldn't have a clue how to re-mount/export their NTFS filesystems using Samba)
[/QUOTE]

And Jean-Pierre replied...
[QUOTE]
This is available in the release candidate advanced ntfs-3g-2010.5.16AR.1, see http://pagesperso-orange.fr/b.andre/advanced-ntfs-3g.html
The option to use is "windows_names", documented in the manual of the said version.
[/QUOTE]

Hopefully this will help a lot in this area, but it would need to be integrated into Ubuntu by someone cleverer than me...

This discussion can be found on the Tuxera site - http://tuxera.com/forum/viewtopic.php?f=2&t=763&p=14549&sid=053f5d02fb19e22aeda1019f6225c76d#p14549

[Rygle (rygle)]

BTW, this issue is possibly a duplicate of another Ubuntu bug. I have also commented about this development there.
https://bugs.launchpad.net/ubuntu/+source/rhythmbox/+bug/318625

[Rygle (rygle)]

Could someone please enable the 'windows_names' switch as the universal default option for mounting NTFS file systems in Ubuntu? I think this would be a good step forward for Ubuntu that is very easy to enable.

[Brian Eaton (brian-speakingincode)]

I like Rygle's proposed solution. Compatibility is a good thing here.

I'd also suggest adding mapchars,iocharset=utf8 as default options for CIFS mounts for the same reason.