ntfs treated by default as POSIX

[Nicolò Chieffo]

Binary package hint: ntfs-3g

version 1:1.616-1

The ntfs filesystem in a default windows istallation has some forbidden characters such as
: \ ?
and others (I don't have the list)
and is not case sensitive

Using ntfs3g it is possible to create files with this names. Those files cannot be read in windows, without Windows Services for Unix.

We are evaluating if ntfs-3g should behave like the old vfat driver that forbids the creation of these files, or should stay by default POSIX.
Anyway I think that ntfs-3g needs an option to control this behaviour

NOTE: microsoft developer ntfs as a POSIX filesystem for interoperability, but added these restrictions in windows to support legacy apps, and because they think that accessing a case sensitive filesystem is a security risk, because the user could execute the troyan EDIT.exe instead of edit.exe (I do not really understand this, but it is reported in the help of Windows Services for Unix)
A windows installation with Windows Services for Unix is not able to manage files which only differ by case, using normal software such as explorer. The appropriate programs installed by WSfU must be used.

[Szabolcs Szakacsits (szaka)]

From ntfs-3g upstream: this bug report is invalid.

NTFS has several file namespaces: DOS, WIN32, POSIX. Linux, as a POSIX OS,
always creates files in the POSIX namespace. Characters in the file name
can be anything except '/' and '\0'. These files are accessible via the WIN
API on Windows only if they don't have "forbidden" characters. To handle
these files on Windows, the same way as on Linux, one needs to install the
Windows Services for Unix (SFU) freely available Microsoft package.

In sort, everything behaves as it was designed. No problem on Linux and all
the files are accessible if SFU is installed on Windows.

If a non-posix file namespace handling is desirable on Linux then I suggest
exporting the NTFS volume via Samba which should have support for
restricting the file namespace to the WIN32 or DOS one.

Thank you for the report, the question will be added to the ntfs-3g support
page.

[Nicolò Chieffo (yelo3)]

I've seen this on the ntfs-3g page, but I don't understand why ntfs
can be accessed in a POISIX way, since it is a filesystem designed for
windows. What are the advanteges? (since people have always used the
vfat filesystem to share files, which is non POSIX at least by
default?). This is an unexpected change! And finally what are the
disadvantages of using the filesystem in the WIN way?

On the other hand I'm glad to discover that there is a fix for
windows. BUT this should *really* be told to the user who uses
ntfs-3g, especially now that there is the spec to add it by default.

Doesn't using samba slow down the acces? I've seen lots of apps that,
before accessing the file, copy it to the local cache, which is a very
"expensive" operation for big files

Finally, I think that the user should have the possibility to choose
(maybe through ntfs-config ?) the first time he uses this driver.

He should be told that using it in the POSIX way, files will be case
sensitive, some characters such as : ? \ and others that previously
were not allowed on the filesystem now are allowed, and that they have
to download and install <B>Windows Services for Unix</B>
And he should be told that if he wants to access the filesystem in the
standard way, there are these restrictions in the filename, and the
case will be non sensitive

[Nicolò Chieffo (yelo3)]

Which component of Windows Services for Unix will enable POSIX
read/write? I would like to install only the needed files to not
waster space and memory

[Szabolcs Szakacsits (szaka)]

On Sat, 7 Jul 2007, Nicolò Chieffo wrote:

> I've seen this on the ntfs-3g page, but I don't understand why ntfs
> can be accessed in a POISIX way, since it is a filesystem designed for
> windows.

No, you're misunderstanding the issues here. NTFS is designed
to be POSIX:
http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/core/fncc_fil_koez.mspx?mfr=true

"Forbidden" characters and case sensitivity is perfectly legal on NTFS.

Your wiki update for WriteSupportForNTFS is misleading. It suggests ntfs-3g
doesn't use NTFS the correct, legal way which is not true. In fact,
Windows restricts access to these files for legacy reason, namely VFAT
can't support these.

> What are the advanteges?

POSIX. You can use it on any other POSIX OS for interoperability. This is
why NTFS was designed to be POSIX. And indeed, ntfs-3g runs fine on Linux,
OS X, FreeBSD, NetBSD, etc.

> (since people have always used the vfat filesystem to share files, which
> is non POSIX at least by default?).

VFAT isn't POSIX at all. NTFS intentionally solved this problem, so one can
use it in its full capacity.

> This is an unexpected change!

No, it's not. Microsoft intentionally made NTFS POISX. There are
governmental and industrial regulations which requires POSIX
for certain solutions and without POSIX compliance Microsoft
couldn't have made important business deals.

> And finally what are the disadvantages of using the filesystem in the WIN
> way?

Quite many things would break and users would be confused why things
don't work.

The question is what is better? Breaking many Linux and other POSIX OSes
applications or fix Windows where the fix is already available?

> On the other hand I'm glad to discover that there is a fix for
> windows. BUT this should *really* be told to the user who uses
> ntfs-3g, especially now that there is the spec to add it by default.

This is Microsoft's job. It's their problem, they restrict users, they need
to solve it.

> Finally, I think that the user should have the possibility to choose
> (maybe through ntfs-config ?) the first time he uses this driver.

Then even more people would be confused why so many programs wouldn't work
on Linux due to these restrictions. Please note that you can't really refer
to VFAT because they are quite different file systems and even Microsoft
stopped supporting it as the main file system, due to very good reasons.

> He should be told that using it in the POSIX way, files will be case
> sensitive, some characters such as : ? \ and others that previously
> were not allowed on the filesystem now are allowed, and that they have
> to download and install <B>Windows Services for Unix</B>
> And he should be told that if he wants to access the filesystem in the
> standard way, there are these restrictions in the filename, and the
> case will be non sensitive

Microsoft always toldl these to the Windows users. I can't see any new or
surprising here. Everybody is keep learning, that's all.

[Nicolò Chieffo (yelo3)]

Well in my opinion you are talking as a very very advanced user, and
I'm instead trying to be near to a normal user which dual boots in
windows and wants his files available to both windows and ubuntu.
(this is what the spec is aimed for, read Use Cases)

I understand that windows forbidden characters is legal in the NTFS
filesystem at all. But they are not legal in a normal windows
installation, and a normal user does not know what's happening, since
these files (without WinSerForUnix, and the normal user does not even
know that this software exists and what it is for) cannot be read.

I updated my wiki comment to be less misleading, but I think this is
still a warning that cannot be ignored

You're talking as most applications will not work if the filesystem is
not POSIX. I instead think that the problem is different and we could
try to answer together to these questions to have a better idea of the
most frequent situation:
1) which partition is normally ntfs?
My answer: the partition where windows resides and maybe another
partition in which documents, videos, music, downloads are stored. For
sure not / or $HOME...
2) which applications are normally used over a ntfs partition?
My answer: openoffice, gedit, totem, nautilus, rhythmbox, firefox
(vim, cat, less, diff, <, > for more advanced users). I cannot think
of other apps now. what would not work in these applications if the
filesystem is not POSIX? Tell me if I forgot something.

Maybe it's not a Microsoft problem... Since we're using their
filesystem, we should do the best to not confuse the normal user

I told you it was an unexpected change and I was using vfat as a
reference because (I think) most normal users were using vfat to
exchange files, and now that ntfs3g is out will for sure switch to
this. But (I think) they expect it works the same way as vfat, because
in windows there is no difference, and because they know they are
using a windows filesystem which has restrictions in windows default
installation. If it behaves differently they might face lots of
problems and most users will not understand which is the problem.

My opinion is that there will be more people confused because files
are not read by windows than the number of people confused because
some apps don't work (since I cannot really think of an application
that does not work).

At finally, Microsoft doesn't tell normal users that the filesystem
restrictions are only for legacy, and that cannot be removed. The user
has to discover this by itself. (this was my case, for example)

Please tell me what you think of my opinions

[Szabolcs Szakacsits (szaka)]

On Sat, 7 Jul 2007, Nicolò Chieffo wrote:
> The ntfs filesystem in a default windows istallation has some forbidden
> characters such as : \ ? and others (I don't have the list) and is not
> case sensitive

No. NTFS doesn't forbid either. They are perfectly legal. This is a
Microsoft documented fact (see URL earlier). It's the WIN32 Windows
subsystem which forbids them.

> Using ntfs3g it is possible to create files with this names. Those files
> cannot be read in windows, without Windows Services for Unix.

No. Any application can access them if they use the POSIX Windows subsystem.
This was described in the Microsoft URL I sent earlier. WSU is just an
example.

> Well in my opinion you are talking as a very very advanced user

I'm the ntfs-3g lead developer, with over five years of extensive experience
on the NTFS field. I understand your points very well but there are millions
of other users who need to be understood as well.

> I'm instead trying to be near to a normal user which dual boots in
> windows and wants his files available to both windows and ubuntu.
> (this is what the spec is aimed for, read Use Cases)

I actively participated in the spec since the beginning ;-)

> I understand that windows forbidden characters is legal in the NTFS
> filesystem at all. But they are not legal in a normal windows
> installation,

I try to explain the situation in a simple way. Suppose you download a RAR
file on Linux but you don't have unrar. Then you would start arguing that
Linux shouldn't allow downloading RAR files because they can't be unrarred
by default. Would this make sense? Not much. The same is happening with
POSIX NTFS. Any program could handle it on Windows if they wanted. But since
people don't complain thus the vendors don't fix it. What you're arguing
now is that let's break Linux too.

> 1) which partition is normally ntfs?
> My answer: the partition where windows resides and maybe another
> partition in which documents, videos, music, downloads are stored. For
> sure not / or $HOME...
> 2) which applications are normally used over a ntfs partition?
> My answer: openoffice, gedit, totem, nautilus, rhythmbox, firefox
> (vim, cat, less, diff, <, > for more advanced users). I cannot think
> of other apps now. what would not work in these applications if the
> filesystem is not POSIX? Tell me if I forgot something.

NTFS is already used for / and $HOME, consequently with any softwares
unless the lack of some driver feature is a showstopper (e.g. shared
writable mmap).

> I told you it was an unexpected change and I was using vfat as a
> reference because (I think) most normal users were using vfat to
> exchange files, and now that ntfs3g is out will for sure switch to
> this. But (I think) they expect it works the same way as vfat, because
> in windows there is no difference, and because they know they are
> using a windows filesystem which has restrictions in windows default
> installation. If it behaves differently they might face lots of
> problems and most users will not understand which is the problem.

NTFS-3G is in use for almost a year, probably even by over a million users
now and there are about only one POSIX...

[Nicolò Chieffo (yelo3)]

Sorry, I didn't want to remove the posix compatibility to ntfs, just
to make an option so that the user could choose what to do. Adding
such option would certainly not affect who wants to use posix ntfs,
and would benefit users who wants to use it in a windows way, because
they use windows as their primary os.

In my opinion most users, except advanced users, would switch to this
option, since they only use the partition as a document storage.

If you really don't want to add this option, I hope that someone will
implement it, since I have never worked with filesystem drivers. Or at
least if ntfs-3g will be included by default, as the spec wants to, a
warning could save users who don't know this. I didn't understand if
you are against this warning, too.

Your example of rar files is not so valid, since you can easily
install rar and use those files...
On the other hand instead it is not easy to discover that to use these
files in windows you need a non-popular software, that is not easy to
install for normal users because it asks some advanced questions

I'm definitely not agaistn posix in ntfs. it is very interesting, and
I will use it now that I know WSfU! Just please inform the users about
this software!

Some questions about POSIX usage:
how can I change permissions and owners? how can I map unix users to
windows users for permissions in windows?

Thank you very much!

[Nicolò Chieffo (yelo3)]

rejected "upstream"