Win32: DIRECTORY case sensitive

[Chisheng Huang]

MS Windows uses case insensitive file names. PROBE-FILE and DELETE-FILE in
SBCL Win32 are case insensitive, as we expect; but, DIRECTORY is case
sensitive sometimes. The following example demonstrates the problem.

Directory d:/test/ has 2 files: AA.C01 and AA.PNG

(directory #P"d:/test/AA.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
(directory #P"d:/test/aa.*") => NIL

(directory #P"d:/test/A*.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
(directory #P"d:/test/a*.*") => NIL

For SBCL Win32, substituting EQUALP for the only EQUAL in MAKE-MATCHER
in sbcl/src/code/filesys.lisp and STRING-EQUAL for the only STRING= in
PATTERN-MATCHES in sbcl/src/code/target-pathname.lisp will make DIRECTORY
case insensitive:
(directory #P"d:/test/aa.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
(directory #P"d:/test/a*.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")

This is SBCL 1.0.33.4 for Win32.

Output from name -a:
MINGW32_NT-5.1 SQUID 1.0.10(0.46/3/2) 2004-03-15 07:17 i686 unknown

*FEATURES*:
(:ASDF :ANSI-CL :COMMON-LISP :SBCL :SB-DOC :SB-TEST :SB-LDB :SB-PACKAGE-LOCKS
 :SB-UNICODE :SB-EVAL :SB-SOURCE-LOCATIONS :IEEE-FLOATING-POINT :X86 :WIN32
 :GENCGC :STACK-GROWS-DOWNWARD-NOT-UPWARD :C-STACK-IS-CONTROL-STACK
 :COMPARE-AND-SWAP-VOPS :UNWIND-TO-FRAME-AND-CALL-VOP :RAW-INSTANCE-INIT-VOPS
 :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-VECTORS
 :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-FIXED-OBJECTS :ALIEN-CALLBACKS
 :CYCLE-COUNTER :INLINE-CONSTANTS :LINKAGE-TABLE :OS-PROVIDES-DLOPEN
 :OS-PROVIDES-PUTWC)

Thanks.

-cph

[Nikodemus Siivola (nikodemus)]

Unless I'm terribly mistaken NTFS volumes are case-sensitive even on Windows. So unless DIRECTORY on Windows has a separate keyword to control case-sensitivity, the default will by necessity be sometimes wrong?

What do Lispworks and Allegro do?

[Nikodemus Siivola (nikodemus)]

Ouch, ouch, ouch.

The thing is, by using the WinAPI function FindFirstFile the right things should happen pretty much by itself.

...but this means rewriting DIRECTORY, pretty much. Again. :/

[Chisheng Huang (cph-chi-square-works)]

Nikodemus Siivola <email address hidden> writes:

> Unless I'm terribly mistaken NTFS volumes are case-sensitive even on
> Windows. So unless DIRECTORY on Windows has a separate keyword to
> control case-sensitivity, the default will by necessity be sometimes
> wrong?
>
> What do Lispworks and Allegro do?

No idea about Allegro. With Lispworks 4.3.7 for Windows and a
directory d:/test/ with 2 files: AA.C01 and AA.PNG,

(directory #p"d:/test/AA.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
(directory #p"d:/test/aa.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")

(directory #p"d:/test/A*.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
(directory #p"d:/test/*.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")

So, Lispworks 4.3.7 for Windows has case-insensitive DIRECTORY.

Thanks a lot for looking into this.

Best wishes,

-cph

[Chisheng Huang (cph-chi-square-works)]

Chisheng Huang <email address hidden> writes:

> Nikodemus Siivola <email address hidden> writes:
>
>> Unless I'm terribly mistaken NTFS volumes are case-sensitive even on
>> Windows. So unless DIRECTORY on Windows has a separate keyword to
>> control case-sensitivity, the default will by necessity be sometimes
>> wrong?
>>
>> What do Lispworks and Allegro do?
>
> No idea about Allegro. With Lispworks 4.3.7 for Windows and a
> directory d:/test/ with 2 files: AA.C01 and AA.PNG,
>
> (directory #p"d:/test/AA.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
> (directory #p"d:/test/aa.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
>
>
> (directory #p"d:/test/A*.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
> (directory #p"d:/test/*.*") => (#P"d:\\test\\AA.C01" #P"d:\\test\\AA.PNG")
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        ^
This should be: (directory #p"d:/test/a*.*")

-cph

[Luis Sergio Oliveira (euluis)]

This is also true for enough-namestring; in windows it should be case insensitive as Allegro CL:

SBCL 2.0.10.119.master.53-8b9b0df87
CL-USER> *features*
(:SWANK :QUICKLISP :ASDF3.3 :ASDF3.2 :ASDF3.1 :ASDF3 :ASDF2 :ASDF :OS-WINDOWS
 :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :X86-64 :GENCGC :64-BIT :ANSI-CL
 :COMMON-LISP :IEEE-FLOATING-POINT :LITTLE-ENDIAN :PACKAGE-LOCAL-NICKNAMES
 :SB-LDB :SB-PACKAGE-LOCKS :SB-SAFEPOINT :SB-SAFEPOINT-STRICTLY :SB-THREAD
 :SB-UNICODE :SBCL :WIN32)
CL-USER> (enough-namestring "/d/USERS/foo/file.txt" "/d/USERS/foo/")
"file.txt"
CL-USER> (enough-namestring "/d/USERS/foo/file.txt" "/d/users/foo/")
"/d/USERS/foo/file.txt"

International Allegro CL Enterprise Edition
10.1 [64-bit Windows] (Nov 12, 2018 10:36)
Copyright (C) 1985-2017, Franz Inc., Oakland, CA, USA. All Rights Reserved.
CL-USER> *features*
(:QUICKLISP :ASDF-PACKAGE-SYSTEM :ASDF3.2 :ASDF3.1 :ASDF3 :ASDF2 :ASDF
 :OS-WINDOWS :ASDF-UNICODE :SWANK :ALLEGRO-CL-ENTERPRISE :IO-TIMEOUT
 :ZLIB-DEFLATE :INFLATE-BUG20472 :ACLDNS :IDE :COMMON-GRAPHICS :RFC8141
 :RFC6874 :RFC3986 :IPV6 :ACL-SOCKET :HIPER-SOCKET :PROFILER
 :NEW-XREF-RELATIONS :|YACC Release 0.98| :COMPILER
 :USE-STRUCTS-IN-COMPILER :CLOS :ATOMIC-SUBWORD-SETF :ATOMIC-SETF
 :VERIFY-CAR-CDR :DYNLOAD :DLWIN :TARGET-MSX86-64 :X86-64 :MSWINDOWS
 :MICROSOFT :MICROSOFT-64 :LITTLE-ENDIAN :ENCAPSULATING-EFS
 :DOUBLE-FLOAT-RANDOM-FIX :RELATIVE-PACKAGE-NAMES :MODULE-VERSIONS
 :IEEE :IEEE-FLOATING-POINT :CONFORMING-IEEE :ICS :COMMON-LISP :ANSI-CL
 :DRAFT-ANSI-CL-2 :X3J13 :ALLEGRO :EXCL :FRANZ-INC :ALLEGRO-VERSION>=
 :ALLEGRO-VERSION= :NEW-ENVIRONMENTS :GSGC :FLAVORS :MULTIPROCESSING
 :OS-THREADS :ALLEGRO-V10.1 :FIXNARGS :64BIT :USE-IN-CASE-MODE
 :SMP-MACROS :SSL-SUPPORT :MSVC2015)
CL-USER> (enough-namestring "/d/USERS/foo/file.txt" "/d/USERS/foo/")
"file.txt"
CL-USER> (enough-namestring "/d/USERS/foo/file.txt" "/d/users/foo/")
"file.txt"

The problem originates from compare-component (in file sbcl/src/code/target-pathname.lisp) by the use of string= when it should use string-equal for windows, which is case insensitive.

[Luis Sergio Oliveira (euluis)]

Relevant email threads in the sbcl-devel mailing list:

- "[Sbcl-devel] Win32: DIRECTORY not case insensitive"; From: Chisheng Huang <cph@ch...> - 2010-03-26 23:19:29; The reporter of this issue.
  https://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/87zl1uu1ye.fsf%40abalone.chi-square-works.com/#msg24867707

- "[Sbcl-devel] Pathnames on MS windows"; From: Anton Kovalenko <anton@sw...> - 2011-08-16 20:26:51; Seems to be one of the SBCL developers
  https://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/87mxf9drwj.fsf%40desktop-ng.home.sw4me.com/#msg27954390

- "[Sbcl-devel] case insensitive pathname and namestring operations in windows - fixable? - proposal to help"; From: Luis Sergio Oliveira <euluis1@gm...> - 2020-12-08 20:12:49; My message, related to my previous comment on this issue
  https://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/CAKOtXSoUK%2BYuG9x2xRyBkqLcoXRZkHejim7b6xA2iC0pX35%3DBg%40mail.gmail.com/#msg37172119

This requires further study before attempting a fix:
- look into Anton Kovalenko message and thread and try to find if some of his proposed work survives.
- how this is dealt by other Common Lisp implementations; e.g., I could check if ACL can deal with case sensitive directories in Windows which seems to a possibility I wasn't aware
- other portable runtimes, such as JVM
- portable Lisp libraries such as UIOP and ASDF