DIRECTORY still resolves symlinks internally with :RESOLVE-SYMLINKS NIL

[Faré]

With the attached patch, I could speed up (asdf:initialize-source-directory) by two orders of magnitude (!) when I (SETF ASDF:*RESOLVE-SYMLINKS* NIL).

I think that as a further cleanup,

1- the CLASSIFY-SYMLINKS flags might be renamed to RESOLVE-SYMLINKS, for consistency. Similarly, &KEY or &OPTIONAL might be used everywhere.

2- You might want to disable symlinks internally even when when RESOLVE-SYMLINKS is true, except at the end, to avoid needless computations.

3- You might want to have a cache (under user control) around some operations so that you don't call readlink and lstat and parse-unix-namestring over and over again on the same files, all the time. With proper caching, there's no reason why operations that return truename on a lot of files should be 2 orders of magnitude slower than without truename.

[Faré (fahree)]

Oops, measuring the wrong thing. The speed up with RESOLVE-SYMLINKS NIL is much more modest than that. Oh well. Further optimizations might help '(see suggestions above).

[Faré (fahree)]

Note that even without wonderful speedups for ASDF, the patch is still a good idea, as it avoids unnecessary (indeed, unwanted) work.

[Faré (fahree)]

Once again this patch is still a good idea. Can some maintainer merge, test and commit?

[pipping (pipping)]

Update: This patch still applies cleanly to sbcl-1.3.9 and appears to fix an issue: Without it, as far as I can tell, there is currently no way to check for all subdirectories of a directory and exclude symlinks. With this patch, (directory "/path/to/dir/*/" :resolve-symlinks nil) gives you just that.

[pipping (pipping)]

Here's a test case. On Linux, I create the following set of files, links, and directories:

$ mkdir /home/pipping/sbcl-test
$ mkdir /home/pipping/sbcl-test/dir
$ mkdir /home/pipping/sbcl-test/.dir
$ mkdir /home/pipping/sbcl-test/dir/dir-in-dir
$ touch /home/pipping/sbcl-test/file
$ touch /home/pipping/sbcl-test/dir/file-in-dir
$ ln -s dir /home/pipping/sbcl-test/link-to-dir
$ ln -s file /home/pipping/sbcl-test/link-to-file
$ ln -s dir/dir-in-dir /home/pipping/sbcl-test/link-to-dir-in-dir
$

Here's an overview over the resulting situation:

$ ls -AlR /home/pipping/sbcl-test/
/home/pipping/sbcl-test/:
total 8
drwxrwxr-x 3 pipping pipping 4096 Sep 29 11:18 dir
drwxrwxr-x 2 pipping pipping 4096 Sep 29 11:05 .dir
-rw-rw-r-- 1 pipping pipping 0 Sep 29 11:06 file
lrwxrwxrwx 1 pipping pipping 3 Sep 29 11:06 link-to-dir -> dir
lrwxrwxrwx 1 pipping pipping 14 Sep 29 11:18 link-to-dir-in-dir -> dir/dir-in-dir
lrwxrwxrwx 1 pipping pipping 4 Sep 29 11:07 link-to-file -> file

/home/pipping/sbcl-test/dir:
total 4
drwxrwxr-x 2 pipping pipping 4096 Sep 29 11:18 dir-in-dir
-rw-rw-r-- 1 pipping pipping 0 Sep 29 11:06 file-in-dir

/home/pipping/sbcl-test/dir/dir-in-dir:
total 0

/home/pipping/sbcl-test/.dir:
total 0
$

Leaving files and links to files alone, we have
- two physical directories, dir and .dir
- two symlinks to directories, link-to-dir and link-to-dir-in-dir

With sbcl-1.3.9.108-e34dd00-linux-x64, I get:

* (directory "/home/pipping/sbcl-test/*/" :resolve-symlinks t)

(#P"/home/pipping/sbcl-test/.dir/" #P"/home/pipping/sbcl-test/dir/"
 #P"/home/pipping/sbcl-test/dir/dir-in-dir/")
*

This is not unexpected (dir could show up either once or twice here since link-to-dir points to dir).

* (directory "/home/pipping/sbcl-test/*/" :resolve-symlinks nil)

(#P"/home/pipping/sbcl-test/.dir/" #P"/home/pipping/sbcl-test/dir/"
 #P"/home/pipping/sbcl-test/link-to-dir-in-dir/"
 #P"/home/pipping/sbcl-test/link-to-dir/")
*

This is unexpected. The fact that #P"/home/pipping/sbcl-test/link-to-dir/" and #P"/home/pipping/sbcl-test/link-to-dir-in-dir/" are contained in this list means that SBCL must have internally dereferenced the link when I thought I had asked it not to. If I apply Faré's patch, I get this instead:

* (directory "/home/pipping/sbcl-test/*/" :resolve-symlinks t)

(#P"/home/pipping/sbcl-test/.dir/" #P"/home/pipping/sbcl-test/dir/"
 #P"/home/pipping/sbcl-test/dir/dir-in-dir/")
* (directory "/home/pipping/sbcl-test/*/" :resolve-symlinks nil)

(#P"/home/pipping/sbcl-test/.dir/" #P"/home/pipping/sbcl-test/dir/")
*

This makes more sense to me.