ASDF on CCL does not follow symlinks in registry

[Piotr Kalinowski]

Environment tested: Max OSX 10.8.2, CCL 1.9, SBCL 1.1.9, ASDF 2.32 and 3.0.2

Reproduction steps:

1. Put a lisp package outside of source-registry and quicklisp. At the time of writing this, you could try the following:

mkdir ~/Sources
cd ~/Sources
git clone https://github.com/sionescu/swap-bytes.git

2. Link it into registry, for example:

ln -sf ~/Sources/swap-bytes ~/.local/share/common-lisp/source

3. Try to load the package with ASDF

Expected: package loads successfully.
Actual: does not work.

This will fail on CCL 1.9, unless you actually copy package source directory, instead of linking it. Linking ASD files does not work too.

It works fine on SBCL 1.1.9 though.

Note that this also works if you put the link in a local projects directory of Quicklisp, because it does not add it to the source registry, but rather scan it itself.

[Faré (fahree)]

Not my problem anymore, but the problem is the unclear semantics of CL:DIRECTORY, that UIOP tries to paper over with UIOP:DIRECTORY*, UIOP:DIRECTORY-FILES and UIOP:SUBDIRECTORIES.

Good luck with
1- deciding on some sensible, portable, semantics.
2- implementing on all of 15 implementations (and/or make a best effort).

[Faré (fahree)]

Patches are, I am sure, welcome by the new maintainer.

[Piotr Kalinowski (pitkali)]

I see. Maybe I'll find some time to take a look.

I guess the problem would go away with xcvb? ;)

[Piotr Kalinowski (pitkali)]

It turns out that CCL, if you disable following links, does not know that links correspond to directories. Instead it always treats them as files, so UIOP:SUBDIRECTORIES does not return them.

[Piotr Kalinowski (pitkali)]

If I fix the UIOP:SUBDIRECTORIES to list both files and directories on CCL, and then filter by which are actually a directories (but this requires checking them with DIRECTORY-EXISTS-P), then the problem disappears. See attached patch.

The question remains though, if UIOP:DIRECTORY-FILES should be modified as well. Currently it will list both files and directories on SBCL and CCL, including symbolic links. However, if a link leads to a directory, SBCL will return a directory pathname for it, whereas on CCL it will be represented as a file pathname.

[Piotr Kalinowski (pitkali)]

I've compared this to LispWorks as well, and it also returns directory pathnames for links, like SBCL, even if I disable following symbolic links. So I attach a patch that will fix DIRECTORY-FILES on CCL to behave the same as SBCL and LW.

[Robert P. Goldman (rpgoldman)]

I am willing to accept this patch and push it. However, I would like to document the behavior we expect and I would like a test case, so that we can check and see that it is applied uniformly (or as near to uniformly as possible).

Let me see if I understand the correspondence:

1. (UIOP:SUBDIRECTORIES x) sometimes does and sometimes does not return a directory pathname for links, L, contained in X, such that L points to a directory. Correct?

2. It is suggested that UIOP:DIRECTORY-FILES should be fixed in the same way.

3. We could create a test case for this by creating an ASDF system somewhere outside the tests/ subdirectory, and linking it into the tests/ subdirectory. This would be a bit of a nuisance, but we could create a system directory in (UIOP:TEMPORARY-DIRECTORY), copy a system definition there, link it back into tests/ and see if it is loadable. We'd have to use RUN-PROGRAM to create a symlink, since I don't believe we have a portable way to do this. I have no idea what the windows semantics should be.

4. We could create more primitive test cases that directly check UIOP:SUBDIRECTORIES and UIOP:DIRECTORY-FILES.

Is this an accurate understanding? If so, I believe we can make this happen, and I can tag a new version of ASDF for testing.

[Piotr Kalinowski (pitkali)]

1. (UIOP:SUBDIRECTORIES x), on CCL 1.9 does *not* list linked directories at all. The directories that are listed are returned as directory pathnames. So if X contains link L to an existing directory, L will not be in the output at all. As a direct consequence, COLLECT-SUB*DIRECTORIES never finds nor recurses into L. Thus none of the ASDs inside L gets detected by ASDF.

2. To be honest, ASDF will pick up all ASDs correctly with UIOP:SUBDIRECTORIES fixed. However, it turns out that because CCL does not check if links point to directories or files (with :FOLLOW-LINK nil, that is), and instead treats them always as files, that they are returned using file pathnames. This is not how it works on SBCL 1.1.9 or LispWorks 6.1, where disabling link following simply results in returning pathnames that do not correspond to true names of link targets.

I thought that the point of UIOP is to abstract away as much implementation specific behaviour as possible. This seems to be one such detail. The client code can deal with this by using DIRECTORY-EXISTS-P, which will behave properly even in case of links and CCL. So it is not really necessary to change DIRECTORY-FILES, but then I feel like it would be a good idea to document this.

However, if this is left like this, I suspect that it will end up with someone writing code that breaks on CCL. But maybe I'm just cynical ;)

3. That sounds like an overkill. I think that testing UIOP:SUBDIRECTORIES will be enough. Although Windows indeed does not have portable way of doing any sort of links. Windows 2000/XP with NTFS has something called junction point, and Vista+ does have proper symbolic links. I actually do have a virtual machine with Windows 7, so I could see how this behaves there.

For portable way of creating links, there seems to be a library called OSICAT that does have this, but it works only on Unix-like systems, so that does not seem to be any better than calling RUN-PROGRAM.

I can look into writing the test cases, although probably not today.

[Robert P. Goldman (rpgoldman)]

I will see what I can do about a test script; I've written a bunch of these by now.

[Faré (fahree)]

Now I remember one of the reasons why I decided to disable follow-symlinks: because it causes massive slowdown when recursing down a filesystem tree, and the stupid lisp implementation computes truenames from each and every file from scratch.

[Robert P. Goldman (rpgoldman)]

Retitled because this happens to me on Linux, as well.

[pipping (pipping)]

I think anyone seriously hoping to fix anything here should take a look at

  https://gitlab.common-lisp.net/asdf/asdf/merge_requests/27

and realise the futility of an attempt.