New keyword :logical-host for DEFSYSTEM

[Juanjo Garcia-Ripoll]

In order to achieve a more declarative ASDF definition we need the loaded files to work without interrogating ASDF at all. This is also needed for building standalone programs which may or may not include ASDF as one of its components.

The syntax is simple: keyword :logical-host works with systems and defines a logical pathname translation for that system using as root the pathname of the system.

foo.asd:

(defsystem :foo
    :logical-host "FOO"
    :components ((:file "foo")))

foo.lisp:

(format t "~&;;; Translation of foo:foo.lisp is ~A"
        (translate-logical-pathname "foo:foo.lisp"))

[Robert P. Goldman (rpgoldman)]

Possible to add logical pathname translation for FOO:FASL; as well? Might be useful for the system to be able to find its binaries...

[Juanjo Garcia-Ripoll (worm)]

A new implementation that allows two keywords

:logical-host
:logical-pathname

Two example systems

$ cat foo.asd
(defsystem :foo
    :logical-pathname "FOO:BASE"
    :components ((:file "faa")))
$ cat faa.lisp
(let ((x "FOO:BASE;SRC;FAA.LISP"))
  (format t "~&;;; Pathname ~S~%;;; translates into ~S"
          x (translate-logical-pathname x)))
(let ((x "FOO:BASE;OUTPUT-FILES;FAA.LISP"))
  (format t "~&;;; Pathname ~S~%;;; translates into ~S"
          x (translate-logical-pathname x)))
(terpri)

This one allows an arbitrary pathname:

$ cat foo2.asd
(defsystem :foo2
    :logical-host "FOO2"
    :components ((:file "faa2")))
$ cat faa2.lisp
(let ((x "FOO:SRC;FAA.LISP"))
  (format t "~&;;; Pathname ~S~%;;; translates into ~S"
          x (translate-logical-pathname x)))
(let ((x "FOO:OUTPUT-FILES;FAA.LISP"))
  (format t "~&;;; Pathname ~S~%;;; translates into ~S"
          x (translate-logical-pathname x)))
(terpri)

[Faré (fahree)]

Considering that :logical-host "FOO" is the same as :logical-pathname "FOO:" and for the sake of not multiplying entities in vain, could we do away with :logical-host and just keep :logical-pathname ?

Problem I foresee with logical pathname translations include:

1- how to distinguish between source files (that should not be translated) and object files (that should be translated), especially when the involved pathname types are not portable, sometimes not unique (think .fas and .lib for clisp!), and possibly even shared (think proprietary object file .o vs compiled object files on some implementations.

2- new namespace to manage and in which to avoid pollution, despite loose coupling. I fear that soon users may have to edit .asd files to adjust logical pathnames. Unless we decide to reuse the ASDF defsystem namespace as part of the logical pathname namespace, as in common-lisp-system:foo;

[Juanjo Garcia-Ripoll (worm)]

The concern about users editing the asd files is out of place. Editing of ASDF files is just as stupid as touching the autoconf files of a library I just downloaded.

The namespace pollution concern is just as out of place. Do you consider package names to be polluting the namespace? We are giving LIBRARY DEVELOPERS, NOT USERS, a way to locate their RESOURCES not their compiled files. Developer will be clever enough to choose unique names that do not lead to collision, just as they do choose package names wisely.

Third, this extension is intended for the EXECUTION of code not as a mean to locate FASL files which are going to be managed by ASDF. The reason I added OUTPUT-FILES is because some of the components will end up producing generated files that might be moved by ASDF due to the unfortunate translation mechanism (for instance C files, shared libraries, etc)

Fourth, the concern about file types is again out of the scope of the extension. This keyword gives developers a means to locate their resource files, whose name they must know, not a means to choose the right extensions for compiled files -- for which there are other mechanisms --. These keywords are equivalent but more transparent than the API you introduced for locating the pathnames of a system and by the same reasoning we should be removing those functions as well.

Fifth, I do not care about dropping :logical-host.

[Faré (fahree)]

I'm unconvinced, but why not?

I think that managing a namespace that is directly mapped from system name would allow for trivial merging of a single logical host.

Allowing the user to specify his own host - why not? But then why not just have :logical-pathname? It doesn't seem to me like it requires that much cleverness to manage the merging, and it simplifies the interface.

Note that if lists of patterns are inefficient to walk, the patterns can be arranged as some kind of trie. Whether implementations do that internally is a different story.

Can this be a contrib, or is there a strong case for making it part of asdf.lisp itself?

[Juanjo Garcia-Ripoll (worm)]

The strong case for making it part of asdf.lisp is that if it is a contrib nobody will use because it will not be available by default. Contribs in other packages are either done so because of stupid political questions (see slime's fancy mode) or because they may hinder functionality, are not portable or have a humongous cost (cffi-grovel)

The point of all this is motivating people to move towards good practices, avoiding use of ASDF's API, system interrogation and all other dependencies that may or may not be available at run time. The point of all this is making system building more and more independent of the libraries and pathname translations going all behind...

I wrote already several thousand words about the utility and interest of this for ASDF's future.

Why do you not answer instead the following question:

  Is there a strong case for making this a contrib and not part of asdf.lisp?

[Faré (fahree)]

I have several reasons to not include it as is in asdf.lisp.

First, general principles:

A- I think ASDF should be designed around a self-containted bootstrapped single-file asdf.lisp, with everything that is not strictly necessary in contribs.

B- For all other features, I'd like to encourage system :asdf-full with plenty of extensions, some controversial, some not, each in their own system, that play well with each other and with people who don't want to use them.

C- I disagree with specific issues in the current design of your particular extension. It is not necessary for the ASDF kernel and has enough arbitrary controversial choices that I feel it's going to be a pain.

Now for specific complaints regarding point C above:

1- I'm not convinced you bring any value as compared to asdf:system-relative-pathname and such. If you want something that builds both with and without ASDF, you still need a setup file somewhere, that could use #+asdf2 or whatever if it wants to do things depending on being compiled with ASDF or not. I know that for me at ITA, logical pathnames TOTALLY DON'T CUT IT, because the paths required in our projects use dots, dashes, mixed cases, and other things not supported by the standard and explicitly excluded by SBCL.

2- If we are to use logical pathnames, I am wary of logical-host pathname pollution, especially when short names can be names of machines, that introduce weird incompatibilities on various machines. What if you name something FOO, and I'm using your application on Windows, trying to access a share \\FOO\myshare on the network? Oops, I get something completely different! If each and every package introduces a new host, this is bound to cause problems eventually.

3- If we are to allow developer-chosen logical pathnames instead of mapping each system to a logical pathname trivially deduced from its name as in :cl-ppcre ==> #p"source-registry:cl-pprce;" then why not offer the option of an arbitrary pathname? Either restrict or let be free, but this intermediate restriction smells bad to me.

[Juanjo Garcia-Ripoll (worm)]

A- Your choice design and it is wrong, since nothing excludes that an initial file knows ands loads the rest of it.

B- Nothing as such exists right now. And there are extensions which are transparent enough that people will not even know whether they use them or not -- for instance, concatenation of FASL files, which is essential for ECL.

C- This was the meat of it.

1- It is not just BUILDING it is RUNNING.
* If all existing libraries begin to depend on the existence of ASDF related functions, it will be impossible to do standalone programs without shipping ASDF with them.
* Moreover, programmers will have to program for ASDF, using ASDF tools to find its stuff.
* Even ASDF itself will not be able to build that software without that depedency.
* Forcing to shipASDF with a program is stupid, introducing a lot of problems because it exists, is there and will do things at boot time that are not wanted by the developers -- looking for configuration files, putting files in arbitary directories not controlled by the application developer, etc.
* So in short, making software depends on ASDF while running is bogus for transparent delivery. And it will become impossible to build other extensions for doing that.

2- and 3- can be coped with. A patch is uploaded that permits arbitrary logical pathnames resolving collisions.

[Faré (fahree)]

Now that there is :defsystem-depends-on and asdf-contrib, could we make this a part of asdf-contrib, assuming there is still demand for it?

[Faré (fahree)]

After having gone through EXTREME pains to make pathnames, including logical pathnames, work as portably as can be in ASDF, I have come to the conclusion that CL pathnames are cesspool of fail, and "logical" pathnames worst of it all. Putting abstractions at all the wrong place.

I categorically refuse to encourage the use of logical pathnames.

The entire concept of registry was invented to work around the inherent failure of logical pathnames.

Please use the source-registry, system-relative-pathname and subpathname.

Bug trashed.

[Robert P. Goldman (rpgoldman)]

I'm also unconvinced by this one. ASDF is designed based on a division of functionality between the installer and the writer of the installed system.

I don't feel like the writer of the installed system knows enough about my local set of logical pathnames, already configured by me, to start messing with it.

I prefer using the ASDF system-relative pathname functions for this.