ASDF should be configurable in user- and system- preference files

[Faré]

So that ASDF may seamlessly install the stuff that the user installed, there ought to be some standard location for configuration files:

~/.config/common-lisp/asdf.conf
/etc/common-lisp/asdf.conf

For simplicity, the configuration file can be a sexp, or even a Lisp file you load - though restricting the language is always a good idea when you can get away with it.

Or something. But anyway, there should be a well-defined way to configure things such that responsibilities are well-established as to who configures what how, and who doesn't have to. Implementers shouldn't have to worry how things are distributed, or distributors how they are implemented.

The API should probably that configuration is read the first time ASDF is used. Or maybe everytime. It should include a way to override the configuration file, so that a managed project may have its own controlled set of libraries independently from what the user specifies by default.

Cf the comment originally in bug https://bugs.launchpad.net/asdf/+bug/485687
See also my rant http://fare.livejournal.com/149264.html

Credits to Stelian Ionescu who clearly explained the necessity of such configuration to me. Blame to me if I didn't get it right.

[Faré (fahree)]

Suggestions:

* use a simple SEXP language readable with READ, using keywords as keywords. This means ASDF can remain lean and not have to include a funky character-level parser.

* So that the file may be shared with other build systems (i.e. XCVB), call it source-registry.conf

* If the user file is found, don't load the system file, except that the user file can explicitly specify
  (:include-system-configuration) or (:include "/etc/common-lisp/source-registry.conf")

* To add a single directory to the registry, specify
  (:directory "/home/USER/cl/cl-blah/")

* To add a whole tree to the registry, specify
  (:tree "/home/USER/cl/")
  ASDF will then recurse into that tree, excluding known traps such as .svn and _darcs.
  Or do we instead want a :recurse t keyword to :directory?

* Conflict resolution: when a tree is specified, detect conflicts and mark them as such instead of picking a winner at random. (XCVB does that and it's very helpful).

* Entries are processed in order. Those that come first in the registry.conf come first in the search order.

* Either ASDF re-loads the registry every time a find-system is performed, or there is an exported way to tell it to re-load its configuration and/or use an alternate one. Maybe an exported special variable to hold the processed configuration. Maybe by default, flush the list of current systems when the configuration is loaded.

* Current *central-registry* does an EVAL of the contents. Possibly we may want to NOT eval by default, only when a :eval t keyword is passed. Or maybe we should evaluate always.

* Do we want to automatically add a trailing / to specified string entries? I say we do.

PS: it is my plan to do just the above for XCVB.

[Robert P. Goldman (rpgoldman)]

Faré wrote:
> Suggestions:
>
> * use a simple SEXP language readable with READ, using keywords as
> keywords. This means ASDF can remain lean and not have to include a
> funky character-level parser.
>
> * So that the file may be shared with other build systems (i.e. XCVB),
> call it source-registry.conf
>
> * If the user file is found, don't load the system file, except that the user file can explicitly specify
> (:include-system-configuration) or (:include "/etc/common-lisp/source-registry.conf")
>
> * To add a single directory to the registry, specify
> (:directory "/home/USER/cl/cl-blah/")

Given that the location of home-directory varies across OSes (notably
between linux and MacOSX), how about we specify that asdf must interpret
"~"?

Once again, I wish that logical-pathnames weren't broken...

Indeed, if we have cases where on one platform a known file will be in
"/etc" and one in, say, "/opt/etc", do we want to add some kind of
rudimentary pathname definer?
>
> * To add a whole tree to the registry, specify
> (:tree "/home/USER/cl/")
> ASDF will then recurse into that tree, excluding known traps such as .svn and _darcs.

and .git...

since we don't know a priori what pooplets might be dropped into the
user's directory trees, I suggest we provide

(:exclude-directories ".git")

Probably :exclude-directories needs to be supported globally and as a
keyword attached to particular :tree commands.

Probably :exclude-directories should do glob pattern matching.

>
> * Conflict resolution: when a tree is specified, detect conflicts and
> mark them as such instead of picking a winner at random. (XCVB does that
> and it's very helpful).
>
> * Entries are processed in order. Those that come first in the
> registry.conf come first in the search order.
>
> * Either ASDF re-loads the registry every time a find-system is
> performed, or there is an exported way to tell it to re-load its
> configuration and/or use an alternate one. Maybe an exported special
> variable to hold the processed configuration. Maybe by default, flush
> the list of current systems when the configuration is loaded.

I don't like the idea of reloading the registry every time a find-system
is done because that will break a lot of legacy code which carefully
crafts the asdf:*central-registry*, and does not have a provision for it
to be reloaded.

I'm also concerned about efficiency. I work on at least one project for
which I end up exploring a very large source tree (so large that without
the equivalent of :exclude-directories, it's infeasible --- even then
the search crashes when tail-recursion optimization is off), yielding a
*central-registry* list that has 124 elements. When you work in the
same repository structure as java programmers.....

>
> * Current *central-registry* does an EVAL of the contents. Possibly we
> may want to NOT eval by default, only when a :eval t keyword is passed.
> Or maybe we should evaluate always.

I don't use this feature, but I'm reluctant to break it for people who do.

* should we add the ability to specify a particular .asd file in the
conf file? I.e., I don't want you to add a directory to the
central-registry; I just wa...

[Robert P. Goldman (rpgoldman)]

Should have been more clear in my last. I think the right solution to the re-configuration issue is to provide an exported function that will flush and reload the configuration.

[Stelian Ionescu (sionescu)]

On Fri, 2009-11-27 at 16:26 +0000, Faré wrote:
> Suggestions:
>
> * use a simple SEXP language readable with READ, using keywords as
> keywords. This means ASDF can remain lean and not have to include a
> funky character-level parser.

I agree

> * So that the file may be shared with other build systems (i.e. XCVB),
> call it source-registry.conf

I doubt that this will work unless you restrict yourself of the lowest
common denominator between ASDF, XCVB and perhaps someday others. Sooner
or later one will grow a feature unsupported by the other and this won't
work any more

> * If the user file is found, don't load the system file, except that the user file can explicitly specify
> (:include-system-configuration) or (:include "/etc/common-lisp/source-registry.conf")

This is not how people(me included) expect this to work. It is a
consolidated practice in *nix world that the set of configuration
directives is built in the following manner, with the latter sources
overriding(or augmenting where applicable) the former:
system rc -> user rc -> environment -> command line

In the Windows world, AFAIK the expectation is identical, with the
system rc being not a file in /etc but a node in the system registry

That said, there are use cases where it is necessary to ignore any or
all of those sources, so it should be possible to opt out

> * To add a single directory to the registry, specify
> (:directory "/home/USER/cl/cl-blah/")
>
> * To add a whole tree to the registry, specify
> (:tree "/home/USER/cl/")
> ASDF will then recurse into that tree, excluding known traps such as .svn and _darcs.
> Or do we instead want a :recurse t keyword to :directory?

If Windows users really want this, fine. I'd vote for :recurse t

> * Conflict resolution: when a tree is specified, detect conflicts and
> mark them as such instead of picking a winner at random. (XCVB does that
> and it's very helpful).

What do you mean by conflicts ?

> * Entries are processed in order. Those that come first in the
> registry.conf come first in the search order.

I agree. This is how $PATH & co. work

> * Either ASDF re-loads the registry every time a find-system is
> performed, or there is an exported way to tell it to re-load its
> configuration and/or use an alternate one. Maybe an exported special
> variable to hold the processed configuration. Maybe by default, flush
> the list of current systems when the configuration is loaded.
>
> * Current *central-registry* does an EVAL of the contents. Possibly we
> may want to NOT eval by default, only when a :eval t keyword is passed.
> Or maybe we should evaluate always.

IMO the format should be stricter: just
(or pathname-designator (member :defaults)) should suffice while being
expressive enough for most purposes that I can think of. Avoiding the
use of eval is a good thing and should be added, at best, only if there
are people who actually need it.

> * Do we want to automatically add a trailing / to specified string
> entries? I say we do.

I agree

>
> PS: it is my plan to do just the above for XCVB.

--
Stelian Ionescu a.k.a. fe[nl]ix
Quidquid latine dictum sit, altum videtur.
http://common-lisp.net/pr...

[Faré (fahree)]

Still working on it, with some preliminary commits to my experimental repository.
http://common-lisp.net/pipermail/asdf-devel/2009-December/000580.html
http://common-lisp.net/gitweb?p=projects/xcvb/asdf.git

[Faré (fahree)]

I believe ASDF 1.595 (in my development repository) is solving these issues.

http://common-lisp.net/gitweb?p=projects/xcvb/asdf.git

[Nathan Bird (ecthellion)]

The docs have an error:
"The source registry will be configured from system configuration file /etc/common-lisp/asdf-output-translations.conf if it exists. "

The code is actually looking for the fles in /etc/asdf-output-translations.conf.

I'm fairly indifferent as to whether the path contains ../common-lisp/.. or not but it would be nice to have docs in sync.

Some other comments:
 * I'd rather have output-translations and source-registries configured in the same file or conf.d/. E.g. arbitrary number of sexps that start with one of (:output-translations :source-registry <some-other-future-config-keyword>), rather than being required to maintain separate files for 2 small forms.
 * Can :inherit-configuration be the default and not a required keyword in every file? As Stelian was saying this tends to be the default behavior in layered configurations like these.

[Faré (fahree)]

Oops, that's a bug in ASDF, to be fixed in 2.111 and 2.004. Thanks for noticing!

As for your comments:

1- not all programs use the two settings. For instance, both XCVB and ASDF use the source-registry but only ASDF uses the asdf-output-translations.

2- I've discussed the inherit-configuration at length before. I'm not convinced that we need or want a default behavior. TEXINPUTS has the default of not inheriting, for instance.

[Faré (fahree)]

Nathan, the issue you reported is fixed in 2.111, 2.004. Should have been its own bug but oh well.

The suggested workaround to support earlier versions of ASDF is to have a symlink from /etc/asdf-output-translations.conf to /etc/common-lisp/asdf-output-translations.conf