Enhancement: add component type for lisp files that are not to be compiled

[Robert P. Goldman]

This is a reasonably common thing to want to do in an ASDF system, and I have seen what must be half a dozen different implementations, many slightly buggy.

There was some discussion of this earlier (in 2009, I believe) in the ASDF-devel mailing list, but one or two approaches proved abortive, and then the matter was dropped.

[Faré (fahree)]

Yes, that would be a good thing to add.

Actually, I was think of having LOAD-OP in general delegate its job on any given CL-SOURCE-FILE to either LOAD-SOURCE-OP or LOAD-FASL-OP based on the component's data.

[Robert P. Goldman (rpgoldman)]

I was thinking about this, and I don't quite get the concept. It seems reasonable that we could delegate, e.g., the PERFORM methods, but there seem like two objections:

1. If we have to delegate a bunch of methods, is delegation really helping us?
2. I'm a little worried about introducing intermediary operation objects for the delegation. Do we have to generate multiple such objects, one for each generic function called? If so, then copying any attached properties from the delegating OPERATION objects to the delegated-to OPERATION objects (top-down properties) and then back from the delegated-to OPERATION objects to the delegating OPERATION objects (properties computed bottom-up) seems like it adds potential for error. Staving off such errors might be more work than simply adding new methods for the LOAD-ONLY-CL-SOURCE-FILE...

[Faré (fahree)]

I think delegating the PERFORM methods is disgusting and defeats the whole dependency model of ASDF. The correct was of delegating an operation is to customize COMPONENT-DEPENDS-ON, so it does the right thing. In the case of LOAD-OP, this right thing could be a a :LOAD-OP-SHOULD initarg to CL-SOURCE-FILE, or some other specialization of COMPONENT-DEPENDS-ON ((O LOAD-OP) (C ...)).

Note that this delegation through COMPONENT-DEPENDS-ON used to be impossible with ASDF 1 and up to ASDF 2.26 due to TRAVERSE systematically propagating all operations downward the component tree. But happily ASDF 2.27 fixed that deep conceptual bug and now it is both possible and desirable.

That said, we should be careful about how to treat "don't compile" files on ECL and other implementations with a compile-and-link model rather than a load-and-dump model. On such implementations, shall we fall back to compiling, or should we link an object for a file that contains a single (LOAD-FROM-STRING #.(READ-FILE-STRING "foo.lisp")) form?

As for the benefits of the delegation model, an important one is decoupling the strategy used by a required system from the strategy used by the requiring system: requiring systems shouldn't have to know the proper strategy for each and every of their direct and indirect required systems, as they currently do. Instead, each system author, who knows best about his system (or should), should be the one to pick the most appropriate strategy for his system. At the same time, there should be sensible implementation-dependent defaults, i.e. system-wide LOAD-FASL-OP on ECL and compile-and-link implementations vs LOAD-OP on load-and-dump implementations; and the end-user should be able to override this default and/or individual system strategies, and if you want to enforce a compile-and-link model like XCVB does, you might for a system-wide LOAD-FASL-OP everywhere.

As for copying slots of OPERATION, that's a tricky topic indeed.

ASDF 1 used to have this ugly (MAKE-SUB-OPERATION o c dep-o dep-c) horror to copy these properties. With ASDF 2.27, I have much simplified it into a (FIND-OPERATION o sub-o) analog to FIND-COMPONENT that preserves the properties; FIND-OPERATION in turn calls a hash-consing MAKE-OPERATION variant of MAKE-INSTANCE for operations using the *OPERATIONS* cache, which is already a much cleaner model.

Most importantly, though, ASDF 3 inherits from ASDF 1 a crucial problem with respect to operations and their slots: COMPONENT-OPERATION-TIME drops all slot information and considers that an operation has been done if any operation of the same type has been done before. This essentially means that slots are supposed not to matter at all as to the semantics of the operation.

Going forward to ASDF 3, this means that either operation slots should be entirely deprecated (my preferred solution so far), or that we should be strict about only storing hash-cons'ed operations as keys in the COMPONENT-OPERATION-TIMES tables. e.g. operations would have a hidden slot "canonical version" which when initialized links to the canonical version for use in such table, and we should then have a protocol for canonicalizing the initarg list. Ouch...

[Faré (fahree)]

I want to mark this bug as "Won't Fix".

This is too alien to the current ASDF model to be implement without a lot of unnatural hacking, especially so on ABCL, ECL, MKCL.

[Faré (fahree)]

Closing as "Won't Fix".

[Robert P. Goldman (rpgoldman)]

I'm going to revert this from "Won't Fix." I see your point, but we support operations in ASDF that are pretty unique to the translate-to-C model, so I don't see any reason why we shouldn't support things that are pretty unique to systems that have a distinction between interpreted and compiled code.

I'm going to call it "opinion," because I'm not going to stamp up and down and demand that it be fixed by someone who doesn't care about it. If someone wants it, though, I'm ready to merge it.