ASDF should handle change in defsystem-depends-on dependencies

[Faré]

When a system listed in defsystem-depends-on changes, ASDF ought to reload that dependency and re-process the .asd file.

This is probably ASDF4 material. Also, a :defsystem-depends-on dependency should be able to add itself and/or some runtime support to the the :depends-on of the systems that use it.

And/or we should drop Common Lisp and adopt more Racket-like modules.

[Robert P. Goldman (rpgoldman)]

What do you mean about saying that a defsystem-depends-on dependency should add itself to the :depends-on of systems that use it. If we have defsystem-depends-on working properly, why would this be necessary/desirable?

[Faré (fahree)]

While debugging the bundle-op's on SBCL, I was experimenting with cffi-grovel (see my pull request for cffi on github), and when I was trying to rebuild osicat without explicitly loading cffi-grovel, it wouldn't re-load cffi-grovel.

I was changing the runtime, and so it would have been enough to just implicitly add cffi-grovel to the :depends-on, but had I been modifying the asdf support a reloading of the .asd file would have been necessary for successful completion.

https://github.com/cffi/cffi/pull/73

[Faré (fahree)]

1- When building a plan, we need to have explicit nodes for reading a defsystem, so that LOAD-OP "foo" depends-on LOAD-ASD-OP "foo".

2- Now, making a plan can recurse into executing subplans, so that planning and execution are interleaved, which is contrary to the initial ASDF 1 model, but a de facto necessity since even before it was semi-formalized by ASDF 2.

3- We must thus detect circularities across levels of plan execution.

4- We must merge the dependencies detected by subplans into the larger plan, efficiently

5- Subplans must partake in the protocol to not re-perform an action twice when requested by multiple (sub)plans, maybe not even plan it twice — yet correctly propagate timestamps.

6- In case some error happens in some subplan, we must correctly roll back any incomplete state.

Your mission, should you accept it...

[Faré (fahree)]

I realize this bug was originally reported by Stelian as bug #1227260.

[Robert P. Goldman (rpgoldman)]

I was rereading this to catch up with long-deferred ASDF tasks. I am not sure I fully understand the notion of adding a LOAD-ASD-OP. On the one hand, this makes perfect sense, on the other hand, it seems to be circular, since the structure of the plan depends on the loading of the ASD file.

If we were to introduce this idea, wouldn't we need something like a meta-plan? Or even a new phase that does not involve plan-then-act. I.e., there would be a first phase to the build process that involves loading all the ASD files -- which we wouldn't pre-plan. We would only plan after we process all the ASD files that need processing.

That would allow us to maintain the plan-then-act structure of ASDF as it is today. I think tearing it all away and replacing with a new structure that interleaves planning and execution is more a job for ASDF 4, which perhaps should be YASDF instead of attempting to continue the current ASDF lineage.

[Faré (fahree)]

Well, it's not *circular*, but it's *staged*: this implies an essential interleaving of plan computation with plan execution.

Note that adding LOAD-ASD-OP does *NOT* *introduce* this staging: it only *acknowledges* and *records* a staging that *already exists*. The "plan-then-act" structure of ASDF is already *a lie*.

And yes, I haven't started to think about what kind of data structure makes this possible, and how that affects cross-compilation.

And yes, this might be an ASDF 4 job.

[Robert P. Goldman (rpgoldman)]

Also, DEFSYSTEM-DEPENDS-ON is fundamentally broken in its interaction with Common Lisp packages. You can't effectively export symbols from an ASDF extension loaded by DEFSYSTEM-DEPENDS-ON into the package in which the DEFSYSTEM form has been read.

This could be fixed by a rewrite of ASDF so that the DEFSYSTEM form is parsed, and not read, but that's more than I propose to put into ASDF 3.

[Faré (fahree)]

OK, so it seems to me that having a *PLAN* object containing the PLAN currently being build and/or executed, and using plan-record-dependency (with a suitable default method), we could properly record the dependencies of loading a asd file (which would still require a load-asd-op or something like that.)

[Faré (fahree)]

OPERATE would then see if *PLAN* is not nil and then call plan-record-dependency (or let the default method be trivial).

[Robert P. Goldman (rpgoldman)]

This is interesting, but absent a solution to DEFSYSTEM-DEPENDS-ON's known problems with the CL package system, I'm reluctant to invest in it further.

My position until now, which this has not changed, is that D-D-O doesn't work, and furthermore, it almost *mandates* namespace pollution by forcing ASDF add-ons into the ASDF package.

If we can't fix that we shouldn't invest in fixing D-D-O.

[Faré (fahree)]

Well I made sure that ASDF accepts strings as class and function designators, to be read and/or interned later, at the many places where packages can't have been created yet.

[Robert P. Goldman (rpgoldman)]

Yes, but that doesn't solve the problem. Assume I have a defsystem that is in package FOO.

Now I DEFSYSTEM-DEPEND-ON some ASDF extension. This extension is implemented in its own namespace, as it should be to avoid namespace collision, ASDF-EXT. Now what? I can't get its exported symbols into package FOO, which has already been created by the defpackage form at the top of my .asd file. I SHOULDN'T jam those exported symbols into the ASDF package, because that's namespace pollution, and I can't protect against naming conflicts.

So the strings don't help, unless you mean that I can arbitrarily put package prefixes into them.

That would help, but now we have to guess every place the programmer might want to stick a string instead of a symbol (inside a :METHOD)? At this point you really have to have a code-walker that looks for strings everywhere, and knows where a string is a string -- for example when it's a pathname or the name of a component -- and not a thing that should be interpreted as a lazily-referenced symbol.

Once you have gone that far, you might as well give up using READ to process the DEFSYSTEM forms and just write your own parser.

Or toss DEFSYSTEM-DEPENDS-ON into the ashheap of history and just put ASDF:LOAD-SYSTEM forms in your .asd files....

[Faré (fahree)]

Yes, you can use a package prefix in those strings. That's the point!

In your methods, your can use call-function (the utility that handles functions for us), or find-symbol* or symbol-call. Or move your functions or of the .asd files.

[Robert P. Goldman (rpgoldman)]

Am I just missing something? I am looking at the DEFSYSTEM GRAMMAR page of the manual, and it clearly talks about using keyword symbols, but I don't see anything in there about using strings.

[Faré (fahree)]

It's probably the manual that's missing something. Sigh. I'll edit it tonight...

[Robert P. Goldman (rpgoldman)]

Can you say when this string capability entered ASDF?

[Faré (fahree)]

The string reading capability first entered ASDF in 2.31.4 5f0f990a in February 2013.
The latest modification seems to be from 50f2420f a bit before 3.1.0.95 in March 2014.
So basically, any version of ASDF 3 in the large supports it.

[Faré (fahree)]

See class-for-type in parse-defsystem.lisp (which I just git blame'd for you).

[Faré (fahree)]

Working on it in my "plan" branch, !37 https://gitlab.common-lisp.net/asdf/asdf/merge_requests/37

[Faré (fahree)]

This is fixed in the "plan" branch. Let's merge it for ASDF 3.3