Syntax control for CL systems

[Faré]

Trying to distill content from these discussionthreads:
http://thread.gmane.org/gmane.lisp.asdf.devel/3883
http://thread.gmane.org/gmane.lisp.asdf.devel/4033

Currently, all components compiled with ASDF promiscuously share whatever readtable is active at the time ASDF:OPERATE is called. This means that if some component side-effects the *readtable* variable and/or the state of the data structure it's bound to, the effects will be seen by all components compiled with ASDF, *including components that do not "depends-on" support functions that the reader output produces. This causes catastrophic failure, when you e.g.

(asdf:load-system :fare-quasiquote)
(named-readtables:in-readtable :fare-quasiquote)
(uiop:run-program "touch somefile-in-the-system-below.lisp") ;; example: one of the files in ASDF itself
(asdf:load-system :some-modified-system-that-doesnt-use-fare-quasiquote) ;; example: ASDF

Then your system will be recompiled using fare-quasiquote for its backquote implementation, except that since it doesn't depends-on fare-quasiquote, the fare-quasiquote runtime will not be there next time you load the file from scratch.

There is nothing fare-quasiquote can do to protect users. Each and every system must be protected, or the REPL readtable will leak back at users.

In practice, the restriction means that
1- to be safe, systems must never change the behavior of any character defined in the CLHS
2- to be safe, systems can never modify the same character as any other system in the build, even systems unrelated to it
3- to be safe, applications must never use two different systems that modify the same character

This strongly limits the kind of readtable modifications permissible.

At the same time, short of a painful audit, it's very hard to know which characters exactly the systems you're trying to depend on do modify, if they do, which makes the whole endeavor very unsafe and has a chilling effect on any attempts to redefine anything, even simple things, even less complex things.

I feel this is wrong in a way similar to the previous :encoding issue: the person who knows which readtable-based dialect of CL [encoding] a system was written in is the author of the system, and he should be able to specify it (default: standard CL [utf-8]). He can't know what dialect [encoding] authors of other systems are using, and has no control on what the user is using at the REPL [meaning of :default].

The solution should be the same: create a flag that enables the new behavior, but disable it by default (easy to enable for those who'd like to). Send notice to all the authors of systems that fail, and notice on all appropriate channels of the upcoming change. After enough heads up, change the default. For the encoding thing, there were all of 20 systems that needed change. Half of them were fixed in a month (actually, most of them in a week). The other half were unmaintained system, and after a year, we made the change, that actually made cl-test-grid much more stable as a result. I expect similar results for this syntax change.

I therefore propose that ASDF shall ensure that each system should get its own readtable by default, separate from other systems. All systems are compiled in standard CL by default, and may explicitly request to be compiled in a different dialect. They are not affected by other systems (unless requested), and not affected by the REPL. They can completely redefine the readtable, yet effects remain local and do not pollute other users, even accidentally, nor are they accidentally polluted by the REPL or anything.