need better asdf support

[Faré]

ASDF integration to SLIME is lacking.

When evaluating a .asd file foo, SLIME should be using
(asdf:load-sysdef "foo" #p"/path/to/foo.asd")
This ensures that the file is loaded in the proper package,
that proper circular-dependency-breaking measures are taken, etc.
It is always wrong to compile-file it to a .fasl,
as many .asd files cannot be compiled, and the fasl will often conflict
with the fasl for a similarly-named lisp file.

When compiling a .lisp file that is part of an asdf system,
SLIME should be using (asdf:perform (make-instance 'asdf:compile-op COMPONENT))
where COMPONENT is the component that corresponds to the file.
This ensures that proper encoding is used, that appropriate :around-compile hooks
are invoked (to setup the correct optimization settings, readtables, package aliases, etc.)
We may have to develop a proper API for going from file to component,
especially when not all .asd files have been loaded yet.

[Stas Boukarev (stassats)]

The first point could be made as the default action when slime-asdf contrib is loaded, as to the second, I wouldn't really expect this to happen, so it's better to make it an optional feature.

[Faré (fahree)]

If you're willing to merge in the changes, I can help with improving the ASDF support.

However, one question remains to me: how old ASDF versions do you want to support?

Many things can be simplified by assuming ASDF 2.000 or newer, and even more things can be simplified if we can assume a relatively recent ASDF (quicklisp has 2.014.6, SBCL from 2011-08-22 or CCL from 2011-07-28 has 2.017, etc).

On the one hand, we should never assume that SLIME runs on super-recent software: debugging, by definition, may happen on software written long ago.

On the other hand, the very idea behind ASDF2 is that you can always upgrade an existing image to use the latest ASDF, even on top of an existing old ASDF.

On the first hand, when you upgrade ASDF, it invalidates previous defsystems in memory, because of potential incompatibilities.

On the second hand, after the defsystems have been invalidated, they can always be reloaded from disk.

On the first hand, you might not always have the exact source used to compile the image version you're debugging.

On the second hand, if you don't have the matching source, then you're not going to have much use for the original system definitions, anyway.

So there. I believe my conclusion is that we should NOT be worrying about ASDF versions that are too old, and that older than 2.000 is probably too old, considering that all maintained implementations have adopted ASDF 2 some time between 2010 and 2011. Hopefully, we can declare anything older than 2.011 obsolete (valid as per 2010-11-28). Actually, I'd really like to declare things before 2.015 obsolete, but that isn't realistic yet, considering that quicklisp still sports 2.014.6 (yikes).

[Faré (fahree)]

Final comment.

We at Google always check in a version of SLIME in our source control;
we build our code with precompiled support for swank,
and we debug with the matching version of SLIME.

This is the only reasonable way to be sure you can debug old versions of your servers: keep around a copy of as much of the toolchain as you need.

With this in mind, it really isn't necessary to support overly old versions of anything.

[Faré (fahree)]

Here's a diff.

It makes a best effort to support old versions of ASDF, but fails if you don't have at least ASDF2 installed. No one should be using ASDF 1, really; not only is it not supported anymore at all, it is a versioning nightmare.

I lightly tested with various old versions including ASDF 2.000, 2.008, 2.009, 2.010, 2.011, 2.012, 2.014.6, by hacking swank-loader to (load "/tmp/asdf.lisp") after I git show 2.000:asdf.lisp > /tmp/asdf.lisp and by commenting out all the asdf loading and upgrading mechanisms in swank-asdf.lisp.

To support specially loading with asdf, I replace compile-file-for-emacs by a version that consults a hook, then I have swank-asdf.lisp use that hook to implement adequate handling of .asd files and files managed by asdf.

Please test, and tell me how I should improve my code so it makes it upstream.

[Faré (fahree)]

NB: my patch was written against
commit 2767c53cb9f0f2b3993d744f9c9ab5d1af44c56f
Author: Stas Boukarev <email address hidden>
Date: Sat Aug 18 16:34:06 2012 +0000
from the git mirror at git://github.com/antifuchs/slime.git

[Faré (fahree)]

The previous patch had a swank.lisp that didn't work without swank-asdf.lisp. This fixes it, plus minor issues.

[Faré (fahree)]

OK, another improvement: this time, swank-asdf.lisp does more work at runtime rather than compile-time, so that if you install a newer asdf, the previously-compiled swank-asdf.lisp won't overwrite new definitions with old compatibility versions.

[Stas Boukarev (stassats)]

I applied it with some modifications. First, it requires *asdf-directory* to be set even if you already have asdf loaded, because it tries to upgrade asdf. That's a not very good idea, upgrading it without the user consent, so I commented it out.
Next, pushing try-compile-file-with-asdf to *compile-file-for-emacs-hook* by default is not good either, it modifies the default behavior significantly, so I commented it out and the users who want to do that should push it in their ~/.swank.lisp

[Faré (fahree)]

It is very important to upgrade ASDF *if* present: because, especially if the initial version of ASDF it old (before 2.014.8 or so), ASDF will not be able to upgrade itself successfully in the middle of loading a system that depends on ASDF. Also, when ASDF upgrades itself to a different version, it invalidates all loaded systems, because that's the conservative thing to do with respect to systems that depend on the guts of ASDF (such as CFFI-Grovel, etc.) and there is no practical way to tell which is which. Therefore, due to these two reasons, the only reasonable time to upgrade ASDF is very early in the game, before you use it to load anything.

Moreover, if someone does NOT want to upgrade ASDF, it is very easy: just do not have an asdf.asd anywhere in your registry. If you DO have an asdf.asd in your registry, I think it is a fair assumption that you want it upgraded.

If you are afraid of this, you may have (defvar *asdf-directory* NIL) and put the upgrade in an
(when *asdf-directory* ...), or (when (or *upgrade-asdf* *asdf-directory*) ...) where *upgrade-asdf* is another variable that defvar's to NIL (or if you follow my advice, to T). If you go this way, the (pushnew *asdf-directory* asdf:*central-registry*) should probably be in a (when *asdf-directory* ...), too.

As for try-compile-file-with-asdf, I admit it's a big change, but I believe a positive change. Maybe we should discuss it on the mailing-list? It's also necessary for some files, such as those that use the :around-compile extension. If the consensus is not to use it as part of the slime-asdf contrib (but why else to use slime-asdf), maybe make that push part of a trivial contrib slime-asdf-compile?

[Faré (fahree)]

And BTW, thanks a lot for your support!

[Stas Boukarev (stassats)]

I prefer to leave it for the user to upgrade ASDF, or anything else. I'd be quite unhappy if it did it silently. That's just a user program for querying ASDF, it shouldn't mess with the system in any way. Now, it may display a warning or something.

Nothing stops anyone from putting
(pushnew 'try-compile-file-with-asdf *compile-file-for-emacs-hook*)

into their ~/.swank.lisp, no need for a separate contrib.