ASDF update by quicklisp client on CMUCL 20c breaks ASDF

[Anton Vodonosov]

After updating quicklisp-client to 2012112500, which comes with ASDF 2.26, CMUCL 20c has the following problem:

CMU Common Lisp 20c release-20c (20C Unicode), running on ubuntu
With core: /home/testgrid/lisps/cmucl-20c/lib/cmucl/lib/lisp-sse2.core
Dumped on: Thu, 2011-11-03 10:30:38+02:00 on lorien2
See <http://www.cons.org/cmucl/> for support information.
Loaded subsystems:
    Unicode 1.28 with Unicode version 6.0.0
    Python 1.1, target Intel x86/sse2
    CLOS based on Gerd's PCL 2010/03/19 15:19:03
* (ql:quickload :alexandria)

Error in KERNEL::UNDEFINED-SYMBOL-ERROR-HANDLER: the function ASDF:FIND-SYSTEM is undefined.
   [Condition of type UNDEFINED-FUNCTION]

Restarts:
  0: [ABORT] Give up on "alexandria"
  1: Return to Top-Level.

[Anton Vodonosov (avodonosov)]

Related quicklisp ticket: https://github.com/quicklisp/quicklisp-client/issues/68

[Zach Beane (xach)]

I get this:

$ lisp
; Loading #P"/home/xach/.cmucl-init.lisp".
;; Loading #P"/home/xach/quicklisp/setup.lisp".
;;; Loading #P"/usr/local/lib/cmucl/lib/contrib/asdf/asdf.sse2f".
; [GC threshold exceeded with 12,007,960 bytes in use. Commencing GC.]
; [GC completed with 1,731,200 bytes retained and 10,276,760 bytes freed.]
; [GC will next occur when at least 13,731,200 bytes are in use.]
; Compiling LAMBDA (.PV-CELL. .NEXT-METHOD-CALL. M ADDED DELETED PLIST .REST-ARG.):
; Compiling Top-Level Form:

Error while trying to load definition for system quicklisp from pathname
/home/xach/quicklisp/quicklisp/quicklisp.asd:
   No matching method for the generic function
   #<STANDARD-GENERIC-FUNCTION #:SYSTEM-SOURCE-FILE (2) {58D56A61}>, when
   called with arguments (#<SYSTEM "quicklisp">).
   [Condition of type LOAD-SYSTEM-DEFINITION-ERROR]

Restarts:
  0: [CONTINUE] Return NIL from load of #P"/home/xach/quicklisp/quicklisp/quicklisp.asd".
  1: Return NIL from load of #P"home:quicklisp/setup.lisp".
  2: Return NIL from load of "home:.cmucl-init".
  3: [ABORT ] Skip remaining initializations.

Debug (type H for help)

("DEFUN* LOAD-SYSDEF" #<PCL::NO-APPLICABLE-METHOD-ERROR {58974635}>)

[Faré (fahree)]

OK, so logging on your machine, and trying to debug things in an emacs M-x shell, I find that:
* Quicklisp loads the builtin ASDF, but finds it's too old.
* Quicklisp tries to compile a new ASDF, but CMUCL borks during compilation (see below).
* Next time around, Quicklisp tries to read the incomplete fasl produced by the previous compilation, and fails.

There is probably a combination of CMUCL compiler bugs behind this:
* CMUCL fails on the defmethod shared-initialize :after, because, says it, operation-forced-not is not a defined function. Yet the function is defined two forms above as an accessor in defclass operation, and it doesn't error out when compiling the file from clean rather than from a situation where a previous asdf was loaded. Presumably, CMUCL fails to properly process the defclass because it believes the defclass already happened.
* Why should an undefined-function in this case be an ERROR and not just a style-warning? That's probably another bug in CMUCL.
* If I fix this problem, then similar issues crop up with %system-source-file.

Quicklisp could be made somewhat more robust against these failures by compiling to a temporary location, then renaming if the compilation was successful. It will still fail, but at least, the second time around the cache won't have been poisoned with a fasl that looks good but isn't.

I have updated ASDF 2.26.5 to work around this CMUCL issue.

[Anton Vodonosov (avodonosov)]

Let me make little conclusion. It seems to be not very trivial job to maintain ASDF upgradable at run time.

As a user I would prefer to have single ASDF version, on all my implementation.

The fact that ASDF is delivered with lisp implementation was caused by something like a chicken and egg problem:
lisp implementation developers wanted to provide module definition system, but without separate installation step.
In result every lisp implementation mainains their own clone of ASDF.

Today, as we have pure lisp delivery system - quicklisp - another option is possible.

Lisp implementations deliver a quicklisp bootstrap file - http://beta.quicklisp.org/quicklisp.lisp.
With this file, users can quicklisp and therefore ASDF.
So we can use the single ASDF across all the lisp implementations, updated regulary.

In short, I wish there will be an option to not load the ASDF provided with CL implementation,
but always use the ASDF provided with quicklisp. This is just simpler for me.

[Faré (fahree)]

Well, in your model, whenever any bug is found in ASDF for some new version of some implementation, then quicklisp must be updated or else becomes incompatible with the implementation. (Recent example: SMP vs non-SMP builds in Allegro, or change in FASL numbering internals in CCL to make it per-architecture, or massive recent fixes for ECL, or upcoming changes in ABCL, etc.)

I don't believe that Xach will like this plan. He only updated ASDF once every year and half, I don't think every month of every two weeks is an option.

[Anton Vodonosov (avodonosov)]

Once a month, the same way as slime or any other library would be good in my point of view.

Maybe it could be and option of quicklisp - not try to (require 'asdf), just use your own ASDF.

Anyway, it's not a call for an action roght now, just thoughts.

[Robert P. Goldman (rpgoldman)]

I would NOT like to see this happen. My company's lisp development relies on us maintaining our own libraries. We think that quicklisp is a great thing for many purposes, but do not want to see reliance on it baked into lisp implementations.

[Faré (fahree)]

Making ASDF self-upgradable was hard, but it was a requirement for ASDF 2, and it's mostly done now, modulo this (now resolved) glitch with CMUCL, and some issues with how CCL or ECL broke old versions of ASDF.

Self-upgrade is now automatically tested as part of our test suite. But I need to reinstall some implementations on my development machine, such as CMUCL.

[Anton Vodonosov (avodonosov)]

Ah, right, we can't insist everyone use quicklisp.

Then different model:
1. Lisp implementation doesn't force me to use ASDF bundled with it.
    Unless I explicitly say (require :asdf), the ASFD bundled with lisp
    implementation is never loaded.

2. Quicklisp has an option to not perform (require 'asdf),
    but immediately load the ASDF provided with quicklisp.
    Maybe this should be the default behaviour rather than an option.

In this model I am free to use whatever ASDF I want. Workflow of
people like Rober is not affected. I can use the ASDF bundled with quicklisp.

A practical obstacle to (1) I am aware of is descirbe here:
https://github.com/quicklisp/quicklisp-client/issues/68#issuecomment-10847593
Lisp implementation (ECL) uses ASDF in it's internals - the sockets
module is an ASDF system. So, the lisp implementation depends
on particular version of ASDF and can't work if I load my version of ASDF.

If I understand correctly, solving this somehow (maybe using tradition
that lisp implementations do not use ASDF internally), will make it
possible for users to work single ASDF version.

----------------------------
I realize it's becoming a bit offtopic for this ticket, but even considering
that making ASDF upgradable is solved already, I am interested to consider
a solution to have single ASDF version.

The whole mechanizm that ASFD is upgradable, and Quicklisp first tests
if ASDF is loaded already, if not then tries (require 'asfd) to load the ASDF version
of the current lisp implementation, if it fails or the loaded ASDF version is too old,
then upgrades it loading it's own ASDF; the whole this mechanizm is
somewhat complex. I don't know a single place where all this is documented,
and it took some time and internet/source code to learn this and be able
to determine, what version of ASDF was used by my program.

[Robert P. Goldman (rpgoldman)]

Isn't this up to the implementation? I mostly use ACL, which has its own DEFSYSTEM, so I must (REQUIRE 'ASDF) to get the bundled ASDF.

OTOH, I think SBCL, at least, may use ASDF to manage its own build, in which case I'm not sure how to make it unavailable (although I imagine that could be managed).

[Faré (fahree)]

1. I don't know any implementation that loads ASDF without you explicitly doing a (require "asdf").

Back in the bad old days of ASDF 1, there was no way to (require "asdf") except maybe on SBCL, no way to guess where to find a proper precompiled asdf for your implementation, and no way to add desired ASDF features (such as output translation or preconfiguration of central registry) without first loading ASDF. Therefore, common-lisp-controller (on debian and gentoo and maybe other distributions) used to dump images with ASDF pre-loaded. This was great when it worked, but was sometimes hell, because it was impossible to load a newer ASDF on top of that (or older, or sometimes even the same), making the simplest build scripts impossible to write portably. These bad old days are past. Please don't take us back there. Looking back, I'm proud of ASDF 2.

2. I suppose your ~/.sbclrc, ~/.cmucl-init.lisp, or whichever file loads your quicklisp could defparameter something (in which package, though?), and quicklisp could consult that variable before it loads asdf. Or quicklisp could have an optional ~/quicklisp/setup-configuration.lisp that is read during setup where you could specify such preferences. At that point though, that becomes a Quicklisp issue, and you should open a ticket on https://github.com/quicklisp/quicklisp-client/issues

3. For next version of ASDF, my proposed solution is that any self-upgrade bugs will be fixed as have been with CMUCL, and that asdf-bundle will be bundled with ASDF, which should solve any and all issues for ECL.

4. I don't think that the current behavior is particularly complex. I don't see what simpler behavior you expect that can work for everyone on the 9 active supported implementations (and optionally the 6 inactive implementations and 1 future announced implementation) in a future-proof way. If you see one, please explain.

5. The current behavior is not less documented than much anything in the CL world. I don't know what documentation you're thinking about, but whichever exists is accepting patches, and whichever doesn't exist is waiting to be written.

[Anton Vodonosov (avodonosov)]

No doubt the work done for ASDF 2 deserves to be proud of.

I thought lisp implementations force user to use particular version of ASDF by REQUIRing ASDF or by depending on features specific to particular ASDF version, because of the ECL problem we see and because ASDF is upgradable (now I glanced again through the asdf papers and see the reason to be upgradable is be able to load ASDF via ASDF).

If so, I can use an ASDF version of my choice.

Another option to prevent Quicklisp REQUIRing ASDF is to load my ASDF before Quicklisp (if ASDF is already loaded and it's version satisfies Quicklisp, then Quicklisp doesn't try to reload ASDF)

[Faré (fahree)]

In the bad old days of ASDF1, the few implementations that provided ASDF each provided their own patched version.

In the days of ASDF2, they all provide a baseline release, except when an urgent patch was required - but return to mainline at next release while mainline quickly merges in the patch. So it's only a matter of how current your ASDF is - don't use something older than what your implementation provides; something newer is encouraged if you need it.

ECL is an exception in that it also loads asdf-bundle (nee asdf-ecl) when you (require "asdf"). This tends to defeat quicklisp, but I'll fix that by merging asdf-bundle into asdf.

Yes, you can prevent Quicklisp from loading a bad ASDF simply by loading yours first. If you're using ECL, you should also load asdf-bundle immediately afterwards.

[Faré (fahree)]

Fix released in 2.27.