wanted: :IMPLEMENTATION keyword for TEST-OP

[Tobias C. Rittweiler]

I want to run a test suite on an implementation I'm not currently working in. For example,
TEST-OP could take an :IMPLEMENTATION argument (with one possible value being :ALL
to test all "registered" implementations -- whatever "registered" means.)

Fare nicely collected sexps describing how to do that for different implementations at

  http://common-lisp.net/gitweb?p=projects/xcvb/xcvb.git;a=blob;f=lisp-invocation.lisp;h=634d49af9169b6a4103a76cabd07901da6ca237b;hb=HEAD

[Faré (fahree)]

Note that if you want to use this in some package other than ASDF (that strictly speaking, must remain a single standalone file), then I'd gladly split lisp-invocation into its own build/system so that it may be used independently from the rest of XCVB.

[Tobias C. Rittweiler (tcr)]

I wouldn't mind if this was part of a contrib [of an imagined contrib system].

[Robert P. Goldman (rpgoldman)]

So presumably this would run a test in an outside process? This would be more generally useful, actually. For example, we have test suites of systems that use some external libraries which do not adequately control their internal states, that really should be run in a separate process.

See my response on the DOC-OP ticket --- you don't need to add :implementation to test-op; just use original-initargs. That's a better solution (although somewhat kludgy) anyway, because it's backward compatible, and ASDF does not have a reliable channel for updates into different implementations.

[Tobias C. Rittweiler (tcr)]

Yes, it'd be run in an outside process. And Fare's "lisp invocation database"
would be the abstraction to construct the relevant shell command.

When I was talking about adding another keyword argument, I didn't mean
to change the signature of PERFORM, or something like that, but to
add that keyword to the ASDF:TEST-SYSTEM function to show its
availability to users.

So it seems this should come in two pieces:

  a) add the possibility to perform an operation in an outside process.

  b) add the possibility to use another implementation as the running
      Lisp in an outside process.

[Nikodemus Siivola (nikodemus)]

This feels out of scope for out-of-the-box ASDF to me. I'm not dead-set against it, but it seems like the notion of "operation" here is very different from ASDF's notion of operation.

What you want seems to be something like

 (do-lisps (lisp *all-lisps*)
   (unless (= 42 (run-lisp lisp
                           '(progn
                              (require :asdf)
                              (asdf:load-system :foo)
                              (asdf:test-system :foo)
                              (fooext:quit 42))))
    (error "~A didn't pass the muster" lisp)))

which to me seems to have very little to do with ASDF.

[Robert P. Goldman (rpgoldman)]

I like this idea as a contrib, but we can't really implement it until we've addressed https://bugs.launchpad.net/asdf/+bug/479478

[Robert P. Goldman (rpgoldman)]

I am milestoning this to ASDF 3 (partly to see if anyone objects). I propose that fixing the TEST-OP is something we have to postpone in the interests of getting this out the door.

If you feel strongly otherwise, please continue the discussion here.

[Faré (fahree)]

Note that lisp-invocation.asd was split from the rest of xcvb.asd (though not the rest of the tarball).

You can therefore use it today already. Doesn't integrate with test-op, but might integrate with another test library?

[Faré (fahree)]

Now that ASDF *does* have a viable "contrib system", in the form of defsystem-depends-on, the asdf-contrib system (that just depends on a list of other systems) and quicklisp, I think resolving this issue can be moved out of ASDF itself, which should remain just the minimal build system to bootstrap other stuff. As big as this "minimal" might be, it doesn't include a test system.

I emphatically don't mean this work shouldn't happen — I'd love for there to be a One True test system for CL. But ASDF just doesn't seem to me (at this point at least) to be the place to do it.

And so I'd like to mark this bug as "Won't Fix" or otherwise "not my problem".

[Faré (fahree)]

Marking as "Won't Fix".

I encourage those who care to develop a suitable ASDF extension.