Comment 3 for bug 479478

I recently bumped up against half of this problem a few times: I needed to programmatically determine whether a test suite passed. My use cases happened to be a script for a script for a git pre-commit hook, and an attempt to add the ability for a test library to test a system in a fresh remote lisp. I anticipate bumping into it again if I set up continuous integration.

(The other half of the problem, in my opinion, is the coordination issue. That is, given a large set of other people's systems how do I programmatically determine which ones pass their tests? There seemed to be considerable excitement on that front in the 2009 thread but I haven't run into this issue personally.)

I looked for a nice way to solve the first half of the issue in the ASDF manual, but found instead a reference that test-op has been discussed in the bug tracker and mailing list -- so here I am.

Proposal:

  1. Make ASDF:TEST-SYSTEM and the corresponding ASDF:OOS call return the value of the associated ASDF:PERFORM call. That would solve my issue nicely.

  2. Separately reconsider whether the second half is still desirable. If so, endorse a return format and ask users to follow it. CL-TEST-GRID seems relevant to this, both for being the largest example of programmatically testing arbitrary CL projects and for having its own test return format, perhaps we could get Anton to comment.

In the previous discussion it was pointed out that some systems are tested in a way such that they would not be able to profitably take advantage of this {particularly systems which use :in-order-to ((test-op (test-op subsystem)))}. I think the answer is "okay, this isn't for those systems. It won't break their current practice."

Here's the previously mentioned pre-commit hook. Note the circuitous logic of sending the output to a string, which I then both read (to get at the value which I would like to make test-system return) and print (to recover the output).

#!/usr/bin/sbcl --script
;; If I want to commit while the tests are failing, the hook requires me to pass a force flag.
(let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
                                       (user-homedir-pathname))))
  (when (probe-file quicklisp-init)
    (load quicklisp-init)))

(sb-ext:quit :unix-status
             (handler-case (let ((string
                                  (with-output-to-string (*standard-output*)
                                    (asdf:test-system 'dishes))))
                             (if (eq (read-from-string (print string)) :ok)
                                 0
                                 1))
               (serious-condition () 1)))