Implement quick-build support

[Faré]

quick-build, like faslpath, has an interesting way of specifying one package per file, and deducing file dependencies from which packages are used or imported from in the file's defpackage header.

It would be nice to have ASDF recognize the same pattern, and thereby allow seamless integration between the two styles. One way to do things would be to model each quick-build file as its own simple system, and have a system search function look in suitably registered paths for them, which could be part of the source-registry if done eagerly, or a separate search function if done lazily.

[Robert P. Goldman (rpgoldman)]

What is quick-build? I just see a pointer to it as "unreleased" on the ASDF page.

I could imagine butchering an outside build system into ASDF (we have a half-baked integration of MAKE into ASDF where I work). I'm not sure there's a big call for that for this alternative, though.

[Drew Crampsie (drewc)]

quick build attached.

it is not really an 'outside' build system per se, it simply COMPILEs and LOADs the files in the order that is specified by the package declaration.

[Faré (fahree)]

For the similar faslpath, see
http://www.cliki.net/faslpath
https://code.google.com/p/faslpath/

[Alastair Bridgewater (alastair-bridgewater)]

As the original author, I'm somewhat qualified to comment on quick-build. It's basically two afternoon's worth of hack, rather than anything really well designed. The version that Drew shared is pretty much up-to-date, and I have only ever used it with SBCL.

Were I to write it again, I would probably use a different representation for the graph and nodes (filenames+dependencies), almost certainly a class-based representation for the nodes if not necessarily the graph overall. I might also add a more flexible method for determining dependencies than simply requiring that the first form in the file be a CL:DEFPACKAGE and parsing it for :USE, :IMPORT-FROM, and :SHADOWING-IMPORT-FROM clauses. I might also allow specifying a "node class" per prefix, which would allow overriding the default mapping from package name to file path and determining the dependencies.

Quick-build fits very well with the code style that I have come to use, largely because it was written after the basics of said style had been settled, and I was at the point of getting tired of maintaining a linear build order, and didn't want to use ASDF. At the same time, it's very much at the level of a personal tool in terms of design (as described above), implementation (as described above, plus the complete and total lack of tests), and documentation (completely nonexistent).

All that said, please don't let me stop you from adopting it, or the ideas that it demonstrates. But if you DO adopt it for ASDF or simply as a public project, please address at least the matter of documentation, if not also customizable mappings from package names to file names and determination of dependencies.

[Faré (fahree)]

OK, see what I pushed in this topic branch:

http://common-lisp.net/gitweb?p=projects/asdf/asdf.git;a=shortlog;h=refs/heads/package-system;js=1

In 92 lines, I give ASDF support for quick-build style packages, except with ASDF integration, correct timestamp propagation (quick-build has the same bug as all free defsystems until ASDF 2), pathname portability, support for uiop:define-package.

Documentation is only in the commit message, so far:

    package-system: support for quick-build style defpackage-based dependencies.

    This is quick-build compatible and fixes lp#1230368.

    To use package-system, just have foo.asd containing
        (defsystem foo :class package-system)
    at the top of your quick-build hierarchy $FOODIR
    for packages whose name start with "FOO/"
    and ASDF will thereafter look for system "foo/bar/baz" in $FOODIR/bar/baz.lisp.

    Such a file will implicitly have its own system defined;
    its dependencies are computed by scanning the file,
    extracting its first defpackage form,
    and using the packages it uses or imports from
    as a as a specification of what systems it depends on.

    You can register packages as belonging to a system with
        (asdf:register-system-packages my-system '(package1 package2))
    Using or importing from a package registered to a given system
    will generate a dependency to the registered system.
    Using or importing from a packages registered to the constant symbol T
    will not generate any dependency.
    Using or importing from a packages that is not registered will generate
    a dependency on a system the name of which is the package name downcased.
    All packages that exist at the time ASDF is initially loaded
    are registered to constant symbol T.

Robert: can you review and hopefully pull into the master branch before the next release?

Alastair: can you review and try in your existing system? I understand ASDF is much bigger than quick-build, but it also does much more, including fixing the "usual" defsystem timestamp propagation bug. Plus, your dependencies mean you probably use ASDF anyway (if you so much as require sb-posix, and don't use my SBCL patch, you're loading ASDF).

[Robert P. Goldman (rpgoldman)]

I'm actually not going to review and pull this into 3.0.3. 3.0.3 is a bugfix release, and this is major new functionality.

This will have to be deferred to 3.1.0, or at least a release candidate for 3.1.0.

Sorry, but I don't have time to review and test, and I would like to push out the existing bug fixes, and maybe one or two more.

[Robert P. Goldman (rpgoldman)]

I answered that too quickly. Another reason that this is not a good choice for 3.0.3, IMO is that it is a major API change and so requires some way of programmers detecting it. That at least means a .x level version number, and I think there's a lot to be said for kicking this can down the road until something we call ASDF 4, so that #+asdf4 checks can be employed.

[Faré (fahree)]

OK, I spawned a system asdf-package-system so the same functionality be usable with older versions of ASDF.

  git://common-lisp.net/projects/asdf/asdf-package-system.git
  http://common-lisp.net/gitweb?p=projects/asdf/asdf-package-system.git

NB: I made modifications based on this system, and rebased the above branch in the asdf.git.

[Faré (fahree)]

For backward compatibility with older versions of ASDF, I recommend
the following:

(defsystem top-of-my-quick-build-style-hierarchy
  :class package-system
  :defsystem-depends-on
  #.(unless (find-class 'package-system nil) '(:asdf-package-system)))

[Faré (fahree)]

I have historically not refrained from adding new features in minor releases. To me major version bumps (e.g. asdf4 vs asdf3) would be to signal significant incompatibility. That said, such a feature might warrant a 3.1 vs a 3.0.3 — though really, it's an add-on.

On the other hand, you're the maintainer, so you get to decide these things now.

As for the functionality test, here's what I recommend (so far):

  :class asdf::package-system
  :defsystem-depends-on #.(unless (find-class 'asdf::package-system nil) '(:asdf-package-system)))

(No asdf:: prefix needed if you (in-package :asdf), but it might be needed if you're merely in the default asdf-user.)

[Faré (fahree)]

As for the refactoring, it's simply renaming defsystem.lisp to parse-defsystem.lisp, to avoid the naming conflict between asdf/defsystem, as defined in asdf.asd and asdf/defsystem.lisp, now renamed asdf/parse-defsystem.lisp.

I also reordered things in the Makefile so subfiles are hopefully concatenated in the same order as in with asdf's monolithic-concatenate-source-op.

[Faré (fahree)]

I believe it's a testimony to the good basic design of ASDF3 that this support could be added in 93 lines of new code, which is half the size of quick-build itself, with a result that eschews the usual pre-ASDF3 defsystem bug and is more portable — taking advantage of the existing body of code.

On the other hand, the reason I didn't push to master is precisely so as not to preempt your prerogatives as the new maintainer in the face of such refactoring as above, however simple it might seem to me. I totally understand that you may take your time to merge my branch.

[Faré (fahree)]

NB: I rebased that branch, twice, to fix bugs. Fixes also available in asdf-package-system.

[Faré (fahree)]

Committed in 3.1.0.1.