Quicklisp-special ASDF repository setup no longer works: bad interaction with ASDF-package system

Bug #1307180 reported by Robert P. Goldman
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ASDF
Won't Fix
Wishlist
Unassigned

Bug Description

Xach uses a bespoke system definition search function. This function substitutes for the standard functions. Unfortunately, that function cannot successfully load systems defined using package systems.

Xach reports:

the affected application is the
one that builds quicklisp.

That application aims to keep total control over the way system names
are mapped to system files, so it can be sure that there aren't systems
implicitly arriving from somewhere else that can't be loaded by
Quicklisp later.

To observe it in action, create a sysdem definition search function that
associates a system name with the file that defines it. Mine is
conceptually something like this like this:

  (defvar *all-known-system-files*
    '(("3bil" . "/path/to/3bil/3bil.asd")
      ("anaphoric-macros" . "/path/to/anaphoric-macros/anaphoric-macros.asd")
      ...
      ("lil" . "/path/to/lisp-interface-library/lil.asd")))

  (defun build-quicklisp-searcher (name)
    (cdr (assoc name name *all-known-system-files* :test 'string=)))

Use that as the sole system definition search function.

In 3.0.3 that sufficed to find the package-system systems like
lil/interface/all. In 3.1 it seems not to, and it seems that the new
package system search function must be present in the search function
list to find them.

Revision history for this message
Robert P. Goldman (rpgoldman) wrote :
Revision history for this message
Robert P. Goldman (rpgoldman) wrote :

We probably need to specify what the protocol is for the system definition search function, and document it. I suspect that once we do this, the problem will turn out to be easily fixable (or we need to figure out a cleaner protocol).

Revision history for this message
Faré (fahree) wrote :

I'm not sure I understand the issue.

If Xach wants a list of all the systems under a package-system, so as to in turn check each of their dependencies, he has to

0- use load-asd to load each of the .asd files he found (I suppose he's already excluding duplicate thirdparty libraries imported into other systems).

1- detect that a system he loaded by scanning all the .asd's is indeed of type package-system.

2- if so, scan for all .lisp files under that hierarchy that are all lowercase and add them to the system list, using e.g. the **/*.lisp pattern, or something similar to asdf::collect-sub*directories-asd-files if he wants to exclude duplicates from source control on an old _darcs/ or .svn/ checkout or similar.

3- for each of them the files he found, he can use find-system and query the dependencies.

A simpler way to do things would be to just remember data about primary systems, but indeed, that would miss out on fine-grained dependency information, particularly from implicit secondary systems that are not directly named from the primary system definition file. Still, that should be pretty simple to achieve, and much better than not supporting asdf-package-system at all.

Revision history for this message
Faré (fahree) wrote :

Note that for the purpose of knowing the .asd file, stripping system names to primary system name is necessary and sufficient (at least with respect to the standard asdf system definition search functions) and so this is something that quicklisp *must* do. There is never a reason to have a non-primary name in the alist that Xach mentions (unless, once again, someone comes up with yet new extensions to asdf and its sysdef earch functions).

Revision history for this message
Robert P. Goldman (rpgoldman) wrote :

I don't think this is for introspection. He said this is for the code that builds quicklisp (I'm not entirely sure what that means). He gets a failure where your lisp-interface-library cannot find its components.

The system-definition-search-function must return, when given a system designator, the .asd file in which it is defined, correct?

So this means that in order to define your own you must do whatever lookup works for simple names and then:

1. Provide a solution for the "slashy names," that will remove everything after the slash and look for the resulting name, recursively and

2. Provide a solution for the package systems, which I don't pretend to understand.

Question: can we (do we?) provide functions for tasks 1 & 2 that outsiders can incorporate into their own system-definition-search-functions?

Alternatively, does FIND-SYSTEM need to be refactored so that manipulations of slashy names and package names happens outside of other processing of system names? If the logic for handling these is independent of the logic for finding foo.asd given "foo" , maybe that refactoring would be a better solution.

Revision history for this message
Faré (fahree) wrote :

I believe this is the code that inspects existing projects and gathers a list of

a- asd files
b- the systems defined by each of those asd files
c- the dependencies between each of these systems

See https://github.com/quicklisp/quicklisp-controller

To reply to your questions:

1. There is already a solution to remove everything after the slash of a system name, it's function primary-system-name.

2. I proposed two solutions for package-system's in my message above, one complete, and one that makes do.

I'm ready to help Xach, but he looks like he distrusts me, so he's on his own. Anyway, indirect bug fixing is not possible. Xach should join the conversation, if he has specific complaints.

Revision history for this message
Zach Beane (xach) wrote :

ASDF 3.1 introduces ASDF/PACKAGE-SYSTEM:SYSDEF-PACKAGE-SYSTEM-SEARCH to ASDF:*SYSTEM-DEFINITION-SEARCH-FUNCTIONS*. Unless it is present, LIL/INTERFACE/ALL is not found. It was not present in the 3.0.3 *s-d-s-f*. This change from 3.0.3 to 3.1.whatever has broken my software. Can the 3.0.3 behavior be restored?

Revision history for this message
Faré (fahree) wrote :

I'm not sure what you're requesting exactly. This function SYSDEF-PACKAGE-SYSTEM-SEARCH is an essential part of ASDF/PACKAGE-SYSTEM. It can't be removed from 3.1 without breaking this new facility.

For compatibility with 3.0.3, systems that depend on the package-system facility use :defsystem-depends-on (:asdf-package-system) which ought to provide this compatibility, including registering said function ASDF/PACKAGE-SYSTEM:SYSDEF-PACKAGE-SYSTEM-SEARCH. Is that not working? I just tested a minute ago, and it worked fine for me.

Are you actually requesting the asdf/package-system facility to be removed from 3.1? I would very much like it to remain, and I'm sure so do drewc and nyef, and possibly other users I don't know of.

You should be able to keep running quicklisp-controller with ASDF 3.0.3; when introspecting a package-system, the asdf-package-system extension will have been loaded by the defsystem form based on the :defsystem-depends-on declaration.

If you want to fully support these systems, my first comment explains how to do it:

0- use load-asd to load each relevant .asd file

1- detect systems that were defined, check if any of them is of type package-system. If you don't want to rely on either asdf 3.1 or asdf-package-system being always loaded, you can use
   (uiop:if-let (ps (uiop:find-symbol* :package-system :asdf/package-system nil)) (typep system ps))

2- if so, scan for all .lisp files under that hierarchy that are all lowercase and add them to the system list, using e.g. the **/*.lisp pattern, or something similar to asdf::collect-sub*directories-asd-files if you want to exclude duplicates from source control on an old _darcs/ or .svn/ checkout or similar.

3- for each of them the files you found, register a secondary package-system, and query its dependencies.

If you don't want to fully support these systems, you should be able to survive by collapsing them under their primary-system-name.

I don't understand how this SYSDEF-PACKAGE-SYSTEM-SEARCH is otherwise interfering with quicklisp-controller.

Revision history for this message
Zach Beane (xach) wrote :

I tried building the Quicklisp world with 3.1, and it does not build lisp-interface-library because of changes in what is required to be present in ASDF:*SYSTEM-DEFINITION-SEARCH-FUNCTIONS* for package-systems to work.

I want ASDF:*SYSTEM-DEFINITION-SEARCH-FUNCTIONS* in 3.1 to have the same required contents as 3.0.3, that is, that the package-system search function is not mandatory to have package-system libraries work with 3.1.

I don't want to write new code to fully support package-systems. They seemed to be supported enough with 3.0.3. Can 3.1 work without a new entry in ASDF:*SYSTEM-DEFINITION-SEARCH-FUNCTIONS* so I don't have to change anything?

Revision history for this message
Faré (fahree) wrote :

The entry *is* present with 3.0.3 — it's just added a bit later, by the defsystem-depends-on.

Is the issue that on 3.0.3, you're adding your entries first, then the asdf-package-system extension adds its own in front of yours, so yours get shadowed, whereas on 3.1, it gets shadowed by yours?

Revision history for this message
Zach Beane (xach) wrote :

That makes sense. My function replaces every other entry right after ASDF is initialized.

Revision history for this message
Faré (fahree) wrote :

Since I don't know what you're trying to do and what your search functions do, I cannot comment on a proper solution.

Revision history for this message
Faré (fahree) wrote :

Why does the shadowing of the search function by yours cause the package-system never to be seen?

Are your search functions trying to return the .asd file for the primary system, and then the actual secondary system never gets found? That's indeed problematic.

Maybe you should put your search functions at the end, but, in these tests, ensure the central-registry and source-registry are empty? Or do you also need them not to be empty?

Revision history for this message
Zach Beane (xach) wrote :

The only entry in asdf:*system-definition-search-functions* is my function. There is no central registry search or source registry search.

If package-system-related systems need a particular search function in the search function list, is it appropriate for them to ensure it's present with pushnew or similar?

Revision history for this message
Faré (fahree) wrote :

"to ensure it's present with pushnew" is precisely what the very last line of asdf/package-system.lisp does.

Unhappily, you subsequently clobber said function.

Instead of wholly clobbering asdf:*system-definition-search-functions*, couldn't you just remove-if from it members of an undesirable subset, i.e. SYSDEF-CENTRAL-REGISTRY-SEARCH and SYSDEF-SOURCE-REGISTRY-SEARCH ? Or, once again, you could (locally) bind *central-registry* to nil and *source-registry* to (make-hash-table :test 'equal).

Since you don't specify what exactly you're trying to do, it's hard to tell how to help.

Revision history for this message
Zach Beane (xach) wrote :

I'm trying to make everything build with 3.1 that built with 3.0.3 without changing my software.

Revision history for this message
Faré (fahree) wrote :

I fear you might have to either make a small change to your software or let the package-system fail.

Clobbering asdf:*system-definition-search-functions* is not modular and not future-proof — and not present-proof, either, it seems. What's wrong with the suggestion to not clobber it but instead just disable central-registry and source-registry in a different way, i.e.

(let ((asdf:*central-registry* nil)
          (asdf::*source-registry* (make-hash-table :test 'equal)))
   ...)

Revision history for this message
Robert P. Goldman (rpgoldman) wrote :

Would it work to do something like:

(setf asdf:*system-definition-search-functions*
  #+asdf3.1
  (list 'sysdef-package-system-search)
  #-asdf3.1
  nil)
;; push quicklisp function onto asdf:*system-definition-search-functions*

I'm afraid I haven't had a chance to look at your code, so I'm not entirely sure what's going on there, but this seems benign.

Revision history for this message
Zach Beane (xach) wrote :

I'm not interested in changing my code at this time. If ASDF does not change, I'll just omit lisp-interface-library until it does.

Revision history for this message
Faré (fahree) wrote :

If your code with unspecified behavior is perfect and other code has to adapt to it, fine. But since it's unspecified, I have no clue how to change either ASDF or lisp-interface-library to adapt to it.

I could do the very ugly thing of adding a special magic case of package-system in search-for-system-definition, but that would really go against the modular design of it all.

You're purposefully disabling package-system as well as all future sysdef search extensions ever to be made part of ASDF, then complaining that systems that depend on them are broken.

Revision history for this message
Zach Beane (xach) wrote :

Ok. I am happy to leave it at that.

Changed in asdf:
importance: Critical → Wishlist
status: New → Won't Fix
Revision history for this message
Faré (fahree) wrote :

Is it quicklisp-controller? Is the code checked in? Do you accept pull requests?

Revision history for this message
Zach Beane (xach) wrote :

I'm going to stick with 3.0.x, which doesn't require any changes to build everything.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.