Bug in use of function for output translations

[Robert P. Goldman]

Jason Miller reports:

I reduced a testcase from the metacopy-with-contextl .asd file. It
defines an output-files method to rename the .fasl so that it can
compile the same lisp files with different features set.

This worked fine when I was using normal output translations, but when I
switched to a :function output translation, it failed. I'm not sure if
this is a bug in asdf, a bug in my output translation function or a bug
in metacopy, so I'm posting it here for some advice.

Test-case is in a tarball below. There is a shell script "test.sh"
which invokes sbcl twice. The first time the output translation is left
unaltered, the second time it specifies a :function translator for all
paths, and this causes an error.

[Faré (fahree)]

Interestingly, if you display (output-files :compile-op '("my-system" "foo")) before and after you (load-system "my-system") you find that it disagrees.

The problem is due to some recent change I did (I'm not sure which) that is causing asdf to think it's not up to date, to actually reload itself, which loses the :around method.

[Faré (fahree)]

This also begs for a test case. Sigh.

[Faré (fahree)]

OK, my understanding of the issue is as follows:

1- The output through apply-output-translations, by being universal, applies to asdf as well as everything. This means asdf can't see its fasl where it now expects it, and decides that it needs to recompile itself.

2- As part of reloading itself, asdf resets all its generic. That's one of the reasons it loads itself before anything else.

3- But ASDF, when it reloads itself, doesn't reload previously registered system definitions unless its schema has changed (which it hasn't since 2.32.13), because it believes all the data will be preserved. The data is preserved indeed, but as we saw above, the methods are not.

Therefore:

A- For jason, the workaround is either (a) to (progn (asdf:upgrade-asdf) (asdf:clear-defined-systems)) right after you define your output-translations function, (you can skip the clear-defined-systems if you promise you upgrade-asdf *BEFORE* you load any .asd file or reference them in any way).

B- For us, we should probably clear-defined-systems always upon upgrade, for the reasons above, even if there was no schema upgrade of the data, because otherwise we will be missing methods. Thus I'm thinking that either cleanup-upgraded-asdf must always run the hooks, or the defun* and friends from uiop/utility must somehow be restricted to only undefine the target function it the version being upgraded from is too old...

[Faré (fahree)]

Once again, I have to significantly revise how ASDF upgrades itself.

This will be overall a simplification (punting, more often), but to do it right will take me some time. There are 16 cases of defun*, and 17 cases of defgeneric*. They must be replaced by simple defun and defgeneric, with the fmakunbound / undefine-function moved to a when-upgrading in upgrade.lisp. The version of the latest defun* is 3.1.7.27, and would protect that block.

[Faré (fahree)]

I have a fix in https://gitlab.common-lisp.net/asdf/asdf/merge_requests/36 but I need to write a test case, which will be painful.

Please don't release 3.1.8 without that fix, since it's a subtle regression.

[pipping (pipping)]

The MR was merged and it included an extension of test/test-sysdef-asdf.script so can this bug be turned into fix-committed as well?

[pipping (pipping)]

(of if it made it into 3.1.8: fix-released)?

[Faré (fahree)]

This was released as part of 3.2.0 (there was no 3.1.8, except as a ECL fork).