unable to reliably customize asdf output translations

[Anton Vodonosov]

Hello.

I want to report here limitation of asdf output translations customization.
This is not a request for a fix, as I found a workaround which works for me.
Reporting this to provide information which might be worth to consider in future
design choices.

My goal is to ensure libraries from private quicklisp installation
tested by cl-test-grid are freshly recompiled.
For this I customize asdf-output-translations for of all the libraries
stored inside the quicklisp directory so that their .fals files
are placed into temporary directory.

So it is simple case: for sources inside this directory place .fals
into that directory.

The official ASDF approach of passing configuration DSL to
asdf:initialize-output-translations fails on some lisp/OSes,
whatever variant of the DSL config I use, because of
the implementation dependent behaviour of cl:pathname-match-p.

The only reliable way we found to customize where ASDF stores the .fasl
files is to redefine the asdf:apply-output-translations function
(thankfully. asdf ensures the functions are never inlined, so I consider
this a relatevely reliable approach).

What I do in my redefined asdf:apply-output-translations? My approach
is not suitable as a generic solution, because to determine if one pathname is a
chile of another I just compare their pathame-directories and is if one is a prefix
of another (as a string) - I rely on the fact that in my case I don't have
weird pathmanes like a/../a/../a.

----

Another small note about configuration DSL. It is strange that
I can specify custom function for the second element of a directive -
TRANSLATION-FUNCTION, but not for the first element - DIRECTORY-DESIGNATOR.
It would be more symmetric if I could pass custom functoin for the directory
designator.

----

As I said, this issue is not pressing for me, I have a workaround.

[Faré (fahree)]

Your problem description is not sufficient for me to take action.

Is your issue due to some implementations not fully implementing cl:pathname-match-p ?

What semantics would you like for your function on the left-hand side of an output-translation ?

Note that we currently sort the output translations by the prefix size of these left-hand-side (search for stable-sort in asdf.lisp). We could either have a prefix plus guard, or give priority to all guards as with an empty prefix.

Can you post your changes to apply-output-translations, so we have a more concrete basis to discuss things?

Thanks.

(Marking the bug report as Incomplete in the meantime.)

[Faré (fahree)]

Anton, is there anything left to do about that bug? Otherwise, I'll resolve as "Won't Fix".

[Anton Vodonosov (avodonosov)]

> Is your issue due to some implementations not fully implementing cl:pathname-match-p ?
I don't know exactly why.

In short, if I want for sources in directory A to have fasls in directory B it is impossible to achieve
with current ASDF output translations.

Sorry I didn't posted examples, but I submitted the ticket half a year after I dealt with this problem.
Unfortunately at the moment when I encountered the problem it was one of a whole tree of problems
laying on the path to particular goal, so I was unable to investigate more.

As I remember translations didn't work by various reasons, depending on situation. I think
one of the cases was that directory disegnator resolved to C:\somewhere and ASDF compared
this with c:\somewhere and cl:pathname-match-p didn't matched them. I don't remember all
the cases.

> What semantics would you like for your function on the left-hand side of an output-translation ?
To answer this question today, I need to learn the ASDF output translations DSL and machinery again.
I think I wanted to be able to specify a function accepting untranslated .fasls and
returning a boolean whether the .fasl matches this output translation entry.

Alternatively, ability to specify a function standing for both left and right side of translation entry would
also be convenient. It accepts untranslated .fasl and returns translated .fasl. Or NIL the function
doesn't consider itself responsible for this .fasl (based on it's directory for example).

> Can you post your changes to apply-output-translations, so we have a more concrete basis to discuss things?

My changes are here, but as I said they are not general enough to applied to official ASDF, only work for me:
https://github.com/cl-test-grid/cl-test-grid/blob/1d426ff14d181144e175a9b020ff1d5770deb0b2/agent/proc-common-asdf.lisp

[Faré (fahree)]

1- ASDF 3 now has a hook to redefine the output-translation function: asdf/driver:*output-translation-function*
Note that you must change it *after* having updated asdf and/or asdf/driver.

2- There have been a *lot* of bug fixes in the past 8 months, including serious ones from 2.26 to 2.27 specifically in the output-translation layer. Adding tests revealed that I was doing something non-portable before that worked on most but not all implementation. I was notably unintentionally failing to wilden some of the pathnames, which makes me surprised the old code worked at all.

3- On which implementation(s) are or were you experiencing issues?

4- I still don't understand what you want to do that isn't done by ASDF. Can you give an example with expected output-files vs what it actually returns?

Please tell me if your issues persist.

[Anton Vodonosov (avodonosov)]

1 - that's cool, I will wait a little until ASDF 3 is widely adopted and then attach my workaround to this hook, instead of redefining asdf:apply-output-translations.

2-3 Unfortunately I don't have an easy way to reproduce this problem. The old code with ASDF output DSL i had is gone.

4
> I still don't understand what you want to do that isn't done by ASDF.
I my workaround I did less than ASDF. Ignored host and device and assumed that a file is a child of my directory
if the file path has the directory as a string prefix.

> Can you give an example with expected output-files vs what it actually returns?
translation entry just didn't matched the file and ASDF returned to default behavior, ignoring my translation.

Feel free to close the ticket. If I meet the problem again or have more information, I will reopen it.

[Anton Vodonosov (avodonosov)]

P. S. can you help me to form an output translation DSL that for sources in directory A will store fasls in directory B?
It will lower a barrier for me to reproduce the problem.

[Faré (fahree)]

You can
 (asdf:initialize-output-translations
         `(:output-translations (,a ,b) :inherit-configuration))
or you can put in your ~/.config/common-lisp/asdf-output-translations.conf the expanded form
 (:output-translations ("/path/to/a" "/path/to/b") :inherit-configuration)
or you can run your program in an environment where you
 export ASDF_OUTPUT_TRANSLATIONS="/path/to/a:/path/to/b:"

[Faré (fahree)]

Marking as WONTFIX for now.

Please reopen if you experience trouble, with more detailed description of the failure.