cl file types on asdf2.014.6, SBCL 1.0.29.11.debian

[Ernst van Waning]

Dear ASDF maintainer,

having downloaded quicklisp, added quicklisp to my init file, I
automatically switched form asdf 1.30 to asdf 2.014.6, which was
perhaps long overdue.

Being one of those whose source files all have the cl extension, I
have trouble to convince asdf2 of that fact. The system I use to make
asdf use cl extentions has the following definition:

;;; system.asd

(in-package :cl-user)

(eval-when (:execute :compile-toplevel)
  (require :asdf))

(use-package :asdf) ;asdf 2.014.6, seems to expect this, but still it assumes .lisp sources
;;; the manual seems to require the use-package, but use-package does not seem to make any difference here

(defclass clcl-source-file (asdf:cl-source-file)
  ((type :initform "cl")))

(defmethod asdf:cl-source-file ((f clcl-source-file) (m asdf:module))
  (declare (ignorable f m))
  "cl")

(asdf:defsystem :symquery
    :version "0.6.0"
    :serial t
    :default-component-class clcl-source-file
    :components ((:file "packages")
                 (:file "symquery")))

(defmethod asdf:source-file-type ((c clcl-source-file)
      (s (eql (asdf:find-system :symquery))))
  "cl")

;;end system.asd

The file above compiles and loads fine with SBCL 1.0.29.11.debian. The modules packages and symquery are both
instances of class clcl-source-file. However, the functions
component-pathname and source-file-explicit-type disagree on the
extension:

CL-USER> (mapcar #'asdf:component-pathname (asdf:module-components (asdf:find-system :symquery)))
(#P"/home/ernst/src/CL/symquery/packages.lisp"
 #P"/home/ernst/src/CL/symquery/symquery.lisp")
CL-USER> (mapcar #'asdf::source-file-explicit-type (asdf:module-components (asdf:find-system :symquery)))
("cl" "cl")
CL-USER> (let ((sq (asdf:find-system :symquery))) (mapcar (lambda (x) (asdf::source-file-type x sq)) (asdf:module-components sq)))
("cl" "cl")

Having followed some days the asdf-devel mailing list, you certainly
have more pressing problems. OTOH, I have tried for days now to find
something to make my source-files with cl-extension load as they did
with asdf 1.130, but so far I didn't succeed. Could you please tell me
what I should do?

Hoping to hear,

Kind regards,

Ernst van Waning

[Robert P. Goldman (rpgoldman)]

I have created and pushed a test case that verifies part of this bug report.

I took the existing test-source-file-type.script and extended it to check the COMPONENT-PATHNAME. Indeed, it does not honor the override of the SOURCE-FILE-TYPE method.

This is in a system that only overrides SOURCE-FILE-TYPE and that does not mess with the :TYPE slot. I leave that for someone else; I am not sure I understand the internals well enough to know whether (a) it's a good idea to mess with the type slot or (b) it's really an error if EXPLICIT-FILE-TYPE is wrong in this case. I leave that for wiser heads.

Note that this entails that ASDF will now fail its tests. Still, since there's a bug, that seems appropriate.

[Robert P. Goldman (rpgoldman)]

<Blush> I was wrong. Turns out Ernst's test case and ours were different. So my test was formulated wrong, and I have had to push a revision. The new test passes, because it does NOT replicate Ernst's claim of a bug in the overriding of SOURCE-FILE-TYPE. </Blush>

I am not sure what accounts for this. Perhaps messing with the TYPE slot has somehow gummed this up. However, COMPONENT-PATHNAME works properly when SOURCE-FILE-TYPE is overridden, as far as I can tell.

The package stuff here seems problematic. Using the package ASDF in CL-USER seems like a very risky thing to do (I know it would pretty badly break on ACL).

Ernst, please compare with test/test-source-file-type-1.asd in the asdf distribution. Do the tests run correctly for you?

Try ripping out the :initform for TYPE in your class redefinition. Does that make COMPONENT-PATHNAME function correctly for you?

I cannot replicate this bug report.

[Faré (fahree)]

Your example works on my machine, using sbcl 1.0.47.debian on amd64. At least, when I copy/paste your "system.asd" into a "symquery.asd", remove the lines from in-package to use-package.

I added tests in the test-suite for this functionality, and it passes in every implementation.

Correction: before 2.014.11 (just committed), it was subtly failing on allegro, due to a conflict with methods from the previous asdf as autoloaded from allegro. Hum. Could it be that you are using a new asdf on top of an old asdf and that this is indeed causing a conflict?

In 2.014.11, I mark gf source-file-type as being redefined, to avoid this issue.

Please retry with 2.014.11.

[Ernst van Waning (evw)]

Dear Faré,

the new version works with clcl-source-files. Thank you very much for
the upgrade.

As I now seem to be joining projects-in-progress (asdf, quicklisp), I'd
like to become a bit more active in that area. Referring to our chat,
let us first see what the barriers to cooperation are exactly before we
declare them to be too high. My barrier to cooperation is that I
probably must become more flexible some old Lisp-habits I developed
(probably without too much thought) over the years and become familiar
with APIs designed by others. First thing is that I must want to
understand new APIs before I can say something useful about them.
Personally, I have always found our community very isolated (too many
Lonely Lispers). Cooperation might counter that, I hope :-)

Thank you & kind regards,

Ernst

PS some comments interspersed bwlow. Happy Easter!

Faré schreef op zo 24-04-2011 om 17:23 [+0000]:
> Your example works on my machine, using sbcl 1.0.47.debian on amd64. At
> least, when I copy/paste your "system.asd" into a "symquery.asd", remove
> the lines from in-package to use-package.

These lines were added out of frustration :-) They are removed again.

> I added tests in the test-suite for this functionality, and it passes in
> every implementation.
>
> Correction: before 2.014.11 (just committed), it was subtly failing on
> allegro, due to a conflict with methods from the previous asdf as
> autoloaded from allegro. Hum. Could it be that you are using a new asdf
> on top of an old asdf and that this is indeed causing a conflict?
>
> In 2.014.11, I mark gf source-file-type as being redefined, to avoid
> this issue.

Downloaded, installed, loaded and after (asdf:upgrade-asdf) it worked.
But I did not undestand your comment about upgrade-asdf during our chat,
My SBCL comes with an old asdf installed and some systems (grovel,
ACLrepl, ...) are defined by that version. Could you please explain?

> Please retry with 2.014.11.
>
>
> ** Changed in: asdf
> Importance: Undecided => Low
>
> ** Changed in: asdf
> Milestone: None => version2.1
>
> ** Changed in: asdf
> Assignee: (unassigned) => Faré (fahree)
>

[Faré (fahree)]

You don't need to run (asdf:upgrade-asdf). By the time you'd run it, it'd be too late, since this problem has to do with older versions of asdf that don't have this function. By the time you have this function, either asdf did the right thing and upgraded correctly already, or it failed and you lose (which unhappily was the case previously wrt source-file-type).

[Ernst van Waning (evw)]

Hello Faré,

now that the cl file extension works, I would like to have the class
clcl-source-file and the method source-file-type separated form my asd
files defining systems, as I use the cl-extension all the time and the
documentation does not seem to forbid that.

With the definitions for the class and method in a file
named ../asdf/clcl.cl, I thought to use :defsystem-depends-on, as in

(defsystem :symquery
    :version "0.6.0"
    :defsystem-depends-on ((:file "../asdf/clcl"))
    :default-component-class clcl-source-file
    :serial t
    :components ((:file "packages")
                 (:file "symquery")))

but that gives an error:
    (during macroexpansion of (DEFSYSTEM :SYMQUERY ...))
    Invalid component designator: (FILE ../asdf/clcl)

With :static-file things go equally wrong. so there must be at least
something I do not fully understand. This can be the notation of
pathnames in asdf (I know mine is wrong), it can
be :defsystem-depends-on or (probably) much more.

Could you please help me solve this? At present, I think I should be
using more Lisp libraries and not rely on my own quick fixes, hence the
effort I take in using asdf2. When I understand better, I could write
or help write documentation about this so it can be added to asdf's
documentation.

Kind regards, hoping to hear

Ernst

Faré schreef op ma 25-04-2011 om 14:14 [+0000]:
> You don't need to run (asdf:upgrade-asdf). By the time you'd run it,
> it'd be too late, since this problem has to do with older versions of
> asdf that don't have this function. By the time you have this function,
> either asdf did the right thing and upgraded correctly already, or it
> failed and you lose (which unhappily was the case previously wrt source-
> file-type).
>
> ** Changed in: asdf
> Status: New => Fix Committed
>

[Robert P. Goldman (rpgoldman)]

On 4/27/11 Apr 27 -8:40 AM, Ernst van Waning wrote:
> Hello Faré,
>
> now that the cl file extension works, I would like to have the class
> clcl-source-file and the method source-file-type separated form my asd
> files defining systems, as I use the cl-extension all the time and the
> documentation does not seem to forbid that.
>
> With the definitions for the class and method in a file
> named ../asdf/clcl.cl, I thought to use :defsystem-depends-on, as in
>
> (defsystem :symquery
> :version "0.6.0"
> :defsystem-depends-on ((:file "../asdf/clcl"))
> :default-component-class clcl-source-file
> :serial t
> :components ((:file "packages")
> (:file "symquery")))
>
> but that gives an error:
> (during macroexpansion of (DEFSYSTEM :SYMQUERY ...))
> Invalid component designator: (FILE ../asdf/clcl)
>
> With :static-file things go equally wrong. so there must be at least
> something I do not fully understand. This can be the notation of
> pathnames in asdf (I know mine is wrong), it can
> be :defsystem-depends-on or (probably) much more.
>
> Could you please help me solve this? At present, I think I should be
> using more Lisp libraries and not rely on my own quick fixes, hence the
> effort I take in using asdf2. When I understand better, I could write
> or help write documentation about this so it can be added to asdf's
> documentation.
>
> Kind regards, hoping to hear
>
> Ernst
>
> Faré schreef op ma 25-04-2011 om 14:14 [+0000]:
>> You don't need to run (asdf:upgrade-asdf). By the time you'd run it,
>> it'd be too late, since this problem has to do with older versions of
>> asdf that don't have this function. By the time you have this function,
>> either asdf did the right thing and upgraded correctly already, or it
>> failed and you lose (which unhappily was the case previously wrt source-
>> file-type).
>>
>> ** Changed in: asdf
>> Status: New => Fix Committed
>>
>

:defsystem-depends-on must name an ASDF system, and not a file.

[Ernst van Waning (evw)]

Dear Robert,

thank you for your quick answer. I will try it out tonight & I hope my
questions are welcome.

Robert P. Goldman schreef op wo 27-04-2011 om 13:47 [+0000]:
> On 4/27/11 Apr 27 -8:40 AM, Ernst van Waning wrote:
> > Hello Faré,
> >
> > now that the cl file extension works, I would like to have the class
> > clcl-source-file and the method source-file-type separated form my asd
> > files defining systems, as I use the cl-extension all the time and the
> > documentation does not seem to forbid that.
> >
> > With the definitions for the class and method in a file
> > named ../asdf/clcl.cl, I thought to use :defsystem-depends-on, as in
> >
> > (defsystem :symquery
> > :version "0.6.0"
> > :defsystem-depends-on ((:file "../asdf/clcl"))
> > :default-component-class clcl-source-file
> > :serial t
> > :components ((:file "packages")
> > (:file "symquery")))
> >
> > but that gives an error:
> > (during macroexpansion of (DEFSYSTEM :SYMQUERY ...))
> > Invalid component designator: (FILE ../asdf/clcl)
> >
> > With :static-file things go equally wrong. so there must be at least
> > something I do not fully understand. This can be the notation of
> > pathnames in asdf (I know mine is wrong), it can
> > be :defsystem-depends-on or (probably) much more.
> >
> > Could you please help me solve this? At present, I think I should be
> > using more Lisp libraries and not rely on my own quick fixes, hence the
> > effort I take in using asdf2. When I understand better, I could write
> > or help write documentation about this so it can be added to asdf's
> > documentation.
> >
> > Kind regards, hoping to hear
> >
> > Ernst
> >
> > Faré schreef op ma 25-04-2011 om 14:14 [+0000]:
> >> You don't need to run (asdf:upgrade-asdf). By the time you'd run it,
> >> it'd be too late, since this problem has to do with older versions of
> >> asdf that don't have this function. By the time you have this function,
> >> either asdf did the right thing and upgraded correctly already, or it
> >> failed and you lose (which unhappily was the case previously wrt source-
> >> file-type).
> >>
> >> ** Changed in: asdf
> >> Status: New => Fix Committed
> >>
> >
>
> :defsystem-depends-on must name an ASDF system, and not a file.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/769634
>
> Title:
> cl file types on asdf2.014.6, SBCL 1.0.29.11.debian
>
> Status in Another System Definition Facility:
> Fix Committed
>
> Bug description:
> Dear ASDF maintainer,
>
> having downloaded quicklisp, added quicklisp to my init file, I
> automatically switched form asdf 1.30 to asdf 2.014.6, which was
> perhaps long overdue.
>
> Being one of those whose source files all have the cl extension, I
> have trouble to convince asdf2 of that fact. The system I use to make
> asdf use cl extentions has the following definition:
>
> ;;; system.asd
>
> (in-package :cl-user)
>
> (eval-when (:execute :compile-toplevel)
> (require :asdf))
>
> (use-package :asdf) ;asdf 2.014.6, seems to expect this, but still it...

[Robert P. Goldman (rpgoldman)]

This is a commonly-enough desired feature that I'm wondering if we shouldn't make "cl" files our first ASDF contrib...

[Ernst van Waning (evw)]

Dear Robert,

thank you for your answer, I hope my extension works by tomorrow,
including my understanding of it. Do you want me to write down my
understanding, so the documentation can be upgraded?

Kind regards,

Ernst

Robert P. Goldman schreef op wo 27-04-2011 om 17:44 [+0000]:
> This is a commonly-enough desired feature that I'm wondering if we
> shouldn't make "cl" files our first ASDF contrib...
>

[Robert P. Goldman (rpgoldman)]

On 4/27/11 Apr 27 -1:32 PM, Ernst van Waning wrote:
> Dear Robert,
>
> thank you for your answer, I hope my extension works by tomorrow,
> including my understanding of it. Do you want me to write down my
> understanding, so the documentation can be upgraded?

Sure, that would be great. I *believe* there is a section in the FAQ
about how to do this --- 12.5.5. If you were to compare this with your
experience and post documentation bugs, that would be very helpful.

[Ernst van Waning (evw)]

Comments below.

Robert P. Goldman schreef op wo 27-04-2011 om 18:55 [+0000]:
> On 4/27/11 Apr 27 -1:32 PM, Ernst van Waning wrote:
> > Dear Robert,
> >
> > thank you for your answer, I hope my extension works by tomorrow,
> > including my understanding of it. Do you want me to write down my
> > understanding, so the documentation can be upgraded?
>
> Sure, that would be great. I *believe* there is a section in the FAQ
> about how to do this --- 12.5.5. If you were to compare this with your
> experience and post documentation bugs, that would be very helpful.
>
Yes, it is 12.5.5, quite short from what I now understand from
scratching the surface thanks to you and Fare. Tomorrow I can spend one
or two hours trying to make the current setup work. If successful, I
can set up some prose to let you know what I did. Maybe I can write by
next Friday, at the earliest.

Kind regards,

Ernst

[Ernst van Waning (evw)]

Dear Robert,

having spent my time on asdf for today, it turned out to be an exercise
in source-registry configuration. System configuration has never been
my favorite pastime. However, working together clearly implies having
conventions. My reason for insisting on asdf2 is that your project is
about cooperation and I want to use and provide software that can be
used by a community, if only to prevent discussions like the curse of
Lisp currently on cll.

So, the asdf-documentation serves again from another perspective and I
will need some more time.

Kind regards,

Ernst

Ernst van Waning schreef op wo 27-04-2011 om 22:52 [+0200]:
> Comments below.
>
> Robert P. Goldman schreef op wo 27-04-2011 om 18:55 [+0000]:
> > On 4/27/11 Apr 27 -1:32 PM, Ernst van Waning wrote:
> > > Dear Robert,
> > >
> > > thank you for your answer, I hope my extension works by tomorrow,
> > > including my understanding of it. Do you want me to write down my
> > > understanding, so the documentation can be upgraded?
> >
> > Sure, that would be great. I *believe* there is a section in the FAQ
> > about how to do this --- 12.5.5. If you were to compare this with your
> > experience and post documentation bugs, that would be very helpful.
> >
> Yes, it is 12.5.5, quite short from what I now understand from
> scratching the surface thanks to you and Fare. Tomorrow I can spend one
> or two hours trying to make the current setup work. If successful, I
> can set up some prose to let you know what I did. Maybe I can write by
> next Friday, at the earliest.
>
> Kind regards,
>
> Ernst
>

[Ernst van Waning (evw)]

Dear Robert,

please find attached section 12.5.5 augmented with my experiences. As
you can see, text is added at the end. I have some comments which are
not intended to be part of the final text. The setup discussed allows
me to load my cl-files. So, "for me it works", but I imagine there may
be things in my text that can be stated in a better or more general way.
Please let me know.

Kind regards,

Ernst

Robert P. Goldman schreef op wo 27-04-2011 om 18:55 [+0000]:
> On 4/27/11 Apr 27 -1:32 PM, Ernst van Waning wrote:
> > Dear Robert,
> >
> > thank you for your answer, I hope my extension works by tomorrow,
> > including my understanding of it. Do you want me to write down my
> > understanding, so the documentation can be upgraded?
>
> Sure, that would be great. I *believe* there is a section in the FAQ
> about how to do this --- 12.5.5. If you were to compare this with your
> experience and post documentation bugs, that would be very helpful.
>

[Robert P. Goldman (rpgoldman)]

I think that this documentation fix is *ALMOST* right. The only problem I see is that it is not sufficiently explicit about how packages are handled. In what package is the clcl-source-file symbol? I think that suppressing the header of the clcl.lisp file (in particular the IN-PACKAGE form) is a mistake here. Suggestion: make a new attachment with just this lisp-file. Ideally, please put a line at the top that says that you are putting it into the public domain...

Thanks!

[Faré (fahree)]

Shouldn't I just have asdf 2.015 export classes cl-source-file.cl and cl-source-file.lsp that use the respective types cl and lsp?

[Robert P. Goldman (rpgoldman)]

On 5/1/11 May 1 -11:09 AM, Faré wrote:
> Shouldn't I just have asdf 2.015 export classes cl-source-file.cl and
> cl-source-file.lsp that use the respective types cl and lsp?
>

I think that might well be the best solution. This is so commonly
desired an extension that it seems silly to make people reimplement it
over and over (possibly with bugs).

But then we should revise the FAQ to indicate how a programmer can check
for a sufficiently-developed version of ASDF in his/her .asd file
(compare with asd-version? just check for an exported symbol? a member
of *features*?).

Best,
r

[Faré (fahree)]

I defined cl-source-file.cl and cl-source-file.lsp in 2.014.14 and modified the documentation. Please see if that satisfies you.

NB: I failed to add a test...

[Faré (fahree)]

I note a bug in your previous setup: you were defining a method on asdf:cl-source-file instead of asdf:source-file-type!

[Faré (fahree)]

I tested cl-source-file.cl manually, and it works.

[Ernst van Waning (evw)]

Dear Fare,

thank yo for the remark -- I deleted all method I had defined on
cl-source-file (blindly copied to get things to work) and
source-file-type (asdf1) and everything seems to work.

I will adapt my text accordingly.

Kind regards,

Ernst

Faré schreef op ma 02-05-2011 om 01:58 [+0000]:
> I note a bug in your previous setup: you were defining a method on asdf
> :cl-source-file instead of asdf:source-file-type!
>

[Ernst van Waning (evw)]

Dear Robert, dear Fare,

please find the adapted text below. Seeing the discussion yo have, my
text may well be obsolete. The main thing I learned was the
configuration and the extension of asdf. As you may remember, I needed
some help here and there :-)

Kind regards,

Ernst

Robert P. Goldman schreef op zo 01-05-2011 om 17:39 [+0000]:
> On 5/1/11 May 1 -11:09 AM, Faré wrote:
> > Shouldn't I just have asdf 2.015 export classes cl-source-file.cl and
> > cl-source-file.lsp that use the respective types cl and lsp?
> >
>
> I think that might well be the best solution. This is so commonly
> desired an extension that it seems silly to make people reimplement it
> over and over (possibly with bugs).
>
> But then we should revise the FAQ to indicate how a programmer can check
> for a sufficiently-developed version of ASDF in his/her .asd file
> (compare with asd-version? just check for an exported symbol? a member
> of *features*?).
>
> Best,
> r
>

[Ernst van Waning (evw)]

Dear Fare,

tried to download v2.014.14, but either I looked with my nose to find
the site or something is wrong.

>From "Getting it" I tried the first option "download just the source for
asdf.lisp,", which gave me v2.14, the tarball gave me v2.14 as well.
After trying with the code we agree upon, it looked for .lisp files,
which was not what I expected. Then, I was left to try the third option
git ..., which gave "fatal: destination path 'asdf' already exists and
is not an empty directory." After opting for "pull the latest from our
git repository (browse)" an error resulted that looked as follows:

XML-parsefout: ongedefinieerde entiteit
Locatie: http://common-lisp.net/gitweb?p=projects/asdf/asdf.git
Regelnummer 41, kolom 20:<div class="title">&nbsp;</div>
-------------------^

which would be in english:
XML-parse error: ondefined entity:
Location http://common-lisp.net/gitweb?p=projects/asdf/asdf.git
Line number 41, column 20: <div class="title">&nbsp;</div>
-------------------^

where the arrow is under the ampersand in ">&nbsp;</div>".

Could you tell me where the latest version is, so I can test and see
what exactly you changed?

Kind regards,

Ernst

Faré schreef op ma 02-05-2011 om 01:53 [+0000]:
> I defined cl-source-file.cl and cl-source-file.lsp in 2.014.14 and
> modified the documentation. Please see if that satisfies you.
>
> NB: I failed to add a test...
>

[Ernst van Waning (evw)]

downloaded v2.014.14. My (old) .asd-file (with :defsystem-depends-on)
loaded correctly. After changing my .asd file to contain
just :default-component-class cl-source-file.cl it worked as well.

Furthermore, I have seen you updated asdf's website. Good :-)

thank you very much,

Kind regards,

Ernst

Faré schreef op ma 02-05-2011 om 01:53 [+0000]:
> I defined cl-source-file.cl and cl-source-file.lsp in 2.014.14 and
> modified the documentation. Please see if that satisfies you.
>
> NB: I failed to add a test...
>

[Faré (fahree)]

Released in 2.015

[Ernst van Waning (evw)]

Dear Fare, Robert,

Thank you very much for your work. Version 2.15 is downloaded and I
intend to use it.

Please regard what I write below in a positive way. To get up to speed
with asdf2, I would like to read a tutorial. Is there one? If not, can
I help write one?

As I have Allegro and SBCL now, I would like to get asdf to place and
find the respective FASLs in directories for Allegro resp. SBCL. From
the documentation
(http://common-lisp.net/project/asdf/asdf.html#Controlling-where-ASDF-saves-compiled-files)
I see that this should be possible. After having read the entire
document several times over the last few weeks, I still have no clear
ideas about how to do this. With due respect, it feels like the
documents I write about my own work :-) When I write about my own work,
I am too deeply involved to write tutorials about it.

So, if there is a tutorial explaining how to start using asdf
effectively, please point me to it; if not I am willing to help you
write it. It is hard for me to imagine I am the only one having
difficulties here.

If asdf is (to be) the de-facto standard for defining systems in CL, a
tutorial will help the CL-community understand and use asdf and perhaps
even use CL-software written by others. I might refer you to recent
discussions on c.l.l. about curses of Lisp: imHo, Lisp is far from
cursed, but needs more accessible documentation. If that is not there,
then we will find tricks or roll our own, which is not desirable if we
want to be a prosperous community.

Hoping to hear,

Kind regards,

Ernst

Faré schreef op ma 09-05-2011 om 17:04 [+0000]:
> Released in 2.015
>
> ** Changed in: asdf
> Status: Fix Committed => Fix Released
>