ASDF treats itself as always loaded, even in contexts where it isn't

[Chris Dusto]

I was running some trial builds for a project with ECL via `asdf:make-build`, but the final binary lacked ASDF. Discussion over IRC resulted in the discovery that ASDF, as a "preloaded system," gets confused; it believes that it included itself in an image, despite not actually doing so.

[Daniel Kochmański (jackdaniel)]

same applies to image-op and program-op – putting #:asdf in depends-on fails to lin libasdf.a with the resulting system.

[Faré (fahree)]

Chris, make-build is not a supported interface (anymore).

Please use program-op and image-op on a system or program-system.

Chris, Daniel, can you provide a test case? All the tests pass on ECL on ASDF 3.2.0.1.

[Daniel Kochmański (jackdaniel)]

As I have pointed out, this bug affects also program-op and image-op.

Chris: make-build is a supported interface on ECL, it's not supported only if you plan to void ECL's warranty by swapping bundled ASDF version.

Regarding test case:

;; test-bin.asd
(defsystem :test-bin
    :serial t
    :depends-on (#:asdf #:alexandria #:usocket)
    :components ((:file "file1")
                 (:file "file2")))

;; file1.lisp
(princ "======================================================================") (terpri)

(princ "We are now executing FILE1.LSP") (terpri)

(defun test-function (x y)
  (format t "~D + ~D is equal to ~D~%" x y (+ x y)))

(princ "TEST-FUNCTION has been created") (terpri)

;; file2.lisp
(princ "We are now executing FILE2.LSP") (terpri)

(princ "Calling TEST-FUNCTION in FILE2.LSP") (terpri)

(test-function 1 1)

(print `(alex ,(alexandria:ensure-list 1)
         sb-s ,sb-bsd-sockets::+AF-LOCAL+
         asdf ,asdf::*asdf-version*))

(princ "Finished") (terpri)

(princ "======================================================================") (terpri)

;; invocation
rlwrap ecl --load "/tmp/asdf.lisp" --eval '(trace c::builder)' --eval '(trace asdf::component-depends-on)' --eval "(asdf:operate 'asdf:program-op :test-bin)" --eval "(ext:quit)"

note that the cache is clean and /tmp/asdf.lisp is the version you have mentioned.

Patch:

From c6f363c33e1ef21c4e8dcb81522b84dde76e69ef Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Daniel=20Kochma=C5=84ski?= <email address hidden>
Date: Tue, 31 Jan 2017 21:17:21 +0100
Subject: [PATCH 1/2] ensure prebuilt-asdf

---
 bundle.lisp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/bundle.lisp b/bundle.lisp
index 7d833fdd..6fd0bc49 100644
--- a/bundle.lisp
+++ b/bundle.lisp
@@ -590,8 +590,8 @@ or of opaque libraries shipped along the source code."))
         `((,lib-op
            ,@(unless (no-uiop c)
                (append (ensure-linkable-system "cmp")
- (or (ensure-linkable-system "uiop")
- (ensure-linkable-system "asdf"))))
+ (ensure-linkable-system "uiop")
+ (ensure-linkable-system "prebuilt-asdf")))
            ,@deps)))))

   (defmethod perform ((o link-op) (c system))
--
2.11.0

[Daniel Kochmański (jackdaniel)]

jack@pandora ~ % /home/jack/.cache/common-lisp/ecl-16.1.3-0476bca3-linux-x64/home/jack/quicklisp/local-projects/dummy-test/test-bin

Condition of type: SIMPLE-ERROR
Package ((UIOP/VERSION . #<UIOP/VERSION package>)) referenced in compiled file
  NIL
but has not been created
Available restarts:

1. (IGNORE) Ignore the error, and try the operation again

Top level in: #<process TOP-LEVEL>.
>

[Chris Dusto (truepikachu)]

My local test case, with vanilla ECL built from source, is at <https://gist.github.com/TruePikachu/06ff1b30117a71a3d5b0a81a9852831a>.

Condition of type: SIMPLE-ERROR
Package ((ASDF/UPGRADE . #<ASDF/UPGRADE package>)) referenced in compiled file
  NIL
but has not been created

[Faré (fahree)]

Daniel, I don't understand your code, and it obviously won't work outside ECL.

Does your prebuilt asdf not include uiop? (That's good if that's the case, but asdf needs to know about it indeed).

[Faré (fahree)]

More generally, can you explain the default asdf installation of ECL?
What is available under what name?
Do I have to care about older versions of ECL in asdf, or should I just assume the latest?

[Chris Dusto (truepikachu)]

At least going by (asdf:asdf-version), the ECL ASDF is 3.1.8.2, probably with a few additions (I know that the #'MAKE-BUILD function has additional keyword arguments for ECL, e.g. `:TYPE :PROGRAM`, but I don't know if that is "by design" for ASDF).

[Daniel Kochmański (jackdaniel)]

Not sure what's there to understand. Two files, one is "vanilla" (not using external dependencies), second makes use of symbols in the dependencies defined: alexandria, sb-bsd-sockets (pulled indirectly by usocket) and asdf:

(print `(alex ,(alexandria:ensure-list 1)
         sb-s ,sb-bsd-sockets::+AF-LOCAL+
         asdf ,asdf::*asdf-version*))

Being able to access a symbol is a proof, that package is available in the program. If you want to, you may wrap this snippet in a function and make it an entry point.

ECL ships with ASDF which has UIOP included, but it's not built-in in the image. That means in particular, that since ECL has a clear separation of build phase and runtime, that having required ASDF for building phase doesn't mean that it will be included in the executable. Depending only on UIOP may pull standalone UIOP from the registry (i.e QL distribution) which doesn't have ASDF, but depending on ASDF picks nothing, because there is no ASDF system.

ECL names prebuilt ASDF system prebuilt-asdf and has it in its lib directory:

(defsystem "prebuilt-asdf" :class asdf::prebuilt-system
        :lib #P"SYS:LIBASDF.A"
        :depends-on ("cmp")
        :components ((:compiled-file "prebuilt-asdf" :pathname #P"SYS:ASDF.FAS")))

I hope that this helps with your understanding of this code. Regarding supporting things, its up to you, each library decides which implementations it wants to support.

[Faré (fahree)]

Why is it called prebuilt-asdf and not asdf? Do you use it with (require "prebuilt-asdf") instead of (require "asdf") ? Are the packages still those of asdf?

[Daniel Kochmański (jackdaniel)]

I don't know – maybe because it is asdf system which is prebuilt and to avoid conflicts? It was that way. prebuilt-asdf has full asdf version bundled with ECL. system-module-pathname returns correct path to the module, it's just not picked up by ASDF.

[Faré (fahree)]

What conflicts are you talking about? As long as you let your asdf at the bottom of the source-registry, or wholly out of it, there will be no conflict.

So, does (require "asdf") work? Is the name always "libasdf.a" ? What about on Windows? Does that work on old versions? Will that work on future versions? What's the API?

[Daniel Kochmański (jackdaniel)]

I'm guessing the intention of the person, who decided to name the system that way, not stating that there is conflict.

What is loaded by require is asdf.fas file, libasdf.a is for an executable inclusion (and is defined in prebuilt-asdf.asd system). This mechanism is something being there at least in ECL 13.5.1. Regarding predicting the future – it's something I'm not capable of. API of what? Require?

[Faré (fahree)]

I investigated a bit.

Daniel, ASDF is supposed to automatically detect asdf.a and you don't need to define a prebuild-asdf.asd, which would be quite awkward to substitute to asdf. The reason why the autodetection failed was because you renamed asdf.a to libasdf.a. I can modify asdf/bundle to find it in 3.2.0.2.

Chris, loading one version of asdf from source then having it look for the asdf.a of the implementation is just wrong. If you're going to upgrade asdf, please do it the right way, i.e. put your asdf.asd in a configured location.

That said, your test case does trigger a different bug in my current working version of asdf (which when explicitly given a dependency on asdf, currently tries to link both asdf/defsystem and asdf, which causes a conflict). So I have more work to do about it for 3.2.0.2.

I have many deadlines the next two weeks, so I cannot guarantee a prompt fix; and then the testing situation with Robert is unclear, so I don't know when next release will be. Hopefully, 3.2.1 in February and 3.3.0 in March, but who knows?

[Daniel Kochmański (jackdaniel)]

I have never renamed it, functionality was flat broken from the beginning. Congrats on fixing the issue.

[Faré (fahree)]

There is now a fix in MR !66 https://gitlab.common-lisp.net/asdf/asdf/merge_requests/66 including a new test case. Thanks for the bug report!

[Faré (fahree)]

The fix was released a few months back in 3.2.1.

Also, make-build will be restored in the upcoming ASDF 3.3.0, though as a deprecated backward-compatibility interface on top of program-op, and we still encourage users to migrate to program-op.