ECL on Mac OS X Yosemite fails ecl-make-image test

[Robert P. Goldman]

Testing: ecl-make-image.script
ecl -norc -load sys:cmp -eval "'(#.(load(string'|script-support.lisp|))#.(asdf-test::load-asdf)#.(asdf-test::frob-packages)#.(asdf-test:run-test-script'|ecl-make-image.script|))"
;;; Loading #P"/usr/local/stow/ecl/lib/ecl-16.0.0/cmp.fas"
;;; Loading "/Users/rpg/lisp/asdf/test/script-support.lisp"
;;; Loading "/Users/rpg/lisp/asdf/build/fasls/ecl/asdf.fas"
Configuring ASDF
Enabling debugging
Being a bit verbose
Comparing directories
ASDF-TEST:*TEST-DIRECTORY* and
 (:PATHNAME-DIRECTORY-PATHNAME (NTH-VALUE 2 (:LOCATE-SYSTEM :TEST-ASDF))) both evaluate to same path:
  #P"/Users/rpg/lisp/asdf/test/"
Frob packages
Running ecl-make-image.script with ecl-16.0.0-unknown-macosx-x64
Test ld-flags in make-image. Should load from ecl-make-image/

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;
;;; BUILDING hello_aux.o FILE
;;;

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;
;;; BUILDING A STANDALONE EXECUTABLE
;;;
;;;
;;; Compiling /Users/rpg/lisp/asdf/test/ecl-make-image/hello.lisp.
;;; OPTIMIZE levels: Safety=3, Space=0, Speed=2, Debug=3
;;;
;;; End of Pass 1.
;;; Finished compiling /Users/rpg/lisp/asdf/test/ecl-make-image/hello.lisp.
;;;
Undefined symbols for architecture x86_64:
  "_hello_string", referenced from:
      __eclKh01T9WuI9OGL_Alcp9G21 in hello.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
TEST ABORTED: Error code 1 when executing
(RUN-PROGRAM "gcc" ("-o"
                    "/Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/hello-tmpM27PKA7O.fas"
                    "-L/usr/local/stow/ecl/lib/"
                    "/private/var/folders/xq/ll1d80qs7hnbw3253yc1_x6m0000gn/T/eclinitIM9iaz.o"
                    "/Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o"
                    "-bundle" "-lecl" "-lpthread" "-lm"))

[Robert P. Goldman (rpgoldman)]

I suspect that ECL may be mistakenly thinking it should be building a 32-bit binary.

[Robert P. Goldman (rpgoldman)]

This does not seem to be an architecture related issue. See the following further analysis:

rpg@rpgoldman-2: ~/lisp/asdf/test $ gcc -o /Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/test.fas -L /usr/local/stow/ecl/lib/ /private/var/folders/xq/ll1d80qs7hnbw3253yc1_x6m0000gn/T/eclinitwwC3Rd.o /Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o -bundle -lecl -lpthread -lm -v
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -dynamic -arch x86_64 -bundle -macosx_version_min 10.11.0 -o /Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/test.fas -L/usr/local/stow/ecl/lib/ /private/var/folders/xq/ll1d80qs7hnbw3253yc1_x6m0000gn/T/eclinitwwC3Rd.o /Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o -lecl -lpthread -lm -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin/libclang_rt.osx.a
Undefined symbols for architecture x86_64:
  "_hello_string", referenced from:
      __eclKh01T9WuI9OGL_rPv3Zi21 in hello.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
rpg@rpgoldman-2: ~/lisp/asdf/test $ nm /Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o
00000000000058b8 b _Cblock
0000000000000190 t _LC1__g0
0000000000000330 s _LC1__g0._ecl_descriptors
00000000000058c0 b _VV
                 U ___stack_chk_fail
                 U ___stack_chk_guard
0000000000000000 T __eclKh01T9WuI9OGL_rPv3Zi21
                 U _cl_symbols
00000000000002f0 s _compiler_data_text
0000000000000300 s _compiler_data_text1
                 U _ecl_cs_overflow
                 U _ecl_function_dispatch
                 U _ecl_make_cfun
                 U _ecl_make_foreign_data
                 U _ecl_make_simple_base_string
                 U _ecl_print
                 U _ecl_process_env
                 U _hello_string
                 U _si_get_sysprop
                 U _si_put_sysprop
                 U _si_select_package
rpg@rpgoldman-2: ~/lisp/asdf/test $ file
Usage: file [-bchikLNnprsvz0] [-e test] [-f namefile] [-F separator] [-m magicfiles] [-M magicfiles] file...
       file -C -m magicfiles
Try `file --help' for more information.
rpg@rpgoldman-2: ~/lisp/asdf/test $ file /Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o
/Users/rpg/lisp/asdf/build/fasls/ecl-16.0.0-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o: Mach-O 64-bit object x86_64
rpg@rpgoldman-2: ~/lisp/asdf/test $ file /private/var/folders/xq/ll1d80qs7hnbw3253yc1_x6m0000gn/T/eclinitwwC3Rd.o
/private/var/folders/xq/ll1d80qs7hnbw3253yc1_x6m0000gn/T/eclinitwwC3Rd.o: Mach-O 64-bit object x86_64
rpg@rpgoldman-2: ~/lisp/asdf/test $ nm /private/var/folders/xq/ll1d80...

[Robert P. Goldman (rpgoldman)]

This bug persists in El Capitan.

[Daniel Kochmański (jackdaniel)]

Did you test it with the 16.1.2 release? I think I've fixed (probably) this with the commit: 55bbfcf ("cmpname: don't randomize init function names for libraries", 2015-10-01).

[Robert P. Goldman (rpgoldman)]

So far I have only been using whatever comes with MacPorts and (K)ubuntu. Just doing all these tests is already quite time-consuming; I don't really have time to build a lot of CL implementations from source.

I could see if the MacPorts folks are interested in shipping a new version. Do you have a snappy explanation of the advantages of 16.1.2? Presumably there's something less specific than "enables the ASDF build-image operation on Mac OS."

[Daniel Kochmański (jackdaniel)]

Fixes bug for building a standalone executables :) full list of changes between 16.0.0 and 16.1.2:
* 16.1.2 changes since 16.0.0

** API changes

   - =si:do-defsetf= accepts optional parameter stores.\\
     New lambda-list:\\
     =(access-fn function &optional (stores-no 1))=.\\
     This change is backward compatible.

   - New MP functions:\\
     =mp:with-rwlock=\\
     =mp:try-get-semaphore= (non-blocking)\\
     =mp:mailbox-try-read= (non-blocking)\\
     =mp:mailbox-try-send= (non-blocking)

   - Added back removed C interfaces\\
     =ecl_import_current_thread=\\
     =ecl_release_current_thread=

   - When =cl-truename= encounters a broken symlink, it returns its path
     instead of signalling a file-error

   - Deprecated variables has been removed
     =c::*suppress-compiler-warnings*=, =c::*suppress-compiler-notes*=

   - Random state might be initialized by a random seed (truncated to
     32bit value) or by a precomputed array.

     Latter is designed to allow reading back the printed random state
     (when printed readably), not as an array to initialize the random
     state.

   - C99 supporting compiler is mandatory for C backend.

   - =COMPILER::*CC_IS_CXX*=: New variable to switch the output extension of
     emitted compiler code to ".cxx" when configured with "--with-c++". This
     eliminates compiler warnings that compiling C++ with a ".c" extension is
     deprecated; this is seen mostly with Clang++.

   - =GRAY:CLOSE= isn't specialized on =T= to preserve compatibility with some
     libraries.

** Enhancements:

   - Added code walker (present in =*features*= as =:walker=)

   - Testing framework initial cleanup

   - Format fallbacks to =prin1= if infinity or NaN are passed to it

   - Annotations are added at runtime (better integration with SLIME)

   - Mersenne-Twister RNG has new 64 bit implementation for appropriate
     machines

   - Add sockets implementation for the android platform

   - Add android build target (official android support)

   - Added Clang-specific pragmas to disable return type, unused value and
     excessive parentheses warnings, which are fairly harmless, but annoying
     and clutter user output.

** Issues fixed:

   - =si:open-unix-socket-stream= accepts both string and base-string
     (automatic coercion is performed)

   - Long form of =DEFSETF= accepts multiple-values as a store forms:

     #+BEGIN_SRC lisp
       (defsetf gah (x) (y z) `(list ,x ,y ,z))
       (setf (gah 3) (values 3 4))
     #+END_SRC

   - Building with single-threaded boehm works if ECL threads are disabled

   - Using labels works with sharp-S-reader

     #+BEGIN_SRC lisp
       (read-from-string
        "(#1=\"Hello\" #S(sharp-s-reader.1.example-struct :A #1#))")
     #+END_SRC

   - Generated C code works well with IEEE 754 infinities
     (regression tests created)

   - User-defined heap sizes can now exceed the size of a fixnum on 32-bit

   - The heap size limit was intended to be 1GB on 32-bit or 4GB on 64-bit
     but inconsistency between =ECL_FIXNUM_BITS= and =FIXNUM_BITS= in the code
     prevented the heap to grow for 64-bit. This now occurs, and a few
     other less visible bugs we...

[Robert P. Goldman (rpgoldman)]

Unfortunately, this still doesn't work on my Mac, although it fails differently (different?):

TEST ABORTED: Error code 1 when executing
(RUN-PROGRAM "/usr/bin/clang" ("-o"
                               "/Users/rpg/lisp/asdf/build/fasls/ecl-16.1.2-unknown-macosx-x64/asdf/test/ecl-make-image/hello-tmp28SYSBFU.fas"
                               "-L/opt/local/lib/"
                               "/private/var/folders/xq/ll1d80qs7hnbw3253yc1_x6m0000gn/T/eclinitEFsy05.o"
                               "/Users/rpg/lisp/asdf/build/fasls/ecl-16.1.2-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o"
                               "-bundle" "-L/opt/local/lib"
                               "-Wl,-headerpad_max_install_names" "-arch"
                               "x86_64" "-L/opt/local/lib"
                               "-Wl,-headerpad_max_install_names" "-arch"
                               "x86_64" "-lecl" "-lpthread" "-lm"))

One thing I note: this is running Apple's compiler /usr/bin/clang, but with MacPorts' library /opt/local/lib.

I also note redundant specification of several flags, although I don't think that will cause problems.

Running by hand:
/usr/bin/clang -o "/Users/rpg/lisp/asdf/build/fasls/ecl-16.1.2-unknown-macosx-x64/asdf/test/ecl-make-image/hello-tmp28SYSBFU.fas" -L /opt/local/lib "/private/var/folders/xq/ll1d80qs7hnbw3253yc1_x6m0000gn/T/eclinitEFsy05.o" "/Users/rpg/lisp/asdf/build/fasls/ecl-16.1.2-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o" "-bundle" "-L/opt/local/lib" -Wl -headerpad_max_install_names -arch x86_64 -L /opt/local/lib -l ecl -lpthread -lm
Undefined symbols for architecture x86_64:
  "_hello_string", referenced from:
      __eclGFF58509LfRJL_Lpiedi21 in hello.o
ld: symbol(s) not found for architecture x86_64

Same error running with MacPorts compiler:

/opt/local/bin/gcc-mp-5 -o "/Users/rpg/lisp/asdf/build/fasls/ecl-16.1.2-unknown-macosx-x64/asdf/test/ecl-make-image/hello-tmp28SYSBFU.fas" -L /opt/local/lib "/private/var/folders/xq/ll1d80qs7hnbw3253yc1_x6m0000gn/T/eclinitEFsy05.o" "/Users/rpg/lisp/asdf/build/fasls/ecl-16.1.2-unknown-macosx-x64/asdf/test/ecl-make-image/hello.o" "-bundle" "-L/opt/local/lib" -headerpad_max_install_names -arch x86_64 -L /opt/local/lib -l ecl -lpthread -lm
Undefined symbols for architecture x86_64:
  "_hello_string", referenced from:
      __eclGFF58509LfRJL_Lpiedi21 in hello.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status