Cannot codesign on macOS: save-lisp-and-die produces non-conforming MachO binaries

[Mark Skilbeck]

1. The binaries produced by SAVE-LISP-AND-DIE do not conform strictly to the MachO format, and as such they cannot be code-signed on macOS.

2.1 Produce a binary:

$ sbcl --eval "(defun hi () (print 'hi))" --eval "(sb-ext:save-lisp-and-die \"hi\" :executable t :toplevel 'hi)"
This is SBCL 2.0.0, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
WARNING: System definition file #P"/Users/mark.skilbeck/hackery/lisp/cl-ppcre/cl-ppcre.asd" contains definition for system "cl-ppcre-test". Please only define "cl-ppcre" and secondary systems with a name starting with "cl-ppcre/" (e.g. "cl-ppcre/test") in that file.
To load "cl-ppcre":
  Load 1 ASDF system:
    cl-ppcre
; Loading "cl-ppcre"
..
[undoing binding stack and other enclosing state... done]
[performing final GC... done]
[defragmenting immobile space... (fin,inst,fdefn,code,sym)=1117+959+19098+19279+26265... done]
[saving current Lisp image into hi:
writing 0 bytes from the read-only space at 0x20000000
writing 384 bytes from the static space at 0x20100000
writing 31031296 bytes from the dynamic space at 0x1000000000
writing 2080768 bytes from the immobile space at 0x20300000
writing 12668928 bytes from the immobile space at 0x21b00000
done]
$ ./hi

HI %

2.2 Try to codesign it:

$ codesign --no-strict -s "YOUR-SIGNING-ID" ./hi
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: ./hi

3. sbcl --version

SBCL 2.0.0

4. uname -a

Darwin skilbecklaptop.local 19.2.0 Darwin Kernel Version 19.2.0: Sat Nov 9 03:47:04 PST 2019; root:xnu-6153.61.1~20/RELEASE_X86_64 x86_64

[Douglas Katzman (dougk)]

You would have to remove ':executable t' from the save-lisp-and-die to produce a conforming binary.
Basically we cheat by assuming that copying arbitrary bytes to the end of an executable file just works. There is other tooling that doesn't like it either, e.g. 'strip'.
See https://bugs.launchpad.net/sbcl/+bug/310108

[Andrew Kravchuk (awkravchuk)]

This is particularly troublesome since nowadays SBCL depends on libzstd, and it's hard to redistribute programs made using it on MacOS since not every Mac user has /opt/local/lib/libzstd.dylib, and install_name_tool chokes on those binaries with exact same error message.

[Stas Boukarev (stassats)]

libzstd is an optional dependency.

[Andrew Kravchuk (awkravchuk)]

While that's technically correct, neither Homebrew nor Macports include an option to turn zstd compression off for SBCL.
I was able to fix this problem by binary patching resulting file with xxd and sed (as opposed to install_name_tool), but it was extremely hackish. Perhaps it is worth to consider complying with Mach-O format internals?