Wanted: a linkkit to produce standalone applications

Bug #794713 reported by Faré
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
SBCL
Incomplete
Low
Unassigned

Bug Description

I have a wishlist item for SBCL: it would be self-enclosed
application delivery into a single executable that includes not only
the Lisp image, but also additional statically compiled C libraries.
CLISP has a "linkkit" for that, and ECL can do it, too.

If I understand what nyef told me, we might have to have some magic
linker script and function annotations to ensure that magic functions
are located at an address known to Lisp code despite new code being
linked into the .text and .data segments. Typically, we'd either
ensure the proper objects files are linked first, or you'd create a
magic segment(s) different from .text and/or .data, that starts at a
magic address, in which to compile the stuff that the Lisp image will
expect to be at a known address.

Is some SBCL hacker interested in doing that? Or maybe only add it to
the SBCL TODO file? Or maybe someone can explain me WHICH functions
and datastructures must be at a fixed address, and then I can try to
write the linker script myself using my experience from bootstrapping
kernels on a new machine.

Notes:
1- Nikodemus reports that everything that's in SB-SYS:*STATIC-FOREIGN-SYMBOLS* needs to stay in
those addresses.
2- Raymond Toy reports that may already have done some/all of this for CMUCL, which may or may not be applicable to SBCL.

Tags: feature
Revision history for this message
Faré (fahree) wrote :

rtoy explains:

"Fred came up with the original idea, but at the time he only knew how to
do it by creating a custom linker script. I talked to him about it and
figured out gcc with a few special arguments would do the same thing.
That allowed me to extend the idea to work on solaris/sparc and even Mac
OS X.

The C side of the stuff is in src/lisp/elf.c and src/lisp/mach-o.c and
src/lisp/save.c, with a little bit of help from the lisp side in
src/code/save.lisp. elf.c and mach-o.c dump the three spaces into valid
elf or mach-o object files.

The script that does the linking is src/tools/linker.sh. It takes the
arguments created by save.c and does the right thing on the various
supported architectures and calls gcc in the right way to tell the
executable that an image is attached and to append the three spaces onto
the end of the C executable."

NB: the current way to checkout CMUCL, not well documented, is:
cvs -d common-lisp.net:/project/cmucl/cvsroot checkout src

Revision history for this message
Attila Lendvai (attila-lendvai) wrote :

i might be way off with this and/or missing key constraints...

but why not just store the .so's in the image, and upon startup write them to /tmp/ and load them from there with absolute paths?

Revision history for this message
Faré (fahree) wrote :

@Attila: your "solution" is about just as much work, and not as effective in the end (not to tell about the security nightmare).

Changed in sbcl:
importance: Undecided → Low
status: New → Triaged
tags: added: feature
Revision history for this message
Faré (fahree) wrote :

After discussing with pkhuong on #sbcl today, I find that trying to relink an executable can be tricky in several ways I didn't foresee.
http://ccl.clozure.com/irc-logs/sbcl/2011-12/sbcl-2011.12.01.txt

1- If you want to reuse existing image files, you want some essential code / data addresses used by sbcl itself to be fixed, which supposes that they be put in their own sections, separate from normal C text and data sections, and not offset by object file headers or non-text/non-data segments. On a GNU platform, a linker script can then make sure these things are in front and/or at a fixed address. On a BSD platform (at least for darwin), it seems you can't fully control what "in front" means, and that you must put things at a fixed address using -segaddr.

2- Things are actually worse, in that if you want to be compatible with an existing image, you can only *add* things, and you may have to add new code and data sections to the end of the existing object (e.g. at the end of it, assuming it doesn't conflict with the heap). For that you must either have kept all the objects with which you build the previous image, or somehow be able to uncompile the current executable into a linkable object. Back in the days, ELK used to do just that using a.out and GNU ld. I don't know how that is portable to other platforms or not. Also, whether you want contiguous zones for each of text and data, or alternate pages of text and data, that will be tricky to manage.

3- Another easier solution might be to regenerate an image using slam.sh, which supposes access to a writable SBCL source directory and all the right shell parameters. Painful, but possible.

Revision history for this message
Faré (fahree) wrote :

4- Yet another solution would be to somehow maintain linking information in the lisp image, and re-link everytime the base C image changes. I have recollections of having been told CMUCL doing something like that back in the day, but that was old second hand information.

Revision history for this message
Nikodemus Siivola (nikodemus) wrote :

One quick-and-dirty option would be to teach save-lisp-and-die how to append shared objects to the core, and dump them in /private/tmp or similar on startup.

Revision history for this message
Faré (fahree) wrote :

akovalenko told me (see same #sbcl transcript) that he already did option #4 in his Windows port of SBCL, effectively implementing a linkkit for his own SBCL porting purposes, so he could frequently recompile the runtime without having to regenerate a core.

There are implications for the FFI layer, and clients (e.g. contribs, CFFI, etc.) may have to be modified to support relinking. akovalenko would know the details.

Revision history for this message
Nikodemus Siivola (nikodemus) wrote :

SBCL already does the linkage-table stuff for shared objects loaded by users (based on the CMUCL code you refer to).

What Anton has done, and which enables relinking the runtime, is getting rid of /all/ static foreign addresses, and always going
through the linkage table.

Revision history for this message
Faré (fahree) wrote :

This bug should probably be closed now. Bazel demonstrated a solution, but it needs to be made more portable. See https://bugs.launchpad.net/sbcl/+bug/1500628

Revision history for this message
Douglas Katzman (dougk) wrote :

Bug reporter says to close, so let's close.

Changed in sbcl:
status: Triaged → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.