Build: Optionally offer an optimized core for REPLs

[Pierre Neidhardt]

Generating a core with pre-loaded packages (namely sb-bsd-sockets, sb-posix,
sb-introspect, sb-cltl2, asdf) is a common optimization technique for faster
REPL startups. The technique is detailed in the SLIME and SLY manuals.

Some may argue that modern machines are fast enough. But from personal
experience even on very fast hardware, I find that it makes for a much smoother
experience, especially for people using SLIME / SLY as their shell :)

In [here](https://github.com/joaotavora/sly/issues/363) I discussed with the SLY
maintainer a more sophisticated technique of on-demand core generations.

He suggested something else: Ask the SBCL developers if we could generate such an
optimized core when building SBCL.

Obviously, the size of the core makes it prohibitive to include it in the "sbcl"
package of the various operating systems. But adding an option could allow
packagers to offer a sbcl-repl-core package which users could install and use
directly, without having to generate one themselves.

What do you think?

[Stas Boukarev (stassats)]

Why does it have to be done by SBCL? Trivial to do yourself.

[Pierre Neidhardt (ambrevar)]

For the following reasons:

- Having a core as an OS package is less effort for the user.
- The core must be regenerated on every SBCL version change.

Automating the core generation on version change is not trivial, I've posted an Emacs function in the aforementioned SLY discussion.

There would be no need for this at all we had the core at hand already.

[Stas Boukarev (stassats)]

SBCL provides all the building blocks, it's up to you to shape them to your liking.
Making a shell script for this is not trivial—it's beyond trivial.

[Pierre Neidhardt (ambrevar)]

The discussion is not about the triviality of achieving something, but about its convenience. If we can provide more convenience to our users, then I believe it's a sane thing to do.

[Attila Lendvai (attila-lendvai)]

after a cursory read, i think we have made a script for this. you can specify an ASDF system name prefix, and it will load every matching ASDF system (but only the systems), and then load all of their dependencies, and save an SBCL core file ready for Slime.

https://hub.darcs.net/hu.dwim/hu.dwim.environment/browse/bin/build-development-image.sh

example usage:

~/common-lisp/hu.dwim.environment/bin/build-development-image.sh hu.dwim

creates the file: ~/.cache/common-lisp/sbcl-2.0.9-linux-x64/hu.dwim_development

and just FTR, this file has the (somewhat messy) emacs side to load it:

https://hub.darcs.net/hu.dwim/hu.dwim.environment/browse/emacs/dwim-init.el

[Attila Lendvai (attila-lendvai)]

> If we can provide more convenience to our users,
> then I believe it's a sane thing to do.

two thoughts on this:

 - not if/when such convenience comes at a disproportionate inconvenience on the shoulders of the project maintainers

 - the unwise use of DWIM behavior can easily lead to long hours of wasted debugging sessions

IMHO, if such a feature is ever to be published and maintained, then it should be part of either ASDF or Slime, because it doesn't depend fundamentally on any specific lisp compiler.

[Pierre Neidhardt (ambrevar)]

You are making complete sense about ASDF: I'll forward the issue there!

In the meantime, I had sent a patch to SLY:
https://github.com/joaotavora/sly/pull/366

But I suspect that an ASDF version would be even shorter and mroe universal in the end.

[Pierre Neidhardt (ambrevar)]

I came to the conclusion that I needed a wrapper script.

The answer is here:

https://gitlab.com/ambrevar/lisp-repl-core-dumper

It's a small tool to easily generate fast-loading core, optionally with
your favourite systems preloaded:

    lisp-repl-core-dumper sbcl alexandria fset

The above generates an image (only if needed) then loads it and drops you in the
REPL.

The tool was a little bit inspired by Attila's link, so thank you Attila for sharing this! :)

Use the above command to set the entry point to a REPL such as SLIME or SLY, it
makes for near instant startup time! This is particularly useful if you use a
Common Lisp REPL as your shell and fire up new REPL instances all the time!

Even if you think that the REPL starts fast enough and that you don't need to
restart your image much, this wrapper still makes for a more comfortable
experience since it's dead simple to use, close to zero setup.

Bonuses:

- Compiler version changes are detected and the image will be automatically
  regenerated.

- Multiple images can be used simultaneously, thus you can easily switch from an
  image with just Alexandria to an image with just Fset.

Feedback welcome!