Memory leak during compilation

[Syll]

When trying to compile the attached file multiple times, sbcl run out of
memory (ldb appears). The number of compilations depends on
dynamic-space-size. These results seem strange, but I am able to
reproduce them...

* sbcl launched with </path/to/sbcl> --dynamic-space-size <memory>
* Compilations launched with (dotimes (i 1000) (print i) (finish-output) (load "test-perf"))

Here is the result :
(<memory> : <number of compilations before failing, with sbcl 2.0.3 / sbcl 2.0.5)
  - 2048 : 1/1
  - 2500 : 1/1
  - 3072 : 32/36
  - 3500 : 7/4 (in that order)
  - 4096 : 10/10
  - 5000 : 47/(stopped after 1000)

----

SBCL 2.0.5
Linux yggdrasil 5.7.7-arch1-1 #1 SMP PREEMPT Wed, 01 Jul 2020 14:53:16 +0000 x86_64 GNU/Linux

[Paul F. Dietz (paul-f-dietz)]

What is the DEBUG optimization setting? There have been cases where that's caused a blowup in memory.

[Syll (syll)]

I didn't touch any optimization setting.

I just launched sbcl with "--dynamic-space-size <memory>" then loaded the file.

It should be default value, but I don't know if there is a mean to query this value.

[Douglas Katzman (dougk)]

Your function is enormous - the compiler conses about 1.3GB while digesting it, and it produces over 200KB of assembly code, and it finds >700 "externally callable" functions inside one function (anything bound in WITH-SIMPLE-RESTART etc is a callable function).

As a practical matter, the compiler is designed to work with typical lisp code, which this is not.
So while there may be an issue with GC not running often enough, there are definitely some scaling issues with regard to compiler internals. i.e. the choices of algorithms aren't designed to deal with this. And perhaps our expansion of WITH-SIMPLE-RESTART is fairly heavyweight.

If you "need" this to work keeping the same shape of that function, I might suggest refactoring it to use a CALL-WITH pattern.
Just defining (TEST-EQ) to expand into `(CALL-WITH-TEST-EQ (lambda () (something)) reduces compiler consing down to 75MB and speeds up compilation by 50x.