sb-alien enum with many keys and large init values causes control stack exhaustion

[Timo Tossavainen]

I ran into this when autogenerating wrappers for C libraries that use enums for #define constants. Apparently the sb-alien enum implementation with a large number of non-contiguous keys and at least one large key exhausts the control stack when generating code for manipulating the enum values.

Test case (input to repl, crashes when compiling set-foo):

(defun key-init (n)
  (let (result)
    (dotimes (i n (nreverse result))
      (push (list (intern (format nil "v~3,'0d" i) "KEYWORD") i) result))))

(eval `(sb-alien:define-alien-type foo (enum foo ,@(key-init 256) (:last 2147483647))))

(defun set-foo ()
  (sb-alien:with-alien ((a foo))
    (setf a :last)
    a))

SBCL 2.2.5

Linux hemlock 5.4.0-121-generic #137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

(:QUICKLISP :ASDF3.3 :ASDF3.2 :ASDF3.1 :ASDF3 :ASDF2 :ASDF :OS-UNIX
 :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :X86-64 :GENCGC :64-BIT :ANSI-CL
 :COMMON-LISP :ELF :IEEE-FLOATING-POINT :LINUX :LITTLE-ENDIAN
 :PACKAGE-LOCAL-NICKNAMES :SB-LDB :SB-PACKAGE-LOCKS :SB-THREAD :SB-UNICODE
 :SBCL :UNIX)

[Timo Tossavainen (ttossava)]

I traced this to the compiler choking on "large" ecases. Making another bug report for that. The sb-alien enum implementation seems potentially very slow for large enums like those used in Vulkan for structure type tags.