SBCL cross compile fails for riscv64 on OpenBSD

[Robert Palm]

Hi,

I try to cross compile SBCL for the riscv64 architecture and use a VisionFive 2 with OpenBSD as a target system. Started it like so:

./cross-make.sh sync root@vf2 /usr/local/sbcl SBCL_ARCH=riscv64

Maybe someone can help to make it work, please?

Thanks!

------------------------------------------------------------------------

+ sh make-host-1.sh
//entering make-host-1.sh
//building cross-compiler, and doing first genesis
This is SBCL 2.3.6, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* T
* target SB-XC:*FEATURES* = (:RISCV :GENCGC :64-BIT :ANSI-CL :ANSI-COMPLIANT-LOAD-TRUENAME :BSD :COMMON-LISP :COMPACT-SYMBOL :COMPARE-AND-SWAP-VOPS :ELF :IEEE-FLOATING-POINT :LITTLE-ENDIAN :MEMORY-BARRIER-VOPS :OPENBSD :OS-PROVIDES-BLKSIZE-T :OS-PROVIDES-DLADDR :OS-PROVIDES-DLOPEN :OS-PROVIDES-POLL :OS-PROVIDES-SUSECONDS-T :OS-THREAD-STACK :PACKAGE-LOCAL-NICKNAMES :SB-DOC :SB-EVAL :SB-FUTEX :SB-LDB :SB-PACKAGE-LOCKS :SB-SOURCE-LOCATIONS :SB-THREAD :SB-UNICODE :SBCL :UNIX)
; Installed READ interceptor
; compiling file "/usr/local/sbcl/src/code/cross-early.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/cross-early.fasl-tmp
; compilation finished in 0:00:00.018
; compiling file "/usr/local/sbcl/src/code/show.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/show.fasl-tmp
; compilation finished in 0:00:00.003
; compiling file "/usr/local/sbcl/src/compiler/early-constantp.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/compiler/early-constantp.fasl-tmp
; compilation finished in 0:00:00.001
; compiling file "/usr/local/sbcl/src/compiler/vop-existsp.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/compiler/vop-existsp.fasl-tmp
; compilation finished in 0:00:00.007
; compiling file "/usr/local/sbcl/src/code/primordial-extensions.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/primordial-extensions.fasl-tmp
; compilation finished in 0:00:00.027
; compiling file "/usr/local/sbcl/src/code/cold-init-helper-macros.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/cold-init-helper-macros.fasl-tmp
; compilation finished in 0:00:00.006
; compiling file "/usr/local/sbcl/src/code/backq.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/backq.fasl-tmp
; compilation finished in 0:00:00.020
; compiling file "/usr/local/sbcl/src/code/cross-byte.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/cross-byte.fasl-tmp
; compilation finished in 0:00:00.010
; compiling file "/usr/local/sbcl/src/code/cross-misc.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/cross-misc.fasl-tmp
; compilation finished in 0:00:00.037
; compiling file "/usr/local/sbcl/src/code/cross-char.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/cross-char.fasl-tmp
; compilation finished in 0:00:00.002
; compiling file "/usr/local/sbcl/src/code/cross-float.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/cross-float.fasl-tmp
; compilation finished in 0:00:00.242
//CROSS-FLOAT DISCREPANCY!
// CACHE: (COMMON-LISP:EXP . #.(MAKE-DOUBLE-FLOAT #x3FF00000 #x0)) -> (#.(MAKE-DOUBLE-FLOAT #x4005BF0A #x8B145769))
// HOST : (2.7182818284590455d0)
; Float-ops cache prefill: 4315 entries
; compiling file "/usr/local/sbcl/src/code/cross-io.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/cross-io.fasl-tmp
; compilation finished in 0:00:00.006
; compiling file "/usr/local/sbcl/src/code/cross-condition.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/cross-condition.fasl-tmp
; compilation finished in 0:00:00.046
; compiling file "/usr/local/sbcl/src/code/uncross.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/uncross.fasl-tmp
; compilation finished in 0:00:00.000
; compiling file "/usr/local/sbcl/src/code/defbangtype.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/defbangtype.fasl-tmp
; compilation finished in 0:00:00.001
; compiling file "/usr/local/sbcl/src/code/early-constants.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/early-constants.fasl-tmp
; compilation finished in 0:00:00.001
; compiling file "/usr/local/sbcl/src/code/function-names.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/code/function-names.fasl-tmp
; compilation finished in 0:00:00.003
; compiling file "/usr/local/sbcl/src/compiler/early-globaldb.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/compiler/early-globaldb.fasl-tmp
; compilation finished in 0:00:00.026
; compiling file "/usr/local/sbcl/src/compiler/generic/parms.lisp" (written 02 AUG 2023 03:59:21 PM):

; wrote /usr/local/sbcl/obj/from-host/src/compiler/generic/parms.fasl-tmp
; compilation finished in 0:00:00.011
; compiling file "/usr/local/sbcl/src/compiler/riscv/parms.lisp" (written 02 AUG 2023 03:59:21 PM):

; file: /usr/local/sbcl/src/compiler/riscv/parms.lisp
; in: DEFCONSTANT +BACKEND-PAGE-BYTES+
; (DEFCONSTANT SB-C:+BACKEND-PAGE-BYTES+)
;
; caught ERROR:
; (during macroexpansion of (DEFCONSTANT +BACKEND-PAGE-BYTES+))
; Error while parsing arguments to DEFMACRO DEFCONSTANT:
; too few elements in
; (+BACKEND-PAGE-BYTES+)
; to satisfy lambda list
; (NAME VALUE &OPTIONAL (DOC)):
; between 2 and 3 expected, but got 1
While evaluating the form starting at line 124, column 0
  of #P"/usr/local/sbcl/make-host-1.lisp":

debugger invoked on a UNBOUND-VARIABLE @21B59F64 in thread
#<THREAD "main thread" RUNNING {1005960073}>:
  The variable +BACKEND-PAGE-BYTES+ is unbound.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE ] Retry using +BACKEND-PAGE-BYTES+.
  1: [USE-VALUE ] Use specified value.
  2: [STORE-VALUE] Set specified value and use it.
  3: [RECOMPILE ] Recompile file "src/compiler/riscv/parms.lisp"
  4: Recompile
  5: [RETRY ] Retry EVAL of current toplevel form.
  6: Ignore error and continue loading file "/usr/local/sbcl/make-host-1.lisp".
  7: [ABORT ] Abort loading file "/usr/local/sbcl/make-host-1.lisp".
  8: [ABORT-BUILD] Abort building SBCL.
  9: Exit debugger, returning to top level.

[Douglas Katzman (dougk)]

This would have to be a considered a new port, for which it's unlikely that any developer has time to help make it work.
You could try changing (at that line which failed, if not others as well) "#+netbsd" to "#+(or netbsd openbsd)", but if you're really feeling brave, just "#+bsd" and hope they all work.

[Robert Palm (r-p-x)]

Thank you Douglas - that was helpful!

I am now here:

//entering make-target-1.sh
//building runtime system and symbol table file
gmake: Entering directory '/usr/local/sbcl/src/runtime'
gmake: Leaving directory '/usr/local/sbcl/src/runtime'
GNUmakefile:50: Config: No such file or directory
gmake: *** No rule to make target 'Config'. Stop.

Does it mean I need to create cpl of new riscv runtime files ?

Config.riscv-bsd
Config.riscv-openbsd
riscv-bsd-os.c
riscv-bsd-os.h

[Douglas Katzman (dougk)]

Yeah, you would need a Config. Some of the BSD patch authors seem to like to have a bit of config that is common to sub-variants of BSD but it's not strictly necessary. You probably want "include Config.generic-openbsd" in Config.riscv-openbsd but I don't know if it is worth making Config.riscv-bsd unless you can guess what common elements would be needed for FreeBSD, NetBSD.
In terms of what will need to go in the C file, I have no helpful advice. It should be some synthesis of the riscv-linux.c file and maybe arm64-bsd-os.c

[Robert Palm (r-p-x)]

Tinkered a bit with it...

My current problem is how to translate following from arm64 to riscv64.

ARM64: https://github.com/openbsd/src/blob/master/sys/arch/arm64/include/signal.h#L51

--------------------------------------------------
struct sigcontext {
 int __sc_unused;
 int sc_mask; /* signal mask to restore */

 unsigned long sc_sp;
 unsigned long sc_lr;
 unsigned long sc_elr;
 unsigned long sc_spsr;
 unsigned long sc_x[30];

 long sc_cookie;
};
--------------------------------------------------

In SBCL these registers are mapped / used in conjuction with the definion in

https://github.com/sbcl/sbcl/blob/master/src/runtime/arm-lispregs.h

in the files

https://github.com/sbcl/sbcl/blob/master/src/runtime/arm64-bsd-os.h#L13

--------------------------------------------------
# define OS_CONTEXT_PC(context) context->sc_elr
--------------------------------------------------

and

https://github.com/sbcl/sbcl/blob/master/src/runtime/arm64-bsd-os.c#L64

--------------------------------------------------
os_context_register_addr(os_context_t *context, int regno)
{
    switch (regno) {
        case reg_LR: return (&context->sc_lr);
        case reg_NSP: return (&context->sc_sp);
        default: return (&context->sc_x[regno]);
    }
}

....

os_context_lr_addr(os_context_t *context)
{
    return os_context_register_addr(context, reg_LR);
}

....

os_context_flags_addr(os_context_t *context)
{
    return (os_context_register_t*)(&context->sc_spsr);
}

--------------------------------------------------

Now on the RISCV side it looks like this:

RISCV64: https://github.com/openbsd/src/blob/master/sys/arch/riscv64/include/signal.h#L48

--------------------------------------------------
struct sigcontext {
 int __sc_unused;
 int sc_mask;

 __register_t sc_ra;
 __register_t sc_sp;
 __register_t sc_gp;
 __register_t sc_tp;
 __register_t sc_t[7];
 __register_t sc_s[12];
 __register_t sc_a[8];
 __register_t sc_sepc;

 /* 64 bit floats as well as integer registers */
 __register_t sc_f[32]; /* floating-point registers */
 __register_t sc_fcsr; /* floating-point control register */

 long sc_cookie;
};

--------------------------------------------------

I think I need to replace (extend ?) the arm64 specific registers with the ones used by riscv.

But how to do that and what do the registers mean at all ?

I can satisfy the compiler and even get over the warm init after commenting out some assembler call in threads.c, but many warnings and at the end an error...

//doing warm init - compilation phase
This is SBCL 2.3.7.37-70eb4eb77-WIP, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
Initial page table:
        Immobile Object Counts
 Gen layout fdefn symbol code Boxed Cons Raw Code SmMix Mixed LgRaw LgCode LgMix Waste% Alloc Trig Dirty GCs Mem-age
  6 0 0 0 0 0 462 0 1835 ...

[Douglas Katzman (dougk)]

check if :sb-thread is in your *features*.
Line #305 of src/cold/shared will add the :os-thread-stack feature for OpenBSD and then you'll get a definition for new_thread_trampoline_switch_stack in thread.c

As to register usage, according to riscv-linux-os.c, they are all sequential in the sigcontext for linux, so it can use "return (os_context_register_t*)&(context->uc_mcontext.__gregs[offset]);" for the definition of os_context_register_addr. Perhaps it works similarly easily on OpenBSD

[Robert Palm (r-p-x)]

Tried to build --without-sb-thread but doesn't seem to have an effect. So I commented out in thread.c

-----------
/*
#ifdef LISP_FEATURE_OS_THREAD_STACK
extern void* funcall1_switching_stack(void*, void *(*fun)(void *));

void* new_thread_trampoline_switch_stack(void* th) {
    return funcall1_switching_stack(th, new_thread_trampoline);
}
#endif
*/
-----------

Maybe I don't need this right now as in linux there doesn't seem to be an assembler file that does that part, too.

Current "blocking" error is this (which seems to be independent from the mapping code below):

"obj/from-xc/src/code/early-defmethod.lisp-obj"
"obj/from-xc/src/code/format.lisp-obj"
"obj/from-xc/src/code/target-format.lisp-obj"
"obj/from-xc/src/code/late-globaldb.lisp-obj"
; in: SB-INT:NAMED-LAMBDA (SB-DISASSEM::INST-PRINTER SB-RISCV-ASM::C.J 1)
; (SB-DISASSEM::LOOKUP-LABEL SB-DISASSEM::|.T9|)
;
; note: deleting unreachable code

; (OR
; (GETHASH SB-DISASSEM::LAB
; (SB-DISASSEM::DSTATE-LABEL-HASH SB-DISASSEM::DSTATE))
; SB-DISASSEM::LAB)
; --> IF
; ==>
; SB-DISASSEM::LAB
;
; note: deleting unreachable code

; (SB-DISASSEM::INST-PRINT-NAME SB-DISASSEM::INST)
;
; note: deleting unreachable code

; (SB-DISASSEM::DSTATE-ARGUMENT-COLUMN SB-DISASSEM::DSTATE)
;
; note: deleting unreachable code

; (SB-DISASSEM::LOCAL-PRINC16 SB-DISASSEM::|.T10|)
;
; note: deleting unreachable code

; (SB-DISASSEM:PRINC16 SB-DISASSEM::THING STREAM)
;
; note: deleting unreachable code

; (SB-DISASSEM:OPERAND SB-DISASSEM::THING SB-DISASSEM::DSTATE)
;
; note: deleting unreachable code
;
; compilation unit finished
; printed 7 notes
; in:
; SB-INT:NAMED-LAMBDA (SB-DISASSEM::INST-PRINTER SB-RISCV-ASM::C.BEQZ 1)
; (OR
; (GETHASH SB-DISASSEM::LAB
; (SB-DISASSEM::DSTATE-LABEL-HASH SB-DISASSEM::DSTATE))
; SB-DISASSEM::LAB)
; --> IF
; ==>
; SB-DISASSEM::LAB
;
; note: deleting unreachable code

; (PRINC ", " STREAM)
;
; note: deleting unreachable code

; (SB-DISASSEM::INST-PRINT-NAME SB-DISASSEM::INST)
;
; note: deleting unreachable code

; (LET* ((SB-DISASSEM::|.T6|
; (THE (UNSIGNED-BYTE 2) (SB-DISASSEM::LOCAL-EXTRACT '#)))
; (SB-DISASSEM::|.T5|
; (THE (UNSIGNED-BYTE 2) (SB-DISASSEM::LOCAL-EXTRACT '#)))
; (SB-DISASSEM::|.T4|
; (THE (UNSIGNED-BYTE 1) (SB-DISASSEM::LOCAL-EXTRACT '#)))
; (SB-DISASSEM::|.T3|
; (THE (UNSIGNED-BYTE 2) (SB-DISASSEM::LOCAL-EXTRACT '#)))
; (SB-DISASSEM::|.T2|
; (THE (UNSIGNED-BYTE 1) (SB-DISASSEM::LOCAL-EXTRACT '#)))
; (SB-DISASSEM::|.T7|
; (SB-DISASSEM::ADJUST-LABEL
; (LIST SB-DISASSEM::|.T2| SB-DISASSEM::|.T3| SB-DISASSEM::|.T4|
; SB-DISASSEM::|.T5| SB-DISASSEM::|.T6|)
; #<FUNCTION SB-RISCV-ASM::USE-CB-LABEL>))
; (SB-DISASSEM::|.T8| (SB-DISASSEM::LOOKUP-LABEL SB-DISASSEM::|.T7|))
; (SB-DISASSEM::|.T1|
; (THE (UNSIGNED-BYTE 3) (SB-DISASSEM::LOCAL-EXTRACT '#))))
; (SB-DISASSEM::LOCAL-PRINT-NAME)
; (SB-DISASSEM::LOCAL-TAB-TO-ARG-COLUMN)
; (SB-D...

[Robert Palm (r-p-x)]

I can see it is coming from

https://github.com/sbcl/sbcl/blob/a6b554af6a7cd8899ae7ccb82317120f7b8a21a7/src/code/cold-init.lisp#L322

and seems to fail in the last format of

https://github.com/sbcl/sbcl/blob/a6b554af6a7cd8899ae7ccb82317120f7b8a21a7/src/compiler/target-disassem.lisp#L1220

(defun !compile-inst-printers ()
  (let ((package sb-assem::*backend-instruction-set-package*)
        (cache (list (list :printer) (list :prefilter) (list :labeller))))
    (do-symbols (symbol package)
      (awhen (get symbol 'instruction-flavors)
        (setf (get symbol 'instructions)
              (collect-inst-variants symbol package it cache))))
    (unless (sb-impl::!c-runtime-noinform-p)
      (format t "~&Disassembler: ~{~D printers, ~D prefilters, ~D labelers~}~%"
              (mapcar (lambda (x) (length (cdr x))) cache)))))

But why ?

[Douglas Katzman (dougk)]

"unreachable code" can be ignored. Those notes always get printed for riscv.
"CORRUPTION WARNING" is serious though.

[Robert Palm (r-p-x)]

I removed the sb-thread from *features* for now (in shared.lisp).

Still some error:

Disassembler: 28 printers, 1 prefilters, 4 labelers
**** 5 ****; Checking #P"/usr/local/sbcl/float-math.lisp-expr"
Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb> fatal error encountered in SBCL pid 4227:
maximum interrupt nesting depth (1024) exceeded

Error opening /dev/tty: Device not configured

ldb>

It's now here in cold-init.lisp:

  ;; The system is finally ready for GC.
  (/show0 "enabling GC")
  (format t "**** 5 ****")
  (setq *gc-inhibit* nil)
  #+sb-thread (finalizer-thread-start)
  (format t "**** 6 ****")
  ;; The show is on.
  (/show0 "going into toplevel loop")
  (handling-end-of-the-world

[Douglas Katzman (dougk)]

try editing gencgc to have these initial values:
 gencgc_verbose = 1
 pre_verify_gen_0 = 1
 verify_gens = 0
And can you attach with gdb and confirm (as I suspect) that it crashed in the garbage collector?

[Robert Palm (r-p-x)]

LINKFLAGS += -Wl,--export-dynamic -Wl,--no-execute-only

Disassembler: 28 printers, 1 prefilters, 4 labelers
Verify before GC [threads] [RO] [static] [dynamic] passed
Verify after GC(0) [threads] [RO] [static] [dynamic] passed
Verify after GC(1) [threads] [RO] [static] [dynamic] passed
Verify after GC(2) [threads] [RO] [static] [dynamic] passed
Verify after GC(3) [threads] [RO] [static] [dynamic] passed
Verify after GC(4) [threads] [RO] [static] [dynamic] passed
Verify after GC(5) [threads] [RO] [static] [dynamic] passed
Next gc when 96792985 bytes have been consed
Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb> fatal error encountered in SBCL pid 52108:
maximum interrupt nesting depth (1024) exceeded

Error opening /dev/tty: Device not configured

[Robert Palm (r-p-x)]

It looks a bit similar to this one.

I did add the

LINKFLAGS += -Wl,--export-dynamic -Wl,--no-execute-only

But seems there is something wrong, still...

[Robert Palm (r-p-x)]

Sorry forgot the link https://bugs.launchpad.net/sbcl/+bug/2024003