Inconsistent prioritization of SB-EXT:RESTRICT-COMPILER-POLICY over PROCLAIM, DECLAIM, and DECLARE forms by platform

[''the Phoeron'' Colin J.E. Lupton]

## Description

This bug was discovered as part of fixing a seemingly unrelated type-declaration based issue for the CL-ISAAC library, when a user had set SB-EXT:RESTRICT-COMPILER-POLICY for 'SAFETY to 3 3 in their SBCL init-file on Ubuntu Linux. The issue in CL-ISAAC seemed to be a conflict with the implementation-dependent features of 'SAFETY geq 2. However, the function in question explicitly declared safety as 0 within its definition, so it was apparent that on Linux the user's compiler-policy restrictions were overriding explicit declarations within the library source-code. This identical behaviour was reproduced on Windows (SBCL 2.3.2) and Linux (SBCL 2.3.4), but could not be reproduced on macOS (SBCL 2.3.4). That is, on macOS the explicit declarations in library source-code were overriding compiler-policy restrictions.

According to my understanding of compiler-policy restrictions, the Linux and Windows behaviour is correct while the macOS behaviour is incorrect. However, the behaviour should be consistent across platforms.

## Test Case

- Add `(SB-EXT:RESTRICT-COMPILER-POLICY 'SAFETY 3 3)` to ~/.sbclrc
- Define a function that declares safety 0, but which would otherwise break this policy, such as:

```lisp
(defun policy-test ()
  (declare (optimize (safety 0)))
  (the integer nil))
```

- And compile

On macOS with SBCL 2.3.4, this will compile without error.

On Linux and Windows this will not compile, explicitly ignoring the local declaration of safety 0.

## Technical Info

My principal dev environment, to which this applies:

sbcl --version =>
SBCL 2.3.4

uname -a =>
Darwin BlackLotusAlpha.ht.home 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64 x86_64

*FEATURES* =>
(:QUICKLISP :ASDF3.3 :ASDF3.2 :ASDF3.1 :ASDF3 :ASDF2 :ASDF :OS-MACOSX :OS-UNIX
 :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :ARENA-ALLOCATOR :X86-64 :GENCGC :64-BIT
 :ANSI-CL :BSD :COMMON-LISP :DARWIN :IEEE-FLOATING-POINT :LITTLE-ENDIAN :MACH-O
 :PACKAGE-LOCAL-NICKNAMES :SB-CORE-COMPRESSION :SB-LDB :SB-PACKAGE-LOCKS
 :SB-THREAD :SB-UNICODE :SBCL :UNIX)

## Additional Notes

Homebrew and Scoop releases are several versions behind, so I'm aware I'm not testing against the latest releases.

[Stas Boukarev (stassats)]

Compile-time type conflicts are detected irrespective of SAFETY. So I have no idea what you are describing.

[''the Phoeron'' Colin J.E. Lupton (thephoeron)]

One would certainly think so. I was pretty confused by this myself, given that the only thing I expect safety optimization to affect is the degree of type-checking preserved in the FASL for runtime, so I have no idea if I characterized the issue correctly. The inconsistent behaviour across platforms didn't help either.

Naturally, I explored all the conjectures specific to CL-ISAAC first. I followed delta debugging, zeroed in on the compiler-policy restriction settings, and came up with a first stab at a falsifiable hypothesis. We need at least one more, but nothing else has come to me yet. At any rate, it was a good enough hypothesis to get CL-ISAAC working on SBCL again.

[Stas Boukarev (stassats)]

Something may be happening, but your analysis and test cases don't demonstrate the problem.

There's a close to 0 chance of cross-platform variance in the compiler, especially when the cpu architecture is the same.

[''the Phoeron'' Colin J.E. Lupton (thephoeron)]

The observable facts contradict this baseline assumption. The only things I can be certain of are that the relevant behaviour is implementation-dependent (the bug only appears in SBCL on any platform) as well as inconsistent by platform (special steps had to be taken to reproduce the bug on a proper subset of supported platforms), because the bug in question did not appear on macOS even after adopting the full configuration parameters of the user reporting the issue, but was immediately reproduced exactly as reported on Linux and Windows. I was able to reproduce the bug on macOS as reported by changing only one line of code: removing the local declaration. This led me to discover the implementation-dependent behaviour of safety, and the rest is pretty much already stated above.

If you'd like a better test case, we can always step back to CL-ISAAC 1.0.7 where the issue is apparent, reproduce the variance by platform, and reanalyze the problem against other Lisp implementations. Other underlying issues may be at play, such as SBCL's internal representation of integers that "overflow" as far as the ISAAC-32 and ISAAC-64 algorithms are concerned, but don't in any other Lisp. That was the source of the type-error bug reported to me by the user, which was already pretty weird considering every other Lisp appeared to be correctly generating 32 and 64 bit integers at that step, but even if they did "overflow" that's what the LOGAND is for, as the modulus. I was going to report this as a separate bug. I've amassed quite a long list of issues I need to report, which I've put off long enough.

Alternatively, I can keep trying to correctly characterize and fix this issue all on my own. I've forked the official mirror of SBCL on GitHub to start the process of contributing, but would appreciate any clarity you can offer on what I can do to support the established workflow.

For the sake of clarity, but tangential to the Issue:

My strategy for bug-testing is based on delta debugging using Abductive Reasoning and Model Theory for my empirical method. I'm well aware that most of the hypotheses generated by abductive inference are going to be nonsense. The intent is to generate enough hypotheses to close the problem space, in the process eliminating irrelevant factors and uncovering hidden variables until the correct system is isolated for the solution space. For most situations this strategy is overkill, so I apply it loosely, relying on experience to decide the granularity. For this situation, it got me to a fix pretty quickly, but feels like an intractable method for the real underlying issue, whatever that might be.

You and I both know from personal experience that compilers are the single most difficult pieces of software to write, maintain, and contribute to. You've been an SBCL maintainer for as long as I can remember. I wrote my first compiler when I was 13, and most recently at my erstwhile start-up Black Brane, a whole framework for language and platform agnostic quantum compilation, as well as quantum variants of Forth, Lisp, and Prolog built on this framework. Basically LLVM for quantum computing.

I'd much rather be working on open-sourcing all...

[Stas Boukarev (stassats)]

I don't need essays, just a reproducible test case.

[''the Phoeron'' Colin J.E. Lupton (thephoeron)]

Fair enough. I gave you one, but if it makes it easier for you I can splice in the relevant code samples here from CL-ISAAC.

On a personal note, Stas, I'm here to contribute, not complain or create more work for you. It took some training but I've got Copilot Chat working well for me. So please don't hesitate to put me to work.