SPECIAL-OPERATOR-P should return true only for the 25 standard special operators

[Jean-Philippe Paradis]

What I do:
(let ((cl-package (find-package '#:cl))
      cl
      others)
  (do-all-symbols (symbol
                   (flet ((process (label symbols)
                            (format t "~2&---~A---" label)
                            (print (length (mapc #'print
                                                 (sort (delete-duplicates symbols)
                                                       #'string<
                                                       :key #'symbol-name))))))
                     (progn (process "OTHERS" others)
                            (process "CL" cl)
                            (values))))
    (when (special-operator-p symbol)
      (if (eq (symbol-package symbol) cl-package)
          (push symbol cl)
          (push symbol others)))))

What happens:
Output:
---OTHERS---
SB-C::%%ALLOCATE-CLOSURES
SB-C::%CLEANUP-FUN
SB-C::%ESCAPE-FUN
SB-C::%FUNCALL
SB-SYS:%PRIMITIVE
SB-C::%WITHIN-CLEANUP
SB-CLTL2:COMPILER-LET
SB-C::GLOBAL-FUNCTION
SB-EXT:TRULY-THE
9

---CL---
BLOCK
CATCH
EVAL-WHEN
FLET
FUNCTION
GO
IF
LABELS
LET
LET*
LOAD-TIME-VALUE
LOCALLY
MACROLET
MULTIPLE-VALUE-CALL
MULTIPLE-VALUE-PROG1
PROGN
PROGV
QUOTE
RETURN-FROM
SETQ
SYMBOL-MACROLET
TAGBODY
THE
THROW
UNWIND-PROTECT
25

What I expected to happen:
SPECIAL-OPERATOR-P should only return true for the 25 standard special operators.

Analysis:
From CLHS SPECIAL-OPERATOR-P:
"Returns true if symbol is a special operator; otherwise, returns false."

From the glossary entry for "special operator":

"one of a fixed set of symbols, enumerated in Figure 3-2, that may appear in the car of a form in order to identify the form as a special form."

I'm aware that special forms can be implemented as macros and vice-versa, nevertheless SPECIAL-OPERATOR-P should return true only for the 25 standard ones. This behavior also makes much more sense for code walkers than the current one. Portable code-walkers do need to treat the 25 special operators specially, while they can completely ignore any extraneous implementation-dependent special operators because they can just be expanded as normal macros (as required by the standard).

SBCL version: 1.0.42
uname -a: Linux dynamorph 2.6.32-30-generic #59-Ubuntu SMP Tue Mar 1 21:30:21 UTC 2011 i686 GNU/Linux

*features*:
(:SWANK :QUICKLISP :SB-BSD-SOCKETS-ADDRINFO :ASDF2 :ASDF :ANSI-CL :COMMON-LISP
 :SBCL :SB-DOC :SB-TEST :SB-LDB :SB-PACKAGE-LOCKS :SB-UNICODE :SB-EVAL
 :SB-SOURCE-LOCATIONS :IEEE-FLOATING-POINT :X86 :UNIX :ELF :LINUX :SB-THREAD
 :LARGEFILE :GENCGC :STACK-GROWS-DOWNWARD-NOT-UPWARD :C-STACK-IS-CONTROL-STACK
 :COMPARE-AND-SWAP-VOPS :UNWIND-TO-FRAME-AND-CALL-VOP :RAW-INSTANCE-INIT-VOPS
 :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-VECTORS
 :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-FIXED-OBJECTS :ALIEN-CALLBACKS
 :CYCLE-COUNTER :INLINE-CONSTANTS :MEMORY-BARRIER-VOPS :LINKAGE-TABLE
 :OS-PROVIDES-DLOPEN :OS-PROVIDES-PUTWC :OS-PROVIDES-SUSECONDS-T)

[Roman Marynchak (roman-marynchak)]

All these are defined by DEF-IR1-TRANSLATOR macro, which marks them as special operators. Can we redefine these nine operators in some other way? This is an interesting question.

[Pascal J. Bourguignon (pjb-informatimago)]

The standard quite clearly allows for implementation specific special operators, but imposes constraints only on special operators named in the CL package.

The problem with those extras special operators is that they don't have a macro definition; see https://bugs.launchpad.net/sbcl/+bug/1032643

With a macro definition for all non standard special operators, generic code walkers can be writen.

[Douglas Katzman (dougk)]

resolving invalid. For comparison:

ABCL
====
Armed Bear Common Lisp 1.7.1
CL-USER(1): (let (list) (do-all-symbols (s) (when (special-operator-p s) (pushnew s list))) (length list))
43

CLISP
=====
Welcome to GNU CLISP 2.49.93+ (2018-02-18) <http://clisp.org/>
[1]> (let (list) (do-all-symbols (s) (when (special-operator-p s) (pushnew s list))) (length list))
40

ECL
===
ECL (Embeddable Common-Lisp) 16.1.3 (git:UNKNOWN)
> (let (list) (do-all-symbols (s) (when (special-operator-p s) (pushnew s list))) (length list))
45

CCL
===
Welcome to Clozure Common Lisp Version 1.10-r16196 (LinuxX8664)!
? (let (list) (do-all-symbols (s) (when (special-operator-p s) (pushnew s list))) (length list))
31

[Jean-Philippe Paradis (hexstream)]

I disagree, the CLHS is very clear with its "one of a fixed set of symbols, enumerated in Figure 3-2" (which I quoted above).

Just because most or all implementations currently get it wrong (and probably accidentally) is not necessarily a good excuse to (gratuitously?) deviate from the standard.

PJB's point is pertinent; users are not even supposed to need to care about non-standard special operators.

At least this is pretty easy to work around, but I would strongly prefer strict conformance to the standard (which is a rare point of almost universal agreement in the community).

This should work on all implementations (but I tested only on SBCL):

(let ((cl-package (find-package '#:cl)))
  (defun my-special-operator-p (symbol)
    (and (special-operator-p symbol)
         (or (eq (symbol-package symbol) cl-package)
             (multiple-value-bind (found status)
                 (find-symbol (symbol-name symbol) cl-package)
               (and (eq found symbol) (eq status :external)))))))

(Sadly, the home package of external symbols in the COMMON-LISP package is not guaranteed to be the COMMON-LISP package, per CLHS 11.1.2.1: http://www.lispworks.com/documentation/HyperSpec/Body/11_aba.htm)

(An alternative implementation would use a hash-table with the 25 expected symbols.)