FREE-VARS walker has trouble with macros vs. special forms on some Lisps

[Patrick Stein]

Reported to iterate-devel mailing list: Robert Goldman rpgoldman at sift.info
Thu Jan 5 13:10:49 PST 2012

http://lists.common-lisp.net/pipermail/iterate-devel/2012-January/000692.html

===========================================
Steve Haflich from Franz kindly clarified to me my misunderstanding of
special-operators and macros.

In particular, he pointed out to me that implementations are free to
implement special operators as macros. Steve writes:

"I assure you that the set of special operators in ACL is _fixed+, and
that there is way for you, the _user_, to extend that set. But this
requirement of the ANS does not require that the set of cl-package
symbols that return true under spcial-operator-p is not larger than
the required operators in the table you quoted from
http://www.franz.com/support/documentation/current/ansicl/subsubsu/specialf.htm

Now see here:
http://www.franz.com/support/documentation/current/ansicl/subsubsu/macrofor.htm
specifically

  An implementation is free to implement a Common Lisp special operator
  as a macro. An implementation is free to implement any macro operator
  as a special operator, but only if an equivalent definition of the
  macro is also provided."

He goes on to explain how this works in ACL:

"In ACL

  user(31): (macro-function 'return)
  #<Function macroexpand-1-transform>

so cl:return names _both_ a special operator and a macro. In our
opinion, this is explicitly recognized by the ANS. (I think I
remember being there when we of X3J13 voted this issue, but I also
think I think I sometimes remember things that never occurred. Your
mileage may vary.)

   Unfortunately, this seemingly innocent divergence from the standard
   breaks the code walker in ITERATE (and may break other code walkers, as
   well).

I haven't examined this particular code walker, but I speculate that
it is checking special-operator-p _before_ checking macro-function. A
_portable_ walker should give the latter priority, which is what Franz
believes the "implementation is free" codicil intends."

Steve's understanding of the code walker in FREE-VARS is correct.
Actually, over and above this, there seems to be a second bug in the
code-walker: the check of (MACRO-FUNCTION (CAR FORM)) is not inside the
branch for (SYMBOLP (CAR FORM)).

So I believe that fixing this bug is not just a matter of pandering to
ACL users, but actually a matter of fixing an incorrect code walker that
could cause problems for arbitrary lisp implementations.

I have attempted to fix this bug by moving the branch for MACRO-FUNCTION
into the branch for SYMBOLP (CAR FORM) and moving it up before the check
for SPECIAL-OPERATOR-P.

See attached patch. I'd be obliged if you could review it for
incorporation. It reduces the number of test failures in Allegro CL to
the same as the number for SBCL: one.

Best,
R

===========================================