wanted: compiler macro expansion in later compilation stages (e.g. after inlining)

[Tobias C. Rittweiler]

Compiler macros seem to be expanded early in compilation; this prevents e.g.

  (declaim (inline matches?))
  (defun matches? (regex)
    #'(lambda (line) (cl-ppcre:scan regex line)))

from triggering the compiler macro on CL-PPCRE:SCAN.

[Nikodemus Siivola (nikodemus)]

This is an interesting notion.

Attached diff shows one way to do this: allow IR1-OPTIMIZE to reconstitute source forms for compiler-macros to look at if any of the arguments is a constant.

The same approach could be extended by annotating the lexenv with derived types of arguments, so compiler-macros could benefit from Python's type derivation.

Stuff to do if this is to go in:

* Annotate the lexenv calling the compiler-macro-function.

* Don't do this more than once per combination unless more constants become apparent during optimization.

[Nikodemus Siivola (nikodemus)]

Better version:

* Augment the environment for the compiler macro to look at.

* Only try once per number of constants available.

Can handle this like this without breaking a sweat:

(defun foo (x)
  (format nil "foo=~S" x))

(defparameter *foo-count* 0)

(define-compiler-macro foo (&whole form x)
  (if (constantp x)
      (progn
        (incf *foo-count*)
        (foo x))
      form))

(declaim (inline bar))
(defun bar (x &key car)
  (list car (foo x)))

(declaim (inline quux))
(defun quux (x y)
  (when y x))

(defun zot ()
  (let ((x (quux "ding" t)))
    (bar (quux x (quux t t)) :car '=)))

(assert (= 1 *foo-count*))

[Nikodemus Siivola (nikodemus)]

Final version. I elected to leave the type information out, since looking at it is not portable anyways, in which case users are probably better off writing deftransforms.

Now the question is, is this actually permitted? Are there circumstances where this can cause things to go wrong?

Reading http://www.lispworks.com/documentation/lw50/CLHS/Body/03_bba.htm makes me think that no-one really considered this as an option. The language of the spec talks about forms, but doesn't go out of its way to say "actual source form" or anything like that.

Currently I can't think of a problem, but still wondering.

[Tobias C. Rittweiler (tcr)]

Implementation that allow querying for type information seem
to follow cltl2 env access; so if you enrich the lexenv with type
information, people would be able to write similiar code across
implementations. Having to go to deftransform means adding
a whole other code path that uses SBCL *internals*.

[Tobias C. Rittweiler (tcr)]

Looking at the patch, the new lexenv defaults to the node-lexenv at that place;
I assume this means that in

  (declaim (inline foo))
  (defun foo (x) (quux x))

  (symbol-macrolet ((%foo% t))
     (foo "bar"))

a compiler macro for QUUX will be able to macroexpand %FOO% and obtain T?

[Nikodemus Siivola (nikodemus)]

Yes, just as if FOO had used a compiler-macro instead of being inline.

[Tobias C. Rittweiler (tcr)]

I have a case where I'd like compiler macros to be retried not only on
a constant argument, but basically in any case a compiler macro depends
on &ENVIRONMENT because the inlining could have caused transplanting
the call form into a new lexenv where a MACROLET or SYMBOL-MACROLET
is active.

I use this stuff to optimize special variable access (the general case) into
lexical variable access by saving information at compile-time about whether
a suitable lexical variable is available.

[James Y Knight (foom)]

Waitasec, isn't that simply *incorrect*? Surely inlining a function shouldn't change its lexenv??

[Tobias C. Rittweiler (tcr)]

It does on SBCL. The time I brought it up the response I got was
that it's actually convenient most of the time. Now I agree. :-)

[Nikodemus Siivola (nikodemus)]

Using the call-site lexenv for inline function OPTIMIZE policy is convenient and defendable.

Inline-expansions seeing macros from the call-site would be just wrong:

 (defun xxx () :fun)

 (declaim (inline foo))
 (defun foo () (xxx))

 (defun bar ()
   (macrolet ((xxx () :macro))
     (foo)))

BAR *must* return :FUN.

Compiler-macro expansions... Hair and pain. Currently

  (define-compiler-macro foo () `(xxx))

and recompiling the BAR above causes it to return :MACRO, which strikes me as deeply suspect. Even if this might be convenient for rare cases, in the large this can only produce subtle and hard to find bugs.