Missing source annotations

[leppie]

Hi

Currently, annotated-case-lambda and annotated-call gives me a good amount of detail. But it seems to get nuked/removed on some macro expansions.

Here is the most obvious one:

Case 1:

(define-syntax define-fx
  (lambda (x)
    (syntax-case x ()
      [(_ (name formals ...) body body* ...)
        (with-syntax (((checks ...)
          (map (lambda (f)
                 (with-syntax ((f f))
                   #'(unless (fixnum? f)
                      (assertion-violation 'name "not a fixnum" f))))
                #'(formals ...))))
          #'(define (name formals ...)
              checks ...
              (let ()
                body body* ...)))])))

(define-fx (fx+ x1 x2)
  (+ x1 x2))

Case 2 (simplest macro):

(define-syntax add
  (syntax-rules ()
    [(_ n ...) (+ n ...)]))

(add 1 2 3) ; <-- annotated call 'loses' info

Case 3:

(define-syntax define*
  (syntax-rules ()
    [(_ n e ...)
      (define n (lambda () e ...))]))

I will add more as I find them, or can reproduce them.

Cheers

leppie

[Abdulaziz Ghuloum (aghuloum)]

Fixed this case in revision 1789.

[Abdulaziz Ghuloum (aghuloum)]

Please reopen as you find more.

[leppie (leppie)]

Hi

This works, thanks, but the behaviour is rather inconsistent now as both the following forms end up as annotated-call:

(define
  (foo x) ; <- this list is the 'area' for the 'lambda'
  (+ x 1))

(define bar
  (lambda (x) (+ x 1)) ; <- this list is the 'area' for the 'lambda'
  )

Now I can live with the above handling, but would prefer that the body of the function be wrapped in the 'area'.

IOW for the first example it will be:

(define (foo x) (+ x 1)) ; <- this list is the 'area' for the 'lambda'

I know I pass psyntax an annotation for that 'area'. But keeping track of it (from what I can see) looks like quite involved.

Thanks

leppie

[Abdulaziz Ghuloum (aghuloum)]

On May 25, 2009, at 9:20 AM, leppie wrote:

> Hi
>
> This works, thanks, but the behaviour is rather inconsistent now as
> both
> the following forms end up as annotated-call:

You mean annotated-lambda, right?

> (define
> (foo x) ; <- this list is the 'area' for the 'lambda'
> (+ x 1))
>
> (define bar
> (lambda (x) (+ x 1)) ; <- this list is the 'area' for the 'lambda'
> )

What exactly is an 'area'? In Ikarus, I only have a starting
position in an annotation, but I presume you have a start and
end positions that constitute this area, right? The annotation
psyntax now uses for lambda is the whole (lambda ---) form
while the one it uses for (define (name . ---) ---) is the name
form. So, your first example should say:

(define (
          foo ; <- this is the area for the lambda
          x)
   (+ x 1))

right?

One way to make it more consistent is to change the lambda
annotation so that it only grabs the "lambda" keyword in the
(lambda ---) form.

> Now I can live with the above handling, but would prefer that the body
> of the function be wrapped in the 'area'.
>
> IOW for the first example it will be:
>
> (define (foo x) (+ x 1)) ; <- this list is the 'area' for the
> 'lambda'

I think this is how it used to be before the commit, right?
The general problem we have is that syntax-case strips out
some syntax information as it destructures the syntax object
and constructs plain pairs that have no syntax information.
(the user too can construct plain pairs using cons, map, ...)
So, after a macro step, we're guaranteed that identifiers have
their annotations intact but we don't have the same guarantee
for lists. This is why I think using the "lambda" keyword as
the annotation is better than using the whole (lambda ---)
form.

Anyways, try to experiment with chi-defun in the expander,
passing different objects (x, kwd, ctxt, body*, ...) to see
what's the best effect for you and I can accommodate.

   (define (chi-defun x r mr)
     (syntax-match x ()
       [(kwd (ctxt . fmls) . body*)
        (let-values (((fmls body)
                      (chi-lambda-clause fmls fmls body* r mr)))
           (build-lambda (syntax-annotation ctxt) fmls body))]))

> I know I pass psyntax an annotation for that 'area'. But keeping track
> of it (from what I can see) looks like quite involved.

For macros, yes. I need to think about it some more. Keep
me posted.

Aziz,,,

[leppie (leppie)]

> What exactly is an 'area'?

It is the start and end position.

> So, your first example should say:

I dont actually track symbols, only lists/vectors. I can see from the code it should pick up the 'foo' symbol, but it I get the actual annotation around it, hence (foo x). Will check to see if it is a bug in my code.

> Anyways, try to experiment with chi-defun in the expander,

I did play a little, but it (the annotation) comes from the macro body.

> For macros, yes.

I will dig a little more :)

Thanks

leppie

[Abdulaziz Ghuloum (aghuloum)]

On May 25, 2009, at 10:40 AM, leppie wrote:

> I dont actually track symbols, only lists/vectors.

I think you *should* keep track of annotations for all
atoms (symbols, numbers, booleans, characters, strings,
etc.).

Aziz,,,

[leppie (leppie)]

> I think you *should* keep track of annotations for all
> atoms (symbols, numbers, booleans, characters, strings,
> etc.).

I think I should too, but have not found an acceptable idea/solution (haven't given it much thought really for a quite a while now).

I have noticed the same 'issue' comes up for annotated-call for procedure calls in macro output (in the original sample, assertion-violation is one). I get a syntax object/list instead of an annotation. Again, I am not really concerned about stepping 'into' macro's (but it might be nice :-) ).

Anyways, I'm gonna play some more :)

Cheers

leppie

[leppie (leppie)]

> I dont actually track symbols, only lists/vectors. I can see from
> the code it should pick up the 'foo' symbol, but it I get the actual
> annotation around it, hence (foo x). Will check to see if it is a bug
> in my code.

It looks like I was indeed cheating for symbols :o) I was mapping the enclosing list's location to them. Time to fix!

[leppie (leppie)]

I have now changed my reader to annotate everything. :)

This has not made things any better on my side however :(

But no worries, I have not made up my mind on how to handle the 'area' (meaning where a procedure entry breakpoint can be set) of a define, lambda and case-lambda.

Cheers

leppie

[leppie (leppie)]

Hi

Eli convinced me the procedure entry is not really useful, due to macros. So it does not really matter how to define a procedure 'body' in terms of define, lambda, case-lambda. (I might need this for the native debugger, still not sure)

On the plus side, I made a little mod to handle case 2.

You will need to expose your 'make-annotation' procedure.

In do-macro-call, replace:

(add-mark (gen-mark) x expr)

with:

(add-mark (gen-mark)
  (if (and (pair? x)
           (stx? expr)
           (annotation? (stx-expr expr)))
      (make-annotation x (annotation-source (stx-expr expr)) x)
      x)
  expr))

This works well for the simple case. It appears to work for multiple macro transforms too.

Not sure how correct this is.

Testing continues!

Cheers

leppie

PS: this hack was purely possible, because of the debugging support! Thanks again :)

[leppie (leppie)]

In chi-defun (with above) you can do:

(build-lambda (syntax-annotation x) fmls body)

and it will return location wrapping the body of the procedure.

[Abdulaziz Ghuloum (aghuloum)]

On May 28, 2009, at 8:14 PM, leppie wrote:

> (add-mark (gen-mark) x expr)
>
> with:
>
> (add-mark (gen-mark)
> (if (and (pair? x)
> (stx? expr)
> (annotation? (stx-expr expr)))
> (make-annotation x (annotation-source (stx-expr expr)) x)
> x)
> expr))

Doesn't look right. I think there are places in the expander where
it assumes that annotations are always inside top-marked stx objects
and if you wrap an annotation around an stx, it gets confused.

Try looking inside the stx-ae* field which contains the trace of all
macro expansions that ultimately lead to the given expression. Maybe
the first or the last one is what you're looking for. Let me know.

Aziz,,,

[leppie (leppie)]

Ok, I'll have a look.

I haven't picked up any issues yet. Runs the PLT R6RS test suite fine.

Is there a sample of how this might cause an error?

As for stx-ae*, that is null for expr, so I dont know how to proceed with that.

Is there perhaps some other way to return a syntax list with an annotation? Perhaps the annotation needs to be wrapped in a syntax object? Could that work?

Cheers

leppie

[leppie (leppie)]

My mistake on case.

Would it be possible to annotate variable references as well?