TRACE doesn't bind I/O control variables effectively

[Richard M Kreuter]

Ran into some troubles using TRACE to debug a routine called in a program that uses some peculiar pretty printing routines. In poking around at it, found the following:

1. TRACE-END-BREAKPONT-FUN currently binds *PRINT-PRETTY* to true, but no other I/O control variables. So TRACE's output of results of the call can be unaesthetic at best, or could dump you into the debugger if you've got a dodgy pretty printer routine.

2. TRACE-END-BREAKPOINT-FUN runs the PRINT-AFTER function in a different dynamic environment, where *PRINT-PRETTY* might be disabled. This could produce differently unaesthetic results, or, say, dump you into the debugger under different circumstances than item 1.

3. TRACE-START-BREAKPOINT-FUN sets up a controlled I/O environment, but doesn't use the pretty printer.

Items 1 and 2 are defects, in that the dynamic environment on the way out of a traced call might be hostile for printing. Reproduction code for both below.

Item 3 is mostly an aesthetic concern around symmetry, though is arguably a functional shortcoming.

Attached is a patch that does the following:

A. Add a new exported variable, SB-DEBUG:*TRACE-PPRINT-DISPATCH*. The initial value is a copy of the initial pprint dispatch table. The user can modify or bind this freely, including assigning it to be the current pretty printer dispatch table.

B. Add an unexported macro, WITH-TRACE-IO-ENVIRONMENT, that sets up a standard/sane I/O environment and that binds *PRINT-PRETTY* to true uses *TRACE-PPRINT-DISPATCH*.

C. Have TRACE-START-BREAKPOINT-FUN and TRACE-END-BREAKPOINT-FUN both use WITH-TRACE-IO-ENVIRONMENT around their normal printing as well as their conditional printing.

D. Finally, move the BREAK calls outside of the controlled I/O environment, so that BREAK gets called in traced function's dynamic environment, not TRACE's dynamic environment. (SBCL's default debugger will rebind everything anyway, but I think the distinction is observable if someone has a custom *DEBUGGER-HOOK*.)

Bug report requirements:

SBCL version: 2.0.6.115-fcaea4fd7-dirty.
uname -a: Darwin <redacted> 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64
*FEATURES*: (:X86-64 :GENCGC :64-BIT :ANSI-CL :BSD :COMMON-LISP :DARWIN :IEEE-FLOATING-POINT :LITTLE-ENDIAN :MACH-O :PACKAGE-LOCAL-NICKNAMES :SB-LDB :SB-PACKAGE-LOCKS :SB-THREAD :SB-UNICODE :SBCL :UNIX)

Reproduction file follows.

(in-package "CL-USER")

;;; Problem 1: TRACE doesn't control the I/O environment when printing
;;; the results of the traced call.

;; Suppose you've got a pretty printer routine that sometimes errors.
;; (One way to sometimes error is to always error.)
(defun pprint-integer-incompetently (stream integer)
  (declare (ignore stream integer))
  (error "Oops!"))

(defvar *my-pprint-dispatch* (copy-pprint-dispatch nil))

(set-pprint-dispatch 'integer 'pprint-integer-incompetently
       0 *my-pprint-dispatch*)

;; Suppose you've got a routine that sometimes returns a value that
;; your pretty printer sometimes chokes on.
(defun foo () 5)

;; Trace that function.
(trace foo)

;; This will TRACE just fine.
(foo)

;; But this will fail in TRACE itself.
(let ((*print-pprint-dispatch* *my-pprint-dispatch*)
      (*print-pretty* t))
  (foo))

;;; Problem 2: TRACE-AFTER routine runs in the user's I/O environment.
;;; This requires a slightly more contrived reproduction.

;; Suppose you're using WITH-STANDARD-IO-SYNTAX as a cheap and
;; cheerfulC-u3M-DELsleazy way to dump some objects to text for
;; READing later, but you want to be vaguely principled about it, so
;; you use WITH-STANDARD-IO-SYNTAX to bind *PRINT-READABLY*, and ensure
;; you give special treatment to anything not standardly readable.

(defvar *pprint-dispatch-table-for-my-data* (copy-pprint-dispatch nil))
(defmacro with-my-io-environment (&body body)
  `(with-standard-io-syntax
     (let ((*read-eval* nil)
    (*print-pprint-dispatch* *pprint-dispatch-table-for-my-data*))
       ,@body)))

;; Hash tables have no standard read syntax, so you make up a syntax
;; for it.
(defun pprint-hash-table (stream hash-table)
  (let ((test-name (find (hash-table-test hash-table) '(eq eql equal equalp)
    :test #'(lambda (x y)
       (eql (coerce x 'function)
     (coerce y 'function)))))
 stuff)
    (maphash #'(lambda (key value) (push (cons key value) stuff)) hash-table)
    (write-string "#H" stream)
    (write (cons test-name stuff) :stream stream)
    nil))
(set-pprint-dispatch 'hash-table 'pprint-hash-table
       0 *pprint-dispatch-table-for-my-data*)

;; Now let's have a function that returns a hash table.
(defun bar ()
  (make-hash-table))

;; Trace it without :PRINT-AFTER
(trace bar)

;; This will print the hash table using my pretty-printer routine
(with-my-io-environment
  (bar))

;; But trace it again with a PRINT-AFTER...
(trace bar :print-after (sb-debug:arg 0))

;; ...and this will break.
(with-my-io-environment
  (bar))

[Attila Lendvai (attila-lendvai)]

if someone gets into patching trace, the debugger, and printing, then i also have this lying around for a long while:

https://github.com/attila-lendvai/sbcl/commit/c497e9fa7343a3e649b30738fdfbca46e8826a43

(it's not directly relevant for this issue, but certainly related)