wanted: use-value restart for decoding errors on streams

[Nikodemus Siivola]

(with-open-file (f "/tmp/foo.tmp" :element-type '(unsigned-byte 8) :direction :output)
  (write-sequence '(102 111 111 98 97 114 240) f))

(with-open-file (f "/tmp/foo.tmp" :external-format :utf-8)
  (let* ((size (file-length f))
         (buffer (make-string size)))
    (read-sequence buffer f)
    buffer))

Observe: no USE-VALUE restart.

[Christophe Rhodes (csr21-cantab)]

Hi,

 status inprogress
 done

This is done in my external-formats tree at
<http://rvw.doc.gold.ac.uk/sullivan/git/sbcl.git>; testing that tree
would be welcome (as there are a couple more things to do before
merging).

Christophe

[Nikodemus Siivola (nikodemus)]

I'm assuming the stuff Christophe was working on is actually now in the tree.

This bug is still open, though, since in the current scheme on things one case set the replacement character once, but cannot handle individual ones or capture decoding errors in the first place.

Adding this doesn't look hard in the current scheme of things, though.

[Christophe Rhodes (csr21-cantab)]

Nikodemus Siivola <email address hidden> writes:

> I'm assuming the stuff Christophe was working on is actually now in the
> tree.

I think so too. (I do have a branch in my local tree, but I think it
got merged and I just forgot to delete the branch. I'll try to check)

> This bug is still open, though, since in the current scheme on things
> one case set the replacement character once, but cannot handle
> individual ones or capture decoding errors in the first place.

Well, the documented interface is through :external-format, but since
it's implemented in terms of handlers and restarts (if I remember
correctly) programmatically doing stuff should be possible too.
Something like
  (defun smart-replacement-function (c)
    (let ((char (code-char (character-encoding-error-code c)))
          (xf (character-coding-error-external-format c)))
      (magic char xf)))
  (handler-bind ((encoding-error #'smart-replacement-function))
    ...)
should already be possible, modulo exported symbols (and lack of MAGIC :-)

> Adding this doesn't look hard in the current scheme of things, though.

Making magic available per-stream might involve a little bit of work.
Can't really remember :-/

Cheers,

Christophe

[Nikodemus Siivola (nikodemus)]

As far as I can tell currently the encoding/decoding errors are always handled by the system before the user handler has a chance to run.

A secondary issue is that the relevant conditions are not documented or exported.