overly diligent input buffer filling by external format input routines

[Faré]

In XCVB, I was trying to communicate to slave processes with Unix named pipes. But for some reason, my SBCL processes were hanging. Examining the source, I saw that it was somehow insisting on either reaching EOF or filling a 4KB buffer before it would return to me when reading from the pipe. That's probably a bug. At least, there should be a way to disable this behavior, and it arguably shouldn't be the default unless SBCL knows for sure that it's dealing with a regular file. (And even then, /proc and other magic filesystems might not be happy with what such insistent buffering.)

To reproduce, run this program and type a few forms. You may copy-paste large amount of whitespace to fill the 4KB buffer.

#!/bin/sh
cd /tmp
\rm -f f0
mknod f0 p

sbcl --eval '(progn (with-open-file (i "/tmp/f0" :direction :input) (loop :for f = (read i nil nil \
nil) :while f :do (format t "Got: ~S~%" f) (finish-output))) (quit))' &
echo "Feed forms to SBCL. It won't process them until either EOF or your fill a 4KB buffer"
cat > f0

[Nikodemus Siivola (nikodemus)]

The issue is with external format input routines not returning until EOF or till the requested amount of characters is read.

Without testing the effect of this properly, it seems to me that

 diff --git a/src/code/fd-stream.lisp b/src/code/fd-stream.lisp
 index 969b177..5145258 100644
 --- a/src/code/fd-stream.lisp
 +++ b/src/code/fd-stream.lisp
 @@ -1546,8 +1546,8 @@
                  (return-from ,in-function total-copied)))
              (setf (buffer-head ibuf) head)
              ;; Maybe we need to refill the stream buffer.
 - (cond ( ;; If there were enough data in the stream buffer, we're done.
 - (= total-copied requested)
 + (cond ( ;; If was data in the stream buffer, we're done.
 + (plusp total-copied)
                     (return total-copied))
                    ( ;; If EOF, we're done in another way.
                     (or (eq decode-break-reason 'eof)

is TRT -- at least it solves the reported case. (Here's hoping launchpad doesn't mangle that too badly...)

[Nikodemus Siivola (nikodemus)]

Attached the above fix as a patch. Builds and passes tests, but haven't yet verified that it fixes the issue for all external formats, or that there aren't any other unforeseen interactions.

[Nikodemus Siivola (nikodemus)]

The above fix is correct. Will commit after 1.0.43.

[Nikodemus Siivola (nikodemus)]

In 1.0.43.6.