overly diligent input buffer filling by external format input routines

Bug #643686 reported by Faré on 2010-09-20
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

In XCVB, I was trying to communicate to slave processes with Unix named pipes. But for some reason, my SBCL processes were hanging. Examining the source, I saw that it was somehow insisting on either reaching EOF or filling a 4KB buffer before it would return to me when reading from the pipe. That's probably a bug. At least, there should be a way to disable this behavior, and it arguably shouldn't be the default unless SBCL knows for sure that it's dealing with a regular file. (And even then, /proc and other magic filesystems might not be happy with what such insistent buffering.)

To reproduce, run this program and type a few forms. You may copy-paste large amount of whitespace to fill the 4KB buffer.

cd /tmp
\rm -f f0
mknod f0 p

sbcl --eval '(progn (with-open-file (i "/tmp/f0" :direction :input) (loop :for f = (read i nil nil \
nil) :while f :do (format t "Got: ~S~%" f) (finish-output))) (quit))' &
echo "Feed forms to SBCL. It won't process them until either EOF or your fill a 4KB buffer"
cat > f0

Nikodemus Siivola (nikodemus) wrote :

The issue is with external format input routines not returning until EOF or till the requested amount of characters is read.

Without testing the effect of this properly, it seems to me that

 diff --git a/src/code/fd-stream.lisp b/src/code/fd-stream.lisp
 index 969b177..5145258 100644
 --- a/src/code/fd-stream.lisp
 +++ b/src/code/fd-stream.lisp
 @@ -1546,8 +1546,8 @@
                  (return-from ,in-function total-copied)))
              (setf (buffer-head ibuf) head)
              ;; Maybe we need to refill the stream buffer.
 - (cond ( ;; If there were enough data in the stream buffer, we're done.
 - (= total-copied requested)
 + (cond ( ;; If was data in the stream buffer, we're done.
 + (plusp total-copied)
                     (return total-copied))
                    ( ;; If EOF, we're done in another way.
                     (or (eq decode-break-reason 'eof)

is TRT -- at least it solves the reported case. (Here's hoping launchpad doesn't mangle that too badly...)

summary: - SBCL insists on buffering named pipes
+ overly diligent input buffer filling by external format input routines
Changed in sbcl:
status: New → Triaged
importance: Undecided → Medium
Nikodemus Siivola (nikodemus) wrote :

Attached the above fix as a patch. Builds and passes tests, but haven't yet verified that it fixes the issue for all external formats, or that there aren't any other unforeseen interactions.

Changed in sbcl:
assignee: nobody → Nikodemus Siivola (nikodemus)
status: Triaged → In Progress
Nikodemus Siivola (nikodemus) wrote :

The above fix is correct. Will commit after 1.0.43.

Nikodemus Siivola (nikodemus) wrote :


Changed in sbcl:
assignee: Nikodemus Siivola (nikodemus) → nobody
status: In Progress → Fix Committed
Changed in sbcl:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers