Broadcast streams with no components can't be closed

Bug #1904722 reported by Richard M Kreuter on 2020-11-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
SBCL
Undecided
Unassigned

Bug Description

The close method for a broadcast stream does nothing if the stream has no components:

--
(with-open-stream (bs (make-broadcast-stream))
    (values (close bs)
            (open-stream-p bs)))
=> T
=> T
--

This is not obviously conforming of SBCL, not a behavior that a program can conformingly benefit from, and not a feature that a program can use very portably (ABCL, Allegro, Clisp, ECL, LispWorks all close the broadcast stream, though CCL and CMUCL don't).

Might as well close the stream.

Noticed this while looking into https://bugs.launchpad.net/sbcl/+bug/1904257. Bug reporting metadata:

$ uname -a
Darwin m5.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64
$ sbcl --version
SBCL 2.0.0
$ sbcl --no-userinit --no-sysinit --non-interactive --eval '(format t "~{~S~%~}" *features*)'
This is SBCL 2.0.0, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
:X86-64
:64-BIT
:ALIEN-CALLBACKS
:ANSI-CL
:AVX2
:BSD
:C-STACK-IS-CONTROL-STACK
:CALL-SYMBOL
:COMMON-LISP
:COMPACT-INSTANCE-HEADER
:COMPARE-AND-SWAP-VOPS
:CYCLE-COUNTER
:DARWIN
:FP-AND-PC-STANDARD-SAVE
:GENCGC
:IEEE-FLOATING-POINT
:IMMOBILE-CODE
:IMMOBILE-SPACE
:INODE64
:INTEGER-EQL-VOP
:LINKAGE-TABLE
:LITTLE-ENDIAN
:MACH-EXCEPTION-HANDLER
:MACH-O
:OS-PROVIDES-BLKSIZE-T
:OS-PROVIDES-DLADDR
:OS-PROVIDES-DLOPEN
:OS-PROVIDES-PUTWC
:OS-PROVIDES-SUSECONDS-T
:PACKAGE-LOCAL-NICKNAMES
:SB-CORE-COMPRESSION
:SB-DOC
:SB-EVAL
:SB-LDB
:SB-PACKAGE-LOCKS
:SB-SIMD-PACK
:SB-SIMD-PACK-256
:SB-SOURCE-LOCATIONS
:SB-THREAD
:SB-UNICODE
:SBCL
:STACK-ALLOCATABLE-CLOSURES
:STACK-ALLOCATABLE-FIXED-OBJECTS
:STACK-ALLOCATABLE-LISTS
:STACK-ALLOCATABLE-VECTORS
:STACK-GROWS-DOWNWARD-NOT-UPWARD
:UNDEFINED-FUN-RESTARTS
:UNIX
:UNWIND-TO-FRAME-AND-CALL-VOP

Douglas Katzman (dougk) wrote :

The broadcast stream with no constituent streams is a singleton object.
This is useful in that it is the lisp equivalent of /dev/null, and various operations might be able to take advantage of detecting that singleton object and doing nothing.
Supposing there were a build-time feature for #+broadcast-stream-always-construct-new-stream, are you seriously suggesting that people would want that? I didn't think so.

Richard M Kreuter (kreuter) wrote :

Oh, I hadn't noticed that SBCL has null-ary MAKE-BROADCAST-STREAM return a singleton. That accounts for the CLOSE behavior.

However, while I see a benefit of having a distinguished "null output" sink, but does it need to be a singleton returned by the (apparent) constructor? A variable in SB-EXT would suffice, and would be portably emulatable on other implementations:

--
(defpackage "MY-PACKAGE"
  #+sbcl
  (import-from "SB-EXT" "*NULL-OUTPUT*") ;say
  ...)

(in-package "MY-PACKAGE")

#-sbcl ; SBCL already has this
(defvar *null-output* (make-broadcast-stream))
--

Anyway, the reason I care is that among some interface invariants that are desirable to have and seem to be intuitive readings of the standard include:

a. when CLOSE runs to completion, the stream is closed afterward, and
b. when a stream is closed, OPEN-STREAM-P returns false.
c. CLOSE does nothing on a closed stream.

Why these invariants? Well, I like to make sure streams get closed eventually, and in programs that create a dynamic number of streams, that's easier to do when CLOSE and OPEN-STREAM-P have those guarantees.

That said, I'd agree that if a distinguished stream is to serve as a global "null output" sink, closing that stream would likely cause trouble, so I can understand why you've made CLOSE a no-op for the singleton broadcast stream.

OTOH, it's already the case that there are a number of distinguished global streams that it's bad to close (the values of *stdin*, *stdout*, *stderr*, *tty*, for example; and if synonym streams get their own distinct open-ness as discussed in 1904257, then any of the standard I/O customization variables). If /that/ problem is worth solving for, ISTM an orthogonal mechanism equivalent to the following could be nice to have.

--
(defvar *never-close-streams* ())

(define-condition never-close-error (error)
  ((stream :initarg :stream :reader never-close-error-stream))
  (:report (lambda (error stream)
             (format stream "~S should not normally be closed."
                     (never-close-error-stream error)))))

(defmethod close :before ((stream stream))
  (when (member stream *never-close-streams*)
    (cerror "Continue" 'never-close-error :stream stream)))

;; User interface.
(defun never-close (stream)
  "Make it an error to try to close STREAM. Note that this does not persist across
SAVE-LISP-AND-DIE; use an *INIT-HOOK* function to reinstate."
  (pushnew stream *never-close-streams*))

;; Then stream-init can put *stdin*, *stdout*, *stderr*, *tty*, and
;; so forth onto *never-close-streams*, and stream-deinit could
;; reset *no-close-streams* to nil.
--

Untested, but if you're interested I can elaborate on this orthogonal mechanism. Let me know?

> so I can understand why you've made CLOSE a no-op for the singleton
> broadcast stream.
>
> OTOH, it's already the case that there are a number of distinguished
> global streams that it's bad to close (the values of *stdin*, *stdout*,
> *stderr*, *tty*, for example; and if synonym streams get their own
> distinct open-ness as discussed in 1904257, then any of the standard I/O
> customization variables). If /that/ problem is worth solving for, ISTM
> an orthogonal mechanism equivalent to the following could be nice to
> have.

CLOSE makes sense only to dispose foreign file descriptors or handles
early.

It does not make sense for pure lisp streams. Especially if they are
layered on top of other streams.

Richard M Kreuter (kreuter) wrote :

Sure, it's true that on SBCL composite streams don't hold onto OS resources that need releasing. But that's orthogonal to whether programs ought to close them.

That composite streams are pure Lisp streams on SBCL is an implementation detail, not part of their standardized interface. For example, what if another implementation, or a future version of SBCL, does a broadcast stream as a wrapper around as a pipe, the other end of which is handled by a different thread? Or as a wrapper around a handle to a foreign structure created and manipulated only through an FFI? Given the permissible variations of implementation strategies, a program that doesn't close its composite streams isn't portable.

Further, 21.1.4 says that closing a component of a composite stream while the stream is open has undefined consequences. So a program that doen't close its composite streams but does eventually close underlying component streams is not conforming.

Not all programs need to be conforming or portable, but IMO SBCL ought to support programs that try to be. Conforming or portable programs are supposed to close streams when they're done using them, and so closing streams ought to behave as standardized.

Tomas Hlavaty (tomas-hlavaty) wrote :

how can you close underlying component streams of Broadcast streams with
no components?

why do you think that Conforming or portable programs are supposed to
close streams when they're done using them?

Richard M Kreuter (kreuter) wrote :
Download full text (6.8 KiB)

Thank you for requesting a clarification.

> how can you close underlying component streams of Broadcast streams
> with no components?

My apologies; I wasn't very clear. I was responding to your proposition that CLOSE only makes sense to dispose of file descriptors or handles, by pointing out that a program should also close any composite stream having a component that the program also wants to close (in order for the program to satisfy 21.1.4).

You're correct that there are a couple of cases where a standard composite stream can have zero components: an empty broadcast stream, and also a concatenated stream that has reached EOF on all of its original components. A program that omits closing those streams won't violate 21.1.4, but is not portable in the ANSI sense (explained below).

Either way, if a program has a collection of streams to deal with, is it supposed to check every stream for type and composition before deciding whether to close it? Or is a program allowed to depend on the standard's requirement that

(progn (close stream) (open-stream-p stream))

closes the stream and returns NIL? (Well, if the form returns, anyway; CLOSE can error.)

Although it is unusual to check whether a stream is still open after closing it, the program I'm working on does so (that's what got me filing 1904722 and then this issue). Here's why: the program is an interpreter that creates an unpredictable number of streams of different types, and can't manage them all using WITH- style forms. So the program does its own bookkeeping, and is supposed to eventually close every stream it creates. But I found a leak, so I added a sanity check to the end of the program:

(assert (notany #'open-stream-p *streams*) (*streams*))

That is, the program does a CLOSE and then an OPEN-STREAM-P on every stream it creates, but at different locations in code. And this gets tripped up by the existence of streams that cannot be closed. I can work around it by filtering out empty broadcast streams on SBCL, but that workaround embeds the assumption that today's observable non-conformance won't go away in the future. If SBCL becomes conformant in this detail in the future, such filtering will introduce a false negative into my checking.

> why do you think that Conforming or portable programs are supposed
> to close streams when they're done using them?

Fair questions. The following is rather pedantic, but you asked. :-)

First I'll describe portability. Here's the "existence proof" that certain kinds of streams ought to get closed in a portable program. (I doubt this is controversial, but I want to make it explicit.)

(1) Some open streams can contain resources that need management to avoid risk of starvation (file descriptors/handles, and maybe also locks, buffers, cursors, and so forth).

(2) Next, some open streams have close-time side effects on the state of the external environment (e.g., deleting or renaming files; or maybe committing or rolling back a database transaction).

(3) Finally, to my knowledge, the standard is silent about whether implementations implicitly close streams that programs don't take measures to close themselves.

Consequently, if a program create...

Read more...

Richard M Kreuter (kreuter) wrote :

Oh well, there's an error in my last comment: LIFO order closing won't work for every program. For example:

(defvar *a-stream*)
(defvar *streams* ())
(push (open "foo") *streams*)
(setq *a-stream* (first *streams*))
(push (make-synonym-stream '*a-stream*) *streams*)
(push (open "bar") *streams*)
(setq *a-stream* (first *streams*))

At this point, *STREAMS* contains 3 streams in most-recently-created-first order, but closing them in that order will violate 21.1.4. Nonetheless, if a program wants to close its streams, it ought to do so in a correct order; that ordering will be program-dependent.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers