Bug #1904722 “Broadcast streams with no components can't be clos...” : Bugs : SBCL

Revision history for this message

Douglas Katzman (dougk) wrote on 2020-11-18:

#1

The broadcast stream with no constituent streams is a singleton object.
This is useful in that it is the lisp equivalent of /dev/null, and various operations might be able to take advantage of detecting that singleton object and doing nothing.
Supposing there were a build-time feature for #+broadcast-stream-always-construct-new-stream, are you seriously suggesting that people would want that? I didn't think so.

Revision history for this message

Richard M Kreuter (kreuter) wrote on 2020-11-19:

#2

Oh, I hadn't noticed that SBCL has null-ary MAKE-BROADCAST-STREAM return a singleton. That accounts for the CLOSE behavior.

However, while I see a benefit of having a distinguished "null output" sink, but does it need to be a singleton returned by the (apparent) constructor? A variable in SB-EXT would suffice, and would be portably emulatable on other implementations:

--
(defpackage "MY-PACKAGE"
  #+sbcl
  (import-from "SB-EXT" "*NULL-OUTPUT*") ;say
  ...)

(in-package "MY-PACKAGE")

#-sbcl ; SBCL already has this
(defvar *null-output* (make-broadcast-stream))
--

Anyway, the reason I care is that among some interface invariants that are desirable to have and seem to be intuitive readings of the standard include:

a. when CLOSE runs to completion, the stream is closed afterward, and
b. when a stream is closed, OPEN-STREAM-P returns false.
c. CLOSE does nothing on a closed stream.

Why these invariants? Well, I like to make sure streams get closed eventually, and in programs that create a dynamic number of streams, that's easier to do when CLOSE and OPEN-STREAM-P have those guarantees.

That said, I'd agree that if a distinguished stream is to serve as a global "null output" sink, closing that stream would likely cause trouble, so I can understand why you've made CLOSE a no-op for the singleton broadcast stream.

OTOH, it's already the case that there are a number of distinguished global streams that it's bad to close (the values of *stdin*, *stdout*, *stderr*, *tty*, for example; and if synonym streams get their own distinct open-ness as discussed in 1904257, then any of the standard I/O customization variables). If /that/ problem is worth solving for, ISTM an orthogonal mechanism equivalent to the following could be nice to have.

--
(defvar *never-close-streams* ())

(define-condition never-close-error (error)
  ((stream :initarg :stream :reader never-close-error-stream))
  (:report (lambda (error stream)
             (format stream "~S should not normally be closed."
                     (never-close-error-stream error)))))

(defmethod close :before ((stream stream))
(when (member stream *never-close-streams*)
(cerror "Continue" 'never-close-error :stream stream)))

;; User interface.
(defun never-close (stream)
"Make it an error to try to close STREAM. Note that this does not persist across
SAVE-LISP-AND-DIE; use an *INIT-HOOK* function to reinstate."
(pushnew stream *never-close-streams*))

;; Then stream-init can put *stdin*, *stdout*, *stderr*, *tty*, and
;; so forth onto *never-close-streams*, and stream-deinit could
;; reset *no-close-streams* to nil.
--

Untested, but if you're interested I can elaborate on this orthogonal mechanism. Let me know?

Oh, I hadn't noticed that SBCL has null-ary MAKE-BROADCAST-STREAM return a singleton. That accounts for the CLOSE behavior.

However, while I see a benefit of having a distinguished "null output" sink, but does it need to be a singleton returned by the (apparent) constructor? A variable in SB-EXT would suffice, and would be portably emulatable on other implementations:

--
(defpackage "MY-PACKAGE"
  #+sbcl
  (import-from "SB-EXT" "*NULL-OUTPUT*") ;say
  ...)

(in-package "MY-PACKAGE")

#-sbcl ; SBCL already has this
(defvar *null-output* (make-broadcast-stream))
--

Anyway, the reason I care is that among some interface invariants that are desirable to have and seem to be intuitive readings of the standard include:

a. when CLOSE runs to completion, the stream is closed afterward, and
b. when a stream is closed, OPEN-STREAM-P returns false.
c. CLOSE does nothing on a closed stream.

Why these invariants? Well, I like to make sure streams get closed eventually, and in programs that create a dynamic number of streams, that's easier to do when CLOSE and OPEN-STREAM-P have those guarantees.

That said, I'd agree that if a distinguished stream is to serve as a global "null output" sink, closing that stream would likely cause trouble, so I can understand why you've made CLOSE a no-op for the singleton broadcast stream.

OTOH, it's already the case that there are a number of distinguished global streams that it's bad to close (the values of *stdin*, *stdout*, *stderr*, *tty*, for example; and if synonym streams get their own distinct open-ness as discussed in 1904257, then any of the standard I/O customization variables). If /that/ problem is worth solving for, ISTM an orthogonal mechanism equivalent to the following could be nice to have.

--
(defvar *never-close-streams* ())

(define-condition never-close-error (error)
  ((stream :initarg :stream :reader never-close-error-stream))
  (:report (lambda (error stream)
             (format stream "~S should not normally be closed."
                     (never-close-error-stream error)))))

(defmethod close :before ((stream stream))
  (when (member stream *never-close-streams*)
    (cerror "Continue" 'never-close-error :stream stream)))

;; User interface.
(defun never-close (stream)
  "Make it an error to try to close STREAM. Note that this does not persist across 
SAVE-LISP-AND-DIE; use an *INIT-HOOK* function to reinstate."
  (pushnew stream *never-close-streams*))

;; Then stream-init can put *stdin*, *stdout*, *stderr*, *tty*, and
;; so forth onto *never-close-streams*, and stream-deinit could
;; reset *no-close-streams* to nil.
--

Untested, but if you're interested I can elaborate on this orthogonal mechanism. Let me know?

Revision history for this message

Tomas Hlavaty (tomas-hlavaty) wrote on 2020-11-19: Re: [Bug 1904722] Re: Broadcast streams with no components can't be closed

#3

> so I can understand why you've made CLOSE a no-op for the singleton
> broadcast stream.
>
> OTOH, it's already the case that there are a number of distinguished
> global streams that it's bad to close (the values of *stdin*, *stdout*,
> *stderr*, *tty*, for example; and if synonym streams get their own
> distinct open-ness as discussed in 1904257, then any of the standard I/O
> customization variables). If /that/ problem is worth solving for, ISTM
> an orthogonal mechanism equivalent to the following could be nice to
> have.

CLOSE makes sense only to dispose foreign file descriptors or handles
early.

It does not make sense for pure lisp streams. Especially if they are
layered on top of other streams.

Revision history for this message

Richard M Kreuter (kreuter) wrote on 2020-11-20:

#4

Sure, it's true that on SBCL composite streams don't hold onto OS resources that need releasing. But that's orthogonal to whether programs ought to close them.

That composite streams are pure Lisp streams on SBCL is an implementation detail, not part of their standardized interface. For example, what if another implementation, or a future version of SBCL, does a broadcast stream as a wrapper around as a pipe, the other end of which is handled by a different thread? Or as a wrapper around a handle to a foreign structure created and manipulated only through an FFI? Given the permissible variations of implementation strategies, a program that doesn't close its composite streams isn't portable.

Further, 21.1.4 says that closing a component of a composite stream while the stream is open has undefined consequences. So a program that doen't close its composite streams but does eventually close underlying component streams is not conforming.

Not all programs need to be conforming or portable, but IMO SBCL ought to support programs that try to be. Conforming or portable programs are supposed to close streams when they're done using them, and so closing streams ought to behave as standardized.

Revision history for this message

Tomas Hlavaty (tomas-hlavaty) wrote on 2020-11-20:

#5

how can you close underlying component streams of Broadcast streams with
no components?

why do you think that Conforming or portable programs are supposed to
close streams when they're done using them?

Revision history for this message

Richard M Kreuter (kreuter) wrote on 2020-11-26:

#6

Download full text (6.8 KiB)

Thank you for requesting a clarification.

> how can you close underlying component streams of Broadcast streams
> with no components?

My apologies; I wasn't very clear. I was responding to your proposition that CLOSE only makes sense to dispose of file descriptors or handles, by pointing out that a program should also close any composite stream having a component that the program also wants to close (in order for the program to satisfy 21.1.4).

You're correct that there are a couple of cases where a standard composite stream can have zero components: an empty broadcast stream, and also a concatenated stream that has reached EOF on all of its original components. A program that omits closing those streams won't violate 21.1.4, but is not portable in the ANSI sense (explained below).

Either way, if a program has a collection of streams to deal with, is it supposed to check every stream for type and composition before deciding whether to close it? Or is a program allowed to depend on the standard's requirement that

(progn (close stream) (open-stream-p stream))

closes the stream and returns NIL? (Well, if the form returns, anyway; CLOSE can error.)

Although it is unusual to check whether a stream is still open after closing it, the program I'm working on does so (that's what got me filing 1904722 and then this issue). Here's why: the program is an interpreter that creates an unpredictable number of streams of different types, and can't manage them all using WITH- style forms. So the program does its own bookkeeping, and is supposed to eventually close every stream it creates. But I found a leak, so I added a sanity check to the end of the program:

(assert (notany #'open-stream-p *streams*) (*streams*))

That is, the program does a CLOSE and then an OPEN-STREAM-P on every stream it creates, but at different locations in code. And this gets tripped up by the existence of streams that cannot be closed. I can work around it by filtering out empty broadcast streams on SBCL, but that workaround embeds the assumption that today's observable non-conformance won't go away in the future. If SBCL becomes conformant in this detail in the future, such filtering will introduce a false negative into my checking.

> why do you think that Conforming or portable programs are supposed
> to close streams when they're done using them?

Fair questions. The following is rather pedantic, but you asked. :-)

First I'll describe portability. Here's the "existence proof" that certain kinds of streams ought to get closed in a portable program. (I doubt this is controversial, but I want to make it explicit.)

(1) Some open streams can contain resources that need management to avoid risk of starvation (file descriptors/handles, and maybe also locks, buffers, cursors, and so forth).

(2) Next, some open streams have close-time side effects on the state of the external environment (e.g., deleting or renaming files; or maybe committing or rolling back a database transaction).

(3) Finally, to my knowledge, the standard is silent about whether implementations implicitly close streams that programs don't take measures to close themselves.

Consequently, if a program create...

Thank you for requesting a clarification.

> how can you close underlying component streams of Broadcast streams
> with no components?

My apologies; I wasn't very clear. I was responding to your proposition that CLOSE only makes sense to dispose of file descriptors or handles, by pointing out that a program should also close any composite stream having a component that the program also wants to close (in order for the program to satisfy 21.1.4).

You're correct that there are a couple of cases where a standard composite stream can have zero components: an empty broadcast stream, and also a concatenated stream that has reached EOF on all of its original components. A program that omits closing those streams won't violate 21.1.4, but is not portable in the ANSI sense (explained below).

Either way, if a program has a collection of streams to deal with, is it supposed to check every stream for type and composition before deciding whether to close it? Or is a program allowed to depend on the standard's requirement that

(progn (close stream) (open-stream-p stream))

closes the stream and returns NIL? (Well, if the form returns, anyway; CLOSE can error.)

Although it is unusual to check whether a stream is still open after closing it, the program I'm working on does so (that's what got me filing 1904722 and then this issue). Here's why: the program is an interpreter that creates an unpredictable number of streams of different types, and can't manage them all using WITH- style forms. So the program does its own bookkeeping, and is supposed to eventually close every stream it creates. But I found a leak, so I added a sanity check to the end of the program:

(assert (notany #'open-stream-p *streams*) (*streams*))

That is, the program does a CLOSE and then an OPEN-STREAM-P on every stream it creates, but at different locations in code. And this gets tripped up by the existence of streams that cannot be closed. I can work around it by filtering out empty broadcast streams on SBCL, but that workaround embeds the assumption that today's observable non-conformance won't go away in the future. If SBCL becomes conformant in this detail in the future, such filtering will introduce a false negative into my checking.

> why do you think that Conforming or portable programs are supposed
> to close streams when they're done using them?

Fair questions. The following is rather pedantic, but you asked. :-)

First I'll describe portability. Here's the "existence proof" that certain kinds of streams ought to get closed in a portable program. (I doubt this is controversial, but I want to make it explicit.)

(1) Some open streams can contain resources that need management to avoid risk of starvation (file descriptors/handles, and maybe also locks, buffers, cursors, and so forth).

(2) Next, some open streams have close-time side effects on the state of the external environment (e.g., deleting or renaming files; or maybe committing or rolling back a database transaction).

(3) Finally, to my knowledge, the standard is silent about whether implementations implicitly close streams that programs don't take measures to close themselves.

Consequently, if a program creates and does not close streams of the kinds that (1) and (2) apply to, and if, per (3), an implementation doesn't implicitly close streams, then that program may work on some implementations or environments but not others. For example, the program might run out of resources and be unable to proceed (because of 1), or the program might complete but leave the external environment in an arbitrary state (because of 2). Such variation disqualifies the program for portability in the ANSI sense, i.e., "required to produce equivalent results and observable side effects in all conforming implementations."

So at least some streams ought to get closed in a portable program.

What about streams that do not need to get closed? Firstly, there's no guarantee that there are streams that do not need to get closed on any particular implementation: any stream could hold onto resources that need to get released for a program to avoid starvation.

Next, even on implementations where some streams do not need to get closed, there's no way for a program to discover whether any particular stream is an ought-to-get-closed or a does-not-need-to-get-closed stream. Although a program can be conditionalized to embed that fact about an implementation, e.g.,

(cond ((and (equal "SBCL" (lisp-implementation-type))
            (typep stream '(not file-stream))) ;ignoring Gray Streams, say.
       ;; don't bother closing
       )
      ...
      ;; When in doubt, close the stream
      (t (close stream :abort abort))) ;assume a lexical ABORT

this approach reduces portability in the ANSI sense, because a conforming implementation might change its approach over time. For example, a future SBCL could change so that composite streams hold onto resources that need to be explicitly released, turning them into ought-to-get-closed streams. If that happens, then the above conditional will leak an ought-to-get-closed stream, reducing the program's predictability. And even if SBCL offered an extension to test for ought-to-get-closed-ness, that extension itself could get deprecated and removed over time, and so we would return to the same reasoning about predictable results over time. IOW, it's impossible for that conditional to be written correctly across all past and future versions of any implementation (at least for those of us who are neither omniscient nor time travellers).

All of that is to say that a program will not have predictable results and side-effects if it treats any streams it creates as do-not-need-to-get-closed streams, even if it does so only in an implementation-dependent conditional branch. So a portable program must treat all streams it creates as ought-to-get-closed streams.

As for a program's conformance, this is a different topic. I overstated things in my last comment: I suppose it turns out that a program that never closes any stream it creates is conforming (though not portable). However, as I mentioned above, a program that ever closes one stream S is not conforming if stream S is a component of any composite stream C due to 21.1.4. However, I think that a program that treats all streams it creates as ought-to-be-closed streams, and that them in an appropriate order will never fail to be conforming in this regard.

In summary, even if there are some do-not-need-to-get-closed streams on one implementation, because their equivalents may be ought-to-be-closed on other implementations (or the same implementation next year), a portable program must treat all streams it creates as ought-to-be-closed streams and make sure to close them. Further, a portable *and* conforming program must close the streams it creates in a suitable order (LIFO will do, but other orderings can too).

Thanks again for your time.

Revision history for this message

Richard M Kreuter (kreuter) wrote on 2020-11-26:

#7

Oh well, there's an error in my last comment: LIFO order closing won't work for every program. For example:

(defvar *a-stream*)
(defvar *streams* ())
(push (open "foo") *streams*)
(setq *a-stream* (first *streams*))
(push (make-synonym-stream '*a-stream*) *streams*)
(push (open "bar") *streams*)
(setq *a-stream* (first *streams*))

At this point, *STREAMS* contains 3 streams in most-recently-created-first order, but closing them in that order will violate 21.1.4. Nonetheless, if a program wants to close its streams, it ought to do so in a correct order; that ordering will be program-dependent.

Revision history for this message

Douglas Katzman (dougk) wrote on 2020-11-29:

#8

Doesn't seem worth debating any more, so I fixed it - https://sourceforge.net/p/sbcl/sbcl/ci/5d2b2424cd9799f8153a5f238740e3b672a3f54f/

It's curious that the spec lists no exceptional situations for WRITE-CHAR, WRITE-STRING, etc.
It should have at least said that an error must be signaled on a closed stream if it meant that.

Douglas Katzman (dougk) on 2020-11-29

Changed in sbcl:
status:	New → Fix Committed

Revision history for this message

Richard M Kreuter (kreuter) wrote on 2020-11-29:

#9

Thank you.

I believe the governing passage for closed streams in is 21.1.1.1.2: "Except as explicitly specified otherwise, the consequences are undefined when a closed stream is used where a stream is called for."

Christophe Rhodes (csr21-cantab) on 2020-12-29

Changed in sbcl:
status:	Fix Committed → Fix Released

SBCL

Broadcast streams with no components can't be closed

Bug Description

Other bug subscribers

Remote bug watches