Streaming API handlers may panic when apiserver shuts down

[Menno Finlay-Smits]

The apiserver tracks the number of active API requests (using a WaitGroup) and when shutting down waits for them to finish before stopping. This WaitGroup is incremented only for Juju normal RPC API but not for non-standard ("streaming") API requests. This means that the goroutines servicing such requests can still be active when the apiserver has closed it *state.State, causing "Session already closed" panics.

This issue has been resolved for the /logsink and /log API handlers (https://github.com/juju/juju/pull/4529) but still exists for the API handlers for tools, charms, backups etc. In order to contribute the apiserver's request WaitGroup they MUST honour the apiserver's tomb, or else they risk blocking the apiserver's shut down. Many of these handlers will not be easy to fix for this reason.

It should be noted that panics in API handlers aren't fatal as the server infrastructure catches the panic and just emits a stack trace. They are however alarming to users and developers.

[Ian Booth (wallyworld)]

Can we just emit the stacktrace if DEBUG or even TRACE level logging is activated? My preference is for TRACE.

[Menno Finlay-Smits (menno.smits)]

Modifying how the stack trace is emitted should be possible. Something inside Juju's API server will have to catch the panic before it bubbles up to the handler within net/http.