Comment 45 for bug 2056193

Logs with "<root>=DEBUG;unit=DEBUG;juju.worker.uniter=TRACE;juju.worker.caas?=TRACE"
https://pastebin.canonical.com/p/WZdbkSS2PF/

One very interesting piece is that we do see:
unit-httprequest-lego-provider-0: 16:25:14 DEBUG juju.worker.uniter.remotestate got action change for httprequest-lego-provider/0: [4 6 24] ok=true
unit-httprequest-lego-provider-0: 16:35:48 DEBUG juju.worker.uniter.remotestate got action change for httprequest-lego-provider/0: [32] ok=true

I'm guessing the number (4 6 24) and (32) is the action tasks that it detected. But we are seeing that there is an action change. (we saw that log message show up after trying to run `juju exec ... -- hostname`)

We see a lot of DEBUG messages about the relation states, and what has already been seen (no relation-created to run, no -joined to run)

As Ian noted in his PR against 3.3, we also see
https://github.com/juju/juju/blob/f6a66aa91eec620f5ac04a19d8c06bef03ae6228/worker/uniter/actions/resolver.go#L59

That there are a couple of conditions `remoteState.ActionsBlocked` or `localState.OutdatedRemoteCharm`, which could cause us to not run any actions (but until his PR would also not generate any logging information for us to know that is happening).

The first condition might be set if:
        // ActionsBlocked is true on CAAS when actions cannot be run due to
        // pod initialization.
        ActionsBlocked bool

The only places that seem to set it are:
worker/uniter/remotestate/watcher.go
183: ActionsBlocked: config.ContainerRunningStatusChannel != nil,
1325: w.current.ActionsBlocked = !runningStatus.Running

The ContainerRunningStatusChannel seems to be about whether this is running locally or remotely. Eg validate considers it an error if:
        if w.ModelType == model.CAAS && !w.Sidecar {
                if w.ContainerRunningStatusChannel != nil &&
                        w.ContainerRunningStatusFunc == nil {
                        return errors.NotValidf("watcher config for CAAS model with nil container running status func")
                }
        }

However, IIRC httprequest-lego-provider *is* a sidecar charm. *And* this only triggers if you had the channel but no valid status func. So there is very little that says anything about when it is ok for it to be nil. (just one particular condition when if it isn't nil you can't have a nil status func.)

And caasoperator sets it when:

                                if op.deploymentMode != caas.ModeOperator {
                                        params.IsRemoteUnit = true
                                        params.ContainerRunningStatusChannel = unitRunningChannels[unitID]

That also doesn't do any checks that unitID is in the map, so it might be just returning nil there.