Comment 22 for bug 2056193

Are you 100% sure comment #16 captures ALL of the logs after an agent restart?

eg I deployed postgresql and stopped the unit agent. I then ran an action which will stay pending since the agent isn't running to process it. I then restarted the agent and you can see there should be an initial message about starting the workers and then an initial message from all the watchers. Note the actions watcher which reports on the pending action. That's what we're looking for here. The entry in the actionsnotification collection should trigger an event from the actions watcher on agent startup, and we need to see what happens next.

We could set "juju.worker.uniter=TRACE" as well just to get everything.

unit-postgresql-0: 11:08:07 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 11:08:07 INFO juju.agent.setup setting logging config to "juju.worker.uniter=DEBUG"
unit-postgresql-0: 11:08:07 WARNING juju.agent.setup developer feature flags enabled: "actions-v2", "developer-mode", "image-metadata", "secrets", "secrets-stores"
unit-postgresql-0: 11:08:07 DEBUG juju.worker.uniter.relation unit "postgresql/0" (leader=true) entered scope for relation "postgresql:database-peers"
unit-postgresql-0: 11:08:07 DEBUG juju.worker.uniter.relation unit "postgresql/0" (leader=true) entered scope for relation "postgresql:restart"
unit-postgresql-0: 11:08:07 DEBUG juju.worker.uniter.relation unit "postgresql/0" (leader=true) entered scope for relation "postgresql:upgrade"
unit-postgresql-0: 11:08:07 DEBUG juju.worker.uniter starting local juju-exec listener on {unix /var/lib/juju/agents/unit-postgresql-0/run.socket <nil>}
unit-postgresql-0: 11:08:07 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 11:08:07 DEBUG juju.worker.uniter juju-exec listener running
unit-postgresql-0: 11:08:07 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate starting secrets rotation watcher
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate starting secret revisions expiry watcher
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate starting obsolete secret revisions watcher (leader=true)
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.secretsrotate got rotate secret changes: []watcher.SecretTriggerChange{}
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got config change for postgresql/0: ok=true, hashes=[426e5e77af179901fdcde601e0edf8d10a6586e6918ccfcbd7baca0361b786f7]
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got unit change for postgresql/0
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.secretrevisionsexpire got revision expiry secret changes: []watcher.SecretTriggerChange{}
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got upgrade series change
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate no upgrade series in progress, reinitializing local upgrade series state
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got leader settings change for postgresql/0: ok=true
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got application change for postgresql/0
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got obsolete secret revisions change for postgresql: []
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate obsolete secret revisions: map[]
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate deleted secrets: []
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got secrets change for postgresql/0: []
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got latest secret info: map[string]secrets.SecretRevisionInfo{}
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate deleted secrets: []
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate obsolete secrets: map[]
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got action change for postgresql/0: [4] ok=true
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got trust config change for postgresql/0: ok=true, hashes=[18cbe02213f17ac8ed6e4e9a5f7636aeb4767855001f64053217a2e5ee21ceb6]
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got address change for postgresql/0: ok=true, hashes=[fbe283e419f2712254935a0b0c635f40d916537bde7e4fb789323f7c41fa7085]
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got storage change for postgresql/0: [pgdata/0] ok=true
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got update status interval change for postgresql/0: ok=true
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.remotestate got relations change for postgresql/0: ok=true: ["postgresql:upgrade" "postgresql:database-peers" "postgresql:restart"]
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.operation running operation run action 4 for postgresql/0
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.operation acquiring machine lock for postgresql/0
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.operation preparing operation "run action 4" for postgresql/0
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.operation executing operation "run action 4" for postgresql/0
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running action get-password
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.runner running action "get-password" on 1
unit-postgresql-0: 11:08:08 DEBUG juju.worker.uniter.runner starting jujuc server {unix @/var/lib/juju/agents/unit-postgresql-0/agent.socket <nil>}
unit-postgresql-0: 11:08:09 DEBUG juju.worker.uniter.operation committing operation "run action 4" for postgresql/0
unit-postgresql-0: 11:08:09 DEBUG juju.worker.uniter.operation lock released for postgresql/0
unit-postgresql-0: 11:08:09 DEBUG juju.worker.uniter.storage next hook op for storage-pgdata-0: {Kind:2 Life:alive Attached:true Location:/var/snap/charmed-postgresql/common}
unit-postgresql-0: 11:08:09 DEBUG juju.worker.uniter no operations in progress; waiting for changes
unit-postgresql-0: 11:08:09 DEBUG juju.worker.uniter [AGENT-STATUS] idle:
unit-postgresql-0: 11:08:09 DEBUG juju.worker.uniter.remotestate storage attachment change for postgresql/0: {storage-pgdata-0 {2 alive true /var/snap/charmed-postgresql/common}}
unit-postgresql-0: 11:08:09 DEBUG juju.worker.uniter.storage next hook op for storage-pgdata-0: {Kind:2 Life:alive Attached:true Location:/var/snap/charmed-postgresql/common}
unit-postgresql-0: 11:08:09 DEBUG juju.worker.uniter no operations in progress; waiting for changes