And here's logs from the first charm that I noticed this for:

$ kubectl logs --tail 1000 -l juju-operator=dex-auth
2020-04-28 13:28:28 INFO juju.cmd supercommand.go:91 running jujud [2.8-rc1 3584 da98e184dd907fe3263b7f098147cd99aba4c73c gc go1.14.2]
2020-04-28 13:28:28 DEBUG juju.cmd supercommand.go:92   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=dex-auth", "--debug"}
2020-04-28 13:28:28 DEBUG juju.agent agent.go:571 read agent config, format "2.0"
2020-04-28 13:28:28 INFO juju.worker.upgradesteps worker.go:70 upgrade steps for 2.8-rc1 have already been run.
2020-04-28 13:28:28 INFO juju.cmd.jujud caasoperator.go:200 caas operator application-dex-auth start (2.8-rc1 [gc])
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "agent" manifold worker started at 2020-04-28 13:28:28.429503399 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "upgrade-steps-gate" manifold worker started at 2020-04-28 13:28:28.429686675 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "clock" manifold worker started at 2020-04-28 13:28:28.429925853 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "api-config-watcher" manifold worker started at 2020-04-28 13:28:28.430132147 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.introspection socket.go:97 introspection worker listening on "@jujud-application-dex-auth"
2020-04-28 13:28:28 DEBUG juju.worker.introspection socket.go:127 stats worker now serving
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "upgrade-steps-flag" manifold worker started at 2020-04-28 13:28:28.439923237 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.apicaller connect.go:128 connecting with old password
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "migration-fortress" manifold worker started at 2020-04-28 13:28:28.451176989 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.api apiclient.go:1105 successfully dialed "wss://172.31.33.208:17070/model/81f1584d-a77f-4f30-8a12-417f4c2fcedd/api"
2020-04-28 13:28:28 INFO juju.api apiclient.go:637 connection established to "wss://172.31.33.208:17070/model/81f1584d-a77f-4f30-8a12-417f4c2fcedd/api"
2020-04-28 13:28:28 INFO juju.worker.apicaller connect.go:158 [81f158] "application-dex-auth" successfully connected to "172.31.33.208:17070"
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "api-caller" manifold worker started at 2020-04-28 13:28:28.655837161 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "migration-minion" manifold worker started at 2020-04-28 13:28:28.666100125 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "log-sender" manifold worker started at 2020-04-28 13:28:28.666290424 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "upgrader" manifold worker started at 2020-04-28 13:28:28.666330008 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "upgrade-steps-runner" manifold worker started at 2020-04-28 13:28:28.667080449 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:584 "upgrade-steps-runner" manifold worker completed successfully
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "migration-inactive-flag" manifold worker started at 2020-04-28 13:28:28.667489517 +0000 UTC
2020-04-28 13:28:28 INFO juju.worker.caasupgrader upgrader.go:112 abort check blocked until version event received
2020-04-28 13:28:28 DEBUG juju.worker.caasupgrader upgrader.go:127 current agent binary version: 2.8-rc1
2020-04-28 13:28:28 INFO juju.worker.caasupgrader upgrader.go:118 unblocking abort check
2020-04-28 13:28:28 INFO juju.worker.migrationminion worker.go:140 migration phase is now: NONE
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "charm-dir" manifold worker started at 2020-04-28 13:28:28.678593991 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "proxy-config-updater" manifold worker started at 2020-04-28 13:28:28.678678165 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.logger logger.go:64 initial log config: "<root>=DEBUG"
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "logging-config-updater" manifold worker started at 2020-04-28 13:28:28.67879184 +0000 UTC
2020-04-28 13:28:28 INFO juju.worker.logger logger.go:118 logger worker started
2020-04-28 13:28:28 DEBUG juju.worker.dependency engine.go:564 "api-address-updater" manifold worker started at 2020-04-28 13:28:28.678830063 +0000 UTC
2020-04-28 13:28:28 DEBUG juju.worker.caasupgrader upgrader.go:150 desired agent binary version: 2.8-rc1
2020-04-28 13:28:28 DEBUG juju.worker.logger logger.go:92 reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING"
2020-04-28 13:34:30 ERROR juju.worker.uniter.context context.go:753 "dex-auth/1" is not the leader but is setting application k8s spec
2020-04-28 13:34:30 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:34:30 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-1/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:34:30 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:34:36 ERROR juju.worker.uniter.context context.go:753 "dex-auth/1" is not the leader but is setting application k8s spec
2020-04-28 13:34:36 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:34:36 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-1/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:34:36 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:34:48 ERROR juju.worker.uniter.context context.go:753 "dex-auth/1" is not the leader but is setting application k8s spec
2020-04-28 13:34:48 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:34:48 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-1/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:34:48 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:34:54 WARNING juju.worker.uniter.operation leader.go:123 we should run a leader-deposed hook here, but we can't yet
2020-04-28 13:35:10 ERROR juju.worker.uniter.context context.go:753 "dex-auth/1" is not the leader but is setting application k8s spec
2020-04-28 13:35:10 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:35:10 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-1/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:35:10 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:36:15 ERROR juju.worker.uniter.context context.go:753 "dex-auth/2" is not the leader but is setting application k8s spec
2020-04-28 13:36:15 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:36:15 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-2/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:36:15 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:36:21 ERROR juju.worker.uniter.context context.go:753 "dex-auth/2" is not the leader but is setting application k8s spec
2020-04-28 13:36:21 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:36:21 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-2/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:36:22 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:36:33 ERROR juju.worker.uniter.context context.go:753 "dex-auth/2" is not the leader but is setting application k8s spec
2020-04-28 13:36:33 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:36:33 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-2/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:36:33 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:36:53 ERROR juju.worker.uniter.context context.go:753 "dex-auth/2" is not the leader but is setting application k8s spec
2020-04-28 13:36:53 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:36:53 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-2/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:36:54 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:37:34 ERROR juju.worker.uniter.context context.go:753 "dex-auth/2" is not the leader but is setting application k8s spec
2020-04-28 13:37:34 ERROR juju-log oidc-client:8: pod-spec-set encountered an error: `ERROR this unit is not the leader`
2020-04-28 13:37:34 ERROR juju-log oidc-client:8: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-dex-auth-2/charm/reactive/dex_auth.py", line 154, in start_charm
    for crd in yaml.safe_load_all(Path("resources/crds.yaml").read_text())
  File "lib/charms/layer/caas_base.py", line 34, in pod_spec_set
    run_hook_command("pod-spec-set", spec)
  File "lib/charms/layer/caas_base.py", line 13, in run_hook_command
    run([cmd], stdout=PIPE, stderr=PIPE, check=True, input=stdin.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

2020-04-28 13:37:34 ERROR juju.worker.uniter.operation runhook.go:136 hook "oidc-client-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2020-04-28 13:38:09 WARNING juju.worker.uniter.operation leader.go:123 we should run a leader-deposed hook here, but we can't yet