Continuous rotation of active unit in K8s charm
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Invalid
|
High
|
Thomas Miller |
Bug Description
I am deploying two charms, dex-auth and oidc-gatekeeper. The deployment works fine in Juju 2.7, and runs into a condition in Juju 2.8 where new units are rotated in, preventing the workload from actually running. Reproducible steps:
juju add-model kubeflow
juju deploy cs:~kubeflow-
juju deploy cs:~kubeflow-
juju relate dex-auth oidc-gatekeeper
juju config oidc-gatekeeper client-
juju config dex-auth static-
juju wait -wv
juju config dex-auth public-
juju config oidc-gatekeeper public-
The steps above work fine on Juju 2.8 until the last two, where `public-url` is configured for both charms. That's where Juju 2.8 starts constantly swapping in new units.
Changed in juju: | |
milestone: | none → 2.8.3 |
status: | New → Triaged |
importance: | Undecided → High |
Changed in juju: | |
milestone: | 2.8.3 → none |
Changed in juju: | |
assignee: | nobody → Thomas Miller (tlmiller) |
The k8s deployment controller for dex-auth keeps toggling between 0 and 1. For deployments, each time a new pod is created, that results in a new Juju unit, hence a new leader. So next step is to figure out what's causing the deployment to kill and restart pods.
In the Juju logs, there is a hook error logged twice
application- dex-auth: 11:58:50 ERROR unit.dex- auth/0. juju-log oidc-client:0: Hook error: reactive/ __init_ _.py", line 74, in main dispatch( restricted= restricted_ mode) reactive/ bus.py" , line 390, in dispatch other_handlers) reactive/ bus.py" , line 359, in _invoke invoke( ) reactive/ bus.py" , line 181, in invoke _action( *args) juju/agents/ unit-dex- auth-0/ charm/reactive/ dex_auth. py", line 60, in start_charm client_ info = endpoint_ from_name( 'oidc-client' ).get_config( ) juju/agents/ unit-dex- auth-0/ charm/hooks/ relations/ oidc-client/ requires. py", line 13, in get_config juju/agents/ unit-dex- auth-0/ charm/hooks/ relations/ oidc-client/ requires. py", line 14, in <listcomp> loads(unit. received_ raw["client_ info"]) python3. 8/json/ __init_ _.py", line 341, in loads
Traceback (most recent call last):
File "lib/charms/
bus.
File "lib/charms/
_invoke(
File "lib/charms/
handler.
File "lib/charms/
self.
File "/var/lib/
oidc_
File "/var/lib/
return [
File "/var/lib/
json.
File "/usr/lib/
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType