k8s charm is removed automatically after errors in pod

[Camille Rodriguez]

I'm currently testing a new charm with the sidecar operator framework. It happened twice that the charm pod would go in error for a few minutes before being able to start correctly, which causes the charm to be removed from the juju status. See details below

At first, I see the charm deploying in the juju status

I notice the pod being in crashbackloop. Here are the logs gathered:
$ m logs metallb-controller-0 -n metallb charm
starting containeragent unit command
containeragent unit "unit-metallb-controller-0" start (2.9.4 [gc])
ERROR "api-caller" manifold worker returned unexpected error: [790fe0] "unit-metallb-controller-0" cannot open api: invalid entity name or password (unauthorized access)
ERROR "api-caller" manifold worker returned unexpected error: [790fe0] "unit-metallb-controller-0" cannot open api: invalid entity name or password (unauthorized access)

Then, it stabilizes and become running/active. However, on the juju side, what I see is this :

Model Controller Cloud/Region Version SLA Timestamp
metallb k8s microk8s/localhost 2.9.4 unsupported 10:28:06-04:00

App Version Status Scale Charm Store Channel Rev OS Address Message
metallb-controller unknown 0/1 metallb-controller local 5 kubernetes

In microk8s :
microk8s.kubectl get all -n metallb camille-ubuntu: Wed Jun 16 10:28:30 2021

NAME READY STATUS RESTARTS AGE
pod/modeloperator-7df7bb7f68-nqsvk 1/1 Running 0 41m
pod/metallb-controller-0 2/2 Running 4 8m34s

In the juju debug-logs

controller-0: 10:19:43 INFO juju.worker.caasapplicationprovisioner.runner start "metallb-controller"
controller-0: 10:19:43 INFO juju.worker.caasapplicationprovisioner.runner stopped "metallb-controller", err: restart immediately
controller-0: 10:19:43 ERROR juju.worker.caasapplicationprovisioner.runner exited "metallb-controller": restart immediately
controller-0: 10:19:43 INFO juju.worker.caasapplicationprovisioner.runner restarting "metallb-controller" in 3s
controller-0: 10:19:46 INFO juju.worker.caasapplicationprovisioner.runner start "metallb-controller"
controller-0: 10:19:46 INFO juju.worker.caasapplicationprovisioner.runner stopped "metallb-controller", err: restart immediately
controller-0: 10:19:46 ERROR juju.worker.caasapplicationprovisioner.runner exited "metallb-controller": restart immediately
controller-0: 10:19:46 INFO juju.worker.caasapplicationprovisioner.runner restarting "metallb-controller" in 3s
controller-0: 10:19:49 INFO juju.worker.caasapplicationprovisioner.runner start "metallb-controller"
controller-0: 10:19:49 INFO juju.worker.caasapplicationprovisioner.runner stopped "metallb-controller", err: restart immediately
controller-0: 10:19:49 ERROR juju.worker.caasapplicationprovisioner.runner exited "metallb-controller": restart immediately
controller-0: 10:19:49 INFO juju.worker.caasapplicationprovisioner.runner restarting "metallb-controller" in 3s
controller-0: 10:19:52 INFO juju.worker.caasapplicationprovisioner.runner start "metallb-controller"
controller-0: 10:19:52 INFO juju.worker.caasapplicationprovisioner.runner stopped "metallb-controller", err: restart immediately
controller-0: 10:19:52 ERROR juju.worker.caasapplicationprovisioner.runner exited "metallb-controller": restart immediately
controller-0: 10:19:52 INFO juju.worker.caasapplicationprovisioner.runner restarting "metallb-controller" in 3s
controller-0: 10:19:55 INFO juju.worker.caasapplicationprovisioner.runner start "metallb-controller"
unit-metallb-controller-0: 10:19:58 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version: cannot set agent version for unit "metallb-co
ntroller/0": not found or dead
unit-metallb-controller-0: 10:19:58 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PAT
H, output: ""
unit-metallb-controller-0: 10:19:58 INFO juju.worker.uniter unit "metallb-controller/0" shutting down: agent should be terminated
unit-metallb-controller-0: 10:19:58 INFO juju.worker.leadership metallb-controller/0 promoted to leadership of metallb-controller
unit-metallb-controller-0: 10:19:59 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version: cannot set agent version for unit "metallb-co
ntroller/0": not found or dead
unit-metallb-controller-0: 10:19:59 INFO juju.worker.leadership metallb-controller/0 promoted to leadership of metallb-controller
unit-metallb-controller-0: 10:19:59 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PAT
H, output: ""
unit-metallb-controller-0: 10:19:59 INFO juju.worker.uniter unit "metallb-controller/0" shutting down: agent should be terminated

unit-metallb-controller-0: 10:20:18 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version: cannot set agent version for unit "metallb-co
ntroller/0": not found or dead
unit-metallb-controller-0: 10:20:18 INFO juju.worker.leadership metallb-controller/0 promoted to leadership of metallb-controller
unit-metallb-controller-0: 10:20:18 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PAT
H, output: ""
unit-metallb-controller-0: 10:20:18 INFO juju.worker.uniter unit "metallb-controller/0" shutting down: agent should be terminated
unit-metallb-controller-0: 10:20:39 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version: cannot set agent version for unit "metallb-co
ntroller/0": not found or dead
unit-metallb-controller-0: 10:20:39 INFO juju.worker.leadership metallb-controller/0 promoted to leadership of metallb-controller
unit-metallb-controller-0: 10:20:39 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PAT
H, output: ""
unit-metallb-controller-0: 10:20:39 INFO juju.worker.uniter unit "metallb-controller/0" shutting down: agent should be terminated

[Jon Seager (jnsgruk)]

Hey Camille

That's a pretty old version of Juju you're running there. Can you upgrade to a later version of 2.9 (i.e. sudo snap refresh juju --channel latest/stable) or even Juju 3.1? You'll need to re-bootstrap your microk8s after to get to the latest controller/agent version.

If you could also show where the charm code is, I can try to reproduce.

Cheers, Jon

[Jon Seager (jnsgruk)]

Oh hang on - I've just seen the timestamp on this bug. Launchpad convinced me this was a new bug! Please let me know if this is still an issue - my guess is that this is no longer valid! Will mark as incomplete.

[Camille Rodriguez (camille.rodriguez)]

Right this is fairly old - we can assume it's no longer an issue and I'll reopen if I ever face it again. Thanks!

[Launchpad Janitor (janitor)]

[Expired for Canonical Juju because there has been no activity for 60 days.]