Inconsistent events on pod restart
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Invalid
|
Undecided
|
Unassigned |
Bug Description
If a pod is deleted with `kubectl delete pod`, these events are fired: stop -> upgrade-charm -> config-changed -> start (consistent with https:/
If a pod is deleted with `sudo microk8s stop` and `sudo microk8s start`, these events are fired: start
Expected behavior:
`upgrade-charm` is not fired when a pod is deleted with `kubectl delete pod` (or `upgrade-charm` is fired after `sudo microk8s stop` and `sudo microk8s start)
Steps to reproduce:
1. juju add-model foo
2. juju deploy mysql-router-k8s --channel 8.0/edge
3. Wait for idle
4. kubectl -n foo delete pod mysql-router-k8s-0
5. Wait for idle
6. sudo microk8s stop
7. sudo microk8s start
8. Wait for idle
9. Check juju debug-log --replay or jhack tail --replay
Versions:
MicroK8s v1.26.4 revision 5219 from snap channel 1.26/stable
Juju 2.9.43-ubuntu-amd64 from snap channel 2.9/edge
Also reproduced on MicroK8s v1.26.4 revision 5222 from snap channel 1.26-strict/stable and Juju 3.2-beta3-
Additional context:
In both pod deletion scenarios, the debug log shows `INFO juju.worker.uniter reboot detected; triggering implicit start hook to notify charm`
Initially discovered in mysql-router-k8s bug report: https:/
Pod churn is different from cluster start / stop. In the latter case, there is no pod churn, juju is killed (can't run the stop hook), and just running the start hook is reasonable when the cluster comes up again - the same pod as before is just being started up. Whereas for pod churn, juju gets a chance to see the pod is deleted and hence can run the stop hook, and then when a replacement pod comes up, it's treated as new and hence the standard new agent hooks run.
Charm hooks are supposed to be idempotent - the same hook can be run multiple times and the charm should do the right thing. The linked issue seems to imply that start hook running twice starts a second instance. This is a charm hook implementation issue.