upgrade k8s charm results in duplicate units
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Canonical Juju |
Fix Released
|
Critical
|
Ian Booth | ||
Bug Description
deploy a stateful k8s charm with 3 units
upgrade the charm such that a new podspec is generated
k8s will progressively terminate and restart each pod
as each pod is restarted, juju will create a new unit instead of associating the restarted pod with its existing unit
looking at the db, the cloudContainer records for each unit correctly record the provider-id, but these are now duplicated, eg there are 2 records with provider-id "mariadb-k8s-1".
juju is supposed to use the provider-id to associate the pod with the unit
Also, 2 of the 3 charm upgrade-hooks errored because only the leader can pod-spec-set, yet the Python code did do a is_leader() check, so there's an issue there.
And looking at pod-0, it has the agent for unit-5, which is one of the duplicate units Juju created.
| description: | updated |
| Ian Booth (wallyworld) wrote | #1 |
| Changed in juju: | |
| assignee: | nobody → Ian Booth (wallyworld) |
| status: | Triaged → In Progress |
| Changed in juju: | |
| milestone: | 2.8-beta1 → 2.7.6 |
| Changed in juju: | |
| status: | In Progress → Fix Committed |
| Changed in juju: | |
| status: | Fix Committed → Fix Released |
It looks like the issue is at least partly caused by the fact that when the pod status is reported back to the controller, the fact that the pods belong to a stateful set is not being set as true. This causes juju to treat the units as stateless and hence delete them as soon as a pod terminates since for stateless deployments, a new pod is created rather than reusing the existing name.