upgrading k8s model from 2.5.7 to 2.6.2 fails

Bug #1830128 reported by Paul Collins on 2019-05-22
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju
High
Yang Kelvin Liu
2.6
High
Yang Kelvin Liu

Bug Description

I just upgraded a k8s model (microk8s 1.14.1) from 2.5.7 to 2.6.2. "juju status" says the model is at 2.6.2, but mariadb-k8s/0 has disappeared:

Model Controller Cloud/Region Version SLA Timestamp
test lxd-controller microk8s-cluster 2.6.2 unsupported 11:36:02+12:00

App Version Status Scale Charm Store Rev OS Address Notes
mariadb-k8s active 0/1 mariadb-k8s jujucharms 1 kubernetes 10.152.183.107

In k8s, things look reasonable; both pods are still there:

[agnew(~)] microk8s.kubectl get all -n test
NAME READY STATUS RESTARTS AGE
pod/mariadb-k8s-0 1/1 Running 0 33m
pod/mariadb-k8s-operator-0 1/1 Running 0 10m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mariadb-k8s ClusterIP 10.152.183.107 <none> 3306/TCP 33m

NAME READY AGE
statefulset.apps/mariadb-k8s 1/1 33m
statefulset.apps/mariadb-k8s-operator 1/1 33m
[agnew(~)]

When I examine the operator's statefulset and pod, they're still on 2.5.7:

[agnew(~)] microk8s.kubectl get statefulset.apps/mariadb-k8s-operator -n test -o yaml | grep image:
        image: jujusolutions/jujud-operator:2.5.7
[agnew(~)] microk8s.kubectl get pod mariadb-k8s-operator-0 -n test -o yaml| grep image:
    image: jujusolutions/jujud-operator:2.5.7
    image: docker.io/jujusolutions/jujud-operator:2.5.7
[agnew(~)]

and the following is logged in machine-0.log on the controller:

2019-05-22 23:25:40 ERROR juju.workers.caasunitprovisioner deployment_worker.go:175 creating or updating StatefulSet: StatefulSet.apps "mariadb-k8s" is invalid: [spec.template.metadata.labels: Invalid value: map[string]string{"juju-application":"mariadb-k8s", "juju-controller-uuid":"541fad46-9ad3-46ad-8841-95397427f197", "juju-model-uuid":"0d2efdac-991a-46d1-89d8-1ce5d0474fb2"}: `selector` does not match template `labels`, spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden]
2019-05-22 23:25:42 WARNING juju.worker.storageprovisioner filesystem_events.go:80 unexpected dead filesystem attachments: [{unit-mariadb-k8s-0 filesystem-0}]
2019-05-22 23:25:42 WARNING juju.worker.storageprovisioner volume_events.go:185 unexpected dead volume attachments: [{unit-mariadb-k8s-0 volume-0}]
(END)

I tried manually changing the operator statefulset to use 2.6.2 and the upgrade seemed to complete successfully, although mariadb-k8s/0 is still MIA.

Looking at machine-0.log again, I notice that the above failed update was to the app's stateful set, not the operator's.

Perhaps whatever change failed was something to do with how the operator tracks its pods, which then blocked the rest of the upgrade.

Tags: k8s Edit Tag help
Ian Booth (wallyworld) on 2019-05-23
tags: added: k8s
Changed in juju:
status: New → Triaged
status: Triaged → In Progress
assignee: nobody → Yang Kelvin Liu (kelvin.liu)
Changed in juju:
milestone: none → 2.6.4
Anastasia (anastasia-macmood) wrote :

Bigger merge PR (https://github.com/juju/juju/pull/10326) brought this change into develop.

Changed in juju:
status: In Progress → Fix Committed
importance: Undecided → High
milestone: 2.6.4 → 2.7-beta1
Yang Kelvin Liu (kelvin.liu) wrote :

2.6 is the first version supports caas model upgrade.
So 2.5 k8s controller can be upgraded to 2.6,
but 2.5 caas model will not be able to be upgraded to 2.6 correctly(because 2.5 operators do not have the knowledge of how to do upgrade).
What we can do is, kubectl delete all operator pods of that model after run `juju upgrade-model`.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers