k8s: cannot successfully deploy unit after upgrade

[Heather Lanigan]

After upgrading a microk8s controller with an existing workload from juju 2.7 to 2.8-beta1, I am unable to successfully deploy units, agent stuck in allocating. The controller has stopped logging as well.

[Heather Lanigan (hmlanigan)]

hit enter too soon. will update with more data shortly.

[Heather Lanigan (hmlanigan)]

$ juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
controller upgrade microk8s/localhost 2.8-beta1.3572 unsupported 17:58:56Z upgraded on "2020-04-23T17:56:51Z"

App Version Status Scale Charm Store Rev OS Address Notes
mariadb-k8s active 1 mariadb-k8s jujucharms 2 kubernetes 10.152.183.11
ubuntu waiting 0/1 ubuntu-plus-k8s local 1 kubernetes agent initializing
ubuntu-plus-k8s active 1 ubuntu-plus-k8s local 0 kubernetes 10.152.183.106

Unit Workload Agent Address Ports Message
mariadb-k8s/0 active idle 10.1.59.38 3306/TCP
ubuntu-plus-k8s/0 waiting idle 10.1.59.36 3306/TCP Hello from start, it is 23/04/2020 17:54:48
ubuntu/0 waiting allocating agent initializing

$ microk8s.kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
controller-upgrade pod/controller-0 2/2 Running 0 10m
controller-upgrade pod/mariadb-k8s-0 1/1 Running 0 12m
controller-upgrade pod/mariadb-k8s-operator-0 1/1 Running 0 10m
controller-upgrade pod/ubuntu-plus-k8s-779d566b87-9mxwh 1/1 Running 0 12m
controller-upgrade pod/ubuntu-plus-k8s-operator-0 1/1 Running 0 10m
kube-system pod/coredns-588fd544bf-2jtwx 1/1 Running 0 100m
kube-system pod/hostpath-provisioner-75fdc8fccd-d8gxl 1/1 Running 0 100m

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
controller-upgrade service/controller-service ClusterIP 10.152.183.67 <none> 17070/TCP 19m
controller-upgrade service/mariadb-k8s ClusterIP 10.152.183.11 <none> 3306/TCP 12m
controller-upgrade service/mariadb-k8s-endpoints ClusterIP None <none> <none> 12m
controller-upgrade service/mariadb-k8s-operator ClusterIP 10.152.183.181 <none> 30666/TCP 12m
controller-upgrade service/ubuntu-plus-k8s ClusterIP 10.152.183.106 <none> 3306/TCP 12m
controller-upgrade service/ubuntu-plus-k8s-operator ClusterIP 10.152.183.207 <none> 30666/TCP 13m
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 102m
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 100m

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
controller-upgrade deployment.apps/ubuntu-plus-k8s 1/1 1 1 12m
kube-system deployment.a...

[Heather Lanigan (hmlanigan)]

https://people.canonical.com/~heather/upgrade-debug-log.gz

[Yang Kelvin Liu (kelvin.liu)]

https://github.com/juju/juju/pull/11544 will be landed to fix a few issues related model upgrade to 2.8.

But there is still one issue related container init process: https://bugs.launchpad.net/juju/+bug/1877515