csi-cinder-controllerplugin CrashLoopBackOff

[Nobuto Murata]

$ juju version
3.3.1-genericlinux-amd64

charmed k8s v1.28.5

The exported bundle will be attached and it's based on
https://ubuntu.com/kubernetes/docs/openstack-integration

After the deployment is completed, csi-cinder-controllerplugin is stuck at CrashLoopBackOff.

$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx-kubernetes-worker default-http-backend-kubernetes-worker-5c79cc75ff-fm4fl 1/1 Running 0 77m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-z6dvp 1/1 Running 1 (75m ago) 76m
kube-system calico-kube-controllers-859bcd68b5-zmn8n 1/1 Running 0 78m
kube-system calico-node-9wp8g 1/1 Running 0 78m
kube-system calico-node-b9gn4 1/1 Running 0 77m
kube-system coredns-59cfb5bf46-mlfcb 1/1 Running 0 79m
kube-system csi-cinder-controllerplugin-684cfb8c48-st68p 5/6 CrashLoopBackOff 22 (3m53s ago) 79m
kube-system csi-cinder-nodeplugin-6vnqv 3/3 Running 0 77m
kube-system csi-cinder-nodeplugin-b2kpz 3/3 Running 0 78m
kube-system kube-state-metrics-78c475f58b-mzxcz 1/1 Running 2 (75m ago) 79m
kube-system metrics-server-v0.6.3-69d7fbfdf8-dvp2f 2/2 Running 0 79m
kube-system openstack-cloud-controller-manager-bhk8z 1/1 Running 3 (75m ago) 77m
kubernetes-dashboard dashboard-metrics-scraper-5dd7cb5fc-9fb9g 1/1 Running 0 79m
kubernetes-dashboard kubernetes-dashboard-7b899cb9d9-xdlhz 1/1 Running 1 (75m ago) 79m

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning Unhealthy 45m (x3 over 70m) kubelet Liveness probe failed: Get "http://192.168.192.198:9808/healthz": dial tcp 192.168.192.198:9808: connect: connection refused
  Warning BackOff 5m52s (x265 over 70m) kubelet Back-off restarting failed container cinder-csi-plugin in pod csi-cinder-controllerplugin-684cfb8c48-st68p_kube-system(1208b47e-f023-4fe6-94b3-cd9bc8afee78)
  Normal Pulled 52s (x18 over 71m) kubelet Container image "rocks.canonical.com:443/cdk/provider-os/cinder-csi-plugin:v1.28.1" already present on machine

[Kevin W Monroe (kwmonroe)]

Thanks for the report. A few things may be going on here:

First, from you bundle, it looks like the machine constraints are quite low (2G mem, 16G disk). I know that's the default for k8s-core; we're addressing those in lp:2053058. I'm concerned that disk/oom issues may be manifesting as failing pods.

Second, there has been quite a bit of o7k integration refactoring in ck8s 1.29 (charms are out; docs are pending publish):

https://github.com/charmed-kubernetes/kubernetes-docs/blob/main/pages/k8s/openstack-integration.md

Third, i see you have the o7k-integrator co-located with kubernetes-control-plane. This isn't a typical supported env and i fear network stack conflicts.

Please let us know if this resolves itself with either:

1) larger machines
2) upgraded openstack integrator charms
3) modified topology to put the integrator on a separate machine

I'm going to set this bug to incomplete and targeting 1.29+ck1 for now. It would be great if you could attach a juju crashdump if you're still able to repro:

https://ubuntu.com/kubernetes/docs/troubleshooting#collecting-debug-information

Thanks!