kube_ovn gets stuck waiting for 1 kube ovn pinger pods to start

[Amjad Chami]

Issue:

A kube-ovn-pinger pod gets stuck, I'm not sure where to look for more information as in pod logs only show:

E1216 23:48:49.988339 990621 config.go:160] failed to get self pod kube-system/kube-ovn-pinger-k6m99: Get "https://10.152.183.1:443/api/v1/namespaces/kube-system/pods/kube-ovn-pinger-k6m99?timeout=15s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1216 23:48:49.988568 990621 klog.go:10] "failed to parse config" err="Get \"https://10.152.183.1:443/api/v1/namespaces/kube-system/pods/kube-ovn-pinger-k6m99?timeout=15s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

and in the journal there's an error message where all attempts at restarting the pod fail with:

pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"pinger\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=pinger pod=kube-ovn-pinger-k6m99_kube-system(5dc7ebfc-3f9a-4ea1-bb2a-d5b4c46d6a0f)\"" pod="kube-system/kube-ovn-pinger-k6m99" podUID="5dc7ebfc-3f9a-4ea1-bb2a-d5b4c46d6a0f"

this bug might be related to: https://bugs.launchpad.net/charm-kube-ovn/+bug/1999443

details:
maas 3.3.5-
juju 3.1.7
ubuntu jammy
charmed-kubernetes 1.28

Kube-OVN:
  Version: v1.11.10
  Build: 2023-08-04_13:27:14
  Commit: git-a11d7e9
  Go Version: go1.19.12
  Arch: amd64

testrun: https://oil-jenkins.canonical.com/artifacts/b439852b-bcaa-4d2f-a5ea-71a63ec96c53/index.html

Crashdump: https://oil-jenkins.canonical.com/artifacts/b439852b-bcaa-4d2f-a5ea-71a63ec96c53/generated/generated/kubernetes-maas/juju-crashdump-kubernetes-maas-2023-12-16-23.46.22.tar.gz