kube-ovn stuck waiting "Waiting to retry configuring Kube-OVN"

[Jeffrey Chang]

The symptom is almost identical to LP#2007162, but since that's fix release in K8s 1.27, I decided to open a new bug.

In SQA run https://solutions.qa.canonical.com/testruns/1945e675-8e5e-48b7-abb9-d38daeb78dbd, which is charmed k8s 1.29 on baremetal maas.
All artifacts in https://oil-jenkins.canonical.com/artifacts/1945e675-8e5e-48b7-abb9-d38daeb78dbd/index.html

kubernetes-control-plane/2 active idle 5/kvm/0 10.246.200.157 6443/tcp
  ceph-csi/2 active idle 10.246.200.157 Unit is ready
  containerd/8 active idle 10.246.200.157 Container runtime available
  kube-ovn/8 waiting idle 10.246.200.157 Waiting to retry configuring Kube-OVN
  ntp/11 active idle 10.246.200.157 123/udp chrony: Ready

and debug log shows
2024-06-27 21:54:08 WARNING unit.kube-ovn/8.update-status logger.go:60 error: deployment "ovn-central" exceeded its progress deadline
2024-06-27 21:54:08 ERROR unit.kube-ovn/8.juju-log server.go:325 Traceback (most recent call last):
  File "./src/charm.py", line 366, in configure_kube_ovn
    self.wait_for_ovn_central()
  File "./src/charm.py", line 720, in wait_for_ovn_central
    self.wait_for_rollout("deployment/ovn-central")
  File "./src/charm.py", line 723, in wait_for_rollout
    self.kubectl(
  File "./src/charm.py", line 490, in kubectl
    return check_output(cmd)
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['kubectl', '--kubeconfig', '/root/.kube/config', 'rollout', 'status', '-n', 'kube-system', 'deployment/ovn-central', '--timeout', '1s']' returned non-zero exit status 1.

[Jeffrey Chang (modern911)]

All 8 occurrences from SolQA are on ARM64, and kube-ovn charm rev 68 with k8s 1.29.

[Jeffrey Chang (modern911)]

Today, Jul 15, we found 2 occurrences on AMD64.
https://solutions.qa.canonical.com/testruns/34d695c7-3737-4d19-9c5e-d172be02ac23
https://solutions.qa.canonical.com/testruns/e670652c-d004-42de-b437-3a5065ce2393