kubeapi-load-balancer failed to reconcile

[Konstantinos Kaskavelis]

Solutions QA team has a failed run where kubeapi-load-balancer stayed in a blocked state with the message: "Failed to reconcile".

The k8s version used for this test run is 1.29, while Juju 3.3/stable has been used for the deployment of kubernetes on top of MAAS 3.4/stable.

From the logs it seems that nginx server fails to restart:

2024-03-08 19:47:38 DEBUG unit.kubeapi-load-balancer/0.juju-log server.go:325 Attempting to ['systemctl', 'restart', 'nginx'] 'nginx' with command ['systemctl', 'restart', 'nginx'].
2024-03-08 19:47:43 DEBUG unit.kubeapi-load-balancer/0.juju-log server.go:325 Job for nginx.service canceled.

2024-03-08 19:47:43 ERROR unit.kubeapi-load-balancer/0.juju-log server.go:325 Could not restart nginx: systemd output: Job for nginx.service canceled.
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/venv/charms/reconciler.py", line 34, in reconcile
    result = self.reconcile_function(event)
  File "./src/charm.py", line 330, in _reconcile
    self._install_load_balancer()
  File "./src/charm.py", line 245, in _install_load_balancer
    self._restart_nginx()
  File "./src/charm.py", line 359, in _restart_nginx
    service_restart(NGINX_SERVICE)
  File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/lib/charms/operator_libs_linux/v1/systemd.py", line 174, in service_restart
    return _systemctl("restart", service_name)
  File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/lib/charms/operator_libs_linux/v1/systemd.py", line 125, in _systemctl
    raise SystemdError(
charms.operator_libs_linux.v1.systemd.SystemdError: Could not restart nginx: systemd output: Job for nginx.service canceled.

Failed run:
https://solutions.qa.canonical.com/testruns/57e729de-62f3-478f-afa6-8523eab77743

Logs (available for two weeks):
https://oil-jenkins.canonical.com/artifacts/57e729de-62f3-478f-afa6-8523eab77743/index.html