Kubernetes Worker Charm

Ingress-daemon-set fails during charm upgrade (1.30/stable -> 1.30/edge)

Bug #2074388 reported by Adam Dyess on 2024-07-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Kubernetes Worker Charm	In Progress	Medium	Adam Dyess	Kubernetes Worker Charm 1.31

Bug Description

During upgrades from ch:arm64/jammy/kubernetes-worker-235 -> ch:arm64/jammy/kubernetes-worker-241

If the kubernetes-worker fails as kubectl command with a 502 GatewayError, it should be treated as a retryable error during the next reconciler hook. Likely the issue is the API server is currently unavailable.

2024-07-29 09:20:20 INFO unit.kubernetes-worker/2.juju-log server.go:325 Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
2024-07-29 09:20:20 WARNING unit.kubernetes-worker/2.config-changed logger.go:60 Error from server (InternalError): an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>") has prevented the request from succeeding
2024-07-29 09:20:20 ERROR unit.kubernetes-worker/2.juju-log server.go:325 Command failed: ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
returncode: 1
stdout:
2024-07-29 09:20:21 INFO unit.kubernetes-worker/2.juju-log server.go:325 Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
2024-07-29 09:20:21 WARNING unit.kubernetes-worker/2.config-changed logger.go:60 Error from server (InternalError): an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>") has prevented the request from succeeding
2024-07-29 09:20:21 ERROR unit.kubernetes-worker/2.juju-log server.go:325 Command failed: ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
returncode: 1
stdout:
2024-07-29 09:20:23 INFO unit.kubernetes-worker/2.juju-log server.go:325 Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
2024-07-29 09:20:23 WARNING unit.kubernetes-worker/2.config-changed logger.go:60 Error from server (InternalError): an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>") has prevented the request from succeeding
2024-07-29 09:20:23 ERROR unit.kubernetes-worker/2.juju-log server.go:325 Command failed: ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
returncode: 1
stdout:
2024-07-29 09:20:27 INFO unit.kubernetes-worker/2.juju-log server.go:325 Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
2024-07-29 09:20:27 WARNING unit.kubernetes-worker/2.config-changed logger.go:60 Error from server (InternalError): an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>") has prevented the request from succeeding
2024-07-29 09:20:27 ERROR unit.kubernetes-worker/2.juju-log server.go:325 Command failed: ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
returncode: 1
stdout:
2024-07-29 09:20:35 INFO unit.kubernetes-worker/2.juju-log server.go:325 Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
2024-07-29 09:20:35 WARNING unit.kubernetes-worker/2.config-changed logger.go:60 Error from server (InternalError): an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>") has prevented the request from succeeding
2024-07-29 09:20:35 ERROR unit.kubernetes-worker/2.juju-log server.go:325 Command failed: ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
returncode: 1
stdout:
2024-07-29 09:20:51 INFO unit.kubernetes-worker/2.juju-log server.go:325 Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
2024-07-29 09:20:51 WARNING unit.kubernetes-worker/2.config-changed logger.go:60 Error from server (InternalError): an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>") has prevented the request from succeeding
2024-07-29 09:20:51 ERROR unit.kubernetes-worker/2.juju-log server.go:325 Command failed: ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
returncode: 1
stdout:
2024-07-29 09:21:23 INFO unit.kubernetes-worker/2.juju-log server.go:325 Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
2024-07-29 09:21:24 WARNING unit.kubernetes-worker/2.config-changed logger.go:60 Error from server (InternalError): an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>") has prevented the request from succeeding
2024-07-29 09:21:24 ERROR unit.kubernetes-worker/2.juju-log server.go:325 Command failed: ['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']
returncode: 1
stdout:
2024-07-29 09:21:24 INFO unit.kubernetes-worker/2.juju-log server.go:325 Status context closed with: []
2024-07-29 09:21:24 ERROR unit.kubernetes-worker/2.juju-log server.go:325 Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/src/kubectl.py", line 20, in kubectl
    return check_output(command).decode("utf-8")
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['kubectl', '--kubeconfig=/root/.kube/config', 'apply', '-f', '/root/cdk/addons/ingress-daemon-set.yaml']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/./src/charm.py", line 453, in <module>
    ops.main(KubernetesWorkerCharm)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/ops/main.py", line 555, in __call__
    return main(charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/charms/reconciler.py", line 45, in reconcile
    self.reconcile_function(event)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/./src/charm.py", line 387, in reconcile
    self._configure_nginx_ingress_controller()
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/./src/charm.py", line 225, in _configure_nginx_ingress_controller
    kubectl("apply", "-f", manifest_path)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/var/lib/juju/agents/unit-kubernetes-worker-2/charm/venv/tenacity/__init__.py", line 326, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0xe66d185cbbe0 state=finished raised CalledProcessError>]