Solutions QA had a failed run due to the kubeapi-load-balancer charm ending up in error state with message: hook failed: "certificates-relation-joined"
The error happened while deploying charmed kubernetes 1.28 on top of MAAS 3.2.10~alpha1- on jammy.
Some logs to give context:
2023-10-11 04:19:44 DEBUG juju.worker.dependency engine.go:603 "metric-collect" manifold worker completed successfully
2023-10-11 04:19:44 DEBUG juju.worker.dependency engine.go:618 "migration-inactive-flag" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-10-11 04:19:44 DEBUG juju.worker.dependency engine.go:618 "hook-retry-strategy" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-10-11 04:19:44 DEBUG juju.worker.dependency engine.go:618 "api-address-updater" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-10-11 04:19:44 DEBUG juju.worker.dependency engine.go:618 "meter-status" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-10-11 04:19:44 DEBUG juju.worker.dependency engine.go:618 "logging-config-updater" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-10-11 04:19:44 DEBUG juju.worker.dependency engine.go:618 "secret-drain-worker" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-10-11 04:19:44 DEBUG juju.worker.dependency engine.go:603 "api-caller" manifold worker completed successfully
2023-10-11 04:19:44 DEBUG unit.kubeapi-load-balancer/0.juju-log server.go:325 certificates:72: tracer: hooks phase, 0 handlers queued
2023-10-11 04:19:44 DEBUG unit.kubeapi-load-balancer/0.juju-log server.go:325 certificates:72: tracer>
tracer: main dispatch loop, 5 handlers queued
tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:website
tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:apiserver
tracer: ++ queue handler hooks/relations/tls-certificates/requires.py:79:joined:certificates
tracer: ++ queue handler reactive/apt.py:50:ensure_package_status
tracer: ++ queue handler reactive/load_balancer.py:230:set_nginx_version
2023-10-11 04:19:44 INFO unit.kubeapi-load-balancer/0.juju-log server.go:325 certificates:72: Invoking reactive handler: reactive/load_balancer.py:230:set_nginx_version
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 ERROR connection is shut down
2023-10-11 04:19:44 ERROR unit.kubeapi-load-balancer/0.juju-log server.go:325 certificates:72: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/reactive/load_balancer.py", line 239, in set_nginx_version
hookenv.application_version_set(version.rstrip())
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 1198, in application_version_set
subprocess.check_call(cmd)
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['application-version-set', b'1.18.0']' returned non-zero exit status 1.
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 Traceback (most recent call last):
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/hooks/certificates-relation-joined", line 22, in <module>
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 main()
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 bus.dispatch(restricted=restricted_mode)
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 _invoke(other_handlers)
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 handler.invoke()
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 self._action(*args)
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/reactive/load_balancer.py", line 239, in set_nginx_version
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 hookenv.application_version_set(version.rstrip())
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 1198, in application_version_set
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 subprocess.check_call(cmd)
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 raise CalledProcessError(retcode, cmd)
2023-10-11 04:19:44 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined logger.go:60 subprocess.CalledProcessError: Command '['application-version-set', b'1.18.0']' returned non-zero exit status 1.
2023-10-11 04:19:45 ERROR juju.worker.uniter.operation runhook.go:180 hook "certificates-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
2023-10-11 04:19:45 DEBUG juju.machinelock machinelock.go:206 created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
2023-10-11 04:19:45 DEBUG juju.machinelock machinelock.go:190 machine lock "machine-lock" released for kubeapi-load-balancer/0 uniter (run relation-joined (72; unit: vault/2) hook)
2023-10-11 04:19:45 DEBUG juju.worker.uniter.operation executor.go:118 lock released for kubeapi-load-balancer/0
2023-10-11 04:19:45 INFO juju.worker.uniter uniter.go:347 unit "kubeapi-load-balancer/0" shutting down: (re)starting watcher: connection is shut down
2023-10-11 04:19:45 DEBUG juju.worker.dependency engine.go:603 "migration-fortress" manifold worker completed successfully
2023-10-11 04:19:45 DEBUG juju.worker.dependency engine.go:618 "uniter" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-10-11 04:19:47 INFO juju unit_agent.go:289 Starting unit workers for "kubeapi-load-balancer/0"
Failed test run: https://solutions.qa.canonical.com/testruns/149c1de6-6cdc-4c1c-bf37-4690a77f6b9c
Logs: https://oil-jenkins.canonical.com/artifacts/149c1de6-6cdc-4c1c-bf37-4690a77f6b9c/index.html
