Soltuions QA team has a failed run where grafana was stuck on an error state with the message:
hook failed: "certificates-relation-changed"
From the logs:
2024-01-15 03:26:21 INFO unit.grafana/0.juju-log server.go:325 certificates:34: Found datasource: {'service_name': 'prometheus', 'type': 'prometheus', 'url': 'http://172.31.44.32:9090', 'description': 'Juju generated source'}
2024-01-15 03:26:21 INFO unit.grafana/0.juju-log server.go:325 certificates:34: Datasource already exist, updating: prometheus - Juju generated source
2024-01-15 03:26:21 ERROR unit.grafana/0.juju-log server.go:325 certificates:34: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/urllib3/connection.py", line 362, in connect
self.sock = ssl_wrap_socket(
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 399, in ssl_wrap_socket
return context.wrap_socket(sock)
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1069, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1338, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 446, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=3000): Max retries exceeded with url: /api/search?type=dash-db (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-grafana-0/charm/reactive/grafana.py", line 730, in configure_sources
generate_prometheus_dashboards(gf_adminpasswd, ds)
File "/var/lib/juju/agents/unit-grafana-0/charm/reactive/grafana.py", line 1043, in generate_prometheus_dashboards
current_dashboards = get_current_dashboards(config["port"], gf_adminpasswd)
File "/var/lib/juju/agents/unit-grafana-0/charm/reactive/grafana.py", line 874, in get_current_dashboards
dash_req = requests.get(
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=3000): Max retries exceeded with url: /api/search?type=dash-db (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
The failure occurred during deployment of COS layer on top of AWS.
Failed run: https://solutions.qa.canonical.com/testruns/3729390a-706a-4fd8-a31d-70f3c21f15e4
Logs can be found here: https://oil-jenkins.canonical.com/artifacts/3729390a-706a-4fd8-a31d-70f3c21f15e4/index.html (available for two weeks)
