Zun with tls enabled is not working on horizon

[Maroš Varchola]

Everything is working correctly with internal/external/backend tls disabled. But, as soon as I enable all tls, (backend, internal, external), I cannot show any zun containers information on horion web ui.

The horizon is throwing this error:

[Sat Feb 05 19:09:47.084332 2022] [wsgi:error] [pid 697:tid 140363056170752] [remote 172.20.0.2:44150] /var/lib/kolla/venv/lib/python3.8/site-packages/horizon/views.py:58: RemovedInDjango40Warning: force_text() is deprecated in favor of force_str().
[Sat Feb 05 19:09:47.084369 2022] [wsgi:error] [pid 697:tid 140363056170752] [remote 172.20.0.2:44150] temp = template.Template(encoding.force_text(self.page_title))
[Sat Feb 05 19:09:47.102905 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] /var/lib/kolla/venv/lib/python3.8/site-packages/openstack_dashboard/api/rest/utils.py:110: RemovedInDjango40Warning: request.is_ajax() is deprecated. See Django 3.1 release notes for more details about this deprecation.
[Sat Feb 05 19:09:47.102934 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] if not request.is_ajax():
[Sat Feb 05 19:09:47.141533 2022] [wsgi:error] [pid 697:tid 140363056170752] [remote 172.20.0.2:44150] /var/lib/kolla/venv/lib/python3.8/site-packages/compressor/signals.py:4: RemovedInDjango40Warning: The providing_args argument is deprecated. As it is purely documentational, it has no replacement. If you rely on this argument as documentation, you can move the text to a code comment or docstring.
[Sat Feb 05 19:09:47.141565 2022] [wsgi:error] [pid 697:tid 140363056170752] [remote 172.20.0.2:44150] post_compress = django.dispatch.Signal(providing_args=['type', 'mode', 'context'])
[Sat Feb 05 19:09:47.193298 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] /usr/lib/python3.8/logging/__init__.py:1088: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
[Sat Feb 05 19:09:47.193327 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] stream.write(msg + self.terminator)
[Sat Feb 05 19:09:47.193342 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] HTTP exception with no status/code
[Sat Feb 05 19:09:47.193350 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] Traceback (most recent call last):
[Sat Feb 05 19:09:47.193366 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/horizon/utils/memoized.py", line 107, in wrapped
[Sat Feb 05 19:09:47.193374 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] value = cache[key] = cache.pop(key)
[Sat Feb 05 19:09:47.193380 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] KeyError: ((<weakref at 0x7fa8c1494130; to 'WSGIRequest' at 0x7fa8c17e6040>,), ())
[Sat Feb 05 19:09:47.193386 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154]
[Sat Feb 05 19:09:47.193391 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] During handling of the above exception, another exception occurred:
[Sat Feb 05 19:09:47.193397 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154]
[Sat Feb 05 19:09:47.193402 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] Traceback (most recent call last):
[Sat Feb 05 19:09:47.193408 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
[Sat Feb 05 19:09:47.193413 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] httplib_response = self._make_request(
[Sat Feb 05 19:09:47.193419 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
[Sat Feb 05 19:09:47.193425 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] self._validate_conn(conn)
[Sat Feb 05 19:09:47.193430 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
[Sat Feb 05 19:09:47.193435 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] conn.connect()
[Sat Feb 05 19:09:47.193441 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/connection.py", line 416, in connect
[Sat Feb 05 19:09:47.193447 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] self.sock = ssl_wrap_socket(
[Sat Feb 05 19:09:47.193452 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
[Sat Feb 05 19:09:47.193458 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] ssl_sock = _ssl_wrap_socket_impl(
[Sat Feb 05 19:09:47.193463 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
[Sat Feb 05 19:09:47.193469 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
[Sat Feb 05 19:09:47.193475 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
[Sat Feb 05 19:09:47.193480 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] return self.sslsocket_class._create(
[Sat Feb 05 19:09:47.193486 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/usr/lib/python3.8/ssl.py", line 1040, in _create
[Sat Feb 05 19:09:47.193491 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] self.do_handshake()
[Sat Feb 05 19:09:47.193497 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
[Sat Feb 05 19:09:47.193503 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] self._sslobj.do_handshake()
[Sat Feb 05 19:09:47.193516 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)
[Sat Feb 05 19:09:47.193523 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154]
[Sat Feb 05 19:09:47.193529 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] During handling of the above exception, another exception occurred:
[Sat Feb 05 19:09:47.193535 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154]
[Sat Feb 05 19:09:47.193541 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] Traceback (most recent call last):
[Sat Feb 05 19:09:47.193546 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
[Sat Feb 05 19:09:47.193552 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] resp = conn.urlopen(
[Sat Feb 05 19:09:47.193558 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
[Sat Feb 05 19:09:47.193565 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] retries = retries.increment(
[Sat Feb 05 19:09:47.193571 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
[Sat Feb 05 19:09:47.193577 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] raise MaxRetryError(_pool, url, error or ResponseError(cause))
[Sat Feb 05 19:09:47.193587 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='opsintvip.sunray.sk', port=9517): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
[Sat Feb 05 19:09:47.193595 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154]
[Sat Feb 05 19:09:47.193601 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] During handling of the above exception, another exception occurred:
[Sat Feb 05 19:09:47.193608 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154]
[Sat Feb 05 19:09:47.193613 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] Traceback (most recent call last):
[Sat Feb 05 19:09:47.193619 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 1022, in _send_request
[Sat Feb 05 19:09:47.193626 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] resp = self.session.request(method, url, **kwargs)
[Sat Feb 05 19:09:47.193632 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
[Sat Feb 05 19:09:47.193639 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] resp = self.send(prep, **send_kwargs)
[Sat Feb 05 19:09:47.193645 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
[Sat Feb 05 19:09:47.193651 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] r = adapter.send(request, **kwargs)
[Sat Feb 05 19:09:47.193657 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/requests/adapters.py", line 517, in send
[Sat Feb 05 19:09:47.193664 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] raise SSLError(e, request=request)
[Sat Feb 05 19:09:47.193696 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] requests.exceptions.SSLError: HTTPSConnectionPool(host='opsintvip.sunray.sk', port=9517): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
[Sat Feb 05 19:09:47.193706 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154]
[Sat Feb 05 19:09:47.193731 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] During handling of the above exception, another exception occurred:
[Sat Feb 05 19:09:47.193739 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154]
[Sat Feb 05 19:09:47.193745 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] Traceback (most recent call last):
[Sat Feb 05 19:09:47.193751 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/openstack_dashboard/api/rest/utils.py", line 127, in _wrapped
[Sat Feb 05 19:09:47.193758 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] data = function(self, request, *args, **kw)
[Sat Feb 05 19:09:47.193764 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zun_ui/api/rest_api.py", line 124, in get
[Sat Feb 05 19:09:47.193769 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] result = client.container_list(request)
[Sat Feb 05 19:09:47.193775 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zun_ui/api/client.py", line 205, in container_list
[Sat Feb 05 19:09:47.193780 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] return zunclient(request).containers.list(limit, marker, sort_key,
[Sat Feb 05 19:09:47.193785 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/horizon/utils/memoized.py", line 109, in wrapped
[Sat Feb 05 19:09:47.193791 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] value = cache[key] = func(*args, **kwargs)
[Sat Feb 05 19:09:47.193797 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zun_ui/api/client.py", line 75, in zunclient
[Sat Feb 05 19:09:47.193803 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] api_version = api_versions.discover_version(c, api_version)
[Sat Feb 05 19:09:47.193809 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zunclient/api_versions.py", line 251, in discover_version
[Sat Feb 05 19:09:47.193815 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] server_start_version, server_end_version = _get_server_version_range(
[Sat Feb 05 19:09:47.193822 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zunclient/api_versions.py", line 242, in _get_server_version_range
[Sat Feb 05 19:09:47.193828 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] version = client.versions.get_current()
[Sat Feb 05 19:09:47.193834 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zunclient/v1/versions.py", line 48, in get_current
[Sat Feb 05 19:09:47.193841 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] for version in self.list():
[Sat Feb 05 19:09:47.193847 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zunclient/v1/versions.py", line 45, in list
[Sat Feb 05 19:09:47.193890 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] return self._list(version_url, "versions")
[Sat Feb 05 19:09:47.193900 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zunclient/common/base.py", line 128, in _list
[Sat Feb 05 19:09:47.193906 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] resp, body = self.api.json_request('GET', url)
[Sat Feb 05 19:09:47.193911 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zunclient/common/httpclient.py", line 368, in json_request
[Sat Feb 05 19:09:47.193917 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] resp = self._http_request(url, method, **kwargs)
[Sat Feb 05 19:09:47.193923 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/zunclient/common/httpclient.py", line 344, in _http_request
[Sat Feb 05 19:09:47.193929 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] resp = self.session.request(url, method,
[Sat Feb 05 19:09:47.193934 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 931, in request
[Sat Feb 05 19:09:47.193940 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] resp = send(**kwargs)
[Sat Feb 05 19:09:47.193945 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 1026, in _send_request
[Sat Feb 05 19:09:47.193950 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] raise exceptions.SSLError(msg)
[Sat Feb 05 19:09:47.193956 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://opsintvip.sunray.sk:9517: HTTPSConnectionPool(host='opsintvip.sunray.sk', port=9517): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
[Sat Feb 05 19:09:47.195764 2022] [wsgi:error] [pid 694:tid 140363056170752] [remote 172.20.0.2:44154] Internal Server Error: /api/zun/containers/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/zun-ui 10.0.0.0rc1 release candidate.