2021-03-12 09:56:47.919024 | 525400d9-0bd3-af49-4717-0000000000fc | TIMING | External deployment Post Deploy tasks | undercloud | 0:33:31.206116 | 0.04s
2021-03-12 09:56:52.902140 | 525400d9-0bd3-af49-4717-000000000108 | TASK | Nova: Manage aggregate and availability zone and add hosts to the zone
lib/python3.6/site-packages/urllib3/util/retry.py\", line 439, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overcloud.mainnetwork.localdomain', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 655, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 514, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='overcloud.mainnetwork.localdomain', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in
run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_os_nova_host_aggregate_payload_bgz0pjcl/ansible_os_nova_host_aggregate_payload.zip/ansible/modules/cloud/openstack/os_nova_host_aggregate.py\", line 182, in <module>\n File \"/tmp/ansible_os_nova_host_aggregate_payload_bgz0pjcl/ansible_os_nova_host_aggregate_payload.zip
/ansible/modules/cloud/openstack/os_nova_host_aggregate.py\", line 123, in main\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_compute.py\", line 1497, in search_aggregates\n aggregates = self.list_aggregates()\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_compute.py\", line 1506, in list_aggregates\n return self.compute.aggregates(allow_unknown_params=True, **filters)\n File \"/usr/lib/python3.6/site-packages/openstack/service_description.py\", line 87, in __get__\n proxy = self._make_proxy(instance)\n File \"/usr/lib/python3.6/site-packages/openstack/service_description.py\", line 262, in _make_proxy\n found_version = temp_adapter.get_api_major_version()\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 354, in get_api_major_version\n return self.session.get_api_major_version(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1276, in get_api_major_version\n return auth.get_api_major_version(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 500, in get_api_major_version\n data = get_endpoint_data(discover_versions=discover_versions)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 208, in get_auth_ref\n return self._plugin.get_auth_ref(session, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/v3/b^[[0;31m2021-03-12 09:56:54.276549 | 525400d9-0bd3-af49-4717-000000000108 | FATAL | Nova: Manage aggregate and availability zone and add hosts to the zone | undercloud | error={"changed": false, "module_stderr": "Failed to discover available identity versions when contacting https://overcloud.mainnetwork.localdomain:13000/v3. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 677, in urlopen\n chunked=chunked,\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 381, in _make_request\n self._validate_conn(conn)\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 978, in _validate_conn\n conn.connect()\n File \"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 371, in connect\n ssl_context=context,\n File \"/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line 384, in ssl_wrap_socket\n return context.wrap_socket(sock, server_hostname=server_hostname)\n File \"/usr/lib64/python3.6/ssl.py\", line 365, in wrap_socket\n _context=self, _session=session)\n File \"/usr/lib64/python3.6/ssl.py\", line 776, in __init__\n self.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 1036, in do_handshake\n self._sslobj.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 648, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 449, in send\n timeout=timeout\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 727, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 439, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overcloud.mainnetwork.localdomain', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwar
The reason is that this playbook only uses the following CA file:
export OS_CACERT="/etc/pki/ca-trust/source/anchors/cm-local-ca.pem"
That pem file only holds the locally created undercloud CA and so it fails.
If you use the system CA which also includes the freeipa CA amongst other things (/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem). It all works.
In fact curling overcloud.mainnetwork.localdomain:13000 from the undercloud works just fine because curl uses the system configured CAs which include the freeipa one and the cert of the overcloud is signed by the freeipa CA.
Note: We only hit this in master because we have not tried nova-az-config with older releases
I filed this because the keystone endpoint creation works just fine, so I think our TLS-e env is more or less working, i.e. this might very well be more of a nova-az-config env problem