trove requests to keystone throwing SSL: CERTIFICATE_VERIFY_FAILED error

[Tom Blackburn]

What happened:
Any calls to the UI or trove cli fail (ie trove list, etc)

2023-10-26 10:44:37.959 1010 ERROR trove.common.wsgi keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate.
Please check that your auth_url is correct. SSL exception connecting to https://10.1.100.70:5000: HTTPSConnectionPool(host='10.1.100.70', port=5000): Max retries exceeded with url: /
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

What you expected to happen:
CA should verify SSL

How to reproduce it (minimal and precise):
kolla-ansible -i all-in-one deploy --tags trove

* OS (e.g. from /etc/os-release):
(venv) ansible@dev-os-01:~/venv$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

* Kernel (e.g. `uname -a`):
Linux dev-os-01 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

* Docker version if applicable (e.g. `docker version`):
docker version
Client: Docker Engine - Community
 Version: 24.0.6
 API version: 1.43
 Go version: go1.20.7
 Git commit: ed223bc
 Built: Mon Sep 4 12:31:44 2023
 OS/Arch: linux/amd64
 Context: default

Server: Docker Engine - Community
 Engine:
  Version: 24.0.6
  API version: 1.43 (minimum version 1.12)
  Go version: go1.20.7
  Git commit: 1a79695
  Built: Mon Sep 4 12:31:44 2023
  OS/Arch: linux/amd64
  Experimental: false
 containerd:
  Version: 1.6.24
  GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version: 1.1.9
  GitCommit: v1.1.9-0-gccaecfc
 docker-init:
  Version: 0.19.0
  GitCommit: de40ad0

* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release):
kolla-ansible installed like so:
pip install git+https://opendev.org/openstack/kolla-ansible@stable/2023.1

* Docker image Install type (source/binary):
binary

* Docker image distribution:
ubuntu

* Are you using official images from Docker Hub or self built?
Docker Hub

* Share your inventory file, globals.yml and other configuration files if relevant
all-in-one (unmodified)

I'm using a self signed cert created with: kolla-ansible -i all-in-one certificates

grep -v '^#' /etc/kolla/globals.yml
workaround_ansible_issue_8743: yes
kolla_base_distro: "ubuntu"

network_interface: "enp65s0f0.100"
neutron_external_interface: "enp65s0f1"
enable_neutron_provider_networks: "yes"
neutron_tenant_network_types: "vxlan"

kolla_internal_vip_address: "10.1.100.70"
keepalived_virtual_router_id: "70"

enable_cinder: "yes"
enable_cinder_backend_lvm: "yes"
openstack_cacert: "/etc/ssl/certs/ca-certificates.crt"

kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_admin_openrc_cacert: "/etc/ssl/certs/ca-certificates.crt"
kolla_copy_ca_into_containers: "yes"
kolla_verify_tls_backend: "yes"
kolla_enable_tls_backend: "yes"

horizon_keystone_multidomain: "yes"
enable_barbican: "yes"

enable_swift: "yes"
enable_swift_s3api: "yes"
enable_octavia: "yes"
octavia_network_interface: v-lbaas
octavia_amp_network:
  name: lb-mgmt-net
  provider_network_type: vlan
  provider_segmentation_id: 1001
  provider_physical_network: physnet1
  external: false
  shared: false
  subnet:
    name: lb-mgmt-subnet
    cidr: "10.1.92.0/24"
    allocation_pool_start: "10.1.92.100"
    allocation_pool_end: "10.1.92.200"
    gateway_ip: "10.1.92.1"
    enable_dhcp: yes
enable_gnocchi: "yes"
enable_cloudkitty: "yes"
cloudkitty_collector_backend: "gnocchi"
cloudkitty_storage_backend: "influxdb"
gnocchi_backend_storage: "file"

enable_magnum: "yes"
enable_cluster_user_trust: true

enable_redis: "yes"
enable_designate: "yes"
designate_ns_record: "csos.in"
enable_trove: "yes"
trove_logging_debug: true

/etc/kolla/trove-api/trove.conf has the cafile set like so:

[keystone_authtoken]
service_type = database
www_authenticate_uri = https://10.1.100.70:5000
project_domain_name = Default
project_name = service
user_domain_name = Default
username = trove
password = _removed_
auth_url = https://10.1.100.70:5000
auth_type = password
cafile = /etc/ssl/certs/ca-certificates.crt
region_name = RegionOne

I have verified the certs are in the container and are correct

docker exec -it trove_api bash

(trove-api)[root@dev-os-01 /]# openssl s_client -connect 10.1.100.70:5000 2>/dev/null < /dev/null | openssl x509 -noout -issuer
issuer=CN = KollaTestCA
(trove-api)[root@dev-os-01 /]# openssl x509 -in /etc/ssl/certs/kolla-customca-root.pem -noout -subject
subject=CN = KollaTestCA

Also verified with python from inside the trove_api container

(trove-api)[root@dev-os-01 /]# python
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://10.1.100.70:5000/')
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.1.100.70', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='10.1.100.70', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))
>>>
>>> r = requests.get('https://10.1.100.70:5000/', verify=False)
/var/lib/kolla/venv/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.100.70'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
>>> r = requests.get('https://10.1.100.70:5000/', verify='/etc/ssl/certs/ca-certificates.crt')
>>> r.text
'{"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://10.1.100.70:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}'
>>> r = requests.get('https://10.1.100.70:5000/', verify='/usr/local/share/ca-certificates/kolla-customca-root.crt')
>>> r.text
'{"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://10.1.100.70:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}'
>>>

If I stomp on the certifi/cacert.pem both python requests & the trove_api will start working
Obviously not a solution, just a fact.

cp -p /var/lib/kolla/venv/lib/python3.10/site-packages/certifi/cacert.pem /var/lib/kolla/venv/lib/python3.10/site-packages/certifi/cacert.pem.OLD
cp /etc/ssl/certs/ca-certificates.crt /var/lib/kolla/venv/lib/python3.10/site-packages/certifi/cacert.pem

(trove-api)[root@dev-os-01 /]# python
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://10.1.100.70:5000/')
>>> r.text
'{"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://10.1.100.70:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}'
>>>

Same ssl error happens with taskmanager

2023-10-26 16:46:00.673 7 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://10.1.100.70:5000: HTTPSConnectionPool(host='10.1.100.70', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))