Cinder

Backup using Swift driver fails when TLS is enabled on internal network

Bug #2037113 reported by Magnus Lööf
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned

Bug Description

When configuring Cinder Backup to use Swift as backup storage, while having TLS enabled on the internal network, the Backup fails to start.

The following can be observed in the logs:

```
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift [-] Can not get Swift capabilities during backup driver initialization.: requests.exceptions.SSLError: HTTPSConnectionPool(host='horizon.openstack.malo.domain.local', port=6780): Max retries exceeded with url: /swift/info (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift Traceback (most recent call last):
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift chunked=chunked,
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift self._validate_conn(conn)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift conn.connect()
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connection.py", line 421, in connect
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift tls_in_tls=tls_in_tls,
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift sock, context, tls_in_tls, server_hostname=server_hostname
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/green/ssl.py", line 445, in wrap_socket
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift return GreenSSLSocket(sock, *a, _context=self, **kw)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/green/ssl.py", line 139, in __init__
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift self.do_handshake()
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/green/ssl.py", line 312, in do_handshake
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift super(GreenSSLSocket, self).do_handshake)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/green/ssl.py", line 161, in _call_trampolining
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift return func(*a, **kw)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift self._sslobj.do_handshake()
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift self._sslobj.do_handshake()
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift During handling of the above exception, another exception occurred:
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift Traceback (most recent call last):
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift timeout=timeout
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift raise MaxRetryError(_pool, url, error or ResponseError(cause))
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='horizon.openstack.malo.domain.local', port=6780): Max retries exceeded with url: /swift/info (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift During handling of the above exception, another exception occurred:
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift Traceback (most recent call last):
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/backup/drivers/swift.py", line 420, in check_for_setup_error
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift conn.get_capabilities()
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/swiftclient/client.py", line 2059, in get_capabilities
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift return get_capabilities((parsed, self.http_conn[1]))
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/swiftclient/client.py", line 1677, in get_capabilities
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift conn.request('GET', parsed.path, '', headers)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/swiftclient/client.py", line 471, in request
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift files=files, **self.requests_args)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/swiftclient/client.py", line 454, in _request
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift return self.request_session.request(*arg, **kwarg)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift resp = self.send(prep, **send_kwargs)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift r = adapter.send(request, **kwargs)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift raise SSLError(e, request=request)
2023-09-21 11:14:37.561 33 ERROR cinder.backup.drivers.swift requests.exceptions.SSLError: HTTPSConnectionPool(host='horizon.openstack.malo.domain.local', port=6780): Max retries exceeded with url: /swift/info (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
```

The following hack seems to fix it:

https://github.com/openstack/cinder/blob/stable/xena/cinder/backup/drivers/swift.py#L407

```
conn = swift.Connection(retries=CONF.backup_swift_retry_attempts,
                preauthurl=CONF.backup_swift_url,
                cacert=CONF.backup_swift_ca_cert_file)
```

Revision history for this message
Magnus Lööf (magnus-loof) wrote :

Possibly related #1888378

It might be the same or a similar problem. However, the suggestion there is to use the Internal network (possibly because there is no TLS there).

But the correct solution is of course:

- Always use TLS between distributed components
- Always configure a requirement to verify the trust

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.