python-swiftclient should retry at least *some* SSLErrors

[Tim Burke]

Currently, our retry logic special-cases SSLErrors to fail quickly: https://github.com/openstack/python-swiftclient/blob/4.2.0/swiftclient/client.py#L1801-L1802

Presumably we did this because if we get back an error about certificate verification (cert expired, cert self-signed), there's no point in retrying -- we expect the same result. However, there are other sorts of errors that might come up; for example,

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='...', port=443): Max retries exceeded with url: ... (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swiftclient/service.py", line 1209, in _download_object_job
    headers, body = conn.get_object(container, obj, **get_args)
  File "/usr/lib/python3/dist-packages/swiftclient/client.py", line 1970, in get_object
    rheaders, body = self._retry(None, get_object, container, obj,
  File "/usr/lib/python3/dist-packages/swiftclient/client.py", line 1856, in _retry
    rv = func(self.url, self.token, *args,
  File "/usr/lib/python3/dist-packages/swiftclient/client.py", line 1306, in get_object
    conn.request(method, path, '', headers)
  File "/usr/lib/python3/dist-packages/swiftclient/client.py", line 470, in request
    self.resp = self._request(method, url, headers=headers, data=data,
  File "/usr/lib/python3/dist-packages/swiftclient/client.py", line 454, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='...', port=443): Max retries exceeded with url: ... (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)')))

Note that this is a particularly disingenuous error message: despite saying "Max retries exceeded" and using our default --retries=5, not one attempt has been made to retry!

[Tim Burke (1-tim-z)]

Ugh. Likely duplicate from 2016 (!): https://bugs.launchpad.net/python-swiftclient/+bug/1641363

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-swiftclient/+/877843

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-swiftclient/+/877843
Committed: https://opendev.org/openstack/python-swiftclient/commit/099dd66265dfdc94e162eba4c1ec3a33b305bc26
Submitter: "Zuul (22348)"
Branch: master

commit 099dd66265dfdc94e162eba4c1ec3a33b305bc26
Author: Tim Burke <email address hidden>
Date: Fri Mar 17 16:14:15 2023 -0700

    Retry on SSLError

    ...provided it isn't something to do with certification validation.

    Closes-Bug: #1641363
    Closes-Bug: #2011731
    Change-Id: Ic3acbf431e444fcd8bc0fe79571fe2db4140cf22

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-swiftclient 4.3.0 release.