Unable to connect to object store with TLS enabled

[Frode Nordahl]

As run by the gss charm with snap support:

The Keystnoe bits appear to dtrt, but when also the Swift API is on a HTTPS URL things blow up.

Tested using cs:~openstack-charmers-next/ceph-radosgw with relation to vault as object store provider.

$ juju run-action --wait glance-simplestreams-sync/1 sync-images
unit-glance-simplestreams-sync-1:
  UnitId: glance-simplestreams-sync/1
  id: "22"
  results:
    Stderr: |
      Traceback (most recent call last):
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
          cnx.do_handshake()
        File "/snap/simplestreams/12/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
          self._raise_ssl_error(self._ssl, result)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
          _raise_current_error()
        File "/snap/simplestreams/12/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
          raise exception_type(errors)
      OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
          chunked=chunked,
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
          self._validate_conn(conn)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
          conn.connect()
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connection.py", line 360, in connect
          ssl_context=context,
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
          return context.wrap_socket(sock)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
          raise ssl.SSLError("bad handshake: %r" % e)
      ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/snap/simplestreams/12/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
          timeout=timeout
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
          method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
          raise MaxRetryError(_pool, url, error or ResponseError(cause))
      urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.16.122.72', port=443): Max retries exceeded with url: /swift/v1/simplestreams (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/snap/simplestreams/12/bin/sstream-mirror-glance", line 185, in <module>
          main()
        File "/snap/simplestreams/12/bin/sstream-mirror-glance", line 161, in main
          tstore = swift.SwiftObjectStore(args.output_swift, region=region)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/simplestreams/objectstores/swift.py", line 78, in __init__
          '.r:*,.rlistings'})
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 1836, in put_container
          query_string=query_string)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 1748, in _retry
          service_token=self.service_token, **kwargs)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 1081, in put_container
          conn.request(method, path, '', req_headers)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 469, in request
          files=files, **self.requests_args)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 452, in _request
          return self.request_session.request(*arg, **kwarg)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
          resp = self.send(prep, **send_kwargs)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
          r = adapter.send(request, **kwargs)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
          raise SSLError(e, request=request)
      requests.exceptions.SSLError: HTTPSConnectionPool(host='172.16.122.72', port=443): Max retries exceeded with url: /swift/v1/simplestreams (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
      /usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py:108: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
        confobj = yaml.load(f)
      /usr/lib/python3/dist-packages/keystoneauth1/adapter.py:235: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1.
        warnings.warn('Using keystoneclient sessions has been deprecated. '
  status: completed
  timing:
    completed: 2020-06-17 06:54:09 +0000 UTC
    enqueued: 2020-06-17 06:54:06 +0000 UTC
    started: 2020-06-17 06:54:06 +0000 UTC

[Paride Legovini (paride)]

Hi Frode,

Just to be sure: this does not happen with the simplestreams .deb from the Ubuntu repos, but it does happen with the snap, everything else being the same. Correct?

[Frankline Ngatia (fngatia)]

20201 Wallaby
@>juju run-action --wait glance-simplestreams-sync/2 sync-images
unit-glance-simplestreams-sync-2:
  UnitId: glance-simplestreams-sync/2
  id: "69"
  message: exit status 1
  results:
    ReturnCode: 1
    Stderr: |
      /usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py:109: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
        confobj = yaml.load(f)
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
          cnx.do_handshake()
        File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in do_handshake
          self._raise_ssl_error(self._ssl, result)
        File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
          _raise_current_error()
        File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
          raise exception_type(errors)
      OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
          httplib_response = self._make_request(
        File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
          self._validate_conn(conn)
        File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
          conn.connect()
        File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
          self.sock = ssl_wrap_socket(
        File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
          return context.wrap_socket(sock)
        File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
          raise ssl.SSLError("bad handshake: %r" % e)
      ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
          resp = conn.urlopen(
        File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
          retries = retries.increment(
        File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
          raise MaxRetryError(_pool, url, error or ResponseError(cause))
      urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.0.0.35', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/...

[Nikolay Vinogradov (nikolay.vinogradov)]

Just hit this bug as well. Using the current stable version of the g-s-s charm [1] with simplestreams snap. SSL CA comes over 'certificates' relation to vault. The problem seems to be that strictly confined 'simplestreams' snap can't access host's truststore where the CA certificate is installed by g-s-s charm.

Workaround is to reinstall the snap with --devmodel

Please fix.

[Nikolay Vinogradov (nikolay.vinogradov)]

[1] https://charmhub.io/glance-simplestreams-sync