Uploading package to server with self-signed certificate on https fails despite adding cert to trust-store

[Mark Cunningham]

On Ubuntu 22.04 with dput version 1.1.0ubuntu2.1, and python3 3.10.x, customers using a self-signed SSL for https are getting the following:

  File "/usr/bin/dput", line 37, in <module>
    sys.exit(load_entry_point('dput==1.1.0+ubuntu2.1', 'console_scripts', 'execute-dput')())
  File "/usr/share/dput/dput/dput.py", line 1235, in main
    upload_methods[method](
  File "/usr/share/dput/dput/methods/https.py", line 18, in upload
    return http.upload(
  File "/usr/share/dput/dput/methods/http.py", line 138, in upload
    conn.endheaders()
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/lib/python3.10/http/client.py", line 1455, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1100, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)

This seems to be an issue in how the SSL for the https connection is validated. Even after adding the self-signed certificate to the trust store with update-ca-certificates, this is not being read by the python code for validation of the cert on the mirror.

The immediate solution has been to modify the main dput file to import the ssl library, and tell it to not validate the certificate for the connection:

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

This is discussed further at the following link:

https://stackoverflow.com/questions/77639570/ssl-verification-problem-when-uploading-a-deb-package-using-dput

This seems like a change in python behavior given this discussion:

https://stackoverflow.com/questions/35569042/ssl-certificate-verify-failed-with-python3

I am not sure what the best path forward is, I would think that ideally there may be an environment variable to tell python to read the certificate from the standard trust-store /etc/ssl/certs/ca-certificates.crt, or otherwise to skip certificate validation, without needing to modify dput directly.

I do not see this happening on 20.04 with python 3.8.x and dput 1.0.3ubuntu1.1, so this seems to be a relatively recent change in behavior.

[Mark Cunningham (mdscunningham)]

Update: after a lot of discussion with Mitch Burton on the Landscape team, he was able to demonstrate this working with a self-signed certificate. We think that this may actually not be strictly an issue with the self-signed SSL, but rather that the name in the cert is not an FQDN, and instead is just the bare hostname.

Upon further testing myself, I swapped the hostname on my test instance from landscape-2310-jammy to landscape-2310-jammy.lxd just as a test. I then updated my /etc/hosts file, the certificates configured in Apache, and imported the newly generated cert into ca-certificates. After this dput worked just fine.

dput lds:ubuntu/jammy/upload hello.changes

D: Splitting host argument out of lds:ubuntu/jammy/upload.
D: Setting host argument.
Checking signature on .changes
gpg: /root/hello.changes: Valid signature from 5E1E964200F3EA3D
Uploading to lds (via https to landscape-2310-jammy.lxd):
  Uploading hello_2.10-2ubuntu4+esm1_amd64.deb: done.
  Uploading hello.changes: done.
Successfully uploaded packages.

This seems to confirm that the issue is not necessarily with dput directly, but in how python's urllib is checking the domain/cert on the connection. This may be something that can be worked around in dput to allow for a bare hostname that is not an FQDN, but either way figured it would be relevant to add this information.