Mirror prober incorrectly reports an invalid SSL certificate

[NetActuate Ops]

We have been noticing a problem with the mirror probing where partway through the probe, it suddenly starts complaining about our mirror having an invalid certificate, which is absolutely not the case.

https://launchpad.net/ubuntu/+mirror/mirror1.cl.netactuate.com-archive
https://launchpad.net/ubuntu/+mirror/mirror1.cl.netactuate.com-release

Here are logs with the failures:
http://launchpadlibrarian.net/486354226/mirror1.cl.netactuate.com-archive-probe-logfile.txt
http://launchpadlibrarian.net/486330936/mirror1.cl.netactuate.com-release-probe-logfile.txt

As you can see from the logs, everything starts off fine, then suddenly one check claims there is an invalid HTTPS certificate, and everything else gets skipped.

You can also see here where everything worked fine:
http://launchpadlibrarian.net/486109160/mirror1.cl.netactuate.com-archive-probe-logfile.txt
http://launchpadlibrarian.net/486094033/mirror1.cl.netactuate.com-release-probe-logfile.txt

Nothing has changed with the certificate between these probes.

"Invalid SSL certificate" can also mean a lot of different things specifically, having more verbose logs of such a failure would be useful.

[Luís Baker (luisbaker)]

My mirror is having the same problem for months
http://launchpadlibrarian.net/590317372/ibakerserver-probe-logfile.txt

[it4innovations (it4i)]

Hi Team,

we have experienced same or similar error about invalid certificate as well but cert has been valid for some time and it expires in October this year-

I could not get newer logs as the prober history errs in timeout.

Br,

Ondrej Filip
IT4Innovations

http://launchpadlibrarian.net/660904400/mirror.it4i.cz-archive-probe-logfile.txt

[Teresa Cancino (hostednode)]

Same problem here,

because: Connection skipped because the server doesn't have a valid HTTPS certificate. It will be retried on the next probing run.

http://launchpadlibrarian.net/667961958/mirror.hnd.cl-archive-probe-logfile.txt

But our mirror have a valid and working certificate

https://mirror.hnd.cl/ubuntu/

% nmap --script ssl-cert -p 443 mirror.hnd.cl
Starting Nmap 7.93 ( https://nmap.org ) at 2023-05-25 23:21 -04
Nmap scan report for mirror.hnd.cl (138.186.10.35)
Host is up (0.0035s latency).
rDNS record for 138.186.10.35: 35.10.186.138.static.hostednode.net

PORT STATE SERVICE
443/tcp open https
| ssl-cert: Subject: commonName=mirror.hnd.cl
| Subject Alternative Name: DNS:mirror.hnd.cl
| Issuer: commonName=R3/organizationName=Let's Encrypt/countryName=US
| Public Key type: rsa
| Public Key bits: 2048
| Signature Algorithm: sha256WithRSAEncryption
| Not valid before: 2023-04-15T23:36:34
| Not valid after: 2023-07-14T23:36:33
| MD5: 525fad578a5ecc3e5b0f2fd00cfc4399

nmap -6 --script ssl-cert -p 443 mirror.hnd.cl

Starting Nmap 6.40 ( http://nmap.org ) at 2023-05-25 23:23 -04
Nmap scan report for mirror.hnd.cl (2803:8240:50:1035::35)
Host is up (0.11s latency).
PORT STATE SERVICE
443/tcp open https
| ssl-cert: Subject: commonName=mirror.hnd.cl
| Issuer: commonName=R3/organizationName=Let's Encrypt/countryName=US
| Public Key type: rsa
| Public Key bits: 2048
| Not valid before: 2023-04-15T23:36:34+00:00
| Not valid after: 2023-07-14T23:36:33+00:00
| MD5: 525f ad57 8a5e cc3e 5b0f 2fd0 0cfc 4399
|_SHA-1: 6d05 e390 16c3 9321 547d fbcb 7298 72c7 8b17 d1b6

Nmap done: 1 IP address (1 host up) scanned in 1.51 seconds
|_SHA-1: 6d05e39016c39321547dfbcb729872c78b17d1b6

[Paul Collins (pjdc)]

This looks to be something peculiar to the mirror prober itself. Even on the machine that runs the mirror prober, I can successfully connect to these hosts from Python:

ubuntu@juju-98d295-prod-launchpad-16:~$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.head('https://mirror.hnd.cl')
<Response [200]>
>>> _

[Paul Collins (pjdc)]

Other reports:
 * https://rt.ubuntu.com/Ticket/Display.html?id=79671
 * https://answers.launchpad.net/launchpad/+question/708953

[Paul Collins (pjdc)]

The state of the machine itself may not even be super relevant, because the prober is invoked with `python3 -S`, i.e. no site init is performed.

[Paul Collins (pjdc)]

Here are some mirrors that are currently not verifying but that can be connected to successfully from the mirror prober host:

== mirror.veesp.com ==

https://launchpadlibrarian.net/709348087/mirror.veesp.com-archive-probe-logfile.txt

but:

ubuntu@juju-98d295-prod-launchpad-16:~$ python3 -c 'import requests, sys ; print(requests.head(sys.argv[1]))' https://mirror.veesp.com/
<Response [200]>
ubuntu@juju-98d295-prod-launchpad-16:~$ curl -siI https://mirror.veesp.com/
HTTP/2 200
alt-svc: h3=":443"; ma=2592000
content-type: text/html; charset=utf-8
server: Caddy
date: Tue, 26 Mar 2024 02:09:20 GMT

ubuntu@juju-98d295-prod-launchpad-16:~$ _

== mirror.its-tps.fr ==

http://launchpadlibrarian.net/721108689/mirror.its-tps.fr-archive-probe-logfile.txt

but:

ubuntu@juju-98d295-prod-launchpad-16:~$ python3 -c 'import requests, sys ; print(requests.head(sys.argv[1]))' https://mirror.its-tps.fr/
<Response [200]>
ubuntu@juju-98d295-prod-launchpad-16:~$ curl -siI https://mirror.its-tps.fr/
HTTP/2 200
alt-svc: h3=":443"; ma=2592000
content-type: text/html; charset=utf-8
server: Caddy
date: Tue, 26 Mar 2024 02:12:05 GMT

ubuntu@juju-98d295-prod-launchpad-16:~$ _

[Guruprasad (lgp171188)]

I verified this in the staging environment and it looks like something is not working in the TLS negotiation between the mirror prober client (Twisted treq) and the affected servers and it is causing the client to error out with a `<class 'twisted.web._newclient.ResponseNeverReceived'>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'tlsv1 alert internal error')]>]` exception.

[Guruprasad (lgp171188)]

I created a minimal reproducer for this issue - https://pastebin.ubuntu.com/p/szGqZpFKn8/ and noticed that this happens only when tunneling the outgoing requests through a proxy. Launchpad is using old versions of Twisted, Treq, pyOpenSSL, cryptography etc. and one or more of those could be contributing to this issue. I will try to upgrade them one by one locally and see if it helps.

[Atlas Internet (atlasinternet)]

hello. i had to disable the HTTPS from the mirrors, because the probe was throwing errors on the validation. any advance on this issue? thanks!