Hey @paelzer, reply to your comment #8

I created an HTTPS-based proxy for testing using the following methods.

openssl req -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -out ca.crt -keyout ca.key -subj "/C=CN/ST=BJ/O=STS/CN=CA"
DOMAIN=quqi.com
openssl genrsa -out $DOMAIN.key
openssl req -new -key $DOMAIN.key -out $DOMAIN.csr -subj "/C=CN/ST=BJ/O=STS/CN=$DOMAIN"
openssl x509 -req -in $DOMAIN.csr -out $DOMAIN.crt -sha256 -CA ca.crt -CAkey ca.key -CAcreateserial -days 3650

sudo apt-get install openssl libssl-dev ssl-cert squid-openssl -y

$ cat /etc/squid/squid.conf
#htpasswd  -c /etc/squid/passwd quqi99
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwd
acl auth_user proxy_auth REQUIRED
http_access allow auth_user
http_access allow all
via off
forwarded_for delete
http_port 127.0.0.1:3128
https_port 127.0.0.1:3129 cert=/etc/squid/cert/quqi.com.crt key=/etc/squid/cert/quqi.com.key

Of course, the customer did not use squid to create an HTTPS-based proxy, they just used an HTTP-based proxy, but they have TLS terminated NLB in front of squid, so the traffic getting to the actual squid instance is unencrypted between the NLB and the ec2 instance. I don't spend time researching how NLB is set up, because whether it has NLB or not, it's an HTTPS-based proxy, so I just used squid for a quick HTTPS-based proxy testing env.

Then we can easily reproduce the problem with urllib, urllib3, requests, nodejs and java

1, urllib version - https://paste.ubuntu.com/p/WPkBKwKKVX/
2, urllib3 version - https://paste.ubuntu.com/p/Pjxq7h8pJ9/
   urllib3 version throws ProxySchemeUnsupported, ProxySchemeUnsupported means tls-over-tls is not supported.
3, requests version - https://paste.ubuntu.com/p/82zMNjdtz7/
   requests version throws ProxySchemeUnsupported as well, TLS in TLS requires support for the 'ssl' module
4, I also tried nodejs and java versions, they don't work, but I don't record test program

However, using curl always works, eg:

1, test http port 3128, it works

$ curl -x http://quqi99:password@127.0.0.1:3128 https://api.snapcraft.io
snapcraft.io store API service - Copyright 2018-2022 Canonical.

2, test https port 3129, it works as well

$ curl --resolve quqi.com:3129:127.0.0.1 --proxy-cacert /etc/squid/cert/ca.crt -x https://quqi99:<email address hidden>:3129 https://api.snapcraft.io
snapcraft.io store API service - Copyright 2018-2022 Canonical.

so then I tested pycurl, and it worked well as well - https://paste.ubuntu.com/p/j6TPBwSC69/

then I found an upstream bug[1]. Normally, the client will use HTTP CONNECT to set up HTTP TUNNEL between the client and proxy, then the proxy set up TCP TUNNEL between the proxy and target, and the tls/https traffic from the client to target will go through these two TUNNELS(HTTP TUNNEL + TCP TUNNEL). but now it seems all python modules(urllib, urllib3, requests) do not support to tunnel one tls/https connection through another tls/https connection. curl doesn't use the python SSL module, the SSL module it is using may support tls-over-tls feature so it works, and curl supports tls over tls since 7.52.0 according to the page - https://curl.se/libcurl/c/CURLOPT_PROXY.html

That's all my findings for your reference so far.

[1] https://bugs.python.org/issue29610