fingerprint SSL check does not work behind a proxy

[Till Kamppeter]

If you have a printer for which a signed binary driver is available via OpenPrinting (newer Epson inkjets) then the download of the signature key file does not work and so the driver installation does not complete.

See this mail from Olaf Meeuwissen from Avasys (they make drivers for Epson):

----------
Hi Till,

Saito-san and I have been looking at how well the automatic download
works when behind a proxy. Short story: it doesn't. The long story
follows below.

Till Kamppeter <email address hidden> writes:

> > for testing the automatic driver download on current Natty [...]
> > apply the patch attached to my previous mail [...]
> > 0001-Activated-and-fixed-testing-mode-for-device-ID-drive.patch
We upgraded Natty on 2011-03-03 and applied your patch to
system-config-printer.

> > cupsctl FileDevice=yes
> > cd /usr/share/system-config-printer/
> > python newprinter.py --setup-printer=file:/tmp/printout
> > --devid="MFG:Epson;MDL:Stylus NX110;"
That's what we did. We've been looking at jockey debug logs, wireshark
network traffic captures and the jockey code trying to figure out where
things went wrong. According to the logs, the query of the OpenPrinting
DB went just fine. It's getting the fingerprint that bombs. Using wget
to fetch the fingerprint in the same environment works fine.
>From the jockey debug logs:

  2011-03-03 13:04:49,904 DEBUG: Querying openprinting.org database...
  2011-03-03 13:04:49,904 DEBUG: ... querying for MFG:Epson;MDL:Stylus NX110;
  2011-03-03 13:04:56,203 DEBUG: OpenPrintingDriverDB: driver driver/epson-stylus-nx110-series info: {'nonfreesoftware': True, 'name': 'epson-stylus-nx110-series', 'license': 'LGPL and SEIKO EPSON CORPORATION SOFTWARE LICENSE AGREEMENT', 'url': 'http://avasys.jp/english/linux_e/', 'thirdpartysupplied': False, 'supportcontacts': [{'url': 'http://avasys.jp/english/linux_e/contact/', 'name': 'AVASYS Corporation', 'level': 'voluntary'}], 'recommended': True, 'functionality': {'text': '100', 'lineart': '100', 'photo': '100', 'graphics': '100'}, 'manufacturersupplied': True, 'licensetext': u'License agreement fo..', 'supplier': 'Seiko Epson Corporation', 'freesoftware': False, 'packages': {'i386': {'epson-inkjet-printer-stylus-nx110-series_1.0.0-1lsb3.2_i386.deb': {'url': 'http://linux.avasys.jp/drivers/lsb/epson-inkjet/stable/debian/dists/lsb3.2/main/binary-i386/epson-inkjet-printer-stylus-nx110-series_1.0.0-1lsb3.2_i386.deb', 'pkgsys': 'deb', 'repositories': {'apt': 'deb http://linux.avasys.jp/drivers/lsb/epson-inkjet/stable/debian/ lsb3.2 main'}, 'version': '1.0.0', 'fingerprint': 'https://linux.avasys.jp/drivers/lsb/epson-inkjet/key/fingerprint', 'release': '1', 'realversion': '1.0.0'}}}, 'shortdescription': 'Epson Inkjet Printer Driver for Linux', 'patents': False}
  2011-03-03 13:04:56,252 WARNING: https://linux.avasys.jp/drivers/lsb/epson-inkjet/key/fingerprint has invalid certificate, ignoring driver: Host 172.xx.xx.xx:8080 returned an invalid certificate (_ssl.c:499: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol)

  2011-03-03 13:04:56,252 DEBUG: Ignoring driver as it does not have a valid GPG fingerprint
  2011-03-03 13:04:56,253 DEBUG: openprinting.org database query finished

We've chased the WARNING statement down to the connect method of
_CertValidatingHTTPSConnection in jockey's verified_https.py

   def connect(self):
        sock = socket.create_connection((self.host, self.port))
        self.sock = ssl.wrap_socket(sock, cert_reqs=ssl.CERT_REQUIRED,
                                          ca_certs=self.ca_certs)
        cert = self.sock.getpeercert()
        hostname = self.host.split(':', 0)[0]
        if not self._validate_certificate_hostname(cert, hostname):
            raise InvalidCertificateException(hostname, cert,
                                              'hostname mismatch')

The ssl.wrap_socket raises an ssl.SSLError. What puzzles us is that
this method creates a connection to the _proxy_ rather than to the host
where the fingerprint resides. The wireshark capture for a wget went
straight to the latter host and negotiated TLSv1 with the proxy on the
fly.

We've also played with passing different ssl_version parameters to
ssl.wrap_socket but that didn't help. We're not sure whether this
method negotiates a usable version but if it doesn't that something that
probably needs fixing as well.

Hope this helps,
----------

Can this get fixed in Natty? Thanks.

[Martin Pitt (pitti)]

I reproduced this with squid.

[Martin Pitt (pitti)]

Note taking for myself. Reproducer:

  sudo apt-get install squid

in trunk:
  sudo http_proxy="http://localhost:3128/" https_proxy="https://localhost:3128/" PYTHONPATH=. backend/jockey-backend --debug

trigger Epson driver search:
  dbus-send --print-reply --dest=com.ubuntu.DeviceDriver /GUI com.ubuntu.DeviceDriver.search_driver string:"printer_deviceid:MFG:EPSON;MDL:Epson ME 320"

[Martin Pitt (pitti)]

Faster reproducer:

# working case without proxy:
$ python -c 'import jockey.detection; from jockey.oslib import OSLib; OSLib.inst = OSLib(); print jockey.detection.OpenPrintingDriverDB._check_get_fingerprint("https://linux.avasys.jp/drivers/lsb/epson-inkjet/key/fingerprint")'
E522 0FB7 014D 0FBD A50D FC2B E5E8 6C00 8AA6 5D56

# failing case with proxy:
$ http_proxy="http://localhost:3128/" https_proxy="https://localhost:3128/" python -c 'import jockey.detection; from jockey.oslib import OSLib; OSLib.inst = OSLib(); print jockey.detection.OpenPrintingDriverDB._check_get_fingerprint("https://linux.avasys.jp/drivers/lsb/epson-inkjet/key/fingerprint")'
WARNING:root:https://linux.avasys.jp/drivers/lsb/epson-inkjet/key/fingerprint has invalid certificate, ignoring driver: Host localhost:3128 returned an invalid certificate (_ssl.c:499: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol)

[Martin Pitt (pitti)]

I added a bunch of test cases, restructured the code a bit, and now switched the implementation to use pycurl instead of jockey's verified_https.py urllib2 handler. I didn't see a way to make this work properly with HTTPS proxies, but with pycurl it works just fine:

$ http_proxy="http://localhost:3128/" https_proxy="https://localhost:3128/" python -c 'import jockey.detection; from jockey.oslib import OSLib; OSLib.inst = OSLib(); print jockey.detection.OpenPrintingDriverDB._check_get_fingerprint("https://linux.avasys.jp/drivers/lsb/epson-inkjet/key/fingerprint")'
E522 0FB7 014D 0FBD A50D FC2B E5E8 6C00 8AA6 5D56

Squid log confirms that the request was proxied and reached the right destination:

1299594547.873 1444 127.0.0.1 TCP_MISS/200 4340 CONNECT linux.avasys.jp:443 - DIRECT/202.41.220.57 -

I'll upload this in a bit. I'd appreciate if you could test this again once it's in Natty, to confirm that it also really works in your setup. Thanks!

[Martin Pitt (pitti)]

Olaf, just saw that you weren't subscribed to this bug report, Till proxied it for you.

I'll upload this in a bit. I'd appreciate if you could test this again once it's in Natty, to confirm that it also really works in your setup. Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package jockey - 0.9.2-0ubuntu1

---------------
jockey (0.9.2-0ubuntu1) natty; urgency=low

  * New upstream bug fix release:
    - Add X.org video driver ABI checking. (This was already cherrypicked
      in an earlier upload)
    - Drop our own verified_https.py and implement GPG fingerprint SSL
      checking with pycurl, which gets along with proxies. (LP: #729185)
    - Some code cleanup and more test cases.
  * debian/control: Add python-pycurl dependency.
 -- Martin Pitt <email address hidden> Tue, 08 Mar 2011 16:08:06 +0100

[Olaf Meeuwissen (olaf.meeuwissen)]

Re #5, we've tested with the fixed code and getting the GPG fingerprint worked fine for us.

Not familiar with the python-pycurl code, I've one little question. The verified_https.py code seemed aimed at adding hostname validation, something HTTPSConnection didn't do. Assuming that was done on purpose, I guess the replacement code should do the same. Does pycurl do the validation? If yes, then I think the fix is OK. If no, then it might need a bit more work.

Now that getting the GPG fingerprint works it time to make GPG proxy aware, see #733023 ;-)

[Martin Pitt (pitti)]

Hello Olaf,

Olaf Meeuwissen [2011-03-11 1:54 -0000]:
> Re #5, we've tested with the fixed code and getting the GPG fingerprint
> worked fine for us.

Nice!

> Not familiar with the python-pycurl code, I've one little question. The
> verified_https.py code seemed aimed at adding hostname validation,
> something HTTPSConnection didn't do. Assuming that was done on purpose,
> I guess the replacement code should do the same. Does pycurl do the
> validation?

Yes, it does by default:

  http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTSSLVERIFYHOST

Both the old (verified_https.py) and the new (pycurl) implementation
are covered by various test cases, amongst them one for a nonmatching
host name:

  http://bazaar.launchpad.net/~jockey-hackers/jockey/trunk/view/head:/tests/detection.py#L896