LP API broken in oneiric with python-httplib2 0.7.0-1

[Abel Deuring]

In oneiric today, someone landed python-httplib2, merged from debian (0.6.0-5 -> 0.7.0-1).
Now, all my scripts are broken:

  ...
  File "/usr/lib/python2.7/dist-packages/launchpadlib/launchpad.py", line 373, in login_anonymously
    version=version)
  File "/usr/lib/python2.7/dist-packages/launchpadlib/launchpad.py", line 198, in __init__
    credentials, service_root, cache, timeout, proxy_info, version)
  File "/usr/lib/python2.7/dist-packages/lazr/restfulclient/resource.py", line 460, in __init__
    self._wadl = self._browser.get_wadl_application(self._root_uri)
  File "/usr/lib/python2.7/dist-packages/lazr/restfulclient/_browser.py", line 299, in get_wadl_application
    response, content = self._request(url, media_type=wadl_type)
  File "/usr/lib/python2.7/dist-packages/lazr/restfulclient/_browser.py", line 242, in _request
    str(url), method=method, body=data, headers=headers)
  File "/usr/lib/python2.7/dist-packages/lazr/restfulclient/_browser.py", line 211, in _request_and_retry
    url, method=method, body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1406, in request
    (response, new_content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python2.7/dist-packages/launchpadlib/launchpad.py", line 126, in _request
    LaunchpadOAuthAwareHttp, self)._request(*args)
  File "/usr/lib/python2.7/dist-packages/lazr/restfulclient/_browser.py", line 130, in _request
    redirections, cachekey)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1188, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1123, in _conn_request
    conn.connect()
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 890, in connect
    self.disable_ssl_certificate_validation, self.ca_certs)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 76, in _ssl_wrap_socket
    cert_reqs=cert_reqs, ca_certs=ca_certs)
  File "/usr/lib/python2.7/ssl.py", line 372, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 132, in __init__
    ciphers)
ssl.SSLError: [Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib

Is it a Launchpad issue? or is python-httplib2 really broken?
please advise.

[Abel Deuring (adeuring)]

Converted from https://answers.launchpad.net/ubuntu/+source/python-httplib2/+question/161415. I don't have yet oneiric installed. Fabien Tassin, who asked the question may be able to provide more information.

[Jean-Baptiste Lallement (jibel)]

Confirmed in Oneiric and reproducible with the code below:

"""
#!/usr/bin/python

from launchpadlib.launchpad import Launchpad

launchpad = Launchpad.login_anonymously('this is a test','production')
u = launchpad.projects['ubuntu']
"""

[Martin Pool (mbp)]

fta pointed out in <https://answers.launchpad.net/ubuntu/+source/python-httplib2/+question/161415> that this is fixed by going back to 0.6.0-5

[Colin Watson (cjwatson)]

Barry, could you have a look at this, please?

[Barry Warsaw (barry)]

Confirmed with the upstream Mercurial clone.

One odd thing about our Oneiric package. It doesn't look like a proper merge of upstream. In particular the CHANGELOG in the package does not match the CHANGELOG in the upstream clone. I haven't check to see if that might have gotten committed after the 0.7.0 release though.

This is interesting because the 0.7.0 news (in the upstream branch, not in the packaging branch) says the attached. disable_ssl_certificate_validation might be a temporary workaround, though I'm not yet sure how to plumb that through launchpadlib, if it's even possible.

0.7.0
    The two major changes in this release are SSL Certificate
    checking and App Engine support. By default the certificates
    of an HTTPS connection are checked, but that can be disabled
    via disable_ssl_certificate_validation. The second change
    is that on App Engine there is a new connection object
    that utilizes the urlfetch capabilities on App Engine, including
    setting timeouts and validating certificates.

    The following issues have been addressed:

    Fixes issue 72. Always lowercase authorization header.
    Fix issue 47. Redirects that become a GET should not have a body.
    Fixes issue 19. Set Content-location on redirected HEAD requests
    Fixes issue 139. Redirect with a GET on 302 regardless of the originating method.
    Fixes issue 138. Handle unicode in headers when writing and retrieving cache entries. Who says headers have to be ASCII!
    Add certificate validation. Work initially started by Christoph Kern.
    Set a version number. Fixes issue # 135.
    Sync to latest version of socks.py
    Add gzip to the user-agent, in case we are making a request to an app engine project: http://code.google.com/appengine/kb/general.html#compression
    Uses a custom httplib shim on App Engine to wrap urlfetch, as opposed
    Add default support for optimistic concurrency on PATCH requests
    Fixes issue 126. IPv6 under various conditions would fail.
    Fixes issue 131. Handle socket.timeout's that occur during send.
    proxy support: degrade gracefully when socket.socket is unavailable

[Barry Warsaw (barry)]

@Jean-Baptiste: Can you try something for me please?

After you get the traceback from the sample code in comment #2, try it exactly the same way again. For me the second time works perfectly (no traceback). I wonder if something's getting cached but not accessed correctly the first time, whereas the second time you run it, the cached file exists and it succeeds.

I think the _ssl traceback is masking a an ENOENT IOError. See this Python issue for why I think that:

http://bugs.python.org/issue9706

and especially this comment:

http://bugs.python.org/issue9706#msg115182

If that's correct, then the bug is probably in launchpadlib.

[Jean-Baptiste Lallement (jibel)]

I've tried your suggestion, but no matter how many times I try, I always get the traceback.
I also tried removing ~/.launchpadlib and with a freshly created profile, it doesn't make any difference.

---
Ubuntu Bug Squad volunteer triager
http://wiki.ubuntu.com/BugSquad

[Barry Warsaw (barry)]

@Jean-Baptiste: thanks, I think that was pebkac on my part, so ignore that. I can reproduce it every time too.

Interestingly, when stepping through httplib2, after the call to _ssl_wrap_socket() I get this error:

SSLError(8, '_ssl.c:503: EOF occurred in violation of protocol')

Hmm... Possibly a problem with Launchpad?

[Barry Warsaw (barry)]

Okay, I think lazr.restfulclient() is simply not prepared to handle certificate validation, which is the new default in httplib2. I have a workaround, which is to pass disable_ssl_certificate_validation=True to the underlying Http object. With this, the example code works. I'll upload a temporary fix to lazr.restfulclient and create a bug task on that project.

[Martin Pool (mbp)]

Seems like there should be a follow on bug that it's vulnerable to mitm.

[Barry Warsaw (barry)]

Perhaps. The thing is, it wasn't doing cert validation before, and with this change it's still not. I think validation would be a new feature for lazr.restfulclient (albeit a useful one!).

[Robert Collins (lifeless)]

I think a new bug is essential; can you propose your existing
workaround to trunk though ?

[Barry Warsaw (barry)]

LP: 798405 is the bug.

Merge proposal for workaround submitted on trunk.

[Barry Warsaw (barry)]

ohai launchpad, please make that LP: #798405

[Martin Pool (mbp)]

Thanks. I know you're not introducing a new bug, only discovering an
existing one, but it's worth having.

[Barry Warsaw (barry)]

Unassigning myself from the lazr.restfulclient task since I cannot push to trunk.

[Francis J. Lacoste (flacoste)]

Barrry, I added you to ~lazr-developers, you should be able to commit changes directly to lazr projects again. (You lost that once you were removed from the ~launchpad team.)

Graham has merged your branch in the mean time though.

[Barry Warsaw (barry)]

On Jun 21, 2011, at 03:58 PM, Francis J. Lacoste wrote:

>Barrry, I added you to ~lazr-developers, you should be able to commit
>changes directly to lazr projects again. (You lost that once you were
>removed from the ~launchpad team.)
>
>Graham has merged your branch in the mean time though.

Thanks, and thanks!
-Barry

[Michael Terry (mterry)]

This also affects ubuntuone-couch, which uses httplib2 to talk to one.ubuntu.com. I'll work on that.

[Michael Terry (mterry)]

Barry, you said bug 798405 was the follow-on bug, but I don't see it. I get a 404 from LP going to that bug. Was there a typo?

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntuone-couch - 0.2.0-0ubuntu4

---------------
ubuntuone-couch (0.2.0-0ubuntu4) oneiric; urgency=low

  * debian/patches/no-ssl-validation.patch:
    - Disable SSL validation because it isn't currently working with
      one.ubuntu.com (LP: #797281)
 -- Michael Terry <email address hidden> Tue, 21 Jun 2011 13:52:46 -0400

[Barry Warsaw (barry)]

@Michael, nope it's there, but the bug is private.

[Roman Yepishev (rye)]

We need to get this http://code.google.com/p/httplib2/source/detail?r=adfecbabd3f9481f059aaa20e69005ff9ea20952 instead of disabling ssl validation altogether.

[Roman Yepishev (rye)]

I created a new bug report for current SSL DNS name check here - LP:839826 - patch has migrated there.

Do not disable SSL checks in your packages due to DNS checks, this is httplib2 bug.