traceback trying to lp-login on Fedora 9

[Gene Czarcinski]

Running Fedora 9 with bzr-1.5-2 "Bazaar encountered an internal error" attempting to do "bzr lp-login gene-czarc

This works fine on Fedora 8 with bzr-1.3.1-8.

bzr --version
Bazaar (bzr) 1.5
  Python interpreter: /usr/bin/python 2.5.1
  Python standard library: /usr/lib64/python2.5
  bzrlib: /usr/lib64/python2.5/site-packages/bzrlib
  Bazaar configuration: /home/gc/.bazaar
  Bazaar log file: /home/gc/.bzr.log

Copyright 2005, 2006, 2007, 2008 Canonical Ltd.
http://bazaar-vcs.org/

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

[James Westby (james-w)]

Hi,

On Debian based systems this problem can be solved/worked
around by installing the ca-certificates package, which includes
the certificate of launchpad's CA.

Is there a similar package that can be installed on Fedora?

Thanks,

James

[Gene Czarcinski (gene-czarc)]

James, thanks for your comment ... you pointed me in the right direction.

This really looks like a Fedora problem rather than bzr but I am leaving this report open for now. I have reported this problem to Fedora's bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=454126

For now, I am circumventing the problem by un-installing python-pycurl but this is not a good solution ... I like debian's solution of providing common ca-certificates.

[Toshio Kuratomi (toshio)]

There is no similar package for Fedora in Fedora 8 or 9. There is also no such package in Red Hat Enterprise Linux or CentOS 4 or 5. There may be such a package in Fedora 10 but I have not looked closely at that package yet.

This seems to be a very valid Bazaar bug.

1) Traceback due to a missing certificate is definitely a bug.
2) If bazaar really needs to guarantee the identity of the server, then it needs to check this in all circumstances, not just when pycurl is installed.
3) If bazaar doesn't actually need to guarantee the identity of the server, then the lack of a verifiable server certificate should be caught (or anticipated) and bazaar should fallback to contacting the server without checking.

[Gene Czarcinski (gene-czarc)]

The more I look, the uglier this gets.

1. To me, it appears that bzr's interface for verifying a cert is to use pycurl and this was true in bzr-1.3.1. Maybe, the presents/absense of pycurl is bzr's "switch" to tell it whether you want cert verification of not .. I can't find any documentation that says anything about this.

2. If certs are always to be verified, then bzr should complain when pycurl is not installed and this would be a bzr bug.

Note: It does not help that I know very little about python and going through python code or C code for python is somewhat difficult.

So, just what is doing the CA verification and issuing the error message?

3. It is not pycurl! I loaded up the pycurl source and found a small program which can be used to invoke it ...
  ./python/curl/__init__.py
Running this with https://launchpad.net I get the same error:
pycurl.error: (60, 'Peer certificate cannot be authenticated ...

This also says that the problem is not bzr.

4. One of the comments above says that, on debian, just install the ca-certificates package. Well, from what I can see, debian and Fedora put the CA certificates in very different locations:

For debian: /usr/share/ca-certificates/ with separate files for each certificate
For Fedora: /etc/pki/tls/certs/ca-bundle.crt as a single file.

Well, the claim is that debian works. I suppose that I could install debian and test this but that seems like a lot of work for me. Does anyone claim to have first-hand experience that bzr lp-login xxx works on debian with pycurl installed?

Does anyone have any "proof" that certificate authentication works under Fedora?

To me, this looks like this is "below" pycurl. In may be in python's urllib or even below that (openssl?). I am trying to find out where that bloody message is being issued from ... so far, no luck.

[Gene Czarcinski (gene-czarc)]

The message comes from curl (libcurl) which python-pycurl interfaces to.

The curl packages does know where the CA certs are located and that it is "bundled".

To me, this says that the CA cert is missing on Fedora.

I am leaving this bug open because there is still the question if bzr should run without pycurl installed.

[Vincent Ladeuil (vila)]

The actual state is rather messy. bzr has two http client implementations one is pycurl based the other urllib based.

pycurl can verify certificates if the underlying libcurl can find the CA certificates.
urllib can't, so far, and just implement the encrypting of the session.

This is actually worked on but at a slow pace.

As you found out there are roughly three parts involved:
- the http implementation support (pycurl or urllib),
- the ssl library
- the CA certificates

Depending on the distro you use the ssl library knows about the CA certificates *or not*.

On windows the situation where even worse as there is no package at all for the CA certificates :)

But that may help you use the following work-around:
- for the pycurl implementation you can specify a CA bundle with CURL_CA_BUNDLE environment variable.

Finally, today, you can use https with either urllib or pycurl and get an encrypted session.

Some people don't care about certificate verification so using urllib is fine.
For those who prefer to verify certificates, pycurl is the way to go, but in that case pycurl (or more generally the underlying libcurl and even sometimes the underlying ssl lib) should be configured with valid CA certificates).

For distributions where CA certificates are directly available, but pycurl not configured to use them transparently, the work-around above is the way to go.

All the above problems have to be addressed in the future bzr urllib implementation for which progress can be watched at lp:~/vila/bzr/https

[James Westby (james-w)]

On Sat, 2008-07-05 at 04:28 +0000, Gene Czarcinski wrote:
> The more I look, the uglier this gets.
>
> 1. To me, it appears that bzr's interface for verifying a cert is to use
> pycurl and this was true in bzr-1.3.1. Maybe, the presents/absense of
> pycurl is bzr's "switch" to tell it whether you want cert verification
> of not .. I can't find any documentation that says anything about this.
>

As vila explained this is more of an implementation detail, as there are
two implementations, one that checks CA certificates, but doesn't have
a nice interface to control that, and the other that doesn't check them
at all yet.

> 2. If certs are always to be verified, then bzr should complain when
> pycurl is not installed and this would be a bzr bug.
>

I believe vila's plan is to make urllib verify them, but provide a
way to allow unverified certificates.

> 4. One of the comments above says that, on debian, just install the ca-
> certificates package. Well, from what I can see, debian and Fedora put
> the CA certificates in very different locations:
>
> For debian: /usr/share/ca-certificates/ with separate files for each certificate
> For Fedora: /etc/pki/tls/certs/ca-bundle.crt as a single file.
>

debian actually symlinks those certificates in to /etc/ssl/certs/ at
install time, so it's more complicated than that. The reason for this
is that the admin can control what is accepted and what isn't by
controlling those symlinks.

> Well, the claim is that debian works. I suppose that I could install
> debian and test this but that seems like a lot of work for me. Does
> anyone claim to have first-hand experience that bzr lp-login xxx works
> on debian with pycurl installed?

Yes, I know this works.

Thanks,

James

[Gene Czarcinski (gene-czarc)]

OK, first the environment variable ... this does not work!

export CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
set
<deleted>
CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
<deleted>
bzr launchpad-login gene-czarc
bzr: ERROR: pycurl.error: (60, 'Peer certificate cannot be authenticated with known CA certificates')

so much for that??
-------------------------------------------------------------------------------------------------
I have done a bunch of tests which are pointing me to curl/libcurl:
https://bugzilla.redhat.com/show_bug.cgi?id=454151

environment: x86_64, Fedora 9 with updates ~ 7/4/08.
----------------------------------------------------
Here are the last tests I ran:

First, I installed "curl" to have a nice command line
interface.

1. run: curl --cacert /etc/pki/tls/certs/ca-bundle.crt https://www.redhat.com
or run: curl https://www.redhat.com
both results: html

2. run: curl --cacert /etc/pki/tls/certs/ca-bundle.crt https://launchpad.net
or run: curl https://launchpad.net
both results: curl: (60) Peer certificate cannot be authenticated ...
-----------------------------------------------------
get new cacert.pem from http://curl.haxx.se/docs/caextract.html
[curl website] ... I was assuming that a cert was missing
-----------------------------------------------------
3. run: curl --cacert ./cacert https://www.redhat.com
result: html

4. run: curl --cacert ./cacert https://launchpad.net
result: html ... looking good!
-----------------------------------------------------
This proves something but I am not sure what because if I copy the new
cacert.pem into /etc/pki/tls/certs/ and try to use it from there, redhat works
but launchpad does not.

OK, one more test: copy /etc/pki/tls/certs/ca-bundle.crt to ./ [this is openssl's]

run curl --cacert ./ca-bundle.crt https://launchpad.net
result: html

Well, to me this proves that the correct cert is installed in Fedora 9's bundle.
-------------------------------------------------------
I have not looked into the libcurl code itself.

However, from the curl.spec in the src.rpm, the configuration looks good:
  --with-ca-bundle=%{_sysconfdir}/pki/tls/certs/ca-bundle.crt

but the there is this:

run: curl-config --ca
result: @CURL_CA_BUNDLE@
--------------------------------------------------------

Any suggestions/test will be appreciated.

[Vincent Ladeuil (vila)]

>>>>> "Gene" == Gene Czarcinski <email address hidden> writes:

    Gene> OK, first the environment variable ... this does not work!
    Gene> export CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
    Gene> set
    Gene> <deleted>
    Gene> CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
    Gene> <deleted>
    Gene> bzr launchpad-login gene-czarc

    Gene> bzr: ERROR: pycurl.error: (60, 'Peer certificate cannot
    Gene> be authenticated with known CA certificates')

I'd guess launchpad uses a certificate that is used by a
Certificate Authority (CA) which is not described in
/etc/pki/tls/certs/ca-bundle.crt.

If you can connect to launchpad.net from a browser, getting the
CA should be easy, then you can check that it appears in
/etc/pki/tls/certs/ca-bundle.crt.

A work-around will then be to get that CA certificate, put it in
a file in the right format and gives *that* file path to
CURL_CA_BUNDLE.

The underlying problem being that the responsibility of
distributing the CA certificates is viewed differently between
distributions and browsers (which are the main users).

[Gene Czarcinski (gene-czarc)]

 Let me try again.

In my message above (https://bugs.launchpad.net/bzr/+bug/245634/comments/9), I mention "cacert.pem" which I downloaded from the curl/libcurl website. I also mention "ca-bundle.crt" which is what is shipped with Fedora 9.

If I run: "curl --cacert ./cacert.pem https://launchpad.net >xxx"
it works and there is html in the xxx file.

If I copy ca-bundle.crt to my local directory and run: "curl --cacert ./ca-bundle.crt https://launchpad.net >xxx"
it also works and there is html in the xxx file.

*** This leads me to conclude that the bundle shipped with Fedora 9 does include the correct CA certificate ***

On the other hand, if I run: "curl https://launchpad.net >xxx"

or "curl --cacert /etc/pki/tls/certs/ca-bundle.crt https://launchpad.net >xxx"

it DOES NOT work and I get: curl: (60) Peer certificate cannot be authenticated with known CA certificates

Furthermore, if I run: "curl -v --cacert ./ca-bundle.crt https://launchpad.net >xxx"
in an attempt to get more diagnostic information, it ALSO DOES NOT WORK! Here I get:

* About to connect() to launchpad.net port 443 (#0)
* Trying 91.189.90.211... connected
* Connected to launchpad.net (91.189.90.211) port 443 (#0)
* CAfile: ./ca-bundle.crt
  CApath: none
* Bad certificate received. Subject = 'CN=launchpad.net,OU=Domain Control Validated,O=launchpad.net', Issuer = '<email address hidden>,CN=Starfield Secure Certification Authority,OU=http://www.starfieldtech.com/repository,O="Starfield Technologies, Inc.",L=Scottsdale,ST=Arizona,C=US'
* NSS error -8179
* Closing connection #0
* Peer certificate cannot be authenticated with known CA certificates

curl: (60) Peer certificate cannot be authenticated with known CA certificates

Just adding "-v" should not make the command fail!

[Toshio Kuratomi (toshio)]

You're describing two bugs here:

One is the failure of bazaar to provide a means to turn "fail-on-unvalidated-server-certificate" on and off. That discussion should continue here.

The other is the problems you're having with curl/libcurl/pycurl on Fedora. Those should probably be discussed on the bug you opened in Fedora's bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=454151

And just FYI, I've tried to reproduce the curl errors you are seeing and have failed to do so. I'll add more information on that to the bz.redhat.com bug.

[James Westby (james-w)]

On Tue, 2008-07-08 at 20:08 +0000, Gene Czarcinski wrote:
> Let me try again.

Wow, that's pretty strange.

Have you filed a bug report against curl in Fedora? I think
they may be better placed to help you.

Thanks,

James