test failures related to CA verification

[Vincent Ladeuil]

Spin-off from bug #614713 to focus on the CA verification failure encountered on gentoo with curl using nss (copy/pasted from https://bugs.launchpad.net/bzr/+bug/614713/comments/15 for easier reference):

What about the NSS problem, particularly as you fail to reproduce it? Shall I open a new bug report for that one? How is verification of the CA intended to work?

I've dug a bit deeper into the code and got this backtrace:
#0 CERT_FindCertIssuer at certvfy.c:276
#1 cert_VerifyCertChainOld at certvfy.c:640
#2 cert_VerifyCertChain at certvfy.c:885
#3 CERT_VerifyCertChain at certvfy.c:894
#4 CERT_VerifyCert at certvfy.c:1490
#5 CERT_VerifyCertNow at certvfy.c:1541
#6 SSL_AuthCertificate at sslauth.c:261
#7 nss_auth_cert_hook at nss.c:635
#8 ssl3_HandleCertificate at ssl3con.c:7902
#9 ssl3_HandleHandshakeMessage at ssl3con.c:8601
#10 ssl3_HandleHandshake at ssl3con.c:8725
#11 ssl3_HandleRecord at ssl3con.c:9064
#12 ssl3_GatherCompleteHandshake at ssl3gthr.c:209
#13 ssl_GatherRecord1stHandshake at sslcon.c:1258
#14 ssl_Do1stHandshake at sslsecur.c:151
#15 SSL_ForceHandshake at sslsecur.c:407
#16 Curl_nss_connect at nss.c:1382
#17 Curl_ssl_connect at sslgen.c:199
#18 Curl_http_connect at http.c:1307
#19 Curl_protocol_connect at url.c:3328
#20 Curl_setup_conn at url.c:5057
#21 Curl_async_resolved at hostasyn.c:145
#22 connect_host at transfer.c:1992
#23 Curl_do_perform at transfer.c:2122
#24 Curl_perform at transfer.c:2268
#25 curl_easy_perform at easy.c:553
#26 do_curl_perform at src/pycurl.c:1024
#27 call_function at Python/ceval.c:3997

That's the place where NSS decides that it does not know the issuer and therefore won't trust the cert either.
http://mxr.mozilla.org/mozilla-central/source/security/nss/lib/certhigh/certvfy.c#276
At that point the certificate in question (CERTCertificate *cert) contains the following data:
subjectName="E=https_server@locahost,CN=127.0.0.1,OU=https server,O=Testing Ltd,L=LocalHost,ST=Internet,C=LH",
<email address hidden>,CN=Master of certificates,OU=VCS,O=Distributed,L=Bazaar,ST=Internet,C=BZ"
Using strace, I see no access to any list of trusted CA certificates on disk, but I might have somehow missed that.

It would be nice to know if your code reaches the location indicated by the backtrace even if you fail to reproduce this issue with nss. If the call to CERT_FindCertIssuer succeeds for you (i.e. doesn't return NULL), we would have to investigate behaviour inside that function more closely. If your function returns NULL as well, the difference must be somewhere after that, and if your system doesn't even reach that function, we would have to look for differences somewhere deeper (i.e. called earlier) in the call stack.

[Martin von Gagern (gagern)]

Digging somewhat deeper, it appears that the CA file cannot be loaded. strace shows a stat to bzr.dev/bzrlib/tests/ssl_certs/ca.crt but no open. Stepping through the curl interface to nss, it appears that curl tries to load libnsspem.so but fails to do so:
https://github.com/bagder/curl/blob/curl-7_21_7/lib/nss.c#L1195

Unfortunately, the warning curl emits at that location doesn't make it to the console or log. Otherwise we would have read this message:
WARNING: failed to load NSS PEM library libnsspem.so. Using OpenSSL PEM certificates will not work.

As that module cannot be loaded, loading the certificate will fail as well:
https://github.com/bagder/curl/blob/curl-7_21_7/lib/nss.c#L394

That's the reason the testing ca certificate will not be available as a trust root. In Fact I find no "libnsspem.so" on my system. Don't know why (yet), but please compare to your systems where this selftest passes.

Btw, in https://bugs.launchpad.net/bzr/+bug/614713/comments/16 Vincent Ladeuil wrote:
> The code use self.cabundle to set pycurl.CAINFO (originally for windows,
> later on for tests too). But reading the doc now, I wonder if
> CURLOPT_ISSUERCERT (aka pycurl.ISSUERCERT) should be used instead for *tests*
> (I think windows still needs to use CAINFO but I may be wrong).

I believe that CAINFO is still the correct thing, even for tests, and we should not switch to ISSUERCERT. The way I read it, CAINFO tells curl: "These are the CA roots I trust, try to verify against any of these". So it accepts a bundle and will choose among them. ISSUERCERT tells it "I definitely want this single cert to bee in the chain". It makes no statement about certs above the given one in the chain. I guess ISSUERCERT is a special case, not likely encountered in the wild, not supported by the bzr command line tool, and should therefore be avoided. CAINFO on the other hand mimics common setups with a set of root certificates, so it's the one we should use for testing as well.

[Martin von Gagern (gagern)]

This issue is not specific to bzr. Filed https://bugs.gentoo.org/show_bug.cgi?id=380119 for curl on gentoo.