SSL trust not system-wide

[dwmw2]

When I install a corporate CA trust root with update-ca-certificates, it doesn't seem to work everywhere. Various things like Firefox, Evolution, Chrome, etc. all fail to trust the newly-installed trusted CA.

This ought to work, and does on other distributions. In p11-kit there is a module p11-kit-trust.so which can be used as a drop-in replacement for NSS's own libnssckbi.so trust root module, but which reads from the system's configured trust setup instead of the hard-coded version.

This allows us to install the corporate CAs just once, and then file a bug against any package that *doesn't* then trust them.

See https://fedoraproject.org/wiki/Features/SharedSystemCertificates for some of the historical details from when this feature was first implemented, but this is all now supported upstream and not at all distribution-specific. There shouldn't be any significant work required; it's mostly just a case of configuring and building it to make use of this functionality. (With 'alternatives' to let you substitute p11-kit-trust.so for the original NSS libnssckbi.so, etc.)

[dwmw2 (dwmw2)]

It does seem that p11-kit-trust.so is working correctly. If I just make a symlink from libnssckbi.so to it, corporate trust installed by update-ca-certificates *does* work in Firefox.

[Christian Ehrhardt  (paelzer)]

Hi dwmw2,
thank you for your bug report and your help to make Ubuntu better.

I beg a pardon as I'm clearly not an expert on this particular area, but I try to sort out the details of this bug report to understand what has to be done.

Currently I understand this as feature request to make update-ca-certificates (almost?) all certificate users in one shot.

The current default config doesn't do that

Thanks for pointing out the links and background to this.

The answer on this thread is what I think the current state is http://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntu
and I understand and agree that to get this as "one shot accept this CA" is a valid feature-request-bug.

I happened to find various similar/related on other projects like firefox for example:
https://bugzilla.mozilla.org/show_bug.cgi?id=620373
https://bugzilla.mozilla.org/show_bug.cgi?id=449498
https://bugzilla.mozilla.org/show_bug.cgi?id=454036
There might be more for others, but it seems to fix the whole thing a Distribution would need to modify all consuming packages to agree on sort of a shared path and mechanism.

Ok, so far I was just trying to wrap my head around this a bit, I guess the next step clearly is the security Teams position on this in general - so I subscribe them for a statement.
Maybe they also know on past or existing approaches to this.

[Christian Ehrhardt  (paelzer)]

@Security Team - do you happen to know about this overall topic and could you share either whatever was the outcome of such discussions in the past or OTOH what you assert on this as a feature request would be?

[dwmw2 (dwmw2)]

The Mozilla bugs you link are a bit of a red herring. They refer to an abortive attempt by Mozilla/NSS to have a 'shared system database' in sql:/etc/pki/nssdb. The idea is that applications specify that as their NSS database and although it's obviously read-only, it automatically adds the user's database from ~/.pki/nssdb as a writeable token. This gets a step towards consistency for all NSS-using applications — but as those bugs note, not even Mozilla's own products are actually using it. You should support that anyway, but it isn't the focus of this bug.

The fix here (which has been working in Fedora for years, since you ask for existing approaches) is to replace NSS's built-in trust root module libnssckbi.so with a symlink to p11-kit-trust.so. Then you get the system's configured trust roots, instead of whatever's hard-coded into that particular instance of libnssckbi.so (and you're shipping multiple potentially different ones of those!)

[dwmw2 (dwmw2)]

I believe we need to update p11-kit to v0.23.4 to make the key pinning work correctly in the recommended configuration, by adding the CKA_NSS_MOZILLA_CA_POLICY attribute.

https://bugs.freedesktop.org/show_bug.cgi?id=99453
https://bugzilla.mozilla.org/show_bug.cgi?id=1324096

[dwmw2 (dwmw2)]

I believe NSS wants these patches backported from 3.30:
https://bugzilla.mozilla.org/show_bug.cgi?id=1334976

Firefox has its own copy of NSS which I think as of Firefox 54 should be fine.
Thunderbird also needs fixing, I think.

[dwmw2 (dwmw2)]

cf. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741005
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=704180
https://lists.freedesktop.org/archives/p11-glue/2013-June/000331.html

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[dwmw2 (dwmw2)]

Any progress on fixing this?

[Tomas Pospisek (tpo-deb)]

Wow, unified CA management would be awesome. No more fiddling around with (and forgetting to correctly install/remove certificates in) various applications (most notably in Firefox, Chromium, wget).

[Christian Ehrhardt  (paelzer)]

No progress on this yet, afaik it is just not high up on anyone's personal task list :-/

[Orion-cora (orion-cora)]

I'm trying to make use of this in Ubuntu 14.04 with p11-kit 0.23.2-5~ubuntu16.04.1, but get the following error:

# trust list
p11-kit: ca-certificates.crt: BEGIN ...: pem block before p11-kit section header
p11-kit: ca-certificates.crt: BEGIN ...: pem block before p11-kit section header

Is p11-kit just too old there?

[Andreas Hasenack (ahasenack)]

This isn't "just" a bug, it's a roadmap item in my view, as many products are affected. It needs a spec, like in the fedora case. I agree that it would be awesome to have this.

[Kevin (kvasko)]

@dwmw2

Were you able to make this work by doing this for firefox?

sudo mv /usr/lib/firefox/libnssckbi.so /usr/lib/firefox/libnssckbi.so.bak
sudo ln -s /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so /usr/lib/firefox/libnssckbi.so

https://askubuntu.com/questions/244582/add-certificate-authorities-system-wide-on-firefox/1036637#1036637

I wasn't able to get it to work. Doing the above doesn't work at all for me. I'm on 18.04.3

[Timo Aaltonen (tjaalton)]

should this be marked as something to fix in focal for the next LTS?

[dwmw2 (dwmw2)]

@kvasko yes, it works here. Are you sure that's the version of libnssckbi.so that is being used? There are lots; I've replaced them all...

[Kevin (kvasko)]

@dwmw2,

I figured out the issue. Long story short, freeipa (which is our CA), when we enroll a PC into the realm, it adds the freeIPA cert to /etc/ssl/certs/ca-certificates.crt like it should, however it also adds other information that it shouldn't.

This results in p11-kit-trust.so blowing parsing errors.

You can read the entire bug report here if you want.

https://pagure.io/freeipa/issue/8106

[Harout S. Hedeshian (harout)]

Like others, I'm manually symlinking .so files on all of my interactive hosts and hoping updates don't break it. IMO this is not a valid workaround.

@ahasenack - I understand this is a roadmap item that would ideally resolve for multiple packages, but it seems that the Mozilla products are the worst offenders at the moment. I don't see anyone requesting anything else in this bug. Would it be possible to at least resolve it for Firefox and Thunderbird? What would it take to get this looked at for the next LTS?

For now, Thunderbird needs this too (and works for me on 18.0.3 LTS):

sudo mv /usr/lib/firefox/libnssckbi.so /usr/lib/firefox/libnssckbi.so.bak
sudo ln -s /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so /usr/lib/firefox/libnssckbi.so
sudo mv /usr/lib/thunderbird/libnssckbi.so /usr/lib/thunderbird/libnssckbi.so.bak
sudo ln -s /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so /usr/lib/thunderbird/libnssckbi.so

[Timo Aaltonen (tjaalton)]

nss should have everything on focal

[Timo Aaltonen (tjaalton)]

p11-kit too

[Timo Aaltonen (tjaalton)]

according to #4 nss should still symlink libnssckbi.so to p11-kit-trust.so

[Olivier Tilloy (osomon)]

It looks like symlinking firefox and thunderbird's own copies of libnssckbi.so to the system-wide p11-kit-trust.so is the proper way to fix this bug, as far as Mozilla's products are concerned.

Before I proceed to doing this, I'd welcome comments from the security team on this approach though, as I suspect I don't understand all the implications.

(an alternative would be building firefox/thunderbird against the system-wide nss, but firefox currently requires 3.50, which isn't yet in focal, and I suspect that requirement is being bumped often, so that wouldn't really work with our distribution model)

[dwmw2 (dwmw2)]

On Thu, 2020-03-19 at 09:44 +0000, Olivier Tilloy wrote:
> It looks like symlinking firefox and thunderbird's own copies of
> libnssckbi.so to the system-wide p11-kit-trust.so is the proper way to
> fix this bug, as far as Mozilla's products are concerned.
>
> Before I proceed to doing this, I'd welcome comments from the security
> team on this approach though, as I suspect I don't understand all the
> implications.
>
> (an alternative would be building firefox/thunderbird against the
> system-wide nss, but firefox currently requires 3.50, which isn't yet in
> focal, and I suspect that requirement is being bumped often, so that
> wouldn't really work with our distribution model)

Right, don't bother trying to replace NSS just for this (although
really, having a single version of NSS on the system *would* be nice).

The interface to libnssckbi.so is a standard PKCS#11 library, and it's
perfectly reasonable to replace that in each of
firefox/thunderbird/chromium individually.

[Marc Deslauriers (mdeslaur)]

Before we switch any software to using p11-kit-trust.so, we need to fix our ca-certificates package to properly handle untrusted or blacklisted certificates. At the moment, I believe they are simply skipped when generating the contents of /usr/share/ca-certificates.

[Timo Aaltonen (tjaalton)]

so what does it require to fix ca-certificates?

[Marc Deslauriers (mdeslaur)]

Looks like Fedora substantially modified the scripts used by ca-certificates to extract untrusted and blacklisted certs. We should probably start by investigating how their package is handling this, what files they are generating, and if they are being properly handled by p11-kit-trust.

[Michael Catanzaro (mike-catanzaro)]

So for the avoidance of doubt, every independent distro has its own custom ca-certificates package with no shared history. I know Debian, Fedora, and openSUSE all have their own completely separate upstreams. Looking at what Fedora does is probably a good idea indeed, just keep in mind it has no shared history with Debian's package. I took a quick look at openSUSE's package and it looks like it has good p11-kit integration as well. Arch uses Fedora; not sure about other independent distros. They all use Mozilla's certificates, but Mozilla doesn't release a package in a way that's directly usable by distros.

Debian's ca-certificates implements certificate blacklisting by putting a ! character at the start of a line in /etc/ca-certificates.conf (which doesn't exist on other distros). Once a certificate is removed, it stays removed, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=743339 which was never fixed.

[Marc Deslauriers (mdeslaur)]

Unfortunately, the ! character at the beginning the the line in ca-certificates.conf is just for blacklisting ca certificates from being imported into the system store, it's not really a backlist that can be used by a crypto library.

[Marco Trevisan (Treviño) (3v1n0)]

Also adding SSSD here, would be easy enough to make its default PAM CA ring to point to /etc/ssl/certs/ca-certificates.crt by default (and change-able in settings) but not sure if we want to go this route as it may make SSSD documentation confusing (as it everywhere mentions /etc/sssd/pki/sssd_auth_ca_db.pem or /etc/sssd/pki/sssd_auth_ca_db.pem).

Maybe a nice way would be to provide a default sssd.conf file that explicitly set that instead of hard-coding it, so we won't break current installations.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Hasenack (ahasenack)]

Related: https://bugs.launchpad.net/ubuntu/+source/crypto-policies/+bug/1926664

(I might create a task here for crypto-policies and close the bug above)