firefox crashes on a FIPS enabled machine

[Vineetha Kamath]

[IMPACT]
firefox is not a FIPS certified library. firefox uses bundled nss and on a machine running FIPS enabled kernel, nss by default goes into FIPS mode if /proc/sys/crypto/fips_enabled=1. This is an untested configuration and since firefox with bundled nss is not a certified library we propose disabling reading the 'fips_enabled' flag and therefore switching the library automatically into FIPS mode. A FIPS customer reported firefox crash on a FIPS enabled system and strace showed it was repeatedly trying to read the fips_enabled flag from the bundled nss before crashing.

The proposed patch disables reading the /proc/sys/crypto/fips_enabled flag. The users of the library however can force nss into FIPS mode via an environment variable. We plan to leave it as is so as not to regress existing users who may be using it.

The issue impacts firefox versions in eoan, disco, bionic and xenial.

lsb_release -rd
Description: Ubuntu Eoan Ermine (development branch)
Release: 19.10

Version: 2:3.45-1ubuntu1

lsb_release -rd
Description: Ubuntu Disco Dingo
Release: 19.04

Version: 2:3.42-1ubuntu2

lsb_release -rd
Description: Ubuntu Bionic Beaver
Release: 18.04

Version: 2:3.35-2ubuntu2.3

lsb_release -rd
Description: Ubuntu 16.04.3 LTS
Release: 16.04

Version: 2:3.28.4-0ubuntu0.16.04

[FIX]
This fix proposes to disable bundled nss in firefox reading proc/sys/crypto/fips_enabled. We only want fips certified modules reading this file and running in fips mode. firefox is not one of our fips certified modules, so should not be reading this along with our fips certified modules to determine whether to run in fips mode.

Users who do want to run the library in FIPS mode can do so by using the environment variable "NSS_FIPS". We propose to leave it as is so as not to regress anyone using this. The user who is using this option should be doing so with the awareness.

[TEST]
Tested on a xenial and bionic desktop ISO running FIPS enabled kernel and in FIPS mode. With the patch fix no crashes were observed when launching firefox browser.
Without the patch fix, firefox crashes.

Tested on a xenial and bionic desktop ISO running non-FIPS generic kernel. With the patch fix, firefox worked as expected and no changes were observed.

[REGRESSION POTENTIAL]
The regression potential for this is small. A FIPS kernel is required to
create /proc/sys/crypto/fips_enabled and it is not available in the standard Ubuntu archive. For users forcing FIPS through environment variable, nothing has changed.

[Vineetha Kamath (vineetha)]

The build log and test runs for eoan build is on my test ppa
https://launchpad.net/~vineetha/+archive/ubuntu/firefox-test/+build/17525936

The build log and test runs for disco build is on my test ppa
https://launchpad.net/~vineetha/+archive/ubuntu/firefox-test/+build/17525851

The build log and test runs for bionic build is on my test ppa
https://launchpad.net/~vineetha/+archive/ubuntu/test-ppa/+build/17524983

The build log and test runs for xenial build is on my test ppa
https://launchpad.net/~vineetha/+archive/ubuntu/firefox-test/+build/17525924

[Vineetha Kamath (vineetha)]

debdiff.xenial

[Vineetha Kamath (vineetha)]

debdiff.bionic

[Vineetha Kamath (vineetha)]

debdiff.disco

[Vineetha Kamath (vineetha)]

debdiff.eoan

[David Negreira (dnegreira)]

Tested the firefox build on Xenial with FIPS enabled and disabled, it works as expected.

[David Negreira (dnegreira)]

Tested the firefox build on Bionic with FIPS enabled and disabled and it is working as expected.

[Vineetha Kamath (vineetha)]

Created attachment 9093608
firefox_nss_disable_fips_enabled_flag.patch

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36

Steps to reproduce:

On a FIPS enabled system, i.e. a system running a FIPS enabled kernel, /proc/sys/crypto/fips_enabled is set to 1. The libraries that are FIPS certified reads this flag to decide if they have to operate in FIPS mode. Firefox's nss bundled code by default reads this flag. Firefox is not one of FIPS certified libraries and should not be reading this flag.

A bug has been filed against Ubuntu firefox package here - https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1843044

Actual results:

On a FIPS enabled system. firefox crashes while starting up. An strace showed that it was repeatedly reading the flag before the crash.

Expected results:

Firefox and its associated nss bundled code are not FIPS certified and hence should not be reading the /proc/sys/crypto/fips_enabled flag. I propose to disable reading that flag.

[Vineetha Kamath (vineetha)]

After applying the patch, no crash was observed on a FIPS enabled system.

[Release-mgmt-account-bot (release-mgmt-account-bot)]

[Bugbug](https://github.com/mozilla/bugbug/) thinks this bug should belong to this component, but please revert this change in case of error.

[Jjones-g (jjones-g)]

Bob, as this is related to NSS and Firefox's FIPS mode, can you take this one?

Reporter: I will note that the patch as-is would need to be reworked to determine whether NSS was built in FIPS mode, rather than commenting out the reads.

[Rrelyea (rrelyea)]

Do not apply this patch as written. Firefox may not be FIPS validated, but NSS itself is. If you want a distribution free of NSS reading the flag, please create a new #define and build environment variable. Reading the FIPS flag on Linux should be default behavior (at least if the NSS FIPS value has been enabled).

This code was specifically added to NSS would automatically go into FIPS mode on systems that are FIPS enabled.

[Jjones-g (jjones-g)]

Comment on attachment 9093608
firefox_nss_disable_fips_enabled_flag.patch

As both above comments said, this would need to be rewritten to make use of our FIPS compile-time options, not unconditionally compile-out FIPS mode, as NSS is absolutely used in FIPS compliant ways regularly.

[Bryan Quigley (bryanquigley)]

Did anyone test trying to get Firefox into FIPS mode (I know that NSS/Firefox hasn't been validated for Ubuntu) - https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/FIPS_Mode_-_an_explanation

[Bryan Quigley (bryanquigley)]

Found the original bug enabling this change here: https://bugzilla.mozilla.org/show_bug.cgi?id=1531267

I tried to enable FIPS on 66/70/73 Nightly and could not get Firefox's Enable FIPS button to work on Ubuntu. Latest Nightly still crashes on Ubuntu. Also tried disabling TLS1.3 and all ciphers except for 1 that's on the FIPS list - still crashes.

[Firefox-3 (firefox-3)]

Alternatively to patching this, what is the modern way to enable FIPS in Firefox? I found these instructions: https://support.mozilla.org/en-US/kb/Configuring%20Firefox%20for%20FIPS%20140-2 but no matter what I do I can't get FIPS enabled - nor will "Enable FIPS" not be grayed out in Security Devices.

[Jjones-g (jjones-g)]

If NSS was built with the FIPS options enabled (`./build.sh --enable-fips`), and is then used with a database set to FIPS mode (`modutil -fips true -dbdir dir`), then Firefox should automatically also go into FIPS mode.

[Nivedita Singhvi (niveditasinghvi)]

We have multiple reports of the latest Firefox not working with
FIPS due to the above ongoing, so we would like to determine
how to fix this as a priority.

We are trying to determine what the best approach to take is
given the Mozilla team's direction to keep the default behavior
of the nss library the same (checking the fips_enabled flag),
and behaving differently if built with an env variable, and
not go with Vineetha's submitted patch.

To get FF to FIPS mode, I suspect on Bionic we will need this
as well:

Bug 1531267:
"FIPS mode should be enabled automatically if the system is in FIPS mode"
Fix in nss version: 3.43
(On Linux, even if /proc/sys/crypto/fips_enabled is 1, one needs to enable database's FIPS mode with modutil.)

On Bionic the nss package version was 2:3.35, which does not
have that fix (Eoan has 2:3.45).

[Victor Tapia (vtapia)]

Created attachment 9120250
nss-stop-fips-query-when-disabled.patch

I'm attaching a patch that uses NSS_FIPS_DISABLED so /proc/sys/crypto/fips_enabled won't be checked when NSS is not built in FIPS mode (without --enable-fips).

[Victor Tapia (vtapia)]

Created attachment 9120251
nss-stop-fips-query-when-disabled.patch

[Dkeeler (dkeeler)]

Victor, are you still interested in working on this bug? Note that we use phabricator to do code review: https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html
Also note that you'll be making changes to nss (https://hg.mozilla.org/projects/nss/), not mozilla-central directly.

(it looks like fixing this bug will address at least some of the failures from bug 1544511)

[Victor Tapia (vtapia)]

Sure, I'm not familiar with the process but will give it a try. Sorry for the late response btw, I've been afk :)

[Victor Tapia (vtapia)]

Created attachment 9123528
Bug 1582169 - Disable reading /proc/sys/crypto/fips_enabled if FIPS is not enabled on build

[Jjones-g (jjones-g)]

Bob, can you take a look at this review when possible? It's pretty simple conditional compilation for FIPS.

[Rrelyea (rrelyea)]

The new patch looks fine, I've r+'ed it. since it's close to the end of the day, I'll push the change later.

bob

[Jjones-g (jjones-g)]

https://hg.mozilla.org/projects/nss/rev/55ba54adfcaea2f984a999a511eec5047462eb57

[Olivier Tilloy (osomon)]

Any chance this fix can be cherry-picked to the firefox 74 branch?

[Jjones-g (jjones-g)]

(In reply to Olivier Tilloy from comment #16)
> Any chance this fix can be cherry-picked to the firefox 74 branch?

It certainly _can_; I don't have any other current ride-along plans for a NSS 3.50 point release, but I'd be happy to add this to the to-do list if we make one. Since on Linux NSS is installed as a system library, we have to release it separately but in lock-step.

If you feel this is sufficient to warrant a point release on its own, could you give me a brief synopsis of why? Thanks!

[Olivier Tilloy (osomon)]

https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1843044 is a downstream (Ubuntu) bug report describing how firefox crashes with a FIPS-enabled kernel (and this is what prompted Victor to contribute this patch).

Given the nature of the problem (a crash), it would be good to have the patch in firefox as early as possible (but we can certainly cherry-pick it and apply it as a distro-patch if it's not making it to firefox 74).

[Bryan Quigley (bryanquigley)]

This was fixed in Firefox 74/75.