openssl fails with out of memory messages while trying to load the FIPS provider in a non-FIPS container on a FIPS host

[Abdurrahman Alp Köken]

I wanted to try the new Ubuntu 24.04 Noble Numbat based .NET docker image and updated the base docker image in our CI pipeline to mcr.microsoft.com/dotnet/sdk:8.0-noble. However, it results in an out-of-memory exception. Based on my investigation, the exception occurs specifically when the update-ca-certificates command is executed. I can also repro the issue with ubuntu:noble image which means it's not specific to .NET docker images. It works fine with Jammy, by the way. The problem likely lies with the Noble base image rather than the .NET image. I'm not sure what changes were made between Jammy and Noble, but it appears that updating certificates consumes a lot of memory in Noble. I adjusted some memory settings in our GitLab runner, but it didn't resolve the issue. I attached all Gitlab Runner shell logs for .NET 8 Jammy, .NET 8 Noble and Ubuntu Noble images.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Xin (simonzx)]

Hi,

I hit the same issue, would like to know if there is an ETA?

[Abdurrahman Alp Köken (akoken)]

Let me narrow down the issue: the problem was resolved when I disabled FIPS on my host. Whatever the underlying cause, it only occurs when FIPS is enabled.

[Kyle Martin (ktm-rincon)]

We are seeing this as well when building an image with ubuntu:24.04 as our base. Like the commenter above, we see the issue on our FIPS system, but do not see it on our non-FIPS system. A workaround or any information is certainly appreciated.

[Kyle Martin (ktm-rincon)]

Here's a relatively minimal example:

```
$ sudo fips-mode-setup --check
FIPS mode is enabled.
$ uname -a
Linux hostname 4.18.0-553.8.1.el8_10.x86_64 #1 SMP Fri Jun 14 03:19:37 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
```

```
FROM ubuntu:24.04 AS builder

RUN . /etc/os-release && \
    DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y --no-install-recommends \
                curl apt-transport-https ca-certificates xz-utils \
                cmake build-essential pkg-config \
                libboost-all-dev libusb-1.0-0-dev \
                python3-mako python3-numpy python3-setuptools python3-requests
```

Ran with

```
docker build .
```

produces attached output

[Henry Coggill (henrycoggill)]

This issue arises because the container is running on top of a FIPS-enabled kernel which provides the /proc/sys/crypto/fips_enabled = 1 flag.

On Noble, openssl sees this flag and expects therefore to be operating in FIPS mode, but at this point the Noble container is not in FIPS mode - we haven't finished developing the FIPS components and haven't released them - and the result is that it throws some errors.

The workaround is to either not run the container on a FIPS-enabled kernel; or to set the environment variable in the Noble container to disable FIPS mode:

ENV OPENSSL_FORCE_FIPS_MODE=0

[Adrien Nader (adrien)]

Steps to reproduce on any machine on Noble or later:

    cp -r /usr/lib/ssl/certs/ .
    OPENSSL_FORCE_FIPS_MODE=1 openssl rehash certs

[Adrien Nader (adrien)]

NB: we can only SRU that to Noble if it first goes in a development version of Ubuntu which will only be possible when development for 25.04 begins.

[Adrien Nader (adrien)]

My plan is to make the library write the following error message when failing to load a provider (it will apply to any provider):

    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)

There was already an error but it was silent until it reached the application which could then assume something inappropriate as we've seen here.

There are a couple constraints.

First, I'm not loading error strings because I'm changing the inside of the library and if they haven't been loaded already, the error message will say "37" rather than "ERR_R_DSO_LIB"/"ERR_LIB_DSO". I plan to work around that by expanding the title of this bug report so that it's easy to find with a search engine. (this might also be because I'm relying on an internal static buffer for formatting because I prefer to avoid adding memory management in an error path)

The other constraint is that I need to "fprintf(stderr, ..." which is a bit uncommon for openssl code but I don't want to pop an error from the error stack like `ERR_print_errors{,_fp}()` does since I'm in the library code and not in the application code. This means the only API usable is `ERR_peek_last_error()`. Not a big issue but that makes for some slightly alien-looking code.

All in all, the output for the `openssl rehash certs` from above will contain the following text, once per cert found (i.e. 147 times here):

    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)
    out of memory

And I hope that one can copy-paste the new error message in a search engine and end directly on this page.

Any thought on this before I integrate this? Rough timeline is that I push that to plucky very soon (tomorrow maybe) and then SRU it in Noble right after (but SRUs take time and this will probably be a staged change until a new openssl security update is released).

[Adrien Nader (adrien)]

For reference, during a build, the testsuite now says the following:

    # Subtest: ../../test/fips_auto_enable_test
    1..1
    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)
    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)
    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)

This makes sense since the testsuite tries to enable the provider and it isn't available. These messages don't appear elsewhere in the build/test log. Therefore things are looking good.

[Abdurrahman Alp Köken (akoken)]

Looks good to me either. This error message is much more reasonable than seeing just `out of memory`. Thanks.