kernel BUG/Oops errors from modprobe while the DRBG has not yet initialized (focal/fips-updates)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
Bionic |
Invalid
|
Undecided
|
Unassigned | ||
Focal |
Fix Released
|
Medium
|
Mauricio Faria de Oliveira | ||
Jammy |
Invalid
|
Undecided
|
Unassigned |
Bug Description
[Impact]
* The Focal FIPS kernel in fips-updates hits kernel BUG/Oops
errors during boot with the FIPS OpenSSL library installed
(but those don't cause issues), when it runs modprobe with
request_module() when looking up crypto algorithms/modules.
* The modprobe command happens to call the OpenSSL library,
and the FIPS version of OpenSSL calls getrandom(),
and the FIPS kernel calls the DRBG for that,
BUT it's _not yet_ initialized that early during boot
when the kernel can run modprobe via request_module().
(e.g., IPv6 initialization time.)
* The issue impacts the kernels in fips-updates only, per:
"UBUNTU: SAUCE: random: Use Crypto API DRBG for urandom in FIPS mode"
which exists in Focal, but not in Xenial/
* The issue only happens with the crypto algorithms, even
if they're built-in (i.e., modprobe is not needed).
[Fix]
* Fall back to CRNG while the DRBG is not yet initialized.
(Marcelo Cerri confirmed it's OK per other discussions.)
* The fix doesn't change the list and details of algorithms
as in /proc/crypto (e.g., name, driver, module, priority)
by the time the DRBG is initialized / initramfs started,
so even though behavior changes, the net effect doesn't.
* (Note: it's not possible to just use an initcall level
earlier than rootfs_initcall() so modprobe isn't there,
because fips_drbg_init() must run _after_ module_init()
level so that crypto_rng_reset() works, even though its
required module is built-in too.
[Test Steps]
* Install the kernel and openssl from fips-updates,
boot with fips=1, check dmesg for BUG/Oops errors:
$ sudo apt install linux-image-fips libssl1.1 # fips-updates
$ sudo vim /etc/default/grub # append fips=1 boot option
$ sudo update-grub && sudo reboot
$ sudo dmesg | grep BUG:
* Check/store the /proc/crypto file for comparisons.
You can boot with break=top as well, to check that
as early as possible, and copy into /run/initramfs/
then exit, to get it later in the rootfs.
[Regression Potential]
* The fix falls back to the regular CRNG for a while
in early boot. The CRNG is used permanently in the
non-FIPS kernels (and in FIPS kernels w/out fips=1),
so the code path is exercised/tested frequently.
* Regressions would most likely occur in calls to
getrandom() before the DRBG is initialized, but
that currently hits a BUG/Oops anyway.
[Original Bug Description]
$ sudo apt install --yes linux-image-fips # fips-updates
$ sudo vim /etc/default/grub # fips=1
$ sudo update-grub && sudo reboot
$ uname -r
5.4.0-1056-fips
$ cat /proc/cmdline
... fips=1
No errors with the original/non-FIPS openssl, because it does NOT call getrandom():
$ dmesg | grep -c BUG:
0
$ dpkg -s libssl1.1 | grep ^Version:
Version: 1.1.1f-1ubuntu2.15
$ strace -e getrandom modprobe --version
kmod version 27
+ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL
+++ exited with 0 +++
But if you install the FIPS openssl, it calls getrandom(), then BUG/Oops happen:
$ sudo apt install libssl1.1 # updates initramfs
$ dpkg -s libssl1.1 | grep ^Version:
Version: 1.1.1f-
$ strace -e getrandom modprobe --version
getrandom(
getrandom(
getrandom(
getrandom(
getrandom(
kmod version 27
+ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL
+++ exited with 0 +++
$ sudo reboot
$ dmesg | grep -c BUG:
22
$ dmesg
...
[ 1.595759] NET: Registered protocol family 10
[ 1.600256] BUG: kernel NULL pointer dereference, address: 0000000000000038
...
[ 1.603829] CPU: 2 PID: 137 Comm: modprobe Not tainted 5.4.0-1056-fips #64-Ubuntu
[ 1.603829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 1.603829] RIP: 0010:urandom_
...
[ 1.603829] Call Trace:
[ 1.603829] __x64_sys_
[ 1.603829] do_syscall_
[ 1.603829] entry_SYSCALL_
...
All BUG/Oops errors are the same:
$ dmesg | grep BUG: | sed 's/^.*BUG:/BUG:/' | uniq -c
22 BUG: kernel NULL pointer dereference, address: 0000000000000038
And they stop after the DRBG is initialized:
[ 3.651566] random: DRBG (drbg_nopr_
[Fix Impact Analysis]
The patch adds a dynamic debug message that can be enabled
in the kernel cmdline, for comparisons (dmesg); we can also
compare /proc/crypto for no changes.
dyndbg="func urandom_read +p"
Also add break=top, so we can copy /proc/crypto and dmesg
right after DRBG is initialized (when initramfs is started).
@ break=top time
suffix=original # or modified
cat /proc/crypto > /run/initramfs/
dmesg > /run/initramfs/
exit
@ login time
sudo -s
cp /run/initramfs/
reboot # next test
There's no difference in the list/details of loaded crypto algorithms at all, with any combination:
# md5sum proc-crypto.*
0b91bd619078fa3
0b91bd619078fa3
The kernel with the fix does not hit BUG/Oops errors:
# grep ^ -m1 dmesg.*
dmesg.modified:[ 0.000000] Linux version 5.4.0-1060-fips ... #68+fipsdrbgnul
dmesg.original:[ 0.000000] Linux version 5.4.0-1060-fips ... #68-Ubuntu ...
# grep -c BUG: dmesg.*
dmesg.modified:0
dmesg.original:22
# grep -c 'random: DRBG uninitialized! crng fallback' dmesg.*
dmesg.modified:110
dmesg.original:0
The 110 number is 5 * 22 calls as modprobe calls getrandom() 5 times (see strace above).
The original kernel has 1 BUG/Oops only because that kills the modprobe task.
summary: |
- kernel BUG/Oops errors from modprobe in early boot while the DRBG is not + kernel BUG/Oops errors from modprobe while the DRBG has not yet initialized (focal/fips-updates) |
information type: | Private → Public |
Changed in linux (Ubuntu Jammy): | |
status: | New → Invalid |
Changed in linux (Ubuntu Bionic): | |
status: | New → Invalid |
Changed in linux (Ubuntu Focal): | |
status: | New → In Progress |
importance: | Undecided → Medium |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
description: | updated |
Changed in linux (Ubuntu): | |
status: | Incomplete → Invalid |
description: | updated |
description: | updated |
description: | updated |
Changed in linux (Ubuntu Focal): | |
status: | In Progress → Fix Committed |
This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 1981487
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.