DRBG initialization failed on some AMD EPYC series CPU and lead to kernel panic (Focal FIPS)

[gerald.yang]

BugLink: https://bugs.launchpad.net/bugs/2069715

[Impact]

Install Focal with FIPS enabled on AMD EPYC 7262 8-Core Processor, kernel panic happens during boot:
[ 3.430477] ima: Allocated hash algorithm: sha1
[ 3.433358] ima: No architecture policies found
[ 3.435785] evm: Initialising EVM extended attributes:
[ 3.438271] evm: security.selinux
[ 3.440265] evm: security.SMACK64
[ 3.442532] evm: security.SMACK64EXEC
[ 3.444753] evm: security.SMACK64TRANSMUTE
[ 3.446900] evm: security.SMACK64MMAP
[ 3.448912] evm: security.apparmor
[ 3.452277] evm: security.ima
[ 3.455549] evm: security.capability
[ 3.457537] evm: HMAC attrs: 0x1
[ 3.461049] PM: Magic number: 12:438:677
[ 3.463502] rtc_cmos 00:03: setting system clock to 2024-06-18T09:40:59 UTC (1718703659)
[ 3.467750] Kernel panic - not syncing: random: Failed to reset DRBG (drbg_nopr_ctr_aes256): -2
[ 3.471060] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.4.0-1100-fips #110-Ubuntu
[ 3.474288] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS 0.0.0 02/06/2015
[ 3.478191] Call Trace:
[ 3.480175] dump_stack+0x6d/0x8b
[ 3.482652] panic+0x114/0x2f6
[ 3.490069] fips_drbg_init+0xa7/0xbd
[ 3.492169] ? chr_dev_init+0x148/0x148
[ 3.494330] do_one_initcall+0x4a/0x200
[ 3.496396] kernel_init_freeable+0x1e6/0x289
[ 3.498967] ? rest_init+0xb0/0xb0
[ 3.500965] kernel_init+0xe/0x110
[ 3.502983] ret_from_fork+0x35/0x40
[ 3.505305] Kernel Offset: 0x35c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 3.509544] ACPI MEMORY or I/O RESET_REG.

It fails to initialize DRBG, this issue doesn't happen on Intel CPU, only on some AMD EPYC series CPU.
Also Jammy with FIPS enabled doesn't have this issue.

[Fix]

It's been fixed by this upstream commit:
commit 552d03a223eda3df84526ab2c1f4d82e15eaee7a
Author: Stephan M<C3><BC>ller <email address hidden>
Date: Sun Nov 21 15:14:20 2021 +0100

    crypto: jitter - consider 32 LSB for APT

    The APT compares the current time stamp with a pre-set value. The
    current code only considered the 4 LSB only. Yet, after reviews by
    mathematicians of the user space Jitter RNG version >= 3.1.0, it was
    concluded that the APT can be calculated on the 32 LSB of the time
    delta. Thi change is applied to the kernel.

    This fixes a bug where an AMD EPYC fails this test as its RDTSC value
    contains zeros in the LSB. The most appropriate fix would have been to
    apply a GCD calculation and divide the time stamp by the GCD. Yet, this
    is a significant code change that will be considered for a future
    update. Note, tests showed that constantly the GCD always was 32 on
    these systems, i.e. the 5 LSB were always zero (thus failing the APT
    since it only considered the 4 LSB for its calculation).

    Signed-off-by: Stephan Mueller <email address hidden>
    Signed-off-by: Herbert Xu <email address hidden>

[Testcase]

On AMD EPYC 7262 8-Core Processor, create a VM and enable FIPS can also reproduce the issue.
I cherry-pick the above upstream commit to Focal FIPS kernel (5.4.0-1100.110-fips), the DRBG init failed message is gone:
[ 3.267954] rtc_cmos 00:03: setting system clock to 2024-06-18T06:58:53 UTC (1718693933)
[ 3.275683] random: random: DRBG (drbg_nopr_ctr_aes256) initialized!
[ 3.279309] md: Waiting for all devices to be available before autodetect

[Where problems could occur]

This commit considers AMD EPYC CPU's RDTSC contains zero in LSB, this won't affect other cases.

[Other Info]

Users also reported the issue here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2045322
https://askubuntu.com/questions/1509977/fips-kernel-panics-with-failed-to-reset-drbg-during-boot

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-fips/5.4.0-1104.114 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-fips' to 'verification-done-focal-linux-fips'. If the problem still exists, change the tag 'verification-needed-focal-linux-fips' to 'verification-failed-focal-linux-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[gerald.yang (gerald-yang-tw)]

Verified on 5.4.0-1104 fips kernel on AMD EPYC 7262 machine and confirmed it fixes the issue.
With 5.4.0-1104 fips kernel, it can boot successfully:

ubuntu@focal:~$ sudo dmesg | grep DRBG
[ 4.121726] random: random: DRBG (drbg_nopr_ctr_aes256) initialized!
ubuntu@focal:~$ uname -a
Linux focal 5.4.0-1104-fips #114-Ubuntu SMP Mon Jul 22 20:48:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@focal:~$