Enabling CONFIG_CRYPTO_STATS causes significant fscrypt performance degradation

Bug #2039576 reported by Shuichi Ihara
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

CONFIG_CRYPTO_STATS is enabled in Jammy 5.15.0-xx as well as 6.2.0-xx kernel by default.

# uname -r
5.15.0-69-generic
root@r100:~# grep CONFIG_CRYPTO_STATS /boot/config-5.15.0-69-generic
CONFIG_CRYPTO_STATS=y

However, that CONFIG_CRYPTO_STATS makes big negative performance impacts.
Lustre (http://www.lustre.com Open source parallel filesystem) supports client encryption based on fscrypt.
I ran some benchmarks of Lustre client encryption on Lustre client with Ubuntu 6.2.0-34-generic kernel below.

1 x client(2 x Gold 6438Y+, 512GB RAM, 2 x HDR200)
Ubuntu22.04 (6.2.0-34-generic)

# fio -name=iotest-w -ioengine=sync -rw=write -blocksize=16m -iodepth=1 -direct=1 -fallocate=none -size=200g -numjobs=128 -group_reporting -directory=/lustre/encrypted -create_serialize=0 -fiename_format='f.$jobnum.$filenum' -scramble_buffers=0 -runtime=60

6.2.0-34-generic, no encryption 48.5GB/s
6.2.0-34-generic, encryption 21.7GB/s
6.2.0-34-generic-noCryptStats, encryption 48.2GB/s

When Lustre client encryption enabled, there were 65% performance degradations compared to non encryption's performance. (21.7GB/s vs 48.5GB/s)
I also tested on 6.2.0-34-generic-noCryptStats which I built custom 6.2.0-34-generic kernel with disabling CONFIG_CRYPTO_STATS.
The result was pretty good. The performance did boost and it was almost same speed against non encryption case. (48.2GB/s vs 48.5GB/s)

I also found same fscrypt performance problem reported in Ceph with RHEL kernel below.
https://<email address hidden>/thread/44GMO5UGOXDZKFSOQMCPPHYTREUEA3ZI/

Eventually, Redhat disabled CONFIG_CRYPTO_STATS in latest RHEL9 kernel
https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/2947
Kernel config option CONFIG_CRYPTO_STATS should be disabled until it is enhanced (BZ#2231850) - https://access.redhat.com/errata/RHSA-2023:5069

I would disable CONFIG_CRYPTO_STATS in all Jammy kernels until proper fixes and new solution is available in upstream kernel.
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Oct 17 21:41 seq
 crw-rw---- 1 root audio 116, 33 Oct 17 21:41 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.4
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 10000M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
MachineType: Giga Computing H263-S63-AAN1-000
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 LC_CTYPE=C.UTF-8
 TERM=vt100
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-6.2.0-34-generic root=UUID=b48b34d5-f2cf-444d-b376-0ffc987891c1 ro
ProcVersionSignature: Ubuntu 6.2.0-34.34~22.04.1-generic 6.2.16
RelatedPackageVersions:
 linux-restricted-modules-6.2.0-34-generic N/A
 linux-backports-modules-6.2.0-34-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.19
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 6.2.0-34-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 08/09/2023
dmi.bios.release: 5.31
dmi.bios.vendor: GIGABYTE
dmi.bios.version: F20
dmi.board.asset.tag: 01234567890123456789AB
dmi.board.name: MS63-HD1-000
dmi.board.vendor: Giga Computing
dmi.board.version: 01000100
dmi.chassis.asset.tag: 01234567890123456789AB
dmi.chassis.type: 23
dmi.chassis.vendor: Giga Computing
dmi.chassis.version: 01234567
dmi.modalias: dmi:bvnGIGABYTE:bvrF20:bd08/09/2023:br5.31:svnGigaComputing:pnH263-S63-AAN1-000:pvr9NH263S63DR000AAN1*:rvnGigaComputing:rnMS63-HD1-000:rvr01000100:cvnGigaComputing:ct23:cvr01234567:sku01234567890123456789AB:
dmi.product.family: Server
dmi.product.name: H263-S63-AAN1-000
dmi.product.sku: 01234567890123456789AB
dmi.product.version: 9NH263S63DR000AAN1*
dmi.sys.vendor: Giga Computing

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2039576

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Shuichi Ihara (sihara) wrote : CurrentDmesg.txt

apport information

tags: added: apport-collected jammy uec-images
description: updated
Revision history for this message
Shuichi Ihara (sihara) wrote : Lspci.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : Lspci-vt.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : Lsusb-v.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : ProcModules.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : UdevDb.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : WifiSyslog.txt

apport information

Revision history for this message
Shuichi Ihara (sihara) wrote : acpidump.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.