kcryptd, big file copy, system became unresponsive

[Janusz Dziedzic]

ubuntu 20.4
Linux eb850 5.4.0-62-generic #70-Ubuntu SMP Tue Jan 12 12:45:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Simple copy big file:
janusz@eb850:~$ ls -al tmp.tar.gz
-rw-r--r-- 1 janusz janusz 3847997532 sty 24 20:09 tmp.tar.gz
janusz@eb850:~$ cp tmp.tar.gz tmp2.tar.gz

GUI apps became unresponsive - slack/teams/firefox ...

Top show all CPU (cores) ~100% usage in WA (io and kcryptd).

nvme0n1 259:0 0 465,8G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part /boot/efi
├─nvme0n1p2 259:2 0 732M 0 part /boot
└─nvme0n1p3 259:3 0 464,6G 0 part
  └─nvme0n1p3_crypt 253:0 0 464,6G 0 crypt
    ├─ubuntu--vg-root 253:1 0 463,6G 0 lvm /
    └─ubuntu--vg-swap_1 253:2 0 980M 0 lvm [SWAP]

Is there any option that will change/tune kcryptd (dm-crypt) to not use all cores?
Today any operation like cp/zip/unzip/git with large files, couse I can't use ubuntu at all.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.14
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: janusz 1801 F.... pulseaudio
 /dev/snd/pcmC0D0p: janusz 1801 F...m pulseaudio
 /dev/snd/pcmC0D6c: janusz 1801 F...m pulseaudio
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2020-04-30 (269 days ago)
InstallationMedia: Ubuntu 18.04.3 LTS "Bionic Beaver" - Release amd64 (20190805)
MachineType: HP HP EliteBook 850 G6
Package: linux (not installed)
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-62-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash elevator=mq-deadline vt.handoff=7
ProcVersionSignature: Ubuntu 5.4.0-62.70-generic 5.4.78
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-62-generic N/A
 linux-backports-modules-5.4.0-62-generic N/A
 linux-firmware 1.187.7
Tags: focal
Uname: Linux 5.4.0-62-generic x86_64
UpgradeStatus: Upgraded to focal on 2020-10-31 (85 days ago)
UserGroups: adm cdrom dip docker lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 02/18/2020
dmi.bios.vendor: HP
dmi.bios.version: R70 Ver. 01.04.05
dmi.board.name: 8549
dmi.board.vendor: HP
dmi.board.version: KBC Version 52.59.00
dmi.chassis.type: 10
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrR70Ver.01.04.05:bd02/18/2020:svnHP:pnHPEliteBook850G6:pvr:rvnHP:rn8549:rvrKBCVersion52.59.00:cvnHP:ct10:cvr:
dmi.product.family: 103C_5336AN HP EliteBook
dmi.product.name: HP EliteBook 850 G6
dmi.product.sku: 6XD81EA#AKD
dmi.sys.vendor: HP

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1912960

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

apport information

[Janusz Dziedzic (janusz-dziedzic)]

Any operation like cp/rm big file:
top - 09:00:33 up 12 min, 1 user, load average: 11,55, 7,23, 3,60
Tasks: 341 total, 1 running, 340 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1,3 us, 7,0 sy, 0,0 ni, 0,0 id, 91,7 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu1 : 5,9 us, 7,3 sy, 0,0 ni, 0,0 id, 86,8 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu2 : 3,7 us, 7,0 sy, 0,3 ni, 27,4 id, 61,5 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu3 : 3,0 us, 6,6 sy, 0,0 ni, 0,0 id, 90,4 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu4 : 2,0 us, 6,7 sy, 0,0 ni, 69,3 id, 22,0 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu5 : 4,7 us, 8,0 sy, 0,7 ni, 0,0 id, 86,7 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu6 : 4,4 us, 5,8 sy, 0,0 ni, 0,0 id, 89,8 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu7 : 2,0 us, 6,7 sy, 0,0 ni, 0,0 id, 91,3 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 31922,9 total, 18111,2 free, 3655,1 used, 10156,6 buff/cache
MiB Swap: 980,0 total, 980,0 free, 0,0 used. 26893,0 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
   3635 janusz 20 0 2841208 302720 117252 S 8,9 0,9 1:10.78 Web Content
   2059 janusz 20 0 4961508 284548 102652 S 6,9 0,9 0:26.83 gnome-shell
   1823 janusz 9 -11 3352488 19036 15332 S 6,3 0,1 0:41.25 pulseaudio
   2631 janusz 20 0 819484 53192 38340 S 5,9 0,2 0:07.68 gnome-terminal-
    454 root 20 0 0 0 0 D 5,6 0,0 0:08.08 kworker/u16:7+kcryptd/253:0
   3452 janusz 20 0 3905104 362944 194092 S 5,6 1,1 0:52.16 firefox
   4667 root 20 0 0 0 0 D 5,6 0,0 0:05.93 kworker/u16:3+kcryptd/253:0
    126 root 20 0 0 0 0 D 5,3 0,0 0:06.50 kworker/u16:1+kcryptd/253:0
    447 root 20 0 0 0 0 D 5,3 0,0 0:07.15 kworker/u16:5+kcryptd/253:0
    450 root 20 0 0 0 0 D 5,3 0,0 0:08.38 kworker/u16:6+kcryptd/253:0 ...

[Janusz Dziedzic (janusz-dziedzic)]

Another example - compilation - PC became unusable ... most of GUI application - don't answer msg only seen ...

top - 13:37:23 up 4:04, 1 user, load average: 56,55, 60,02, 43,35
Tasks: 430 total, 1 running, 429 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1,7 us, 1,3 sy, 0,0 ni, 0,0 id, 96,0 wa, 0,0 hi, 1,0 si, 0,0 st
%Cpu1 : 3,0 us, 2,7 sy, 0,0 ni, 0,0 id, 94,0 wa, 0,0 hi, 0,3 si, 0,0 st
%Cpu2 : 3,7 us, 1,0 sy, 0,0 ni, 9,3 id, 86,0 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu3 : 3,5 us, 2,1 sy, 0,0 ni, 0,0 id, 94,3 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu4 : 3,6 us, 2,6 sy, 0,0 ni, 0,0 id, 93,8 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu5 : 2,0 us, 1,7 sy, 0,0 ni, 0,0 id, 96,3 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu6 : 3,8 us, 1,7 sy, 0,0 ni, 94,5 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu7 : 2,0 us, 1,0 sy, 0,0 ni, 0,0 id, 97,0 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 31925,3 total, 4748,0 free, 6415,8 used, 20761,6 buff/cache
MiB Swap: 980,0 total, 980,0 free, 0,0 used. 23416,2 avail Mem

[Marcelo Cerri (mhcerri)]

Hi, Janusz.

Thank you for reporting this bug.

Can you please test if the same problem also happens with the 5.8 hwe kernel? You can install it with `apt install linux-generic-hwe-20.04`.

[Janusz Dziedzic (janusz-dziedzic)]

anusz@eb850:~$ uname -a
Linux eb850 5.8.0-41-generic #46~20.04.1-Ubuntu SMP Mon Jan 18 17:52:23 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
janusz@eb850:~$

Same problems with 5.8.

Attached screen when remove big file

[Janusz Dziedzic (janusz-dziedzic)]

Same problems with 5.11rc4 - seems like linux kernel issue when dm-crypt used. I suspect because of best performance. But in my case, and probably most GUI users, disk performance isn't most important.
Today it is funny, can't run compilation in case I know will have Teams telco soon :)

Do we know which linux mailing group handle dm-crypt issues?

[Janusz Dziedzic (janusz-dziedzic)]

Finally I did:

- get new 2TB M2 SSD
- install fresh ubuntu 20.04 (1TB not crypted for system, 1TB crypted)
- no issues with crypted partition

After that don't have this problem.

So, possible issue was:
 - smth wrong when upgrade from 18.04 -> 20.04
 - my old m2 SSD (512GB)
 - or combination2

I think we can close this one.