CIFS Mounts Keep Load Average at n+1

[Doug Harple]

Today I noticed that my two primary Ubuntu boxes were running at high load. After investigation, I discovered at that unmounting their CIFS shares alleviating the high load.

I rebooted to the previous kernel, linux-modules-6.2.0-39-generic, and saw that the load was low after everything had settled after startup.

I rebooted to the current kernel, linux-modules-6.5.0-14-generic, and again saw the high load.

The load appears to be n+1, where n is the number of CIFS mounts. (Based on a sample of two boxes.)

I confirmed that the default mount options (e.g. the ones not specified in fstab) are the same between the two kernels.

This *might* also be slowing down external (USB disk access. I'm still investigating that.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-6.5.0-14-generic 6.5.0-14.14~22.04.1
ProcVersionSignature: Ubuntu 6.5.0-14.14~22.04.1-generic 6.5.3
Uname: Linux 6.5.0-14-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Fri Jan 12 19:47:25 2024
InstallationDate: Installed on 2020-10-01 (1198 days ago)
InstallationMedia: Ubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: linux-signed-hwe-6.5
UpgradeStatus: Upgraded to jammy on 2023-01-01 (376 days ago)

[Doug Harple (dharple)]

Mount options (all names and IPs redacted):

# mount | grep mount-name | sed -e's/,/\n,/g'
//server-name/mount-name on /mnt/mount-name type cifs (rw
,relatime
,vers=3.1.1
,cache=strict
,username=client-name
,domain=domain-name
,uid=0
,noforceuid
,gid=0
,noforcegid
,addr=192.1.1.123
,file_mode=0600
,dir_mode=0700
,seal
,soft
,nounix
,serverino
,mapposix
,mfsymlinks
,rsize=4194304
,wsize=4194304
,bsize=1048576
,echo_interval=60
,actimeo=1
,closetimeo=1)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mikael Andersson (manderss100)]

Just want to add I'm seeing this as well, kernel 6.2 had a load average of less than 1 most of the time, after 6.5 update I see load average of 5.x with processors mostly idle. I have 4 CIFS mounts which might be the cause of this, I have yet to try to debug this. I'm running Ubuntu 22.04 in a Hyper-V machine.

[Doug Harple (dharple)]

Following up on my initial comment: please ignore my comment about accessing USB drives. It isn't relevant.

[Makoto Naka (mslash)]

I experienced the same behaviour on

Operating System: Ubuntu 23.10
          Kernel: Linux 6.5.0-15-generic
    Architecture: x86-64 (Intel N100 4CPU)

And the following mount options:

//<nas ip>/paperless /share/smb/paperless cifs credentials=/root/.smbcredentials,multichannel,iocharset=utf8 0 0

9 cifs mounts in total with the same options and a totally idle system goes from 0 to 10 Load average

Nfs mounts are not producing this bug, it's strictly cifs related.

Before mounts:

cat /proc/loadavg
0.00 0.26 3.15 1/244 203137

After mounts:

cat /proc/loadavg
9.99 7.37 5.39 1/252 203256

That said all other metrics cpu/io/memory... are totally fine, just a calculation bug I suppose.

[Makoto Naka (mslash)]

Same with Linux 6.5.0-17-generic kernel update

[mark avery (marave)]

This issue is fixed in the mainline kernel, 6.5.10 and higher.

Re Possible bug report kernel 6.5.06.5.1 high load when CIFS share is mounted (cifsd-cfid-laundromat inD state)
https://lore.kernel<email address hidden>/T/

The relevant patches:
"smb3: allow controlling length of time directory entries are cached with dir leases"
https://github.com/torvalds/linux/commit/238b351d0935

"smb3: allow controlling maximum number of cached directories"
https://github.com/torvalds/linux/commit/6a50d71d0fff

"smb3: do not start laundromat thread when dir leases disabled"
https://github.com/torvalds/linux/commit/2da338ff752a

"smb: client: make laundromat a delayed worker"
https://github.com/torvalds/linux/commit/e95f3f744650

"smb: client: prevent new fids from being removed by laundromat"
https://github.com/torvalds/linux/commit/81ba10959970

[Thomas Opfer (t.o)]

This seems to be fixed in recent kernels. (Currently not seeing this issue with 6.5.0-35-generic.)

[Doug Harple (dharple)]

Yes. I don't know how to close the bug or mark it as resolved. I don't see an option for it.