evict_inodes inode xxx, i_count = 1, was skipped!

[Laurent Bonnaud]

Hi,

during boot the kernel logs the following messages:

# dmesg | grep evict_inodes
[ 12.784805] evict_inodes inode 00000000d69da69b, i_count = 1, was skipped!
[ 12.784810] evict_inodes inode 000000008b9a7c55, i_count = 1, was skipped!
[ 12.784811] evict_inodes inode 000000008ff8e64c, i_count = 1, was skipped!
[ 12.784811] evict_inodes inode 00000000194d080e, i_count = 1, was skipped!
[ 12.784812] evict_inodes inode 0000000009b77b7b, i_count = 1, was skipped!
[ 12.811217] evict_inodes inode 00000000ec873c71, i_count = 1, was skipped!
[ 12.811222] evict_inodes inode 0000000072bf501d, i_count = 1, was skipped!
[ 12.811223] evict_inodes inode 00000000b42b829a, i_count = 1, was skipped!
[ 12.811224] evict_inodes inode 00000000c7c5b4ef, i_count = 1, was skipped!
[ 13.412667] evict_inodes inode 0000000061a23a39, i_count = 1, was skipped!
[ 13.412671] evict_inodes inode 00000000b42b829a, i_count = 1, was skipped!
[ 13.412671] evict_inodes inode 0000000072bf501d, i_count = 1, was skipped!
[ 13.412672] evict_inodes inode 00000000c7c5b4ef, i_count = 1, was skipped!
[ 13.412673] evict_inodes inode 00000000ec873c71, i_count = 1, was skipped!

Those messages are not displayed when I boot my system with a mainline Linux kernel.
Moreover, in the mainline Linux source code, I do not find the lines where such a message could be printed.

Therefore, those messages are probably caused by an Ubuntu patch.

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-5-generic 6.5.0-5.5
ProcVersionSignature: Ubuntu 6.5.0-5.5-generic 6.5.0
Uname: Linux 6.5.0-5-generic x86_64
ApportVersion: 2.27.0-0ubuntu2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: bonnaudl 4827 F.... wireplumber
 /dev/snd/seq: bonnaudl 4808 F.... pipewire
CasperMD5CheckResult: unknown
CurrentDesktop: KDE
Date: Sun Sep 24 14:39:10 2023
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-6.5.0-5-generic root=/dev/mapper/MonVolume2-UbuntuRacine ro vsyscall=none security=apparmor tpm_tis.interrupts=0 preempt=full split_lock_detect=off quiet splash vt.handoff=7
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-6.5.0-5-generic N/A
 linux-backports-modules-6.5.0-5-generic N/A
 linux-firmware 20230919.git3672ccab-0ubuntu2
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/26/2022
dmi.bios.release: 7.16
dmi.bios.vendor: INSYDE Corp.
dmi.bios.version: 1.07.16NRTR4
dmi.board.asset.tag: Tag 12345
dmi.board.name: NS50_70MU
dmi.board.vendor: TUXEDO
dmi.board.version: Applicable
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: Notebook
dmi.chassis.version: N/A
dmi.ec.firmware.release: 7.7
dmi.modalias: dmi:bvnINSYDECorp.:bvr1.07.16NRTR4:bd05/26/2022:br7.16:efr7.7:svnTUXEDO:pnTUXEDOInfinityBookS15Gen6:pvrNotApplicable:rvnTUXEDO:rnNS50_70MU:rvrApplicable:cvnNotebook:ct10:cvrN/A:skuNotApplicable:
dmi.product.family: Not Applicable
dmi.product.name: TUXEDO InfinityBook S 15 Gen6
dmi.product.sku: Not Applicable
dmi.product.version: Not Applicable
dmi.sys.vendor: TUXEDO

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Thomas Lamprecht (t-lamprecht)]

This came indeed in via a cherry-pick made by Ubuntu [0], that was an early attempt to fix a bug that was actually present in the ceph layer, as confirmed by the original patch author[1], and fixed actually there now [2].

[0]: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/mantic/commit/?h=master-prep&id=a53dba9297be9597eac7b17738723bd44bac97ea

[1]: https://<email address hidden>/

[2]: https://patchwork.kernel<email address hidden>/

[Dewet Diener (dewet)]

I don't use ceph but I'm seeing this spammed across my Ubuntu 23.10 machines. Is it a false alarm then?

[Henry Liebenthal (minusman)]

Is this bug already fixed and in the update channels?

[Thomas F. (derherrfreitag)]

These messages get triggered randomly (I suspect on unmounting NFS mounts, which are managed by systemd automounter) on my Ubuntu 23.10 machines and end up in systemd journal and kern.log (gigabytes of messages), system uses 4 CPUs completely and gets unresponsive.

[Andrey Arapov (andrey-arapov)]

Can confirm.
I am seeing the same "evict_inodes" messages, whenever am inserting or removing the USB drive.
As well as when I boot up my laptop (Lenovo X1 Carbon gen10)

```
[Mon Nov 20 16:39:27 2023] usb 3-9: USB disconnect, device number 18
[Mon Nov 20 16:39:27 2023] evict_inodes inode 00000000aee04a8d, i_count = 1, was skipped!
```

Ubuntu 23.10 on 6.5.0-10-generic Linux kernel.

(Haven't seen these in Ubuntu 23.04 with older Linux kernel 6.2)

[Espen (espensa)]

I'm also getting these testing on 23.10 6.5.0-10-generic and here the issue is triggered by enabling the docker service.

[Gordon Lack (gordon-lack)]

These messages get displayed when a btrfs file system is unmounted (note they appear 16s after the mount, which would be the unmount).

I have one that is mounted (and unmounted) every 15 minutes and these show up at every unmount.

....
[1254622.716595] BTRFS: device label Lacknet-Freq devid 1 transid 447446 /dev/sdb1 scanned by (udev-worker) (641179)
[1255506.889750] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm
[1255506.889760] BTRFS info (device sdb1): disk space caching is enabled
[1255506.894559] BTRFS info (device sdb1): enabling ssd optimizations
[1255506.894565] BTRFS info (device sdb1): auto enabling async discard
[1255522.602675] evict_inodes inode 00000000d0474694, i_count = 1, was skipped!
[1255522.602679] evict_inodes inode 00000000eed5437e, i_count = 1, was skipped!
[1255522.602681] evict_inodes inode 0000000034452673, i_count = 1, was skipped!
[1255522.602682] evict_inodes inode 000000006cef2072, i_count = 1, was skipped!
[1255522.602683] evict_inodes inode 0000000037f015ca, i_count = 1, was skipped!
[1255522.602684] evict_inodes inode 00000000dcab3d4c, i_count = 1, was skipped!
[1255522.602685] evict_inodes inode 000000000267b235, i_count = 1, was skipped!
[1255522.602686] evict_inodes inode 00000000a76faacf, i_count = 1, was skipped!
[1255522.602687] evict_inodes inode 000000007448e65b, i_count = 1, was skipped!
[1255522.602688] evict_inodes inode 0000000092488dbd, i_count = 1, was skipped!
[1255522.602689] evict_inodes inode 00000000309d70e6, i_count = 1, was skipped!
[1255522.604942] evict_inodes inode 000000002f985da7, i_count = 1, was skipped!
[1255522.605046] evict_inodes inode 000000008c056c2d, i_count = 1, was skipped!
[1255522.768786] BTRFS: device label Lacknet-Freq devid 1 transid 447449 /dev/sdb1 scanned by (udev-worker) (641763)
[1256406.952345] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm
[1256406.952355] BTRFS info (device sdb1): disk space caching is enabled
[1256406.960879] BTRFS info (device sdb1): enabling ssd optimizations
[1256406.960885] BTRFS info (device sdb1): auto enabling async discard
[1256422.696551] evict_inodes inode 000000001d632894, i_count = 1, was skipped!
[1256422.696556] evict_inodes inode 00000000f9a54040, i_count = 1, was skipped!
[1256422.696557] evict_inodes inode 00000000f6ce5746, i_count = 1, was skipped!
[1256422.696558] evict_inodes inode 0000000049a39285, i_count = 1, was skipped!
.....

[tomeq_ (t-glowacki)]

It happens each time btrfs usb external drive is mounted, also happens for ext4 - the same drive, same usb cage. Using systemd-automount.

Linux nas 6.5.0-14-generic

Bus 002 Device 002: ID 174c:1153 ASMedia Technology Inc. ASM1153 SATA 3Gb/s bridge

[397017.384016] BTRFS: device fsid 98b21b16-ff76-4606-b091-117dd57fc825 devid 1 transid 1629 /dev/sdf1 scanned by mount (88452)
[397017.385233] BTRFS info (device sdf1): using crc32c (crc32c-intel) checksum algorithm
[397017.385251] BTRFS info (device sdf1): using free space tree
[397099.871588] evict_inodes inode 00000000ec798847, i_count = 1, was skipped!
[397099.871593] evict_inodes inode 00000000cdbf17cc, i_count = 1, was skipped!
[397099.871594] evict_inodes inode 0000000090c65438, i_count = 1, was skipped!
[397099.871596] evict_inodes inode 00000000846a60bd, i_count = 1, was skipped!
[397099.871598] evict_inodes inode 0000000023687c6b, i_count = 1, was skipped!
[397099.871598] evict_inodes inode 00000000bb13ceca, i_count = 1, was skipped!
[397099.871602] evict_inodes inode 0000000060410da0, i_count = 1, was skipped!
[397099.871603] evict_inodes inode 00000000bfed6dcf, i_count = 1, was skipped!
[397099.871603] evict_inodes inode 000000009348343d, i_count = 1, was skipped!
[397099.871607] evict_inodes inode 00000000a5321882, i_count = 1, was skipped!
[397099.871611] evict_inodes inode 0000000049062c28, i_count = 1, was skipped!
[397099.871611] evict_inodes inode 00000000f6e484ef, i_count = 1, was skipped!

[Torsten Krah (tkrah)]

I do have the same messages (6.5.0-14-generic #14~22.04.1-Ubuntu) in the dmesg after booting, but I don't use ceph nor do I have an usb drive connected.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Marc Dietrich (marvin24)]

seems to be fixed in linux-image-6.5.0-16-generic (frpm kernel ppa).

[Tyler Stachecki (tstachecki)]

I'm seeing this with test VMs running locally -- it's definitely not a Ceph issue.

[bol (bol)]

This issue appears on my Thinkpad T-series as well when saving or removing a state with zsysd. In my case on a LUKS-encrypted (not sure if it matters) drive with kernel linux-image-6.5.0-14-generic.

```# zsysctl save --system
ZSys is adding automatic system snapshot to GRUB menu
Successfully saved as "autozsys_xnh1ue"
```
```journalctl -f
Jan 17 18:48:41 <hostname> kernel: evict_inodes inode 000000006a2639d6, i_count = 1, was skipped!
Jan 17 18:48:41 <hostname> kernel: evict_inodes inode 000000006a2639d6, i_count = 1, was skipped!```

[Christian Kujau (christiank)]

Same here during boot, running 6.5.0-15-generic on Ubuntu 23.10:

$ dmesg | grep -C3 evict
[ 10.769204] EXT4-fs (xvda1): mounted filesystem 446b2699-dd26-4b84-9ada-d0cb48d21c93 r/w with ordered data mode. Quota mode: none.
[ 11.061845] SGI XFS with ACLs, security attributes, realtime, quota, no debug enabled
[ 11.069288] XFS (xvdb): Mounting V5 Filesystem b2136bf4-6edc-4fb2-8ad9-1fc236c2671d
[ 11.109335] evict_inodes inode 00000000f37fffda, i_count = 1, was skipped!
[ 11.109345] evict_inodes inode 00000000e97e4ee2, i_count = 1, was skipped!
[ 11.109348] evict_inodes inode 0000000092d4f6b3, i_count = 1, was skipped!
[ 11.109351] XFS (xvdb): Ending clean mount

---- However, mounting succeeds and no ill effects so far:

$ mount | grep xfs
/dev/xvdb on /home type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

$ xfs_info /dev/xvdb
meta-data=/dev/xvdb isize=512 agcount=4, agsize=1638400 blks
         = sectsz=4096 attr=2, projid32bit=1
         = crc=1 finobt=1, sparse=1, rmapbt=1
         = reflink=1 bigtime=1 inobtcount=1 nrext64=0
data = bsize=4096 blocks=6553600, imaxpct=25
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=16384, version=2
         = sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

[Thomas (t.c)]

I also see that message - not only at boot - my dmesg is full of it:

[1017116.534368] evict_inodes inode 0000000077d96b5a, i_count = 1, was skipped!
[1017116.534369] evict_inodes inode 00000000973ed623, i_count = 1, was skipped!
[1017116.534370] evict_inodes inode 00000000a18d9813, i_count = 1, was skipped!
[1017116.534371] evict_inodes inode 000000001a74c3a2, i_count = 1, was skipped!
[1017116.534372] evict_inodes inode 000000007ccea03e, i_count = 1, was skipped!
[1017116.534373] evict_inodes inode 0000000097fd1857, i_count = 1, was skipped!
[1017116.534374] evict_inodes inode 00000000b8584b66, i_count = 1, was skipped!
[1017116.534375] evict_inodes inode 00000000d9438af2, i_count = 1, was skipped!
[1017116.534376] evict_inodes inode 00000000a46aa80b, i_count = 1, was skipped!

root@comp:~# dpkg -l|grep `uname -r`
ii linux-headers-6.5.0-15-generic 6.5.0-15.15~22.04.1 amd64 Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
ii linux-image-6.5.0-15-generic 6.5.0-15.15~22.04.1 amd64 Signed kernel image generic
ii linux-modules-6.5.0-15-generic 6.5.0-15.15~22.04.1 amd64 Linux kernel extra modules for version 6.5.0 on 64 bit x86 SMP
ii linux-modules-extra-6.5.0-15-generic 6.5.0-15.15~22.04.1 amd64 Linux kernel extra modules for version 6.5.0 on 64 bit x86 SMP

I also dont use ceph directly - maybe it is involved with docker or zfs ....

[Gordon Lack (gordon-lack)]

kernel 6.5.0-17-generic has just arrived in 23.10 (Mantic) and that seems to clear the issue.

mount/unmount an external btrfs filesystem no longer produces the messages.

[Bin Li (binli)]

On 22.04.4 with 6.5.0-oem-1009 kernel, we also met this issue during shutdown.

Feb 23 15:20:36 P3-Tiny-Ref3-1 kernel: evict_inodes inode 0000000020f5b07f, i_count = 1, was skipped!
Feb 23 15:20:36 P3-Tiny-Ref3-1 kernel: evict_inodes inode 000000003d74f469, i_count = 1, was skipped!

After upgraded to 6.5.0-oem-1014 this issue is gone, from the source code, I found the commit below was removed.

https://git.launchpad.net/~ubuntu-
kernel/ubuntu/+source/linux/+git/mantic/commit/?id=a53dba9297be9597eac7b17738723bd44bac97ea

[Bin Li (binli)]

commit 3c9a7d086da4a403d72ab8df8d924f7247882653
Author: Dimitri John Ledkov <email address hidden>
Date: Fri Oct 27 18:53:00 2023 +0200

    Revert "UBUNTU: SAUCE: ceph: make sure all the files successfully put before unmounting"

    BugLink: https://bugs.launchpad.net/bugs/2041613

    This reverts commit a53dba9297be9597eac7b17738723bd44bac97ea.

    Signed-off-by: Dimitri John Ledkov <email address hidden>
    Acked-by: Andrea Righi <email address hidden>
    Acked-by: Roxana Nicolescu <email address hidden>
    Signed-off-by: Stefan Bader <email address hidden>

[Laurent Bonnaud (laurent-bonnaud)]

This bug is no longer present in kernel 6.8.0-22-generic from Ubuntu noble.
Thanks for the fix!

[Jan Schermer (zviratko)]

I have just seen this on 6.8.0-22-generic on a libvirt host running serveral k8s instances in KVM. Their drivers are backed by ZFS.

I just switched them to cache=writeback for disks, that might trigger this issue. It happened when they were idle but at a time (8AM) when I suspect some crons ran.

This message just happened again when I was deleting the VMs (virsh destroy) and their disks (zfs destroy).