zfs_arc_max not working anymore in zfs 0.8.1

[BertN45]

In the past I could limit the size of L1ARC by specifying "options zfs zfs_arc_max=3221225472" in /etc/modprobe.d/zfs.conf. I tried even to fill /sys/module/zfs/parameters/zfs_arc_max directly, but none of those methods limits the size of L1ARC. It worked nicely in zfs 0.7.x.

Nowadays I use a nvme drive with 3400 MB/s read throughput and I see not much difference between e.g booting the system from nvme and rebooting the system from L1ARC. Having a 96-99% hit rate for the L1ARC, I like to free up some memory, thus avoiding some delay of freeing memory from L1ARC while loading a VM.

-------------------------------------------------------------------------
UPDATE:

The system only limits the L1ARC, if we directly after the login write the zfs_arc_max value to the file /sys/module/zfs/parameters/zfs_arc_max.

------------------------------------------------------------------------

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: zfsutils-linux 0.8.1-1ubuntu14.1
ProcVersionSignature: Ubuntu 5.3.0-23.25-generic 5.3.7
Uname: Linux 5.3.0-23-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu8.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Fri Nov 29 06:19:15 2019
InstallationDate: Installed on 2019-11-25 (3 days ago)
InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
SourcePackage: zfs-linux
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.sudoers.d.zfs: [inaccessible: [Errno 13] Permission denied: '/etc/sudoers.d/zfs']

[Colin Ian King (colin-king)]

Maybe I'm missing something here. I tested this out on an Eoan server:

$ dmesg | grep ZFS
[ 10.821332] ZFS: Loaded module v0.8.1-1ubuntu14.3, ZFS pool version 5000, ZFS filesystem version 5

$ uname -a
Linux eaon-amd64 5.3.0-42-generic #34-Ubuntu SMP Fri Feb 28 05:49:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I added a zfs_arc_max setting into the zfs.conf:

cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=50331648

rebooted and I get:

cat /sys/module/zfs/parameters/zfs_arc_max
50331648

So this is setting the L1ARC size as I expected. Can you explain further the issue you are seeing?

[BertN45 (lammert-nijhof)]

Yes you did miss something. Of course the value is changed in /etc/modprobe.d/zfs.conf. I changed it myself! However zfs is ignoring that value and is using considerably more memory for L1ARC.

If I wanted to limit the size of L1ARC, I had to change the value in /sys/module/zfs/parameters/zfs_arc_max.

[Colin Ian King (colin-king)]

Just for clarification, how are you determining that zfs is using more memory for the L1ARC?

[BertN45 (lammert-nijhof)]

I display it constantly using conky (see attachment) and conky is using

${color}L1ARC / ${color0}Hits%: ${color}${alignr}(${exec cat /proc/spl/kstat/zfs/arcstats | grep -m 1 uncompressed_size | awk '{printf "%4.2f",$3/1073741824}'}) ${exec cat /proc/spl/kstat/zfs/arcstats | grep -m 1 size | awk '{printf "%4.2f",$3/1073741824}'}GiB of 2.00GiB - ${color0}${exec arc_summary | grep -m 1 Actual | awk '{print int($8+0.5)}'}%

${color5}${execbar cat /proc/spl/kstat/zfs/arcstats | grep -m 1 size | awk '{printf int($3/21474836)}'}

[Colin Ian King (colin-king)]

I've experimented with this on eoan and focal (ZFS 0.8.1 and ZFS 0.8.3) on a 4GB VM image. I set the /etc/modprobe.d/zfs.conf as follows:

options zfs zfs_arc_max=134217728
# 128 MB

And rebooted. I then exercised the zfs with various greps and git logs on the linux git repository while running:

while true; do cat /proc/spl/kstat/zfs/arcstats | grep -m 1 size | awk '{ print $3 / (1024 * 1024) }'; sleep 1; done

The output shows that one does get arc sizes greater than the 128MB while doing a lot of I/O but it settles down to sub 128MB once the I/O activity is finished:

2.93295
2.93295
2.93295
63.8354
131.395
127.909
148.942
128.498
138.224
145.271
133.244
134.122
138.132
144.814
129.4
134.788
135.666
140.428
143.811
146.734
132.323
136.709
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
130.353
130.34
130.34
130.34
138.63
140.16
139.826
139.826
139.826
139.921
139.921
133.32
122.35
115.929
111.941
110.478
111.219
111.219

Two observations:

1. The driver does some sanity checks; if zfs_arc_max is too small (< 64MB) or too large (> physical memory size) then the setting is ignored and the default is used. One can double check if the setting has been updated by checking with c_max, e.g.:

grep c_max /proc/spl/kstat/zfs/arcstats
c_max 4 134217728

If c_max is not the same as zfs_arc_max then the zfs_arc_max has worked. Can you sanity check that?

2. With a high amount of throughput the arcstats show that the "size" stat can overshot the zfs_arc_max setting but once I/O has completed it will setting down to the threshold set. This setting is not updated directly and is only sync'd during an arc_kstat update. So maybe it's a snapshot that maybe transient and possibly not completely up to date. Keeping the stats exactly up to date would cause a performance bottleneck, so the stats are aggregated - I believe these values are not exactly 100% correct as there is a performance impact in gathering and summing the stats.

Perhaps you could experiment with a low zfs_arc_max setting (e.g. 128MB 134217728) and see if this works OK and then bumping up the value to see if it stops working; then we can work out why it's going wrong at that point.

[BertN45 (lammert-nijhof)]

I moved to Ubuntu 20.04 and I do not have that problem anymore.

I stopped with writing directly after the login the value to the file /sys/module/zfs/parameters/zfs_arc_max.

I now only rely on "options zfs zfs_arc_max=3221225472" in /etc/modprobe.d/zfs.conf.

Problem solved in 20.04 and I tried it by loading various VMs including Windows 10 and run the updates for those VMs. The L1ARC size never exceeded 3.01 GB of the 3.00 GB I allowed. I knew it would exceed for a short period the max value, but in 19.10 at the time of the bug report, it did exceed it sometimes with 1 GB or more.

I use and used conky to display L1ARC values constantly; L1ARC size uncompressed; L1ARC size compressed; max allowed vale and L1ARC hit rate and I have a tweaked bar display that would allow up to 102% :)

[Colin Ian King (colin-king)]

I'll close this bug report if that's OK.

[BertN45 (lammert-nijhof)]

OK, close the bug-report.

On Thu, 2020-03-26 at 22:47 +0000, Colin Ian King wrote:
> I'll close this bug report if that's OK.
>

[Adam Lucke (quantenschaum)]

I would like to reopen this issue. I installed Ubuntu 20.04.1 LTS Server (no zfs root, 16GB RAM) and added zfs volumes. I tried to limit the arc with

`options zfs zfs_arc_max=134217728` in `/etc/modprobe.d/zfs.conf` to 128MiB (just to try)

I updated the initial ramdisk with `update-initramfs -u` and rebooted the machine.

Then I observe, that the arc still grows to ~6GB > 128M.

```
al@nas:~$ cat /etc/modprobe.d/zfs.conf; cat /sys/module/zfs/parameters/zfs_arc_max; cat /proc/spl/kstat/zfs/arcstats

options zfs zfs_arc_max=134217728

134217728

12 1 0x01 98 26656 9388908203 858454750163
name type data
hits 4 2341389
misses 4 69038
demand_data_hits 4 409755
demand_data_misses 4 10337
demand_metadata_hits 4 1892418
demand_metadata_misses 4 11520
prefetch_data_hits 4 20609
prefetch_data_misses 4 36648
prefetch_metadata_hits 4 18607
prefetch_metadata_misses 4 10533
mru_hits 4 332923
mru_ghost_hits 4 0
mfu_hits 4 1981310
mfu_ghost_hits 4 0
deleted 4 21
mutex_miss 4 0
access_skip 4 0
evict_skip 4 99
evict_not_enough 4 0
evict_l2_cached 4 0
evict_l2_eligible 4 201728
evict_l2_ineligible 4 34816
evict_l2_skip 4 0
hash_elements 4 70022
hash_elements_max 4 70117
hash_collisions 4 2316
hash_chains 4 1152
hash_chain_max 4 2
p 4 4173713920
c 4 8316876800
c_min 4 519804800
c_max 4 8316876800
size 4 6496367160
compressed_size 4 5849741824
uncompressed_size 4 6240316928
overhead_size 4 416625152
hdr_size 4 23995776
data_size 4 5993091584
metadata_size 4 273275392
dbuf_size 4 44426616
dnode_size 4 120015552
bonus_size 4 41562240
anon_size 4 18524672
anon_evictable_data 4 0
anon_evictable_metadata 4 0
mru_size 4 4014986752
mru_evictable_data 4 3841052672
mru_evictable_metadata 4 24693760
mru_ghost_size 4 0
mru_ghost_evictable_data 4 0
mru_ghost_evictable_metadata 4 0
mfu_size 4 2232855552
mfu_evictable_data 4 1694498304
mfu_evictable_metadata 4 15691264
mfu_ghost_size 4 0
mfu_ghost_evictable_data 4 0
mfu_ghost_evictable_metadata 4 0
l2_hits 4 0
l2_misses 4 0
l2_feed...

[Adam Lucke (quantenschaum)]

The `zfs_arc_max` setting *is* respected when I set it to a value of `1073741824` = 1GiB

```
cat /etc/modprobe.d/zfs.conf; cat /sys/module/zfs/parameters/zfs_arc_max; cat /proc/spl/kstat/zfs/arcstats |grep ^size -B3
options zfs zfs_arc_max=1073741824

1073741824

c 4 1073741824
c_min 4 519804800
c_max 4 1073741824
size 4 1086124448
```

The manpage `man zfs-module-parameters` says

```
zfs_arc_max (ulong)
Max arc size of ARC in bytes. If set to 0 then it will consume 1/2 of system RAM. This value must be at least 67108864 (64 megabytes).

This value can be changed dynamically with some caveats. It cannot be set back to 0 while running and reducing it below the current ARC size will not cause the ARC to shrink without memory pressure to induce shrinking.

Default value: 0.
```

Before I had set it to `134217728`, more than 64MiB, but it was not respected.

I also tried setting `zfs_arc_max` to `519804800` (495.7MiB, where ever this number comes from), the value reported as `c_min` and it *not* respected.

So, the minimum arc size seems to be around 1GB and *not* 64MB, as the manpage claims. (documentation lies, the code does not)

[Richard Laager (rlaager)]

The limit in the code does seem to be 64 MiB. I'm not sure why this isn't working. I am not even close to an expert on this part of OpenZFS, so all I can suggest is to file a bug report upstream: https://github.com/openzfs/zfs/issues/new