Issue with L1ARC for uncompressed datasets

[BertN45]

I have done some testing with Ubuntu 21.04 and OpenZFS. I mostly run Virtual Machines and I noticed a huge difference in performance between the VM disk IO, between the same VM stored on a lz4 compressed dataset and on an uncompressed dataset. The difference in the VM disk throughput is a factor 40 to 90 dependent on the type of disk IO test. My host OS was Ubuntu 21.04 and the Guest has been Windows 7 using ChrystalDiskMark.

Looking at the the size of L1ARC that I monitored with Conky, that L1ARC size did not change, when I used the VM on an uncompressed dataset. It only flipped say ~100 MB during running that VM, so it showed 3.26 GB and seconds later 3.15 GB and it always flipped between these two values. The hosts had space enough free to increase the L1ARC. My wild guess was that the L1ARC has not been used, only the small part for uncompressed records. Maybe it is a bug, maybe a feature or maybe something else.

I have a detailed description in the next discourse: https://discourse.destinationlinux.network/t/ubuntu-21-04-and-bleeding-edge-openzfs-2-0/3621/3

The interesting part is the third entry and for some context the first and second entries.

ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: zfsutils-linux 2.0.2-1ubuntu5
ProcVersionSignature: Ubuntu 5.11.0-13.14-generic 5.11.7
Uname: Linux 5.11.0-13-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu62
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Sun Apr 11 18:34:01 2021
InstallationDate: Installed on 2019-11-30 (498 days ago)
InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
SourcePackage: zfs-linux
UpgradeStatus: Upgraded to hirsute on 2021-03-24 (18 days ago)