Comment 2 for bug 1971699

A user has ran into a storage performance issue, and it has been traced back to CONFIG_INTEL_IOMMU_DEFAULT_ON being enabled.

They have a Dell XtremIO FC storage server, running VMWare ESXi 7.0U3, with 4x pvscsi controllers (2 LUNs per controller), running a single large LV with 8-way 1MB stripe.

There are Ubuntu 20.04 and 22.04 VMs running on the same server. The VMs are attached to the storage unit through iSCSI.

When running the following FIO script:

[write1]
filename=/dev/xio2data/prd01
readwrite=randwrite
random_generator=tausworthe64
blocksize=8K
ioengine=libaio
iodepth=32
direct=1
runtime=30
numjobs=8
group_reporting
invalidate=0

On 20.04 with 5.4.0-109-generic they see:

WRITE: bw=1010MiB/s (1059MB/s), 1010MiB/s-1010MiB/s (1059MB/s-1059MB/s), io=29.6GiB (31.8GB), run=30007-30007msec

They note a 2%-3% %SYS load.

On 22.04 with 5.15.0-27-generic they see:

WRITE: bw=568MiB/s (595MB/s), 568MiB/s-568MiB/s (595MB/s-595MB/s), io=16.6GiB (17.9GB), run=30007-30007msec

They note a consistent 18%-19% %SYS load, and poor performance.

It seems VMWare emulates a IOMMU, and their VMs were configuring 8 IOMMU groups for their PCI devices, and it was significantly impacting iSCSI performance.

When intel_iommu=off was supplied on the kernel command line, the performance issue is no longer present. This is a suitable workaround for the time being.

This user is in agreement with and asking for CONFIG_INTEL_IOMMU_DEFAULT_ON to be disabled.