[UBUNTU 24.04] IOMMU DMA mode changed in kernel config causes massive throughput degradation for PCI-related network workloads

[bugproxy]

Symptom:
Comparing Ubuntu 24.04 (kernelversion: 6.8.0-31-generic) against Ubuntu 22.04, all of our PCI-related network measurements on LPAR show massive throughput degradations (up to -72%). This shows for almost all workloads and numbers of connections, detereorating with the number of connections increasing. Especially drastic is the drop for a high number of parallel connections (50 and 250) and for small and medium-size transactional workloads. However, also for streaming-type workloads the degradation is clearly visible (up to 48% degradation).

Problem:
With kernel config setting CONFIG_IOMMU_DEFAULT_DMA_STRICT=y, IOMMU DMA mode changed from lazy to strict, causing these massive degradations.
Behavior can also be changed with a kernel commandline parameter (iommu.strict) for easy verification.

The issue is known and was quickly fixed upstream in December 2023, after being present for little less than two weeks.
Upstream fix: https://github.com/torvalds/linux/commit/b2b97a62f055dd638f7f02087331a8380d8f139a

Repro:
rr1c-200x1000-250 with rr1c-200x1000-250.xml:

<?xml version="1.0"?>
<profile name="TCP_RR">
        <group nprocs="250">
                <transaction iterations="1">
                        <flowop type="connect" options="remotehost=<remote IP> protocol=tcp tcp_nodelay" />
                </transaction>
                <transaction duration="300">
                        <flowop type="write" options="size=200"/>
                        <flowop type="read" options="size=1000"/>
                </transaction>
                <transaction iterations="1">
                        <flowop type="disconnect" />
                </transaction>
        </group>
</profile>

0) Install uperf on both systems, client and server.
1) Start uperf at server: uperf -s
2) Start uperf at client: uperf -vai 5 -m uperf-profile.xml

3) Switch from strict to lazy mode using kernel commandline parameter iommu.strict=0.
4) Repeat steps 1) and 2).

Example:
For the following example, we chose the workload named above (rr1c-200x1000-250):

iommu.strict=1 (strict): 233464.914 TPS
iommu.strict=0 (lazy): 835123.193 TPS

[Frank Heimes (fheimes)]

I just had a look at the Ubuntu kernel noble master-next tree and can find commit:
"Revert "s390: update defconfigs"" under the hash b2b97a62f055
and I can see that it got reverted with that:
$ git show b2b97a62f055 | grep CONFIG_IOMMU_DEFAULT_DMA_STRICT
    CONFIG_IOMMU_DEFAULT_DMA_STRICT option needs to be disabled.
-CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
-CONFIG_IOMMU_DEFAULT_DMA_STRICT=y

But git also tells me that it is in since kernel v6.8 and with that since the first ubuntu 6.8 kernel we had: Ubuntu-6.8.0-6 -- so should also be in Ubuntu-6.8.0-31.
But it does not seem to be reflected in the kernel options of the Ubuntu kernel:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble
$ uname -a
Linux hwe0008 6.8.0-31-generic #31-Ubuntu SMP Sat Apr 20 00:14:26 UTC 2024 s390x s390x s390x GNU/Linux
$ grep CONFIG_IOMMU_DEFAULT_DMA_STRICT /boot/config-6.8.0-31-generic
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
also not in the current, updated kernel:
Linux hwe0008 6.8.0-36-generic #36-Ubuntu SMP Mon Jun 10 09:59:13 UTC 2024 s390x s390x s390x GNU/Linux
$ grep CONFIG_IOMMU_DEFAULT_DMA_STRICT /boot/config-6.8.0-31-generic
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
For some reason the change in the upstream commit was not taken over into the Ubuntu kernel configs ...

[Kleber Sacilotto de Souza (kleber-souza)]

Upstream commit b2b97a62f055dd638f7f02087331a8380d8f139a changes the s390x defconfig's, which don't impact the Ubuntu kernel configs.

Looking at the configs for the 22.04 generic kernel, 'CONFIG_IOMMU_DEFAULT_DMA_STRICT=y' seems to be set for s390x on all Ubuntu 5.15 generic kernel builds as well. Therefore it doesn't seem the config has regressed, and although changing the kernel parameter during the boot time improves the performance it is likely not the only aspect causing the degradation.

Can IBM please confirm which 22.04 kernel version doesn't show the performance degradation and what are the config values for 'CONFIG_IOMMU_DEFAULT_DMA_STRICT' and 'CONFIG_IOMMU_DEFAULT_DMA_LAZY' on the running kernel?

Before changing the kernel config I would like to gather some more information to assess the consequences of the change and whether we need to make other changes as well.

Thank you.

[Frank Heimes (fheimes)]

Btw. I just checked the jammy kernel config options and jammy/22.04 had and still has strict=yes:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
$ uname -a
Linux maasrrc1 5.15.0-112-generic #122-Ubuntu SMP Thu May 23 08:10:47 UTC 2024 s390x s390x s390x GNU/Linux
ubuntu@maasrrc1:~$ grep CONFIG_IOMMU_DEFAULT_DMA_STRICT /boot/config-5.15.0-112-generic
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
$ grep CONFIG_IOMMU_DEFAULT_DMA_STRICT /boot/config-5.15.0-112-generic
I also checked older jammy kernels and strict was always yes.

So this has obviously never changed from jammy to noble!

Have you modified/tweaked the setting manually on your side before testing on jammy?!

[Frank Heimes (fheimes)]

<Looks like Kleber and me commented in parallel ...>