On systems with IOMMU enabled, every streaming DMA mapping involves an IOVA to be allocated and freed. For small mappings, IOVA sizes are normally cached, so IOVA allocations complete in a reasonable time. For larger mappings, things can be significantly slower to the point where softlockups occur due to lock contention
on iova_rbtree_lock.
commit 9257b4a206fc ("iommu/iova: introduce per-cpu caching to iova allocation")
introduced a scalable IOVA cache mechanism, that helps increase performance up
to 128kb mappings.
On systems that do larger streaming DMA mappings, e.g. a NVMe device with:
/sys/block/nvme0n1/queue/max_hw_sectors_kb
2048
The 2048kb mapping takes significantly longer, causing lock contention on the iova_rbtree_lock, as other resources such as ethernet NICs are also trying to
acquire the lock.
A workaround is to disable IOMMU with "iommu=off amd_iommu=off" on the kernel command line.
[Fix]
The fix is to clamp max_hw_sectors to the optimised IOVA size that still fits in the cache, so allocation and freeing of the IOVA is faster, during streaming DMA mapping.
The fix requires two dependency commits, which introduces a function to find the optimal value, dma_opt_mapping_size().
The dependencies are present in 6.0-rc1 and later.
The fix itself simply changes max_hw_sectors from dma_max_mapping_size() to dma_opt_mapping_size(). The fix needs a backport, as setting dev->ctrl.max_hw_sectors moved from nvme_reset_work() in 5.15 to nvme_pci_alloc_dev() in later releases.
The system needs to be extremely busy. So busy infact, that we cannot reproduce it in lab environments, and only in production.
The systems that hit this issue have 64 cores, ~90%+ sustained CPU usage, ~90%+ sustained memory usage, high disk I/O, and nearly saturated network throughput with 100Gb NICs.
The NVMe disk MUST have /sys/block/nvme0n1/queue/max_hw_sectors_kb greater than
128kb, in this case, 2048kb.
Leave the system at sustained load until IOVA allocations slow to a halt and soft or hardlockups occur, waiting for iova_rbtree_lock.
If you install the kernel and leave it running, the soft lockups will no longer occur.
[Where problems could occur]
We are changing the value of max_hw_sectors_kb for NVMe devices, for systems with IOMMU enabled. For those without IOMMU or IOMMU disabled, it will remain the same as it is now.
The value is the minimum between the maximum supported by hardware, and the largest that fits into cache. For some workloads, this might have a small impact on performance, due to the need to split up larger IOVA allocations into multiple smaller ones, but there should be a larger net gain due to IOVA allocations now fitting into the cache, and completing much faster than a single large one.
If a regression were to occur, users could disable the IOMMU as a workaround.
BugLink: https:/ /bugs.launchpad .net/bugs/
[Impact]
On systems with IOMMU enabled, every streaming DMA mapping involves an IOVA to be allocated and freed. For small mappings, IOVA sizes are normally cached, so IOVA allocations complete in a reasonable time. For larger mappings, things can be significantly slower to the point where softlockups occur due to lock contention
on iova_rbtree_lock.
commit 9257b4a206fc ("iommu/iova: introduce per-cpu caching to iova allocation")
introduced a scalable IOVA cache mechanism, that helps increase performance up
to 128kb mappings.
On systems that do larger streaming DMA mappings, e.g. a NVMe device with:
/sys/block/ nvme0n1/ queue/max_ hw_sectors_ kb
2048
The 2048kb mapping takes significantly longer, causing lock contention on the iova_rbtree_lock, as other resources such as ethernet NICs are also trying to
acquire the lock.
We hit the following soft lockup:
watchdog: BUG: soft lockup - CPU#60 stuck for 24s! spin_unlock_ irqrestore+ 0x25/0x30 timeout+ 0x82/0xc0 free+0x170/ 0x170 fn+0x2e/ 0x120 softirq+ 0x433/0x4c0 event+0x21/ 0x30 program_ event+0xab/ 0x130 0xdd/0x2ee rcu+0x7d/ 0xa0 apic_timer_ interrupt+ 0x80/0x90 apic_timer_ interrupt+ 0x1b/0x20 spin_unlock_ irqrestore+ 0x25/0x30 iova+0x1d8/ 0x1f0 iova_fast+ 0x5c/0x3a0 dma_alloc_ iova.isra. 0+0x128/ 0x170 0x1ab/0x4b0 dma_map_ sg+0x1a4/ 0x4c0 map_sg_ attrs+0x72/ 0x80 sg_attrs+ 0xe/0x20 data+0xde/ 0x800 [nvme] cpu_khz+ 0x10/0x10 rq+0xaf/ 0x1f0 [nvme] load_avg_ se+0x2a2/ 0x2c0 mq_try_ issue_directly+ 0x15b/0x200 request_ issue_directly+ 0x51/0xa0 try_issue_ list_directly+ 0x7f/0xf0 sched_insert_ requests+ 0xa4/0xf0 flush_plug_ list+0x103/ 0x1c0 plug_list+ 0xe3/0x110 submit_ bio+0x29d/ 0x600 bio+0x1e5/ 0x220 block_valid+ 0x9f/0xc0 bio_noacct+ 0xac/0x2c0 bio+0x50/ 0x140 readpages+ 0x6a2/0xe20 page_state+ 0x6b/0xb0 +0x37/0x40 0x95/0x280 ra_unbounded+ 0x161/0x220 cache_ra+ 0x3d/0x50 readahead+ 0x137/0x330 async_ra+ 0xa6/0xd0 get_pages+ 0x224/0x660 get_pages+ 0x9e/0x660 read+0xbe/ 0x410 file_read_ iter+0xe5/ 0x150 read_iter+ 0x5b/0x190 read+0x110/ 0x1a0 0x102/0x1a0 0x71/0xa0 sys_pread64+ 0x1e/0x30 network_ ops_symbols+ 0xc4de/ 0xf750 [falcon_ lsm_pinned_ 15907] 64+0x5c/ 0xc0 64+0x69/ 0xc0 64+0x69/ 0xc0 SYSCALL_ 64_after_ hwframe+ 0x61/0xcb
CPU: 60 PID: 608304 Comm: segment-merger- Tainted: P W EL 5.15.0-76-generic #83~20.04.1-Ubuntu
RIP: 0010:_raw_
Call Trace:
<IRQ>
fq_flush_
? fq_ring_
call_timer_
run_timer_
? lapic_next_
? clockevents_
__do_softirq+
irq_exit_
sysvec_
</IRQ>
<TASK>
asm_sysvec_
RIP: 0010:_raw_
...
alloc_
alloc_
iommu_
? __kmalloc+
iommu_
__dma_
dma_map_
nvme_map_
? recalibrate_
? ktime_get+0x46/0xc0
nvme_queue_
? __update_
__blk_
blk_mq_
blk_mq_
blk_mq_
blk_mq_
blk_flush_
blk_mq_
__submit_
? ext4_inode_
submit_
? xa_load+0x61/0xa0
submit_
ext4_mpage_
? __mod_lruvec_
ext4_readahead
read_pages+
page_cache_
do_page_
ondemand_
page_cache_
filemap_
? filemap_
filemap_
generic_
ext4_file_
new_sync_
vfs_read+
ksys_pread64+
__x64_
unload_
do_syscall_
? do_syscall_
? do_syscall_
entry_
A workaround is to disable IOMMU with "iommu=off amd_iommu=off" on the kernel command line.
[Fix]
The fix is to clamp max_hw_sectors to the optimised IOVA size that still fits in the cache, so allocation and freeing of the IOVA is faster, during streaming DMA mapping.
The fix requires two dependency commits, which introduces a function to find the optimal value, dma_opt_ mapping_ size().
commit a229cc14f339531 1b899e5e582b71e fa8dd01df0 mapping_ size() /github. com/torvalds/ linux/commit/ a229cc14f339531 1b899e5e582b71e fa8dd01df0
Author: John Garry <email address hidden>
Date: Thu Jul 14 19:15:24 2022 +0800
Subject: dma-mapping: add dma_opt_
Link: https:/
commit 6d9870b7e5def24 50e21316515b9ef c0529204dd opt_mapping_ size() /github. com/torvalds/ linux/commit/ 6d9870b7e5def24 50e21316515b9ef c0529204dd
Author: John Garry <email address hidden>
Date: Thu Jul 14 19:15:25 2022 +0800
Subject: dma-iommu: add iommu_dma_
Link: https:/
The dependencies are present in 6.0-rc1 and later.
The fix itself simply changes max_hw_sectors from dma_max_ mapping_ size() to dma_opt_ mapping_ size(). The fix needs a backport, as setting dev->ctrl. max_hw_ sectors moved from nvme_reset_work() in 5.15 to nvme_pci_ alloc_dev( ) in later releases.
commit 3710e2b056cb92a d816e4d79fa54a6 a5b6ad8cbd /github. com/torvalds/ linux/commit/ 3710e2b056cb92a d816e4d79fa54a6 a5b6ad8cbd
Author: Adrian Huang <email address hidden>
Date: Fri Apr 21 16:08:00 2023 +0800
Subject: nvme-pci: clamp max_hw_sectors based on DMA optimized limitation
Link: https:/
The fix is present in 6.4-rc3 and later.
[Testcase]
The system needs to be extremely busy. So busy infact, that we cannot reproduce it in lab environments, and only in production.
The systems that hit this issue have 64 cores, ~90%+ sustained CPU usage, ~90%+ sustained memory usage, high disk I/O, and nearly saturated network throughput with 100Gb NICs.
The NVMe disk MUST have /sys/block/ nvme0n1/ queue/max_ hw_sectors_ kb greater than
128kb, in this case, 2048kb.
Leave the system at sustained load until IOVA allocations slow to a halt and soft or hardlockups occur, waiting for iova_rbtree_lock.
A test kernel is available in the following ppa:
https:/ /launchpad. net/~mruffell/ +archive/ ubuntu/ sf374805- test
If you install the kernel and leave it running, the soft lockups will no longer occur.
[Where problems could occur]
We are changing the value of max_hw_sectors_kb for NVMe devices, for systems with IOMMU enabled. For those without IOMMU or IOMMU disabled, it will remain the same as it is now.
The value is the minimum between the maximum supported by hardware, and the largest that fits into cache. For some workloads, this might have a small impact on performance, due to the need to split up larger IOVA allocations into multiple smaller ones, but there should be a larger net gain due to IOVA allocations now fitting into the cache, and completing much faster than a single large one.
If a regression were to occur, users could disable the IOMMU as a workaround.