Page fault in RDMA ODP triggers BUG_ON during MMU notifier registration
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
New
|
Undecided
|
Unassigned | ||
Focal |
Fix Released
|
Undecided
|
Chengen Du |
Bug Description
[Impact]
When a page fault is triggered in RDMA ODP, it registers an MMU notifier during the process.
Unfortunately, an error arises due to a race condition where the mm is released while attempting to register a notifier.
==========
Oct 14 23:38:32 bnode001 kernel: [1576115.901880] kernel BUG at mm/mmu_
Oct 14 23:38:32 bnode001 kernel: [1576115.909129] RSP: 0000:ffffbd3def
Oct 14 23:38:32 bnode001 kernel: [1576115.912689] RAX: ffffa11635d20000 RBX: ffffa0f913ba5800 RCX: 0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.912691] RDX: ffffffffc0b666f0 RSI: ffffffffc0b601c7 RDI: ffffa0f913ba5850
Oct 14 23:38:32 bnode001 kernel: [1576115.913564] RAX: 0000000000000000 RBX: ffffffffc0b5a060 RCX: 0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.913565] RDX: 0000000000000007 RSI: ffffa1152ed3c400 RDI: ffffa1102dcd4300
Oct 14 23:38:32 bnode001 kernel: [1576115.914431] RBP: ffffbd3defcb7c88 R08: ffffa1163f4f50e0 R09: ffffa11638c072c0
Oct 14 23:38:32 bnode001 kernel: [1576115.914432] R10: ffffa0fd99a00000 R11: 0000000000000000 R12: ffffa1152c923b80
Oct 14 23:38:32 bnode001 kernel: [1576115.915263] RBP: ffffbd3def843cb0 R08: ffffa1163f7350e0 R09: ffffa11638c072c0
Oct 14 23:38:32 bnode001 kernel: [1576115.915265] R10: ffffa1088d000000 R11: 0000000000000000 R12: ffffa1102dcd4300
Oct 14 23:38:32 bnode001 kernel: [1576115.916079] R13: ffffa1152c923b80 R14: ffffa1152c923bf8 R15: ffffa114f8127800
Oct 14 23:38:32 bnode001 kernel: [1576115.916080] FS: 000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.917705] R13: ffffa1152ed3c400 R14: ffffa1152ed3c478 R15: ffffa1101cbfbc00
Oct 14 23:38:32 bnode001 kernel: [1576115.917706] FS: 000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.918506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 14 23:38:32 bnode001 kernel: [1576115.918508] CR2: 00007f94146af5e0 CR3: 0000001722472004 CR4: 0000000000760ee0
Oct 14 23:38:32 bnode001 kernel: [1576115.919301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 14 23:38:32 bnode001 kernel: [1576115.919302] CR2: 00007f32f0a2dc80 CR3: 0000001f9f1fc004 CR4: 0000000000760ee0
Oct 14 23:38:32 bnode001 kernel: [1576115.920082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.920084] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Oct 14 23:38:32 bnode001 kernel: [1576115.920850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.921604] PKRU: 55555554
Oct 14 23:38:32 bnode001 kernel: [1576115.921605] Call Trace:
Oct 14 23:38:32 bnode001 kernel: [1576115.922354] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Oct 14 23:38:32 bnode001 kernel: [1576115.922355] PKRU: 55555554
Oct 14 23:38:32 bnode001 kernel: [1576115.923112] mmu_notifier_
Oct 14 23:38:32 bnode001 kernel: [1576115.923867] Call Trace:
Oct 14 23:38:32 bnode001 kernel: [1576115.923870] ? mmu_notifier_
Oct 14 23:38:32 bnode001 kernel: [1576115.924645] ib_umem_
Oct 14 23:38:32 bnode001 kernel: [1576115.925409] ib_umem_
Oct 14 23:38:32 bnode001 kernel: [1576115.926161] pagefault_
Oct 14 23:38:32 bnode001 kernel: [1576115.926906] pagefault_
Oct 14 23:38:32 bnode001 kernel: [1576115.927651] pagefault_
Oct 14 23:38:32 bnode001 kernel: [1576115.928393] pagefault_
Oct 14 23:38:32 bnode001 kernel: [1576115.929131] mlx5_ib_
Oct 14 23:38:32 bnode001 kernel: [1576115.929866] mlx5_ib_
Oct 14 23:38:32 bnode001 kernel: [1576115.930610] process_
Oct 14 23:38:32 bnode001 kernel: [1576115.931351] process_
Oct 14 23:38:32 bnode001 kernel: [1576115.932084] worker_
Oct 14 23:38:32 bnode001 kernel: [1576115.932813] worker_
Oct 14 23:38:32 bnode001 kernel: [1576115.933543] kthread+0x104/0x140
Oct 14 23:38:32 bnode001 kernel: [1576115.934272] kthread+0x104/0x140
Oct 14 23:38:32 bnode001 kernel: [1576115.934986] ? process_
Oct 14 23:38:32 bnode001 kernel: [1576115.934988] ? kthread_
Oct 14 23:38:32 bnode001 kernel: [1576115.935687] ? process_
Oct 14 23:38:32 bnode001 kernel: [1576115.935689] ? kthread_
Oct 14 23:38:32 bnode001 kernel: [1576115.936387] ret_from_
Oct 14 23:38:32 bnode001 kernel: [1576115.936389] ---[ end trace 1823b59637af552f ]---
Oct 14 23:38:32 bnode001 kernel: [1576115.937077] ret_from_
==========
[Fix]
There is an upstream patch that fixes this issue:
==========
commit a4e63bce1414df7
Author: Jason Gunthorpe <email address hidden>
Date: Thu Feb 27 13:41:18 2020 +0200
RDMA/odp: Ensure the mm is still alive before creating an implicit child
==========
The patch has been implemented to modify the behavior by calling mmget() around the registration, thereby ensuring it is held to avoid the race condition.
[Test Plan]
This is a race condition issue and may not be easy to reproduce.
The test plan involves running on a system with InfiniBand, triggering the RDMA ODP page fault path to check if everything works as expected.
[Where problems could occur]
The patch calls mmget_not_zero() before registering the MMU notifier and puts it after registration is done.
This change may not affect the execution result but ensures that the mm will not be released during registration.
The risk associated with adopting this patch can be judged as low.
CVE References
Changed in linux (Ubuntu Focal): | |
assignee: | nobody → Chengen Du (chengendu) |
Changed in linux (Ubuntu Focal): | |
status: | New → In Progress |
Changed in linux (Ubuntu Focal): | |
status: | In Progress → Fix Committed |
This bug is awaiting verification that the linux/5.4.0-171.189 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification- needed- focal-linux' to 'verification- done-focal- linux'. If the problem still exists, change the tag 'verification- needed- focal-linux' to 'verification- failed- focal-linux' .
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/ /wiki.ubuntu. com/Testing/ EnableProposed for documentation how to enable and use -proposed. Thank you!