Page fault in RDMA ODP triggers BUG_ON during MMU notifier registration

Bug #2046534 reported by Chengen Du
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
New
Undecided
Unassigned
Focal
Fix Released
Undecided
Chengen Du

Bug Description

[Impact]
When a page fault is triggered in RDMA ODP, it registers an MMU notifier during the process.
Unfortunately, an error arises due to a race condition where the mm is released while attempting to register a notifier.
==========
Oct 14 23:38:32 bnode001 kernel: [1576115.901880] kernel BUG at mm/mmu_notifier.c:255!
Oct 14 23:38:32 bnode001 kernel: [1576115.909129] RSP: 0000:ffffbd3def843c90 EFLAGS: 00010246
Oct 14 23:38:32 bnode001 kernel: [1576115.912689] RAX: ffffa11635d20000 RBX: ffffa0f913ba5800 RCX: 0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.912691] RDX: ffffffffc0b666f0 RSI: ffffffffc0b601c7 RDI: ffffa0f913ba5850
Oct 14 23:38:32 bnode001 kernel: [1576115.913564] RAX: 0000000000000000 RBX: ffffffffc0b5a060 RCX: 0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.913565] RDX: 0000000000000007 RSI: ffffa1152ed3c400 RDI: ffffa1102dcd4300
Oct 14 23:38:32 bnode001 kernel: [1576115.914431] RBP: ffffbd3defcb7c88 R08: ffffa1163f4f50e0 R09: ffffa11638c072c0
Oct 14 23:38:32 bnode001 kernel: [1576115.914432] R10: ffffa0fd99a00000 R11: 0000000000000000 R12: ffffa1152c923b80
Oct 14 23:38:32 bnode001 kernel: [1576115.915263] RBP: ffffbd3def843cb0 R08: ffffa1163f7350e0 R09: ffffa11638c072c0
Oct 14 23:38:32 bnode001 kernel: [1576115.915265] R10: ffffa1088d000000 R11: 0000000000000000 R12: ffffa1102dcd4300
Oct 14 23:38:32 bnode001 kernel: [1576115.916079] R13: ffffa1152c923b80 R14: ffffa1152c923bf8 R15: ffffa114f8127800
Oct 14 23:38:32 bnode001 kernel: [1576115.916080] FS: 0000000000000000(0000) GS:ffffa1163f4c0000(0000) knlGS:0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.917705] R13: ffffa1152ed3c400 R14: ffffa1152ed3c478 R15: ffffa1101cbfbc00
Oct 14 23:38:32 bnode001 kernel: [1576115.917706] FS: 0000000000000000(0000) GS:ffffa1163f700000(0000) knlGS:0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.918506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 14 23:38:32 bnode001 kernel: [1576115.918508] CR2: 00007f94146af5e0 CR3: 0000001722472004 CR4: 0000000000760ee0
Oct 14 23:38:32 bnode001 kernel: [1576115.919301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 14 23:38:32 bnode001 kernel: [1576115.919302] CR2: 00007f32f0a2dc80 CR3: 0000001f9f1fc004 CR4: 0000000000760ee0
Oct 14 23:38:32 bnode001 kernel: [1576115.920082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.920084] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Oct 14 23:38:32 bnode001 kernel: [1576115.920850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 14 23:38:32 bnode001 kernel: [1576115.921604] PKRU: 55555554
Oct 14 23:38:32 bnode001 kernel: [1576115.921605] Call Trace:
Oct 14 23:38:32 bnode001 kernel: [1576115.922354] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Oct 14 23:38:32 bnode001 kernel: [1576115.922355] PKRU: 55555554
Oct 14 23:38:32 bnode001 kernel: [1576115.923112] mmu_notifier_get_locked+0x5f/0xe0
Oct 14 23:38:32 bnode001 kernel: [1576115.923867] Call Trace:
Oct 14 23:38:32 bnode001 kernel: [1576115.923870] ? mmu_notifier_get_locked+0x79/0xe0
Oct 14 23:38:32 bnode001 kernel: [1576115.924645] ib_umem_odp_alloc_child+0x15a/0x290 [ib_core]
Oct 14 23:38:32 bnode001 kernel: [1576115.925409] ib_umem_odp_alloc_child+0x15a/0x290 [ib_core]
Oct 14 23:38:32 bnode001 kernel: [1576115.926161] pagefault_mr+0x312/0x5d0 [mlx5_ib]
Oct 14 23:38:32 bnode001 kernel: [1576115.926906] pagefault_mr+0x312/0x5d0 [mlx5_ib]
Oct 14 23:38:32 bnode001 kernel: [1576115.927651] pagefault_single_data_segment.isra.0+0x284/0x490 [mlx5_ib]
Oct 14 23:38:32 bnode001 kernel: [1576115.928393] pagefault_single_data_segment.isra.0+0x284/0x490 [mlx5_ib]
Oct 14 23:38:32 bnode001 kernel: [1576115.929131] mlx5_ib_eqe_pf_action+0x7d5/0x990 [mlx5_ib]
Oct 14 23:38:32 bnode001 kernel: [1576115.929866] mlx5_ib_eqe_pf_action+0x7d5/0x990 [mlx5_ib]
Oct 14 23:38:32 bnode001 kernel: [1576115.930610] process_one_work+0x1eb/0x3b0
Oct 14 23:38:32 bnode001 kernel: [1576115.931351] process_one_work+0x1eb/0x3b0
Oct 14 23:38:32 bnode001 kernel: [1576115.932084] worker_thread+0x4d/0x400
Oct 14 23:38:32 bnode001 kernel: [1576115.932813] worker_thread+0x4d/0x400
Oct 14 23:38:32 bnode001 kernel: [1576115.933543] kthread+0x104/0x140
Oct 14 23:38:32 bnode001 kernel: [1576115.934272] kthread+0x104/0x140
Oct 14 23:38:32 bnode001 kernel: [1576115.934986] ? process_one_work+0x3b0/0x3b0
Oct 14 23:38:32 bnode001 kernel: [1576115.934988] ? kthread_park+0x90/0x90
Oct 14 23:38:32 bnode001 kernel: [1576115.935687] ? process_one_work+0x3b0/0x3b0
Oct 14 23:38:32 bnode001 kernel: [1576115.935689] ? kthread_park+0x90/0x90
Oct 14 23:38:32 bnode001 kernel: [1576115.936387] ret_from_fork+0x1f/0x40
Oct 14 23:38:32 bnode001 kernel: [1576115.936389] ---[ end trace 1823b59637af552f ]---
Oct 14 23:38:32 bnode001 kernel: [1576115.937077] ret_from_fork+0x1f/0x40
==========

[Fix]
There is an upstream patch that fixes this issue:
==========
commit a4e63bce1414df7ab6eb82ca9feb8494ce13e554
Author: Jason Gunthorpe <email address hidden>
Date: Thu Feb 27 13:41:18 2020 +0200

    RDMA/odp: Ensure the mm is still alive before creating an implicit child
==========
The patch has been implemented to modify the behavior by calling mmget() around the registration, thereby ensuring it is held to avoid the race condition.

[Test Plan]
This is a race condition issue and may not be easy to reproduce.
The test plan involves running on a system with InfiniBand, triggering the RDMA ODP page fault path to check if everything works as expected.

[Where problems could occur]
The patch calls mmget_not_zero() before registering the MMU notifier and puts it after registration is done.
This change may not affect the execution result but ensures that the mm will not be released during registration.
The risk associated with adopting this patch can be judged as low.

Chengen Du (chengendu)
Changed in linux (Ubuntu Focal):
assignee: nobody → Chengen Du (chengendu)
Chengen Du (chengendu)
Changed in linux (Ubuntu Focal):
status: New → In Progress
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-171.189 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux' to 'verification-done-focal-linux'. If the problem still exists, change the tag 'verification-needed-focal-linux' to 'verification-failed-focal-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-v2 verification-needed-focal-linux
Revision history for this message
Chengen Du (chengendu) wrote :

The verification of jammy (5.4.0-171.189) was conducted using a custom approach due to the absence of an InfiniBand environment for testing the patch.
The patch specifically altered the count of mm->mm_users within a function.
To ensure that this modification did not impact the original logic, a kernel module was developed.
The module replicated the content of ib_umem_odp_alloc_child but substituted InfiniBand-related functions with other unaffected custom functions for testing purposes.
===
#include <linux/init.h>
#include <linux/module.h>
#include <linux/sched/mm.h>
#include <linux/slab.h>

MODULE_DESCRIPTION("Test");
MODULE_LICENSE("GPL");

int inner_test(void) {
 return 0;
}

int test(void) {
 int ret;
 char *odp_data = kzalloc(8, GFP_KERNEL);
 if (!odp_data)
  return -ENOMEM;
 if (!mmget_not_zero(current->mm)) {
  ret = -EFAULT;
  goto out_free;
 }

 ret = inner_test();
 if (ret)
  goto out_mmput;
 mmput(current->mm);
 return 0;

out_mmput:
 mmput(current->mm);
out_free:
 kfree(odp_data);
 return ret;
}

static int test_init(void) { return test(); }

static void test_exit(void) {}

module_init(test_init);
module_exit(test_exit);
===

Executing this kernel module allows us to verify that the count of mm_users aligns with expectations, validated through the use of the atomic_read function.

tags: added: verification-done-focal-linux
removed: verification-needed-focal-linux
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (16.4 KiB)

This bug was fixed in the package linux - 5.4.0-171.189

---------------
linux (5.4.0-171.189) focal; urgency=medium

  * focal/linux: 5.4.0-171.189 -proposed tracker (LP: #2048282)

  * Packaging resync (LP: #1786013)
    - [Packaging] remove helper scripts
    - [Packaging] update annotations scripts
    - debian/dkms-versions -- update from kernel-versions (main/2024.01.08)

  * Page fault in RDMA ODP triggers BUG_ON during MMU notifier registration
    (LP: #2046534)
    - RDMA/odp: Ensure the mm is still alive before creating an implicit child

  * Hotplugging SCSI disk in QEMU VM fails (LP: #2047382)
    - Revert "PCI: acpiphp: Reassign resources on bridge if necessary"

  * CVE-2023-6040
    - netfilter: nf_tables: Reject tables of unsupported family

  * kernel_selftests failures on kernel-P10d-LPAR10.ppc64el.10
    (LP: #2032641)
    - selftests: Skip TM tests on synthetic TM implementations

  * [Debian] autoreconstruct - Do not generate chmod -x for deleted files
    (LP: #2045562)
    - [Debian] autoreconstruct - Do not generate chmod -x for deleted files

  * CVE-2023-6931
    - perf/core: Add a new read format to get a number of lost samples
    - perf: Fix perf_event_validate_size()
    - perf: Fix perf_event_validate_size() lockdep splat

  * CVE-2023-6932
    - ipv4: igmp: fix refcnt uaf issue when receiving igmp query packet

  * CVE-2023-6606
    - smb: client: fix OOB in smbCalcSize()

  * CVE-2023-45863
    - kobject: Fix slab-out-of-bounds in fill_kobj_path()

  * Focal update: v5.4.259 upstream stable release (LP: #2043724)
    - RDMA/cxgb4: Check skb value for failure to allocate
    - lib/test_meminit: fix off-by-one error in test_pages()
    - pwm: hibvt: Explicitly set .polarity in .get_state()
    - HID: logitech-hidpp: Fix kernel crash on receiver USB disconnect
    - quota: Fix slow quotaoff
    - net: prevent address rewrite in kernel_bind()
    - drm: etvnaviv: fix bad backport leading to warning
    - drm/msm/dsi: skip the wait for video mode done if not applicable
    - ravb: Fix up dma_free_coherent() call in ravb_remove()
    - ieee802154: ca8210: Fix a potential UAF in ca8210_probe
    - mlxsw: fix mlxsw_sp2_nve_vxlan_learning_set() return type
    - xen-netback: use default TX queue size for vifs
    - drm/vmwgfx: fix typo of sizeof argument
    - ixgbe: fix crash with empty VF macvlan list
    - net: nfc: fix races in nfc_llcp_sock_get() and nfc_llcp_sock_get_sn()
    - nfc: nci: assert requested protocol is valid
    - workqueue: Override implicit ordered attribute in
      workqueue_apply_unbound_cpumask()
    - dmaengine: stm32-mdma: abort resume if no ongoing transfer
    - usb: xhci: xhci-ring: Use sysdev for mapping bounce buffer
    - net: usb: dm9601: fix uninitialized variable use in dm9601_mdio_read
    - usb: dwc3: Soft reset phy on probe for host
    - usb: musb: Get the musb_qh poniter after musb_giveback
    - usb: musb: Modify the "HWVers" register address
    - iio: pressure: bmp280: Fix NULL pointer exception
    - iio: pressure: dps310: Adjust Timeout Settings
    - iio: pressure: ms5611: ms5611_prom_is_valid false negative bug
    - mcb: remove is_added flag from mcb_device struct
   ...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-oracle/5.4.0-1117.126 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-oracle' to 'verification-done-focal-linux-oracle'. If the problem still exists, change the tag 'verification-needed-focal-linux-oracle' to 'verification-failed-focal-linux-oracle'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-oracle-v2 verification-needed-focal-linux-oracle
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-raspi/5.4.0-1102.114 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-raspi' to 'verification-done-focal-linux-raspi'. If the problem still exists, change the tag 'verification-needed-focal-linux-raspi' to 'verification-failed-focal-linux-raspi'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-raspi-v2 verification-needed-focal-linux-raspi
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.