Cephfs + fscache: unable to handle kernel NULL pointer dereference at 0000000000000000 IP: jbd2__journal_start+0x22/0x1f0

Bug #1783246 reported by Daniel Axtens
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned

Bug Description

SRU Justification
-----------------

[Impact]
Certain sequences of file system operations on a cephfs volume backed by fscache with an ext4 store can cause a kernel BUG:

[ 5818.932770] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 5818.934354] IP: jbd2__journal_start+0x33/0x1e0
...
[ 5818.962490] Call Trace:
[ 5818.963055] ? ext4_writepages+0x5d5/0xf40
[ 5818.963884] __ext4_journal_start_sb+0x6d/0x120
[ 5818.964994] ext4_writepages+0x5d5/0xf40
[ 5818.965991] ? __enqueue_entity+0x5c/0x60
[ 5818.966791] ? check_preempt_wakeup+0x130/0x240
[ 5818.967679] do_writepages+0x4b/0xe0
[ 5818.968625] ? ext4_mark_inode_dirty+0x1d0/0x1d0
[ 5818.969526] ? do_writepages+0x4b/0xe0
[ 5818.970493] ? ext4_statfs+0x114/0x260
[ 5818.971267] __filemap_fdatawrite_range+0xc1/0x100
[ 5818.972425] ? __filemap_fdatawrite_range+0xc1/0x100
[ 5818.973385] filemap_write_and_wait+0x31/0x90
[ 5818.974461] ext4_bmap+0x8c/0xe0
[ 5818.975150] cachefiles_read_or_alloc_pages+0x1bf/0xd90 [cachefiles]
[ 5818.976718] ? _cond_resched+0x19/0x40
[ 5818.977482] ? wake_up_bit+0x42/0x50
[ 5818.978227] ? fscache_run_op.isra.8+0x4c/0x80 [fscache]
[ 5818.979249] __fscache_read_or_alloc_pages+0x1d3/0x2e0 [fscache]
[ 5818.980397] ceph_readpages_from_fscache+0x6c/0xe0 [ceph]
[ 5818.981630] ceph_readpages+0x49/0x100 [ceph]
[ 5818.982691] __do_page_cache_readahead+0x1c9/0x2c0
[ 5818.983628] ? __cap_is_valid+0x21/0xb0 [ceph]
[ 5818.984526] ondemand_readahead+0x11a/0x2a0
[ 5818.985374] ? ondemand_readahead+0x11a/0x2a0
[ 5818.986825] page_cache_async_readahead+0x71/0x80
[ 5818.987751] generic_file_read_iter+0x784/0xbf0
[ 5818.988663] ? ceph_put_cap_refs+0x1c4/0x330 [ceph]
[ 5818.989620] ? page_cache_tree_insert+0xe0/0xe0
[ 5818.990519] ceph_read_iter+0x106/0x820 [ceph]
[ 5818.991818] new_sync_read+0xe4/0x130
[ 5818.992588] __vfs_read+0x29/0x40
[ 5818.993504] vfs_read+0x8e/0x130
[ 5818.994192] SyS_read+0x55/0xc0
[ 5818.994870] do_syscall_64+0x73/0x130
[ 5818.995632] entry_SYSCALL_64_after_hwframe+0x3d/0xa2

[Fix]
Cherry-pick 5d988308283ecf062fa88f20ae05c52cce0bcdca from upstream.

This patch stops cephfs from reusing current->journal for its own internal use, which means that it's valid when ext4 uses it via fscache.

[Testcase]
A user has been using the following test case:
( cat /proc/fs/fscache/stats > ~/test.log; i=0; while true; do
    touch small; echo 3 > /proc/sys/vm/drop_caches & md5sum small; let "i++"; if ! (( $i % 1000 )); then
        echo "Test iteration $i done" >> ~/test.log; cat /proc/fs/fscache/stats >> ~/test.log;
    fi;
done ) > ~/nohup.out 2>&1

(It boils down to "touch file; drop caches; read file")
Without the patch, this fails very quickly - usually the first time, always within a few iterations. With the patch, the user ran this loop for over 60 hours without incident.

[Regression potential]
The change is not trivial, but is limited to cephfs, and has been in mainline since v4.16. So the risk of regression is well contained.

Changed in linux (Ubuntu Bionic):
status: New → Fix Committed
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Revision history for this message
David Coronel (davecore) wrote :

I have confirmation from a user who has done verification for this kernel. Changing to verification-done-bionic.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Daniel Axtens (daxtens)
Changed in linux (Ubuntu):
assignee: Daniel Axtens (daxtens) → nobody
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (35.6 KiB)

This bug was fixed in the package linux - 4.15.0-33.36

---------------
linux (4.15.0-33.36) bionic; urgency=medium

  * linux: 4.15.0-33.36 -proposed tracker (LP: #1787149)

  * RTNL assertion failure on ipvlan (LP: #1776927)
    - ipvlan: drop ipv6 dependency
    - ipvlan: use per device spinlock to protect addrs list updates
    - SAUCE: fix warning from "ipvlan: drop ipv6 dependency"

  * ubuntu_bpf_jit test failed on Bionic s390x systems (LP: #1753941)
    - test_bpf: flag tests that cannot be jited on s390

  * HDMI/DP audio can't work on the laptop of Dell Latitude 5495 (LP: #1782689)
    - drm/nouveau: fix nouveau_dsm_get_client_id()'s return type
    - drm/radeon: fix radeon_atpx_get_client_id()'s return type
    - drm/amdgpu: fix amdgpu_atpx_get_client_id()'s return type
    - platform/x86: apple-gmux: fix gmux_get_client_id()'s return type
    - ALSA: hda: use PCI_BASE_CLASS_DISPLAY to replace PCI_CLASS_DISPLAY_VGA
    - vga_switcheroo: set audio client id according to bound GPU id

  * locking sockets broken due to missing AppArmor socket mediation patches
    (LP: #1780227)
    - UBUNTU SAUCE: apparmor: fix apparmor mediating locking non-fs, unix sockets

  * Update2 for ocxl driver (LP: #1781436)
    - ocxl: Fix page fault handler in case of fault on dying process

  * netns: unable to follow an interface that moves to another netns
    (LP: #1774225)
    - net: core: Expose number of link up/down transitions
    - dev: always advertise the new nsid when the netns iface changes
    - dev: advertise the new ifindex when the netns iface changes

  * [Bionic] Disk IO hangs when using BFQ as io scheduler (LP: #1780066)
    - block, bfq: fix occurrences of request finish method's old name
    - block, bfq: remove batches of confusing ifdefs
    - block, bfq: add requeue-request hook

  * HP ProBook 455 G5 needs mute-led-gpio fixup (LP: #1781763)
    - ALSA: hda: add mute led support for HP ProBook 455 G5

  * [Bionic] bug fixes to improve stability of the ThunderX2 i2c driver
    (LP: #1781476)
    - i2c: xlp9xx: Fix issue seen when updating receive length
    - i2c: xlp9xx: Make sure the transfer size is not more than
      I2C_SMBUS_BLOCK_SIZE

  * x86/kvm: fix LAPIC timer drift when guest uses periodic mode (LP: #1778486)
    - x86/kvm: fix LAPIC timer drift when guest uses periodic mode

  * Please include ax88179_178a and r8152 modules in d-i udeb (LP: #1771823)
    - [Config:] d-i: Add ax88179_178a and r8152 to nic-modules

  * Nvidia fails after switching its mode (LP: #1778658)
    - PCI: Restore config space on runtime resume despite being unbound

  * Kernel error "task zfs:pid blocked for more than 120 seconds" (LP: #1781364)
    - SAUCE: (noup) zfs to 0.7.5-1ubuntu16.3

  * CVE-2018-12232
    - PATCH 1/1] socket: close race condition between sock_close() and
      sockfs_setattr()

  * CVE-2018-10323
    - xfs: set format back to extents if xfs_bmap_extents_to_btree

  * change front mic location for more lenovo m7/8/9xx machines (LP: #1781316)
    - ALSA: hda/realtek - Fix the problem of two front mics on more machines
    - ALSA: hda/realtek - two more lenovo models need fixup of MIC_LOCATION

  * Cephfs + fscache: unab...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.