Cephfs + fscache: unable to handle kernel NULL pointer dereference at 0000000000000000 IP: jbd2__journal_start+0x22/0x1f0

Bug #1783246 reported by Daniel Axtens on 2018-07-24
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)

Bug Description

SRU Justification

Certain sequences of file system operations on a cephfs volume backed by fscache with an ext4 store can cause a kernel BUG:

[ 5818.932770] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 5818.934354] IP: jbd2__journal_start+0x33/0x1e0
[ 5818.962490] Call Trace:
[ 5818.963055] ? ext4_writepages+0x5d5/0xf40
[ 5818.963884] __ext4_journal_start_sb+0x6d/0x120
[ 5818.964994] ext4_writepages+0x5d5/0xf40
[ 5818.965991] ? __enqueue_entity+0x5c/0x60
[ 5818.966791] ? check_preempt_wakeup+0x130/0x240
[ 5818.967679] do_writepages+0x4b/0xe0
[ 5818.968625] ? ext4_mark_inode_dirty+0x1d0/0x1d0
[ 5818.969526] ? do_writepages+0x4b/0xe0
[ 5818.970493] ? ext4_statfs+0x114/0x260
[ 5818.971267] __filemap_fdatawrite_range+0xc1/0x100
[ 5818.972425] ? __filemap_fdatawrite_range+0xc1/0x100
[ 5818.973385] filemap_write_and_wait+0x31/0x90
[ 5818.974461] ext4_bmap+0x8c/0xe0
[ 5818.975150] cachefiles_read_or_alloc_pages+0x1bf/0xd90 [cachefiles]
[ 5818.976718] ? _cond_resched+0x19/0x40
[ 5818.977482] ? wake_up_bit+0x42/0x50
[ 5818.978227] ? fscache_run_op.isra.8+0x4c/0x80 [fscache]
[ 5818.979249] __fscache_read_or_alloc_pages+0x1d3/0x2e0 [fscache]
[ 5818.980397] ceph_readpages_from_fscache+0x6c/0xe0 [ceph]
[ 5818.981630] ceph_readpages+0x49/0x100 [ceph]
[ 5818.982691] __do_page_cache_readahead+0x1c9/0x2c0
[ 5818.983628] ? __cap_is_valid+0x21/0xb0 [ceph]
[ 5818.984526] ondemand_readahead+0x11a/0x2a0
[ 5818.985374] ? ondemand_readahead+0x11a/0x2a0
[ 5818.986825] page_cache_async_readahead+0x71/0x80
[ 5818.987751] generic_file_read_iter+0x784/0xbf0
[ 5818.988663] ? ceph_put_cap_refs+0x1c4/0x330 [ceph]
[ 5818.989620] ? page_cache_tree_insert+0xe0/0xe0
[ 5818.990519] ceph_read_iter+0x106/0x820 [ceph]
[ 5818.991818] new_sync_read+0xe4/0x130
[ 5818.992588] __vfs_read+0x29/0x40
[ 5818.993504] vfs_read+0x8e/0x130
[ 5818.994192] SyS_read+0x55/0xc0
[ 5818.994870] do_syscall_64+0x73/0x130
[ 5818.995632] entry_SYSCALL_64_after_hwframe+0x3d/0xa2

Cherry-pick 5d988308283ecf062fa88f20ae05c52cce0bcdca from upstream.

This patch stops cephfs from reusing current->journal for its own internal use, which means that it's valid when ext4 uses it via fscache.

A user has been using the following test case:
( cat /proc/fs/fscache/stats > ~/test.log; i=0; while true; do
    touch small; echo 3 > /proc/sys/vm/drop_caches & md5sum small; let "i++"; if ! (( $i % 1000 )); then
        echo "Test iteration $i done" >> ~/test.log; cat /proc/fs/fscache/stats >> ~/test.log;
done ) > ~/nohup.out 2>&1

(It boils down to "touch file; drop caches; read file")
Without the patch, this fails very quickly - usually the first time, always within a few iterations. With the patch, the user ran this loop for over 60 hours without incident.

[Regression potential]
The change is not trivial, but is limited to cephfs, and has been in mainline since v4.16. So the risk of regression is well contained.

Changed in linux (Ubuntu Bionic):
status: New → Fix Committed
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
David Coronel (davecore) wrote :

I have confirmation from a user who has done verification for this kernel. Changing to verification-done-bionic.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Daniel Axtens (daxtens) on 2018-08-16
Changed in linux (Ubuntu):
assignee: Daniel Axtens (daxtens) → nobody
Launchpad Janitor (janitor) wrote :
Download full text (35.6 KiB)

This bug was fixed in the package linux - 4.15.0-33.36

linux (4.15.0-33.36) bionic; urgency=medium

  * linux: 4.15.0-33.36 -proposed tracker (LP: #1787149)

  * RTNL assertion failure on ipvlan (LP: #1776927)
    - ipvlan: drop ipv6 dependency
    - ipvlan: use per device spinlock to protect addrs list updates
    - SAUCE: fix warning from "ipvlan: drop ipv6 dependency"

  * ubuntu_bpf_jit test failed on Bionic s390x systems (LP: #1753941)
    - test_bpf: flag tests that cannot be jited on s390

  * HDMI/DP audio can't work on the laptop of Dell Latitude 5495 (LP: #1782689)
    - drm/nouveau: fix nouveau_dsm_get_client_id()'s return type
    - drm/radeon: fix radeon_atpx_get_client_id()'s return type
    - drm/amdgpu: fix amdgpu_atpx_get_client_id()'s return type
    - platform/x86: apple-gmux: fix gmux_get_client_id()'s return type
    - vga_switcheroo: set audio client id according to bound GPU id

  * locking sockets broken due to missing AppArmor socket mediation patches
    (LP: #1780227)
    - UBUNTU SAUCE: apparmor: fix apparmor mediating locking non-fs, unix sockets

  * Update2 for ocxl driver (LP: #1781436)
    - ocxl: Fix page fault handler in case of fault on dying process

  * netns: unable to follow an interface that moves to another netns
    (LP: #1774225)
    - net: core: Expose number of link up/down transitions
    - dev: always advertise the new nsid when the netns iface changes
    - dev: advertise the new ifindex when the netns iface changes

  * [Bionic] Disk IO hangs when using BFQ as io scheduler (LP: #1780066)
    - block, bfq: fix occurrences of request finish method's old name
    - block, bfq: remove batches of confusing ifdefs
    - block, bfq: add requeue-request hook

  * HP ProBook 455 G5 needs mute-led-gpio fixup (LP: #1781763)
    - ALSA: hda: add mute led support for HP ProBook 455 G5

  * [Bionic] bug fixes to improve stability of the ThunderX2 i2c driver
    (LP: #1781476)
    - i2c: xlp9xx: Fix issue seen when updating receive length
    - i2c: xlp9xx: Make sure the transfer size is not more than

  * x86/kvm: fix LAPIC timer drift when guest uses periodic mode (LP: #1778486)
    - x86/kvm: fix LAPIC timer drift when guest uses periodic mode

  * Please include ax88179_178a and r8152 modules in d-i udeb (LP: #1771823)
    - [Config:] d-i: Add ax88179_178a and r8152 to nic-modules

  * Nvidia fails after switching its mode (LP: #1778658)
    - PCI: Restore config space on runtime resume despite being unbound

  * Kernel error "task zfs:pid blocked for more than 120 seconds" (LP: #1781364)
    - SAUCE: (noup) zfs to 0.7.5-1ubuntu16.3

  * CVE-2018-12232
    - PATCH 1/1] socket: close race condition between sock_close() and

  * CVE-2018-10323
    - xfs: set format back to extents if xfs_bmap_extents_to_btree

  * change front mic location for more lenovo m7/8/9xx machines (LP: #1781316)
    - ALSA: hda/realtek - Fix the problem of two front mics on more machines
    - ALSA: hda/realtek - two more lenovo models need fixup of MIC_LOCATION

  * Cephfs + fscache: unab...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers