Cephfs + fscache: unable to handle kernel NULL pointer dereference at 0000000000000000 IP: jbd2__journal_start+0x22/0x1f0
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned | ||
Bionic |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
SRU Justification
-----------------
[Impact]
Certain sequences of file system operations on a cephfs volume backed by fscache with an ext4 store can cause a kernel BUG:
[ 5818.932770] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 5818.934354] IP: jbd2__journal_
...
[ 5818.962490] Call Trace:
[ 5818.963055] ? ext4_writepages
[ 5818.963884] __ext4_
[ 5818.964994] ext4_writepages
[ 5818.965991] ? __enqueue_
[ 5818.966791] ? check_preempt_
[ 5818.967679] do_writepages+
[ 5818.968625] ? ext4_mark_
[ 5818.969526] ? do_writepages+
[ 5818.970493] ? ext4_statfs+
[ 5818.971267] __filemap_
[ 5818.972425] ? __filemap_
[ 5818.973385] filemap_
[ 5818.974461] ext4_bmap+0x8c/0xe0
[ 5818.975150] cachefiles_
[ 5818.976718] ? _cond_resched+
[ 5818.977482] ? wake_up_
[ 5818.978227] ? fscache_
[ 5818.979249] __fscache_
[ 5818.980397] ceph_readpages_
[ 5818.981630] ceph_readpages+
[ 5818.982691] __do_page_
[ 5818.983628] ? __cap_is_
[ 5818.984526] ondemand_
[ 5818.985374] ? ondemand_
[ 5818.986825] page_cache_
[ 5818.987751] generic_
[ 5818.988663] ? ceph_put_
[ 5818.989620] ? page_cache_
[ 5818.990519] ceph_read_
[ 5818.991818] new_sync_
[ 5818.992588] __vfs_read+
[ 5818.993504] vfs_read+0x8e/0x130
[ 5818.994192] SyS_read+0x55/0xc0
[ 5818.994870] do_syscall_
[ 5818.995632] entry_SYSCALL_
[Fix]
Cherry-pick 5d988308283ecf0
This patch stops cephfs from reusing current->journal for its own internal use, which means that it's valid when ext4 uses it via fscache.
[Testcase]
A user has been using the following test case:
( cat /proc/fs/
touch small; echo 3 > /proc/sys/
echo "Test iteration $i done" >> ~/test.log; cat /proc/fs/
fi;
done ) > ~/nohup.out 2>&1
(It boils down to "touch file; drop caches; read file")
Without the patch, this fails very quickly - usually the first time, always within a few iterations. With the patch, the user ran this loop for over 60 hours without incident.
[Regression potential]
The change is not trivial, but is limited to cephfs, and has been in mainline since v4.16. So the risk of regression is well contained.
Changed in linux (Ubuntu Bionic): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu): | |
assignee: | Daniel Axtens (daxtens) → nobody |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification- needed- bionic' to 'verification- done-bionic' . If the problem still exists, change the tag 'verification- needed- bionic' to 'verification- failed- bionic' .
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/ /wiki.ubuntu. com/Testing/ EnableProposed for documentation how to enable and use -proposed. Thank you!