NULL pointer dereference during writeback

Bug #1730374 reported by Knickers Brown
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

This could be part of CVE-2016-3070

Happened during dd of 40GB LV:

$ sudo dd if=/dev/vg0/gentoo of=/dev/vg1/gentoo

I also used SIGUSR1 on the dd to monitor it occasionally. There was other activity at the time as well.

It appears that an attempt was made to fix this in 4.10
https://patchwork.kernel.org/patch/9614353/
but it is still happening in 4.13

Nov 06 04:19:13 lakshmi kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
Nov 06 04:19:13 lakshmi kernel: IP: locked_inode_to_wb_and_lock_list+0x26/0x110
Nov 06 04:19:13 lakshmi kernel: PGD 0
Nov 06 04:19:14 lakshmi kernel: P4D 0
Nov 06 04:19:14 lakshmi kernel:
Nov 06 04:19:14 lakshmi kernel: Oops: 0000 [#1] SMP
Nov 06 04:19:14 lakshmi kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_
Nov 06 04:19:14 lakshmi kernel: async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 hid_g
Nov 06 04:19:14 lakshmi kernel: CPU: 0 PID: 67 Comm: kswapd0 Not tainted 4.13.0-16-generic #19-Ubuntu
Nov 06 04:19:14 lakshmi kernel: Hardware name: System manufacturer System Product Name/M4A89GTD-PRO/USB3, BIOS 3029 07/05/2012
Nov 06 04:19:14 lakshmi kernel: task: ffff93db5adc8000 task.stack: ffffb03b01d94000
Nov 06 04:19:14 lakshmi kernel: RIP: 0010:locked_inode_to_wb_and_lock_list+0x26/0x110
Nov 06 04:19:14 lakshmi kernel: RSP: 0018:ffffb03b01d97bc8 EFLAGS: 00010292
Nov 06 04:19:14 lakshmi kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff93d9758dfe28
Nov 06 04:19:14 lakshmi kernel: RDX: 0000000000000001 RSI: 0000000005080020 RDI: ffff93d9758dea58
Nov 06 04:19:14 lakshmi kernel: RBP: ffffb03b01d97bf0 R08: 0000000000000000 R09: ffff93d9cd655468
Nov 06 04:19:14 lakshmi kernel: R10: 0000000000000228 R11: 0000000000000000 R12: ffff93d9758dea58
Nov 06 04:19:14 lakshmi kernel: R13: ffff93d9758deae0 R14: 0000000000000000 R15: 0000000000000059
Nov 06 04:19:14 lakshmi kernel: FS: 0000000000000000(0000) GS:ffff93db77c00000(0000) knlGS:0000000000000000
Nov 06 04:19:14 lakshmi kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 06 04:19:14 lakshmi kernel: CR2: 0000000000000000 CR3: 00000001d7e08000 CR4: 00000000000006f0
Nov 06 04:19:14 lakshmi kernel: Call Trace:
Nov 06 04:19:14 lakshmi kernel: inode_io_list_del+0x23/0x50
Nov 06 04:19:14 lakshmi kernel: evict+0x55/0x1a0
Nov 06 04:19:14 lakshmi kernel: dispose_list+0x39/0x50
Nov 06 04:19:14 lakshmi kernel: prune_icache_sb+0x5a/0x80
Nov 06 04:19:14 lakshmi kernel: super_cache_scan+0x134/0x1b0
Nov 06 04:19:14 lakshmi kernel: shrink_slab.part.48+0x1d6/0x3d0
Nov 06 04:19:14 lakshmi kernel: shrink_slab+0x1b/0x30
Nov 06 04:19:14 lakshmi kernel: shrink_node+0x11e/0x300
Nov 06 04:19:14 lakshmi kernel: kswapd+0x2cc/0x750
Nov 06 04:19:14 lakshmi kernel: kthread+0x125/0x140
Nov 06 04:19:14 lakshmi kernel: ? mem_cgroup_shrink_node+0x180/0x180
Nov 06 04:19:14 lakshmi kernel: ? kthread_create_on_node+0x70/0x70
Nov 06 04:19:14 lakshmi kernel: ret_from_fork+0x25/0x30
Nov 06 04:19:14 lakshmi kernel: Code: 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 56 41 55 41 54 53 4c 8d af 88 00 00 00 49 89 fc
Nov 06 04:19:14 lakshmi kernel: RIP: locked_inode_to_wb_and_lock_list+0x26/0x110 RSP: ffffb03b01d97bc8
Nov 06 04:19:14 lakshmi kernel: CR2: 0000000000000000
Nov 06 04:19:14 lakshmi kernel: ---[ end trace 5aa11bcf674e53cc ]---

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: linux-image-4.13.0-16-generic 4.13.0-16.19
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic x86_64
ApportVersion: 2.20.7-0ubuntu3.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: ksta 3322 F.... pulseaudio
 /dev/snd/controlC0: ksta 3322 F.... pulseaudio
CurrentDesktop: ubuntu:GNOME
Date: Mon Nov 6 06:59:50 2017
HibernationDevice: RESUME=/dev/mapper/vg1-swap
InstallationDate: Installed on 2017-11-05 (0 days ago)
InstallationMedia: Ubuntu-Server 17.10 "Artful Aardvark" - Release amd64 (20171017.1)
MachineType: System manufacturer System Product Name
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 nouveaufb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.13.0-16-generic root=/dev/mapper/vg1-root ro
RelatedPackageVersions:
 linux-restricted-modules-4.13.0-16-generic N/A
 linux-backports-modules-4.13.0-16-generic N/A
 linux-firmware 1.169
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/05/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 3029
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: M4A89GTD-PRO/USB3
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3029:bd07/05/2012:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnM4A89GTD-PRO/USB3:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.family: To Be Filled By O.E.M.
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

CVE References

Revision history for this message
Knickers Brown (metta-crawler) wrote :
description: updated
description: updated
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

It doesn't appear the patch mentioned ever landed in mainline.

To see if this was fixed another way, would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.14 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc8

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
importance: Undecided → Medium
tags: added: kernel-da-key
Revision history for this message
Knickers Brown (metta-crawler) wrote :

I'd love to have time to work on this kernel bug but I have fires all over the place with 17.10.

I have a dual-boot setup between 16.04 and 17.10. The 17.10 took over nine minutes to boot originally. After finding out about Debian_bug_867368 I got it to boot in over three minutes. It still gets stuck booting and shutting down in other areas.

Revision history for this message
Knickers Brown (metta-crawler) wrote :

Eventually I'll have time to test the upstream kernel.

Revision history for this message
Knickers Brown (metta-crawler) wrote :
Download full text (3.5 KiB)

I was unable to reproduce this on 4.13 until Nov 23 when it happened again.

This time the logs show that I was handling PDF files via Nautilus which doesn't imply as much disk I/O as the previous 40G dd did.

A quote from:
https://patchwork.kernel.org/patch/9614361/
{{ I am able to reproduce the problem by adding articial [artificial] delays in
__blkdev_put() and writeback_sb_inodes(). Please see the repro patch
below: }}

Nov 23 06:20:38 lakshmi kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
Nov 23 06:20:38 lakshmi kernel: IP: locked_inode_to_wb_and_lock_list+0x26/0x110
Nov 23 06:20:38 lakshmi kernel: PGD 0
Nov 23 06:20:39 lakshmi kernel: P4D 0
Nov 23 06:20:39 lakshmi kernel:
Nov 23 06:20:39 lakshmi kernel: Oops: 0000 [#1] SMP
Nov 23 06:20:39 lakshmi kernel: Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs vhost_net vhost tap uas usb_storage xt_C
Nov 23 06:20:39 lakshmi kernel: ppdev lp parport ip_tables x_tables autofs4 xfs btrfs raid10 raid456 async_raid6_recov async_memcpy
Nov 23 06:20:39 lakshmi kernel: CPU: 2 PID: 67 Comm: kswapd0 Not tainted 4.13.0-16-generic #19-Ubuntu
Nov 23 06:20:39 lakshmi kernel: Hardware name: System manufacturer System Product Name/M4A89GTD-PRO/USB3, BIOS 3029 07/05/2012
Nov 23 06:20:39 lakshmi kernel: task: ffff99631ae28000 task.stack: ffffb98741d98000
Nov 23 06:20:39 lakshmi kernel: RIP: 0010:locked_inode_to_wb_and_lock_list+0x26/0x110
Nov 23 06:20:39 lakshmi kernel: RSP: 0018:ffffb98741d9bbc8 EFLAGS: 00010292
Nov 23 06:20:39 lakshmi kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff9961358ddc68
Nov 23 06:20:39 lakshmi kernel: RDX: 0000000000000001 RSI: 0000000005080020 RDI: ffff9961358dea58
Nov 23 06:20:39 lakshmi kernel: RBP: ffffb98741d9bbf0 R08: 0000000000000000 R09: ffff99629957efc8
Nov 23 06:20:39 lakshmi kernel: R10: 0000000000000228 R11: ffff99629957cd90 R12: ffff9961358dea58
Nov 23 06:20:39 lakshmi kernel: R13: ffff9961358deae0 R14: 0000000000000000 R15: 00000000000000e3
Nov 23 06:20:39 lakshmi kernel: FS: 0000000000000000(0000) GS:ffff996337c80000(0000) knlGS:0000000000000000
Nov 23 06:20:39 lakshmi kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 23 06:20:39 lakshmi kernel: CR2: 0000000000000000 CR3: 0000000292009000 CR4: 00000000000006e0
Nov 23 06:20:39 lakshmi kernel: Call Trace:
Nov 23 06:20:39 lakshmi kernel: inode_io_list_del+0x23/0x50
Nov 23 06:20:39 lakshmi kernel: evict+0x55/0x1a0
Nov 23 06:20:39 lakshmi kernel: dispose_list+0x39/0x50
Nov 23 06:20:39 lakshmi kernel: prune_icache_sb+0x5a/0x80
Nov 23 06:20:39 lakshmi kernel: super_cache_scan+0x134/0x1b0
Nov 23 06:20:39 lakshmi kernel: shrink_slab.part.48+0x1d6/0x3d0
Nov 23 06:20:39 lakshmi kernel: shrink_slab+0x1b/0x30
Nov 23 06:20:39 lakshmi kernel: shrink_node+0x11e/0x300
Nov 23 06:20:39 lakshmi kernel: kswapd+0x2cc/0x750
Nov 23 06:20:39 lakshmi kernel: kthread+0x125/0x140
Nov 23 06:20:39 lakshmi kernel: ? mem_cgroup_shrink_node+0x180/0x180
Nov 23 06:20:39 lakshmi kernel: ? kthread_create_on_node+0x70/0x70
Nov 23 06:20:39 lakshmi kernel: ret_from_fork+0x25/0x30
Nov 23 06:20:39 lakshmi kernel: Code: 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 56 41 ...

Read more...

Revision history for this message
Knickers Brown (metta-crawler) wrote :
Revision history for this message
Knickers Brown (metta-crawler) wrote :

It's likely that this was either fixed upstream or the root cause is from a patch that was put on the 4.13 kernels that the 4.14 kernels don't have.

Nov 05 22:15:30 lakshmi kernel: Linux version 4.13.0-16-generic (buildd@lcy01-02) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu2)) #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC 2017 (Ubuntu 4.13.0-16.19-generic 4.13.4)
Nov 06 04:19:13 lakshmi kernel: BUG: unable to handle kernel NULL pointer dereference at (null)

Nov 06 04:53:36 lakshmi kernel: Linux version 4.13.0-16-generic (buildd@lcy01-02) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu2)) #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC 2017 (Ubuntu 4.13.0-16.19-generic 4.13.4)
Nov 12 14:21:07 lakshmi kernel: BUG: unable to handle kernel paging request at 0000000000001000

Nov 16 08:03:01 lakshmi kernel: Linux version 4.13.0-16-generic (buildd@lcy01-02) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu2)) #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC 2017 (Ubuntu 4.13.0-16.19-generic 4.13.4)
Nov 23 06:20:38 lakshmi kernel: BUG: unable to handle kernel NULL pointer dereference at (null)

Nov 23 07:12:24 lakshmi kernel: Linux version 4.13.0-17-generic (buildd@lcy01-amd64-011) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3)) #20-Ubuntu SMP Mon Nov 6 10:04:08 UTC 2017 (Ubuntu 4.13.0-17.20-generic 4.13.8)

Nov 28 06:45:45 lakshmi kernel: Linux version 4.14.2-041402-generic (kernel@tangerine) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3)) #201711240330 SMP Fri Nov 24 08:32:52 UTC 2017
Dec 01 19:56:41 lakshmi kernel: Linux version 4.14.3-041403-generic (kernel@tangerine) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3)) #201711300431 SMP Thu Nov 30 09:32:55 UTC 2017
Dec 08 14:14:31 lakshmi kernel: Linux version 4.14.4-041404-generic (kernel@tangerine) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3)) #201712050630 SMP Tue Dec 5 11:34:06 UTC 2017
Dec 13 06:32:14 lakshmi kernel: Linux version 4.14.5-041405-generic (kernel@kathleen) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3)) #201712101332 SMP Sun Dec 10 13:33:10 UTC 2017

Revision history for this message
Knickers Brown (metta-crawler) wrote :

It happened again.

$ uname -r
4.14.9-041409-generic

Dec 29 14:32:14 lakshmi kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
Dec 29 14:32:14 lakshmi kernel: IP: locked_inode_to_wb_and_lock_list+0x20/0x110
Dec 29 14:32:14 lakshmi kernel: PGD 0 P4D 0
Dec 29 14:32:14 lakshmi kernel: Oops: 0000 [#1] SMP
Dec 29 14:32:14 lakshmi kernel: Modules linked in: uas usb_storage nfnetlink bluetooth ecdh_generic isofs xt_CHECKSUM
Dec 29 14:32:14 lakshmi kernel: lp parport ip_tables x_tables autofs4 xfs btrfs zstd_compress raid10 raid456 async_ra
Dec 29 14:32:14 lakshmi kernel: CPU: 4 PID: 67 Comm: kswapd0 Not tainted 4.14.9-041409-generic #201712251541
Dec 29 14:32:14 lakshmi kernel: Hardware name: System manufacturer System Product Name/M4A89GTD-PRO/USB3, BIOS 3029
Dec 29 14:32:14 lakshmi kernel: task: ffff9ab5e4c195c0 task.stack: ffffa9a5c1da0000
Dec 29 14:32:14 lakshmi kernel: RIP: 0010:locked_inode_to_wb_and_lock_list+0x20/0x110
Dec 29 14:32:14 lakshmi kernel: RSP: 0018:ffffa9a5c1da3c38 EFLAGS: 00010292
Dec 29 14:32:14 lakshmi kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffa9a5c1da3cb0
Dec 29 14:32:14 lakshmi kernel: RDX: 0000000000000001 RSI: 0000000005080020 RDI: ffff9ab3f58dea58
Dec 29 14:32:14 lakshmi kernel: RBP: ffff9ab3f58dea58 R08: 0000000000000000 R09: ffff9ab3f5992b38
Dec 29 14:32:14 lakshmi kernel: R10: 0000000000000000 R11: 0000000000000228 R12: ffff9ab3f58deae0
Dec 29 14:32:14 lakshmi kernel: R13: ffff9ab5da2d9000 R14: 0000000000000000 R15: 0000000000000265
Dec 29 14:32:14 lakshmi kernel: FS: 0000000000000000(0000) GS:ffff9ab5f7d00000(0000) knlGS:0000000000000000
Dec 29 14:32:14 lakshmi kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 29 14:32:14 lakshmi kernel: CR2: 0000000000000000 CR3: 000000041a0cb000 CR4: 00000000000006e0
Dec 29 14:32:14 lakshmi kernel: Call Trace:
Dec 29 14:32:14 lakshmi kernel: inode_io_list_del+0x1e/0x40
Dec 29 14:32:14 lakshmi kernel: evict+0x50/0x190
Dec 29 14:32:14 lakshmi kernel: dispose_list+0x35/0x50
Dec 29 14:32:14 lakshmi kernel: prune_icache_sb+0x52/0x70
Dec 29 14:32:14 lakshmi kernel: super_cache_scan+0x124/0x1a0
Dec 29 14:32:14 lakshmi kernel: shrink_slab.part.50+0x1de/0x3d0
Dec 29 14:32:14 lakshmi kernel: shrink_node+0x123/0x310
Dec 29 14:32:14 lakshmi kernel: kswapd+0x29f/0x6f0
Dec 29 14:32:14 lakshmi kernel: ? mem_cgroup_shrink_node+0x190/0x190
Dec 29 14:32:14 lakshmi kernel: kthread+0x118/0x130
Dec 29 14:32:14 lakshmi kernel: ? kthread_create_on_node+0x70/0x70
Dec 29 14:32:14 lakshmi kernel: ret_from_fork+0x1f/0x30
Dec 29 14:32:14 lakshmi kernel: Code: 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 55 41 54 4c 8d a7 88 00 00 00
Dec 29 14:32:14 lakshmi kernel: RIP: locked_inode_to_wb_and_lock_list+0x20/0x110 RSP: ffffa9a5c1da3c38
Dec 29 14:32:14 lakshmi kernel: CR2: 0000000000000000
Dec 29 14:32:14 lakshmi kernel: ---[ end trace 40202702d10d2b9b ]---

Changed in linux (Ubuntu):
status: Incomplete → New
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
description: updated
tags: added: kernel-bug
Revision history for this message
Knickers Brown (metta-crawler) wrote :

I no longer am using the motherboard, CPU and RAM that this happened on.

I am using 4.15.0-13-generic on 18.04 now.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.