Kernel bug caused by AUFS

Bug #1513157 reported by frew on 2015-11-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Medium
Unassigned

Bug Description

Basically, when I write "a lot" of data while in a docker container I can consistently crash my machine. Here are some logs that I gathered when trying to figure this out:

Nov 04 08:58:54 zfp kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
Nov 04 08:58:54 zfp kernel: IP: [<ffffffffc0b5a93b>] au_write_pre+0x8b/0x110 [aufs]
Nov 04 08:58:54 zfp kernel: PGD 3b64ba067 PUD 3b5e32067 PMD 0
Nov 04 08:58:54 zfp kernel: Oops: 0000 [#1] SMP
Nov 04 08:58:54 zfp kernel: Modules linked in: rfcomm veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs bnep arc4 snd_hda_codec_hdmi intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_realtek snd_hda_codec_generic kvm_intel kvm snd_hda_intel iwlmvm snd_hda_codec mac80211 snd_hda_core snd_hwdep snd_pcm uvcvideo iwlwifi serio_raw snd_seq_midi lpc_ich snd_seq_midi_event videobuf2_vmalloc videobuf2_memops thinkpad_acpi snd_rawmidi videobuf2_core v4l2_common nvram videodev cfg80211 media snd_seq joydev input_leds snd_seq_device snd_timer btusb btrtl btbcm btintel bluetooth rtsx_pci_ms snd memstick mei_me mei soundcore
Nov 04 08:58:54 zfp kernel: ie31200_edac edac_core shpchp mac_hid sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc parport_pc ppdev lp parport autofs4 drbg ansi_cprng algif_skcipher af_alg dm_crypt hid_generic hid_thingm usbhid hid rtsx_pci_sdmmc crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse nouveau i915 ahci libahci rtsx_pci mxm_wmi e1000e ttm i2c_algo_bit drm_kms_helper ptp drm pps_core wmi video
Nov 04 08:58:54 zfp kernel: CPU: 3 PID: 23904 Comm: mysqld Not tainted 4.2.0-16-generic #19-Ubuntu
Nov 04 08:58:54 zfp kernel: Hardware name: LENOVO 20AN006LUS/20AN006LUS, BIOS GLET77WW (2.31 ) 01/27/2015
Nov 04 08:58:54 zfp kernel: task: ffff8803b5aae040 ti: ffff8803b71e4000 task.ti: ffff8803b71e4000
Nov 04 08:58:54 zfp kernel: RIP: 0010:[<ffffffffc0b5a93b>] [<ffffffffc0b5a93b>] au_write_pre+0x8b/0x110 [aufs]
Nov 04 08:58:54 zfp kernel: RSP: 0018:ffff8803b71e7c58 EFLAGS: 00010286
Nov 04 08:58:54 zfp kernel: RAX: 0000000000000000 RBX: ffff8803b72e7500 RCX: 0000000000000000
Nov 04 08:58:54 zfp kernel: RDX: 0000000100000000 RSI: 0000000000000000 RDI: ffff8803becff208
Nov 04 08:58:54 zfp kernel: RBP: ffff8803b71e7cd8 R08: ffff88042b2d3200 R09: ffff8803a71ea458
Nov 04 08:58:54 zfp kernel: R10: 00007ff42b18c000 R11: ffff88042acfcb00 R12: ffff8803b71e7cf0
Nov 04 08:58:54 zfp kernel: R13: 0000000000000001 R14: ffff8803a7344300 R15: ffff8803b72e6400
Nov 04 08:58:54 zfp kernel: FS: 00007ff3fa5a6700(0000) GS:ffff88043e2c0000(0000) knlGS:0000000000000000
Nov 04 08:58:54 zfp kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 04 08:58:54 zfp kernel: CR2: 0000000000000098 CR3: 00000003b64b9000 CR4: 00000000001406e0
Nov 04 08:58:54 zfp kernel: Stack:
Nov 04 08:58:54 zfp kernel: ffff8803a7344300 0000070300000004 ffff880397a5a600 ffff8803b7980e00
Nov 04 08:58:54 zfp kernel: 0000000000000000 ffff8803a7344480 ffff880397a5a3c0 ffff88042b2d3200
Nov 04 08:58:54 zfp kernel: ffff8803b5aae040 00000000ca877142 ffff8803b71e7cd8 ffff8803a71ea458
Nov 04 08:58:54 zfp kernel: Call Trace:
Nov 04 08:58:54 zfp kernel: [<ffffffffc0b5b1ac>] aufs_write_iter+0x4c/0x100 [aufs]
Nov 04 08:58:54 zfp kernel: [<ffffffff81325abd>] ? security_file_permission+0x3d/0xc0
Nov 04 08:58:54 zfp kernel: [<ffffffffc0b5b160>] ? aufs_splice_write+0x110/0x110 [aufs]
Nov 04 08:58:54 zfp kernel: [<ffffffff8124a44f>] aio_run_iocb+0x27f/0x2e0
Nov 04 08:58:54 zfp kernel: [<ffffffff810e080d>] ? call_rcu_sched+0x1d/0x20
Nov 04 08:58:54 zfp kernel: [<ffffffff812bfe4e>] ? ext4_discard_preallocations+0x38e/0x4b0
Nov 04 08:58:54 zfp kernel: [<ffffffff8124b278>] ? do_io_submit+0x198/0x4f0
Nov 04 08:58:54 zfp kernel: [<ffffffff811de4ba>] ? kmem_cache_alloc+0x1ca/0x200
Nov 04 08:58:54 zfp kernel: [<ffffffff8124b32b>] do_io_submit+0x24b/0x4f0
Nov 04 08:58:54 zfp kernel: [<ffffffff8124b5e0>] SyS_io_submit+0x10/0x20
Nov 04 08:58:54 zfp kernel: [<ffffffff817ef9f2>] entry_SYSCALL_64_fastpath+0x16/0x75
Nov 04 08:58:54 zfp kernel: Code: 74 10 49 8b 87 d0 00 00 00 0f b6 40 30 41 88 44 24 08 49 8b 87 d0 00 00 00 48 8b 58 38 f0 48 ff 43 38 4d 85 e4 74 0f 48 8b 43 20 <48> 8b 80 98 00 00 00 49 89 04 24 45 85 ed 75 63 31 f6 4c 89 f7
Nov 04 08:58:54 zfp kernel: RIP [<ffffffffc0b5a93b>] au_write_pre+0x8b/0x110 [aufs]
Nov 04 08:58:54 zfp kernel: RSP <ffff8803b71e7c58>
Nov 04 08:58:54 zfp kernel: CR2: 0000000000000098
Nov 04 08:58:54 zfp kernel: ---[ end trace 8adeb327cb22564d ]---
Nov 04 08:58:57 zfp kernel: BUG: unable to handle kernel paging request at 0000000000003078
Nov 04 08:58:57 zfp kernel: IP: [<ffffffff8122fc59>] vfs_fsync_range+0x9/0xb0
Nov 04 08:58:57 zfp kernel: PGD 3b64ba067 PUD 3b5e32067 PMD 0
Nov 04 08:58:57 zfp kernel: Oops: 0000 [#2] SMP
Nov 04 08:58:57 zfp kernel: Modules linked in: rfcomm veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs bnep arc4 snd_hda_codec_hdmi intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_realtek snd_hda_codec_generic kvm_intel kvm snd_hda_intel iwlmvm snd_hda_codec mac80211 snd_hda_core snd_hwdep snd_pcm uvcvideo iwlwifi serio_raw snd_seq_midi lpc_ich snd_seq_midi_event videobuf2_vmalloc videobuf2_memops thinkpad_acpi snd_rawmidi videobuf2_core v4l2_common nvram videodev cfg80211 media snd_seq joydev input_leds snd_seq_device snd_timer btusb btrtl btbcm btintel bluetooth rtsx_pci_ms snd memstick mei_me mei soundcore
Nov 04 08:58:57 zfp kernel: ie31200_edac edac_core shpchp mac_hid sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc parport_pc ppdev lp parport autofs4 drbg ansi_cprng algif_skcipher af_alg dm_crypt hid_generic hid_thingm usbhid hid rtsx_pci_sdmmc crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse nouveau i915 ahci libahci rtsx_pci mxm_wmi e1000e ttm i2c_algo_bit drm_kms_helper ptp drm pps_core wmi video
Nov 04 08:58:57 zfp kernel: CPU: 1 PID: 23902 Comm: mysqld Tainted: G D 4.2.0-16-generic #19-Ubuntu
Nov 04 08:58:57 zfp kernel: Hardware name: LENOVO 20AN006LUS/20AN006LUS, BIOS GLET77WW (2.31 ) 01/27/2015
Nov 04 08:58:57 zfp kernel: task: ffff8803b5aac4c0 ti: ffff8803b64d4000 task.ti: ffff8803b64d4000
Nov 04 08:58:57 zfp kernel: RIP: 0010:[<ffffffff8122fc59>] [<ffffffff8122fc59>] vfs_fsync_range+0x9/0xb0
Nov 04 08:58:57 zfp kernel: RSP: 0018:ffff8803b64d7f00 EFLAGS: 00010246
Nov 04 08:58:57 zfp kernel: RAX: 0000000000003000 RBX: ffff8803b72e6401 RCX: 0000000000000000
Nov 04 08:58:57 zfp kernel: RDX: 7fffffffffffffff RSI: 0000000000000000 RDI: ffff8803b72e6400
Nov 04 08:58:57 zfp kernel: RBP: ffff8803b64d7f38 R08: 0000100000000004 R09: 0000000000000b90
Nov 04 08:58:57 zfp kernel: R10: ffff8803b72e6438 R11: ffff88042acfcb00 R12: ffff8803b72e6400
Nov 04 08:58:57 zfp kernel: R13: 0000000000000000 R14: 00007ff3fcff3cf0 R15: 0000000000000172
Nov 04 08:58:57 zfp kernel: FS: 00007ff3fcff4700(0000) GS:ffff88043e240000(0000) knlGS:0000000000000000
Nov 04 08:58:57 zfp kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 04 08:58:57 zfp kernel: CR2: 000000000040b410 CR3: 00000003b64b9000 CR4: 00000000001406e0
Nov 04 08:58:57 zfp kernel: Stack:
Nov 04 08:58:57 zfp kernel: ffffffff8122fd5d 0000000000000000 000000007c494ce8 0000000000000032
Nov 04 08:58:57 zfp kernel: 0000000000000172 0000000000000000 0000000051eb851f ffff8803b64d7f48
Nov 04 08:58:57 zfp kernel: ffffffff81230000 00007ff3fcff3cd0 ffffffff817ef9f2 00000000563a394e
Nov 04 08:58:57 zfp kernel: Call Trace:
Nov 04 08:58:57 zfp kernel: [<ffffffff8122fd5d>] ? do_fsync+0x3d/0x70
Nov 04 08:58:57 zfp kernel: [<ffffffff81230000>] SyS_fsync+0x10/0x20
Nov 04 08:58:57 zfp kernel: [<ffffffff817ef9f2>] entry_SYSCALL_64_fastpath+0x16/0x75
Nov 04 08:58:57 zfp kernel: Code: 04 25 28 00 00 00 75 07 48 83 c4 18 5b 5d c3 e8 5e b5 e4 ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 47 28 <48> 8b 40 78 48 85 c0 0f 84 8b 00 00 00 55 48 89 e5 41 56 41 55
Nov 04 08:58:57 zfp kernel: RIP [<ffffffff8122fc59>] vfs_fsync_range+0x9/0xb0
Nov 04 08:58:57 zfp kernel: RSP <ffff8803b64d7f00>
Nov 04 08:58:57 zfp kernel: CR2: 0000000000003078
Nov 04 08:58:57 zfp kernel: ---[ end trace 8adeb327cb22564e ]---

I am pretty sure that if you were to just create a container and write /dev/urandom to some file at a reasonable rate you could repro this. I haven't tried because this is my work machine and I'm having trouble justifying crashing my machine a 5th time today :)

Let me know if there are more details you need.

Thanks.

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-4.2.0-16-generic 4.2.0-16.19
ProcVersionSignature: Ubuntu 4.2.0-16.19-generic 4.2.3
Uname: Linux 4.2.0-16-generic x86_64
ApportVersion: 2.19.1-0ubuntu4
Architecture: amd64
Date: Wed Nov 4 09:27:07 2015
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=729dd129-ad72-4d34-87d2-b698d6f581ae
InstallationDate: Installed on 2015-06-16 (140 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Release amd64 (20150422)
MachineType: LENOVO 20AN006LUS
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.2.0-16-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-4.2.0-16-generic N/A
 linux-backports-modules-4.2.0-16-generic N/A
 linux-firmware 1.149
SourcePackage: linux
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
UpgradeStatus: Upgraded to wily on 2015-11-02 (1 days ago)
dmi.bios.date: 01/27/2015
dmi.bios.vendor: LENOVO
dmi.bios.version: GLET77WW (2.31 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20AN006LUS
dmi.board.vendor: LENOVO
dmi.board.version: 0B98401 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrGLET77WW(2.31):bd01/27/2015:svnLENOVO:pn20AN006LUS:pvrThinkPadT440p:rvnLENOVO:rn20AN006LUS:rvr0B98401WIN:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 20AN006LUS
dmi.product.version: ThinkPad T440p
dmi.sys.vendor: LENOVO

frew (frioux) wrote :

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.3 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3-unstable/

Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Incomplete
frew (frioux) wrote :

This issue started when I upgraded from 15.04 to 15.11. Thus it did not happen with 3.19.

I tried to reproduce with 4.3 but I cannot, because AUFS is applied by canonical and is thus not in the upstream kernel.

Please advise on how to continue from here.

Launchpad Janitor (janitor) wrote :

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

Changed in linux (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers