Ubuntu
linux package

Kernel bug caused by AUFS

Bug #1513157 reported by frew
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Expired
Medium
Unassigned

Bug Description

Basically, when I write "a lot" of data while in a docker container I can consistently crash my machine. Here are some logs that I gathered when trying to figure this out:

Nov 04 08:58:54 zfp kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
Nov 04 08:58:54 zfp kernel: IP: [<ffffffffc0b5a93b>] au_write_pre+0x8b/0x110 [aufs]
Nov 04 08:58:54 zfp kernel: PGD 3b64ba067 PUD 3b5e32067 PMD 0
Nov 04 08:58:54 zfp kernel: Oops: 0000 [#1] SMP
Nov 04 08:58:54 zfp kernel: Modules linked in: rfcomm veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs bnep arc4 snd_hda_codec_hdmi intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_realtek snd_hda_codec_generic kvm_intel kvm snd_hda_intel iwlmvm snd_hda_codec mac80211 snd_hda_core snd_hwdep snd_pcm uvcvideo iwlwifi serio_raw snd_seq_midi lpc_ich snd_seq_midi_event videobuf2_vmalloc videobuf2_memops thinkpad_acpi snd_rawmidi videobuf2_core v4l2_common nvram videodev cfg80211 media snd_seq joydev input_leds snd_seq_device snd_timer btusb btrtl btbcm btintel bluetooth rtsx_pci_ms snd memstick mei_me mei soundcore
Nov 04 08:58:54 zfp kernel: ie31200_edac edac_core shpchp mac_hid sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc parport_pc ppdev lp parport autofs4 drbg ansi_cprng algif_skcipher af_alg dm_crypt hid_generic hid_thingm usbhid hid rtsx_pci_sdmmc crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse nouveau i915 ahci libahci rtsx_pci mxm_wmi e1000e ttm i2c_algo_bit drm_kms_helper ptp drm pps_core wmi video
Nov 04 08:58:54 zfp kernel: CPU: 3 PID: 23904 Comm: mysqld Not tainted 4.2.0-16-generic #19-Ubuntu
Nov 04 08:58:54 zfp kernel: Hardware name: LENOVO 20AN006LUS/20AN006LUS, BIOS GLET77WW (2.31 ) 01/27/2015
Nov 04 08:58:54 zfp kernel: task: ffff8803b5aae040 ti: ffff8803b71e4000 task.ti: ffff8803b71e4000
Nov 04 08:58:54 zfp kernel: RIP: 0010:[<ffffffffc0b5a93b>] [<ffffffffc0b5a93b>] au_write_pre+0x8b/0x110 [aufs]
Nov 04 08:58:54 zfp kernel: RSP: 0018:ffff8803b71e7c58 EFLAGS: 00010286
Nov 04 08:58:54 zfp kernel: RAX: 0000000000000000 RBX: ffff8803b72e7500 RCX: 0000000000000000
Nov 04 08:58:54 zfp kernel: RDX: 0000000100000000 RSI: 0000000000000000 RDI: ffff8803becff208
Nov 04 08:58:54 zfp kernel: RBP: ffff8803b71e7cd8 R08: ffff88042b2d3200 R09: ffff8803a71ea458
Nov 04 08:58:54 zfp kernel: R10: 00007ff42b18c000 R11: ffff88042acfcb00 R12: ffff8803b71e7cf0
Nov 04 08:58:54 zfp kernel: R13: 0000000000000001 R14: ffff8803a7344300 R15: ffff8803b72e6400
Nov 04 08:58:54 zfp kernel: FS: 00007ff3fa5a6700(0000) GS:ffff88043e2c0000(0000) knlGS:0000000000000000
Nov 04 08:58:54 zfp kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 04 08:58:54 zfp kernel: CR2: 0000000000000098 CR3: 00000003b64b9000 CR4: 00000000001406e0
Nov 04 08:58:54 zfp kernel: Stack:
Nov 04 08:58:54 zfp kernel: ffff8803a7344300 0000070300000004 ffff880397a5a600 ffff8803b7980e00
Nov 04 08:58:54 zfp kernel: 0000000000000000 ffff8803a7344480 ffff880397a5a3c0 ffff88042b2d3200
Nov 04 08:58:54 zfp kernel: ffff8803b5aae040 00000000ca877142 ffff8803b71e7cd8 ffff8803a71ea458
Nov 04 08:58:54 zfp kernel: Call Trace:
Nov 04 08:58:54 zfp kernel: [<ffffffffc0b5b1ac>] aufs_write_iter+0x4c/0x100 [aufs]
Nov 04 08:58:54 zfp kernel: [<ffffffff81325abd>] ? security_file_permission+0x3d/0xc0
Nov 04 08:58:54 zfp kernel: [<ffffffffc0b5b160>] ? aufs_splice_write+0x110/0x110 [aufs]
Nov 04 08:58:54 zfp kernel: [<ffffffff8124a44f>] aio_run_iocb+0x27f/0x2e0
Nov 04 08:58:54 zfp kernel: [<ffffffff810e080d>] ? call_rcu_sched+0x1d/0x20
Nov 04 08:58:54 zfp kernel: [<ffffffff812bfe4e>] ? ext4_discard_preallocations+0x38e/0x4b0
Nov 04 08:58:54 zfp kernel: [<ffffffff8124b278>] ? do_io_submit+0x198/0x4f0
Nov 04 08:58:54 zfp kernel: [<ffffffff811de4ba>] ? kmem_cache_alloc+0x1ca/0x200
Nov 04 08:58:54 zfp kernel: [<ffffffff8124b32b>] do_io_submit+0x24b/0x4f0
Nov 04 08:58:54 zfp kernel: [<ffffffff8124b5e0>] SyS_io_submit+0x10/0x20
Nov 04 08:58:54 zfp kernel: [<ffffffff817ef9f2>] entry_SYSCALL_64_fastpath+0x16/0x75
Nov 04 08:58:54 zfp kernel: Code: 74 10 49 8b 87 d0 00 00 00 0f b6 40 30 41 88 44 24 08 49 8b 87 d0 00 00 00 48 8b 58 38 f0 48 ff 43 38 4d 85 e4 74 0f 48 8b 43 20 <48> 8b 80 98 00 00 00 49 89 04 24 45 85 ed 75 63 31 f6 4c 89 f7
Nov 04 08:58:54 zfp kernel: RIP [<ffffffffc0b5a93b>] au_write_pre+0x8b/0x110 [aufs]
Nov 04 08:58:54 zfp kernel: RSP <ffff8803b71e7c58>
Nov 04 08:58:54 zfp kernel: CR2: 0000000000000098
Nov 04 08:58:54 zfp kernel: ---[ end trace 8adeb327cb22564d ]---
Nov 04 08:58:57 zfp kernel: BUG: unable to handle kernel paging request at 0000000000003078
Nov 04 08:58:57 zfp kernel: IP: [<ffffffff8122fc59>] vfs_fsync_range+0x9/0xb0
Nov 04 08:58:57 zfp kernel: PGD 3b64ba067 PUD 3b5e32067 PMD 0
Nov 04 08:58:57 zfp kernel: Oops: 0000 [#2] SMP
Nov 04 08:58:57 zfp kernel: Modules linked in: rfcomm veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs bnep arc4 snd_hda_codec_hdmi intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_realtek snd_hda_codec_generic kvm_intel kvm snd_hda_intel iwlmvm snd_hda_codec mac80211 snd_hda_core snd_hwdep snd_pcm uvcvideo iwlwifi serio_raw snd_seq_midi lpc_ich snd_seq_midi_event videobuf2_vmalloc videobuf2_memops thinkpad_acpi snd_rawmidi videobuf2_core v4l2_common nvram videodev cfg80211 media snd_seq joydev input_leds snd_seq_device snd_timer btusb btrtl btbcm btintel bluetooth rtsx_pci_ms snd memstick mei_me mei soundcore
Nov 04 08:58:57 zfp kernel: ie31200_edac edac_core shpchp mac_hid sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc parport_pc ppdev lp parport autofs4 drbg ansi_cprng algif_skcipher af_alg dm_crypt hid_generic hid_thingm usbhid hid rtsx_pci_sdmmc crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse nouveau i915 ahci libahci rtsx_pci mxm_wmi e1000e ttm i2c_algo_bit drm_kms_helper ptp drm pps_core wmi video
Nov 04 08:58:57 zfp kernel: CPU: 1 PID: 23902 Comm: mysqld Tainted: G D 4.2.0-16-generic #19-Ubuntu
Nov 04 08:58:57 zfp kernel: Hardware name: LENOVO 20AN006LUS/20AN006LUS, BIOS GLET77WW (2.31 ) 01/27/2015
Nov 04 08:58:57 zfp kernel: task: ffff8803b5aac4c0 ti: ffff8803b64d4000 task.ti: ffff8803b64d4000
Nov 04 08:58:57 zfp kernel: RIP: 0010:[<ffffffff8122fc59>] [<ffffffff8122fc59>] vfs_fsync_range+0x9/0xb0
Nov 04 08:58:57 zfp kernel: RSP: 0018:ffff8803b64d7f00 EFLAGS: 00010246
Nov 04 08:58:57 zfp kernel: RAX: 0000000000003000 RBX: ffff8803b72e6401 RCX: 0000000000000000
Nov 04 08:58:57 zfp kernel: RDX: 7fffffffffffffff RSI: 0000000000000000 RDI: ffff8803b72e6400
Nov 04 08:58:57 zfp kernel: RBP: ffff8803b64d7f38 R08: 0000100000000004 R09: 0000000000000b90
Nov 04 08:58:57 zfp kernel: R10: ffff8803b72e6438 R11: ffff88042acfcb00 R12: ffff8803b72e6400
Nov 04 08:58:57 zfp kernel: R13: 0000000000000000 R14: 00007ff3fcff3cf0 R15: 0000000000000172
Nov 04 08:58:57 zfp kernel: FS: 00007ff3fcff4700(0000) GS:ffff88043e240000(0000) knlGS:0000000000000000
Nov 04 08:58:57 zfp kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 04 08:58:57 zfp kernel: CR2: 000000000040b410 CR3: 00000003b64b9000 CR4: 00000000001406e0
Nov 04 08:58:57 zfp kernel: Stack:
Nov 04 08:58:57 zfp kernel: ffffffff8122fd5d 0000000000000000 000000007c494ce8 0000000000000032
Nov 04 08:58:57 zfp kernel: 0000000000000172 0000000000000000 0000000051eb851f ffff8803b64d7f48
Nov 04 08:58:57 zfp kernel: ffffffff81230000 00007ff3fcff3cd0 ffffffff817ef9f2 00000000563a394e
Nov 04 08:58:57 zfp kernel: Call Trace:
Nov 04 08:58:57 zfp kernel: [<ffffffff8122fd5d>] ? do_fsync+0x3d/0x70
Nov 04 08:58:57 zfp kernel: [<ffffffff81230000>] SyS_fsync+0x10/0x20
Nov 04 08:58:57 zfp kernel: [<ffffffff817ef9f2>] entry_SYSCALL_64_fastpath+0x16/0x75
Nov 04 08:58:57 zfp kernel: Code: 04 25 28 00 00 00 75 07 48 83 c4 18 5b 5d c3 e8 5e b5 e4 ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 47 28 <48> 8b 40 78 48 85 c0 0f 84 8b 00 00 00 55 48 89 e5 41 56 41 55
Nov 04 08:58:57 zfp kernel: RIP [<ffffffff8122fc59>] vfs_fsync_range+0x9/0xb0
Nov 04 08:58:57 zfp kernel: RSP <ffff8803b64d7f00>
Nov 04 08:58:57 zfp kernel: CR2: 0000000000003078
Nov 04 08:58:57 zfp kernel: ---[ end trace 8adeb327cb22564e ]---

I am pretty sure that if you were to just create a container and write /dev/urandom to some file at a reasonable rate you could repro this. I haven't tried because this is my work machine and I'm having trouble justifying crashing my machine a 5th time today :)

Let me know if there are more details you need.

Thanks.

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-4.2.0-16-generic 4.2.0-16.19
ProcVersionSignature: Ubuntu 4.2.0-16.19-generic 4.2.3
Uname: Linux 4.2.0-16-generic x86_64
ApportVersion: 2.19.1-0ubuntu4
Architecture: amd64
Date: Wed Nov 4 09:27:07 2015
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=729dd129-ad72-4d34-87d2-b698d6f581ae
InstallationDate: Installed on 2015-06-16 (140 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Release amd64 (20150422)
MachineType: LENOVO 20AN006LUS
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.2.0-16-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-4.2.0-16-generic N/A
 linux-backports-modules-4.2.0-16-generic N/A
 linux-firmware 1.149
SourcePackage: linux
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
UpgradeStatus: Upgraded to wily on 2015-11-02 (1 days ago)
dmi.bios.date: 01/27/2015
dmi.bios.vendor: LENOVO
dmi.bios.version: GLET77WW (2.31 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20AN006LUS
dmi.board.vendor: LENOVO
dmi.board.version: 0B98401 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrGLET77WW(2.31):bd01/27/2015:svnLENOVO:pn20AN006LUS:pvrThinkPadT440p:rvnLENOVO:rn20AN006LUS:rvr0B98401WIN:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 20AN006LUS
dmi.product.version: ThinkPad T440p
dmi.sys.vendor: LENOVO

Revision history for this message
frew (frioux) wrote :
Revision history for this message
Brad Figg (brad-figg) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.3 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3-unstable/

Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Incomplete
Revision history for this message
frew (frioux) wrote :

This issue started when I upgraded from 15.04 to 15.11. Thus it did not happen with 3.19.

I tried to reproduce with 4.3 but I cannot, because AUFS is applied by canonical and is thus not in the upstream kernel.

Please advise on how to continue from here.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

Changed in linux (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.