Memory allocation failure, presumably in FUSE

Bug #1398465 reported by Philipp Kern on 2014-12-02
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Medium
Unassigned

Bug Description

I just tried out linux-lts-utopic on trusty and hit a few backtraces. We make intensive use of FUSE and the FS operation I conducted on FUSE hung post-backtrace.

[ 1951.617813] ------------[ cut here ]------------
[ 1951.617818] kernel BUG at /build/buildd/linux-lts-utopic-3.16.0/mm/slub.c:3380!
[ 1951.617819] invalid opcode: 0000 [#1] SMP
[ 1951.617821] Modules linked in: des_generic nfsv3 nfsv4 vmnet(OE) parport_pc vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) xt_addrtype ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat bridge stp llc aufs netconsole configfs nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_NFLOG xt_comment xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_connmark xt_conntrack nf_conntrack xt_mark cuse iptable_filter ip_tables x_tables rpcsec_gss_krb5 nfnetlink_log nfnetlink autofs4 rfcomm bnep bluetooth 6lowpan_iphc binfmt_misc snd_hda_codec_hdmi hp_wmi sparse_keymap intel_rapl x86_pkg_temp_thermal snd_hda_codec_realtek intel_powerclamp coretemp snd_hda_codec_generic kvm_intel kvm ppdev crct10dif_pclmul crc32_pclmul snd_hda_intel ghash_clmulni_intel aesni_intel snd_hda_controller aes_x86_64 lrw snd_hda_codec gf128mul glue_helper ablk_helper cryptd snd_hwdep lp snd_pcm parport serio_raw sb_edac edac_core snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq nvidia(POE) snd_seq_device snd_timer lpc_ich snd drm ioatdma dca soundcore wmi tpm_infineon mac_hid nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache nls_iso8859_1 hid_generic usbhid hid isci firewire_ohci e1000e psmouse(OE) libsas firewire_core ahci libahci ptp crc_itu_t scsi_transport_sas pps_core [last unloaded: vmnet]
[ 1951.617874] CPU: 2 PID: 13631 Comm: srcfsd Tainted: P OE 3.16.0-25-generic #33-Ubuntu
[ 1951.617876] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v03.65 12/19/2013
[ 1951.617877] task: ffff88061f5cb2f0 ti: ffff880800cdc000 task.ti: ffff880800cdc000
[ 1951.617878] RIP: 0010:[<ffffffff811b61a3>] [<ffffffff811b61a3>] kfree+0x133/0x140
[ 1951.617884] RSP: 0018:ffff880800cdfbf8 EFLAGS: 00010246
[ 1951.617885] RAX: 00ffff0000000400 RBX: ffff880000007833 RCX: 000000018020001f
[ 1951.617886] RDX: 00ffff0000000000 RSI: ffffea0002716f80 RDI: ffff880000007833
[ 1951.617887] RBP: ffff880800cdfc10 R08: ffff88009c5bf400 R09: 000000018020001f
[ 1951.617888] R10: ffffea00000001c0 R11: ffffffff812bfd95 R12: ffff8807addaf800
[ 1951.617889] R13: ffffffff812bfda1 R14: 0000000000020010 R15: ffff880800cdfd58
[ 1951.617890] FS: 00007fccdb950700(0000) GS:ffff88082fc40000(0000) knlGS:0000000000000000
[ 1951.617891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1951.617892] CR2: 0000000008a89ff0 CR3: 00000007c8fb7000 CR4: 00000000000407e0
[ 1951.617893] Stack:
[ 1951.617894] ffff8807a1e296c0 ffff8807addaf800 0000000000000000 ffff880800cdfc28
[ 1951.617896] ffffffff812bfda1 ffff8807a1e296c0 ffff880800cdfc58 ffffffff812bfe88
[ 1951.617897] ffff8807a1e296c0 ffff8807addaf800 ffffffff812c71c0 0000000000020010
[ 1951.617899] Call Trace:
[ 1951.617904] [<ffffffff812bfda1>] fuse_request_free+0x31/0x50
[ 1951.617906] [<ffffffff812bfe88>] fuse_put_request+0xc8/0x110
[ 1951.617908] [<ffffffff812c71c0>] ? fuse_short_read+0x160/0x160
[ 1951.617910] [<ffffffff812bffdc>] request_end+0x10c/0x1c0
[ 1951.617911] [<ffffffff812c1436>] fuse_dev_do_write+0x826/0xd10
[ 1951.617914] [<ffffffff811d36e0>] ? do_sync_read+0x90/0x90
[ 1951.617916] [<ffffffff812c1c79>] fuse_dev_write+0x69/0x80
[ 1951.617918] [<ffffffff811d37bc>] do_sync_readv_writev+0x4c/0x80
[ 1951.617920] [<ffffffff811d510d>] do_readv_writev+0x1bd/0x240
[ 1951.617922] [<ffffffff811d5219>] vfs_writev+0x39/0x50
[ 1951.617924] [<ffffffff811d534a>] SyS_writev+0x4a/0xd0
[ 1951.617927] [<ffffffff817690ad>] system_call_fastpath+0x1a/0x1f
[ 1951.617928] Code: 49 8b 02 31 f6 f6 c4 40 74 04 41 8b 72 68 4c 89 d7 e8 92 3d fb ff eb 8f 4c 8b 50 30 48 8b 10 80 e6 80 4c 0f 44 d0 e9 32 ff ff ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 f1
[ 1951.617946] RIP [<ffffffff811b61a3>] kfree+0x133/0x140
[ 1951.617948] RSP <ffff880800cdfbf8>
[ 1951.617950] ---[ end trace a783e9182f5a2f00 ]---
[ 1953.884500] general protection fault: 0000 [#2] SMP
[ 1953.884505] Modules linked in: des_generic nfsv3 nfsv4 vmnet(OE) parport_pc vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) xt_addrtype ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat bridge stp llc aufs netconsole configfs nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_NFLOG xt_comment xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_connmark xt_conntrack nf_conntrack xt_mark cuse iptable_filter ip_tables x_tables rpcsec_gss_krb5 nfnetlink_log nfnetlink autofs4 rfcomm bnep bluetooth 6lowpan_iphc binfmt_misc snd_hda_codec_hdmi hp_wmi sparse_keymap intel_rapl x86_pkg_temp_thermal snd_hda_codec_realtek intel_powerclamp coretemp snd_hda_codec_generic kvm_intel kvm ppdev crct10dif_pclmul crc32_pclmul snd_hda_intel ghash_clmulni_intel aesni_intel snd_hda_controller aes_x86_64 lrw snd_hda_codec gf128mul glue_helper ablk_helper cryptd snd_hwdep lp snd_pcm parport serio_raw sb_edac edac_core snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq nvidia(POE) snd_seq_device snd_timer lpc_ich snd drm ioatdma dca soundcore wmi tpm_infineon mac_hid nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache nls_iso8859_1 hid_generic usbhid hid isci firewire_ohci e1000e psmouse(OE) libsas firewire_core ahci libahci ptp crc_itu_t scsi_transport_sas pps_core [last unloaded: vmnet]
[ 1953.884561] CPU: 9 PID: 13908 Comm: g4 Tainted: P D OE 3.16.0-25-generic #33-Ubuntu
[ 1953.884562] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v03.65 12/19/2013
[ 1953.884564] task: ffff8806995fa8c0 ti: ffff88061ecf4000 task.ti: ffff88061ecf4000
[ 1953.884565] RIP: 0010:[<ffffffff8116c017>] [<ffffffff8116c017>] set_page_dirty+0x17/0x60
[ 1953.884571] RSP: 0018:ffff88061ecf7d48 EFLAGS: 00010206
[ 1953.884572] RAX: 0000ffffffff0000 RBX: ffffea000000168e RCX: ffff8807a1e2a308
[ 1953.884573] RDX: 0000000000000000 RSI: ffff8807a1e2a380 RDI: ffffea000000168e
[ 1953.884574] RBP: ffff88061ecf7d50 R08: 0000000000000293 R09: 0000000000000000
[ 1953.884575] R10: 0000000000000113 R11: 0000000000000006 R12: ffffea000000168e
[ 1953.884576] R13: ffff8807a1e2a380 R14: 0000000000000001 R15: ffff8807a1e2a358
[ 1953.884577] FS: 00007f89cf67fc80(0000) GS:ffff88082fd20000(0000) knlGS:0000000000000000
[ 1953.884578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1953.884579] CR2: 00000000017fe3d0 CR3: 000000061f36c000 CR4: 00000000000407e0
[ 1953.884580] Stack:
[ 1953.884581] ffffea000000168e ffff88061ecf7d70 ffffffff8116c085 ffff88061ecf7e10
[ 1953.884583] 0000000000000000 ffff88061ecf7da8 ffffffff812c669f 0000000000020000
[ 1953.884584] ffff88061ecf7e90 0000000000000000 0000000000001000 ffff8807a1e2a220
[ 1953.884586] Call Trace:
[ 1953.884589] [<ffffffff8116c085>] set_page_dirty_lock+0x25/0x50
[ 1953.884591] [<ffffffff812c669f>] fuse_release_user_pages.isra.19+0x4f/0x70
[ 1953.884593] [<ffffffff812c877d>] fuse_direct_io+0x39d/0x570
[ 1953.884595] [<ffffffff812c8a24>] __fuse_direct_read+0x44/0x60
[ 1953.884596] [<ffffffff812c8a92>] fuse_direct_read+0x52/0x60
[ 1953.884599] [<ffffffff811d3f48>] vfs_read+0x98/0x170
[ 1953.884601] [<ffffffff811d4bc6>] SyS_read+0x46/0xb0
[ 1953.884605] [<ffffffff817690ad>] system_call_fastpath+0x1a/0x1f
[ 1953.884605] Code: 00 00 49 0f 4e db e9 56 ff ff ff 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 53 48 89 fb e8 ae 1d 01 00 48 85 c0 74 23 <48> 8b 40 78 48 8b 40 18 f0 80 63 02 fb 48 c7 c2 70 4e 20 81 48
[ 1953.884624] RIP [<ffffffff8116c017>] set_page_dirty+0x17/0x60
[ 1953.884625] RSP <ffff88061ecf7d48>
[ 1953.884628] ---[ end trace a783e9182f5a2f01 ]---
[ 1967.296324] BUG: unable to handle kernel paging request at ffffffff81767496
[ 1967.296330] IP: [<ffffffff811624b5>] __delete_from_page_cache+0xd5/0x2c0
[ 1967.296335] PGD 1c16067 PUD 1c17063 PMD 16001e1
[ 1967.296338] Oops: 0003 [#3] SMP
[ 1967.296340] Modules linked in: des_generic nfsv3 nfsv4 vmnet(OE) parport_pc vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) xt_addrtype ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat bridge stp llc aufs netconsole configfs nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_NFLOG xt_comment xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_connmark xt_conntrack nf_conntrack xt_mark cuse iptable_filter ip_tables x_tables rpcsec_gss_krb5 nfnetlink_log nfnetlink autofs4 rfcomm bnep bluetooth 6lowpan_iphc binfmt_misc snd_hda_codec_hdmi hp_wmi sparse_keymap intel_rapl x86_pkg_temp_thermal snd_hda_codec_realtek intel_powerclamp coretemp snd_hda_codec_generic kvm_intel kvm ppdev crct10dif_pclmul crc32_pclmul snd_hda_intel ghash_clmulni_intel aesni_intel snd_hda_controller aes_x86_64 lrw snd_hda_codec gf128mul glue_helper ablk_helper cryptd snd_hwdep lp snd_pcm parport serio_raw sb_edac edac_core snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq nvidia(POE) snd_seq_device snd_timer lpc_ich snd drm ioatdma dca soundcore wmi tpm_infineon mac_hid nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache nls_iso8859_1 hid_generic usbhid hid isci firewire_ohci e1000e psmouse(OE) libsas firewire_core ahci libahci ptp crc_itu_t scsi_transport_sas pps_core [last unloaded: vmnet]
[ 1967.296394] CPU: 2 PID: 6966 Comm: chrome Tainted: P D OE 3.16.0-25-generic #33-Ubuntu
[ 1967.296395] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v03.65 12/19/2013
[ 1967.296396] task: ffff8807fb4e0a30 ti: ffff8800b5a18000 task.ti: ffff8800b5a18000
[ 1967.296397] RIP: 0010:[<ffffffff811624b5>] [<ffffffff811624b5>] __delete_from_page_cache+0xd5/0x2c0
[ 1967.296400] RSP: 0018:ffff8800b5a1bba0 EFLAGS: 00010046
[ 1967.296401] RAX: ffffffff81767492 RBX: 0000000000000003 RCX: 000000000000002c
[ 1967.296402] RDX: 0000000000000002 RSI: ffff88070000782c RDI: ffff88071d7570f0
[ 1967.296403] RBP: ffff8800b5a1bbe8 R08: ffff88070000782c R09: ffff88071b7b8920
[ 1967.296404] R10: ffff8800b5a1bbb8 R11: 0000000000000220 R12: ffffea001c0001c0
[ 1967.296404] R13: ffff88071d7570e8 R14: 0000000000000000 R15: ffff88071d7570f0
[ 1967.296406] FS: 00007f4fcf960a00(0000) GS:ffff88082fc40000(0000) knlGS:0000000000000000
[ 1967.296407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1967.296408] CR2: ffffffff81767496 CR3: 00000000b5a97000 CR4: 00000000000407e0
[ 1967.296409] Stack:
[ 1967.296410] 000000000000002c ffff88070000782c ffffffff81767492 ffff88071d7570e8
[ 1967.296411] ffffea001c0001c0 ffff88071d7570e8 0000000000000000 ffff88071d757100
[ 1967.296413] 00000000000001ba ffff8800b5a1bc18 ffffffff811626de ffffea001c0001c0
[ 1967.296415] Call Trace:
[ 1967.296419] [<ffffffff81767492>] ? mutex_lock+0x12/0x2f
[ 1967.296422] [<ffffffff811626de>] delete_from_page_cache+0x3e/0xa0
[ 1967.296425] [<ffffffff8117095b>] truncate_inode_page+0x5b/0x90
[ 1967.296427] [<ffffffff8117b75b>] shmem_undo_range+0x2bb/0x670
[ 1967.296429] [<ffffffff8117bb24>] shmem_truncate_range+0x14/0x30
[ 1967.296430] [<ffffffff8117bbf7>] shmem_evict_inode+0xb7/0x140
[ 1967.296434] [<ffffffff811eecf4>] evict+0xb4/0x180
[ 1967.296436] [<ffffffff811ef4d5>] iput+0xf5/0x180
[ 1967.296437] [<ffffffff811ea748>] __dentry_kill+0x1a8/0x200
[ 1967.296439] [<ffffffff811ea845>] dput+0xa5/0x180
[ 1967.296441] [<ffffffff811d5c28>] __fput+0x188/0x220
[ 1967.296442] [<ffffffff811d5d0e>] ____fput+0xe/0x10
[ 1967.296446] [<ffffffff8108e107>] task_work_run+0xa7/0xe0
[ 1967.296450] [<ffffffff81012f67>] do_notify_resume+0x97/0xb0
[ 1967.296451] [<ffffffff8176936a>] int_signal+0x12/0x17
[ 1967.296452] Code: 0f 85 b0 00 00 00 83 c3 01 83 fb 03 74 08 48 8b 45 c8 eb dd 66 90 41 f6 c6 01 48 8b 45 d0 0f 85 e7 01 00 00 4c 89 30 48 8b 45 c8 <83> 68 04 01 4d 85 f6 0f 84 b8 01 00 00 48 8b 45 c8 83 68 04 80
[ 1967.296471] RIP [<ffffffff811624b5>] __delete_from_page_cache+0xd5/0x2c0
[ 1967.296473] RSP <ffff8800b5a1bba0>
[ 1967.296474] CR2: ffffffff81767496
[ 1967.296476] ---[ end trace a783e9182f5a2f02 ]---

So far I hit it once, so this is sort of the canary. I'll update the bug if I hit it again. Also I'll go and uninstall vmware-workstation.

Philipp Kern, thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command, as it will automatically gather debugging information, in a terminal:
apport-collect 1398465

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs .

affects: linux-lts-utopic (Ubuntu) → linux (Ubuntu)
Changed in linux (Ubuntu):
importance: Undecided → Low
status: New → Incomplete
Changed in linux (Ubuntu):
importance: Low → Medium
tags: added: kernel-da-key utopic
Philipp Kern (pkern) wrote :

Seems fixed in -26. That one didn't crash for me yet while -25 crashed within hours. The changelog looks quite horrible to me with quite a few memory corruptions, including thp (on "always" for me), compound page freeing, FUSE, and vfs. Let's close this unless I see it again on -26.

Changed in linux (Ubuntu):
status: Incomplete → Invalid
status: Invalid → Won't Fix

Not Won't FIx as outlined in https://wiki.ubuntu.com/Bugs/Status .

Changed in linux (Ubuntu):
status: Won't Fix → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers