Comment 20 for bug 1505948

Revision history for this message
Robert Doebbelin (2-robert-3) wrote : Re: [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)

Great, thanks!

Robert
Am 11.03.2016 15:01 schrieb "Seth Forshee" <email address hidden>:

> On Fri, Mar 11, 2016 at 01:03:32PM -0000, Robert Doebbelin wrote:
> > Thank you Seth for taking a close look at the problem and my proposed
> > fix. As mentioned on the mailing list my test runs fine now with the two
> > fixes.
> >
> > However, I prefer your fix as it prevents us from running into this
> > issue again. Our test system is happily installing VMs for two hours now
> > using your build. Please propose your patch.
>
> I'm not subscribed to fuse-devel and hadn't refreshed the mailing list
> thread so I didn't realize that you had discovered that the hang was
> unrelated. That's good.
>
> I'm happy to send the patches, I'll go ahead and send both my patch and
> your iocb patch after I make sure it all applies/builds okay on 4.5.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1505948
>
> Title:
> Memory arena corruption with FUSE (was Memory allocation failure
> crashes kernel hard, presumably related to FUSE)
>
> Status in linux package in Ubuntu:
> Confirmed
> Status in linux source package in Wily:
> Confirmed
> Status in linux package in Fedora:
> Unknown
>
> Bug description:
> Hello everybody,
>
> Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our
> setup when trying to start a Qemu process on top of a fuse-based
> mount. Here is an example stacktrace:
>
> [ 739.807817] BUG: unable to handle kernel paging request at
> ffff8800a4104ea0
> [ 739.840201] IP: [<ffffffff811cc95a>] kmem_cache_alloc_trace+0x7a/0x1f0
> [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0
> [ 739.890418] Oops: 0000 [#1] SMP
> [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre
> ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa
> ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter
> xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit
> nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack
> ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi
> x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
> glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost
> macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac
> 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek
> ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1
> sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure
> hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core
> hpsa mdio wmi
> [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted
> 4.2.0-040200-generic #201508301530
> [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
> [ 740.416827] task: ffff882f8e958dc0 ti: ffff882f28c20000 task.ti:
> ffff882f28c20000
> [ 740.451672] RIP: 0010:[<ffffffff811cc95a>] [<ffffffff811cc95a>]
> kmem_cache_alloc_trace+0x7a/0x1f0
> [ 740.494047] RSP: 0018:ffff882f28c23c68 EFLAGS: 00010286
> [ 740.518425] RAX: 0000000000000000 RBX: 00000000000000d0 RCX:
> 00000000000026b3
> [ 740.551611] RDX: 00000000000026b2 RSI: 00000000000000d0 RDI:
> ffff882fbf407840
> [ 740.584846] RBP: ffff882f28c23ca8 R08: 0000000000019920 R09:
> ffffe8d000200ab0
> [ 740.618287] R10: ffffffff812e8dcd R11: ffffea00bca0ac00 R12:
> 00000000000000d0
> [ 740.651320] R13: ffff882fbf407840 R14: ffff8800a4104ea0 R15:
> ffff882fbf407840
> [ 740.684195] FS: 00007f2642ffd700(0000) GS:ffff882fbfa00000(0000)
> knlGS:0000000000000000
> [ 740.722030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 740.749469] CR2: ffff8800a4104ea0 CR3: 0000002f26f83000 CR4:
> 00000000001426e0
> [ 740.783390] Stack:
> [ 740.792577] ffffffff812e8dcd 0000000000000048 0000000000000002
> ffff882f908c8468
> [ 740.827003] 0000000001bef000 ffff882f928e4600 ffff882f28c23e48
> ffff882f28c23d70
> [ 740.860971] ffff882f28c23d38 ffffffff812e8dcd 0000000000000001
> ffff882f908c8300
> [ 740.894994] Call Trace:
> [ 740.906211] [<ffffffff812e8dcd>] ? fuse_direct_IO+0xdd/0x280
> [ 740.932940] [<ffffffff812e8dcd>] fuse_direct_IO+0xdd/0x280
> [ 740.958866] [<ffffffff8117750e>] generic_file_direct_write+0x9e/0x150
> [ 740.989318] [<ffffffff812e96bc>] fuse_file_write_iter+0x15c/0x2e0
> [ 741.017725] [<ffffffff811e94a7>] __vfs_write+0xa7/0xf0
> [ 741.041787] [<ffffffff811e9b09>] vfs_write+0xa9/0x190
> [ 741.065307] [<ffffffff811ea9d9>] SyS_pwrite64+0x69/0xa0
> [ 741.090141] [<ffffffff81085b57>] ? SyS_rt_sigprocmask+0x67/0xb0
> [ 741.135924] [<ffffffff817a8e32>] entry_SYSCALL_64_fastpath+0x16/0x75
> [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6
> 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b
> 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63
> [ 741.306817] RIP [<ffffffff811cc95a>]
> kmem_cache_alloc_trace+0x7a/0x1f0
>
> The problem has also been documented by somebody else in the Fedora
> bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310
>
> This behaviour is 100% reproducible. I have asked the fuse-devel
> mailinglist for advice, but up to this point with no success:
>
> http://sourceforge.net/p/fuse/mailman/message/34537139/
>
> We are still investigating if this issue is also happening with 4.0
> and will add the information to this bug report once we have it. Any
> help on debugging will be greatly appreciated.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+subscriptions
>

--

--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender