remove_proc_entry+0x139/0x1b0() -- name 'fs/nfsfs'

Bug #1376245 reported by Stéphane Loeuillet on 2014-10-01
162
This bug affects 30 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Medium
Unassigned
Trusty
Undecided
Stefan Bader
Utopic
Medium
Unassigned

Bug Description

Ubuntu 14.04 amd64

ii linux-image-3.13.0-37-generic 3.13.0-37.64 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP

sloeuillet@SLoeuillet-DE107:/sfr$ uname -a
Linux SLoeuillet-DE107 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Today, I encountered 2 quite identical stack traces in my dmesg, related to nfs :

[368973.412120] WARNING: CPU: 1 PID: 15607 at /build/buildd/linux-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x139/0x1b0()
[368973.412122] name 'fs/nfsfs'
[368973.412123] Modules linked in: cdc_acm nfsv3 rpcsec_gss_krb5 8021q garp stp mrp llc cuse bnep rfcomm bluetooth nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc binfmt_misc fscache hp_wmi sparse_keymap snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm radeon snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq sp5100_tco snd_seq_device serio_raw snd_timer ttm drm_kms_helper snd kvm edac_core soundcore drm edac_mce_amd k8temp i2c_algo_bit i2c_piix4 tpm_infineon shpchp wmi mac_hid parport_pc ppdev lp parport hid_generic usbhid hid tg3 psmouse ptp pps_core floppy ahci libahci
[368973.412164] CPU: 1 PID: 15607 Comm: kworker/u8:1 Tainted: G W 3.13.0-37-generic #64-Ubuntu
[368973.412166] Hardware name: Hewlett-Packard HP Compaq dc5850 Small Form Factor/3029h, BIOS 786F6 v03.14 11/15/2011
[368973.412171] Workqueue: netns cleanup_net
[368973.412172] 0000000000000009 ffff88006a875c80 ffffffff8171ed09 ffff88006a875cc8
[368973.412176] ffff88006a875cb8 ffffffff8106773d 0000000000000000 0000000000000005
[368973.412179] ffffffffa05168a8 ffff88011b418b30 0000000000000100 ffff88006a875d18
[368973.412182] Call Trace:
[368973.412188] [<ffffffff8171ed09>] dump_stack+0x45/0x56
[368973.412192] [<ffffffff8106773d>] warn_slowpath_common+0x7d/0xa0
[368973.412195] [<ffffffff810677ac>] warn_slowpath_fmt+0x4c/0x50
[368973.412198] [<ffffffff81229839>] remove_proc_entry+0x139/0x1b0
[368973.412217] [<ffffffffa04f64e2>] nfs_fs_proc_net_exit+0x62/0x70 [nfs]
[368973.412225] [<ffffffffa04fc5b2>] nfs_net_exit+0x12/0x20 [nfs]
[368973.412228] [<ffffffff8161b409>] ops_exit_list.isra.1+0x39/0x60
[368973.412231] [<ffffffff8161bc90>] cleanup_net+0x110/0x250
[368973.412235] [<ffffffff810839c2>] process_one_work+0x182/0x450
[368973.412237] [<ffffffff810847b1>] worker_thread+0x121/0x410
[368973.412240] [<ffffffff81084690>] ? rescuer_thread+0x430/0x430
[368973.412243] [<ffffffff8108b492>] kthread+0xd2/0xf0
[368973.412246] [<ffffffff8108b3c0>] ? kthread_create_on_node+0x1c0/0x1c0
[368973.412249] [<ffffffff8172f77c>] ret_from_fork+0x7c/0xb0
[368973.412251] [<ffffffff8108b3c0>] ? kthread_create_on_node+0x1c0/0x1c0
[368973.412253] ---[ end trace bb78fe56a3dcb678 ]---

[368973.448199] WARNING: CPU: 1 PID: 15607 at /build/buildd/linux-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x139/0x1b0()
[368973.448201] name 'fs/nfsfs'
[368973.448203] Modules linked in: cdc_acm nfsv3 rpcsec_gss_krb5 8021q garp stp mrp llc cuse bnep rfcomm bluetooth nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc binfmt_misc fscache hp_wmi sparse_keymap snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm radeon snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq sp5100_tco snd_seq_device serio_raw snd_timer ttm drm_kms_helper snd kvm edac_core soundcore drm edac_mce_amd k8temp i2c_algo_bit i2c_piix4 tpm_infineon shpchp wmi mac_hid parport_pc ppdev lp parport hid_generic usbhid hid tg3 psmouse ptp pps_core floppy ahci libahci
[368973.448250] CPU: 1 PID: 15607 Comm: kworker/u8:1 Tainted: G W 3.13.0-37-generic #64-Ubuntu
[368973.448252] Hardware name: Hewlett-Packard HP Compaq dc5850 Small Form Factor/3029h, BIOS 786F6 v03.14 11/15/2011
[368973.448257] Workqueue: netns cleanup_net
[368973.448259] 0000000000000009 ffff88006a875c80 ffffffff8171ed09 ffff88006a875cc8
[368973.448263] ffff88006a875cb8 ffffffff8106773d 0000000000000000 0000000000000005
[368973.448265] ffffffffa05168a8 ffff88011b418b30 0000000000000100 ffff88006a875d18
[368973.448268] Call Trace:
[368973.448275] [<ffffffff8171ed09>] dump_stack+0x45/0x56
[368973.448279] [<ffffffff8106773d>] warn_slowpath_common+0x7d/0xa0
[368973.448283] [<ffffffff810677ac>] warn_slowpath_fmt+0x4c/0x50
[368973.448286] [<ffffffff81229839>] remove_proc_entry+0x139/0x1b0
[368973.448313] [<ffffffffa04f64e2>] nfs_fs_proc_net_exit+0x62/0x70 [nfs]
[368973.448322] [<ffffffffa04fc5b2>] nfs_net_exit+0x12/0x20 [nfs]
[368973.448325] [<ffffffff8161b409>] ops_exit_list.isra.1+0x39/0x60
[368973.448327] [<ffffffff8161bc90>] cleanup_net+0x110/0x250
[368973.448331] [<ffffffff810839c2>] process_one_work+0x182/0x450
[368973.448334] [<ffffffff810847b1>] worker_thread+0x121/0x410
[368973.448337] [<ffffffff81084690>] ? rescuer_thread+0x430/0x430
[368973.448341] [<ffffffff8108b492>] kthread+0xd2/0xf0
[368973.448343] [<ffffffff8108b3c0>] ? kthread_create_on_node+0x1c0/0x1c0
[368973.448346] [<ffffffff8172f77c>] ret_from_fork+0x7c/0xb0
[368973.448349] [<ffffffff8108b3c0>] ? kthread_create_on_node+0x1c0/0x1c0
[368973.448351] ---[ end trace bb78fe56a3dcb679 ]---

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-37-generic 3.13.0-37.64
ProcVersionSignature: Ubuntu 3.13.0-37.64-generic 3.13.11.7
Uname: Linux 3.13.0-37-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.4
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: sloeuillet 2501 F.... pulseaudio
CurrentDesktop: Unity
Date: Wed Oct 1 14:44:17 2014
HibernationDevice: RESUME=UUID=f74ad7c2-cbec-4b9b-8b27-d1b624315ca8
InstallationDate: Installed on 2013-09-16 (380 days ago)
InstallationMedia: Ubuntu 12.04.3 LTS "Precise Pangolin" - Release amd64 (20130820.1)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
MachineType: Hewlett-Packard HP Compaq dc5850 Small Form Factor
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-37-generic root=/dev/sda1 ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-37-generic N/A
 linux-backports-modules-3.13.0-37-generic N/A
 linux-firmware 1.127.7
RfKill:

SourcePackage: linux
UpgradeStatus: Upgraded to trusty on 2014-04-28 (156 days ago)
dmi.bios.date: 11/15/2011
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: 786F6 v03.14
dmi.board.name: 3029h
dmi.board.vendor: Hewlett-Packard
dmi.chassis.type: 4
dmi.chassis.vendor: Hewlett-Packard
dmi.modalias: dmi:bvnHewlett-Packard:bvr786F6v03.14:bd11/15/2011:svnHewlett-Packard:pnHPCompaqdc5850SmallFormFactor:pvr:rvnHewlett-Packard:rn3029h:rvr:cvnHewlett-Packard:ct4:cvr:
dmi.product.name: HP Compaq dc5850 Small Form Factor
dmi.sys.vendor: Hewlett-Packard

Stéphane Loeuillet (leroutier) wrote :

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Joseph Salisbury (jsalisbury) wrote :

Did this issue just start happening with the -37 kernel? Did it also happen with the -36 kernel?

Changed in linux (Ubuntu):
importance: Undecided → Medium
tags: added: kernel-da-key
Joseph Salisbury (jsalisbury) wrote :

May be related to the following commit:

commit 781ea515c9ef873359364957d111848963f661b6
Author: Eric W. Biederman <email address hidden>
Date: Thu Jul 31 04:35:20 2014 -0700

    NFS: Fix /proc/fs/nfsfs/servers and /proc/fs/nfsfs/volumes

Onestone (onestone) wrote :

I have the same trace in dmesg. I'm 99% positive it started happening with the -37 kernel.

Onestone (onestone) wrote :

Note: I don't have any NFS mounts. The stack trace is dumped despite that.

Currently seeing this whenever I exit a Docker container on:

Linux jenkins-1 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Jussi Saarinen (jms) wrote :

I can reproduce this regularly on two physical computers but I haven't seen this at all on two virtual machines (vmware guests running ubuntu server 14.04.01). All running kernel 3.13.0-37-generic (64bit)

Gionn (giovanni.toraldo) wrote :

For me happens on 3.13.0-37-generic, not using docker but VirtualBox

gadLinux (gad-aguilardelgado) wrote :
Download full text (3.8 KiB)

Hi,

for me happening in Linux red-compute 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux.

It seems this has something related to openvswitch that seems to trigger the error more often. But every bootup I have this on my log:

[ 437.773703] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 458.240318] ------------[ cut here ]------------
[ 458.240326] WARNING: CPU: 0 PID: 157 at /build/buildd/linux-3.13.0/fs/proc/generic.c:522 remove_proc_entry+0x19f/0x1b0()
[ 458.240328] remove_proc_entry: removing non-empty directory 'fs/nfsfs', leaking at least 'volumes'
[ 458.240330] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables veth xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi rfcomm rbd libceph nfsd auth_rpcgss openvswitch gre vxlan ip_tunnel binfmt_misc nfs_acl nfs lockd sunrpc fscache dm_crypt xfs snd_usb_audio snd_usbmidi_lib fglrx(POX) gspca_ov534 gspca_main videodev wacom joydev kvm_amd kvm dm_multipath scsi_dh serio_raw edac_core edac_mce_amd k10temp snd_hda_codec_hdmi btusb snd_hda_codec_realtek bluetooth snd_hda_intel snd_hda_codec snd_seq_midi snd_seq_midi_event snd_hwdep snd_pcm snd_rawmidi snd_seq snd_page_alloc snd_seq_device snd_timer snd soundcore sp5100_tco i2c_piix4 amd_iommu_v2 asus_atk0110 mac_hid parport_pc ppdev lp parport btrfs libcrc32c raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear dm_mirror dm_region_hash dm_log hid_generic hid_logitech_dj usbhid hid pata_acpi psmouse pata_atiixp pata_marvell e1000e firewire_ohci ahci libahci firewire_core ptp floppy pps_core crc_itu_t r8169 mii wmi
[ 458.240393] CPU: 0 PID: 157 Comm: kworker/u12:8 Tainted: P OX 3.13.0-37-generic #64-Ubuntu
[ 458.240395] Hardware name: System manufacturer System Product Name/M4A79T Deluxe, BIOS 3503 05/06/2011
[ 458.240398] Workqueue: netns cleanup_net
[ 458.240400] 0000000000000009 ffff88031964bc80 ffffffff8171ed09 ffff88031964bcc8
[ 458.240403] ffff88031964bcb8 ffffffff8106773d ffff88031d22cc80 0000000000000005
[ 458.240405] ffffffffa0ffb8a8 ffff8800cfb60130 ffff88031d22ccf5 ffff88031964bd18
[ 458.240408] Call Trace:
[ 458.240412] [<ffffffff8171ed09>] dump_stack+0x45/0x56
[ 458.240415] [<ffffffff8106773d>] warn_slowpath_common+0x7d/0xa0
[ 458.240419] [<ffffffff810677ac>] warn_slowpath_fmt+0x4c/0x50
[ 458.240421] [<ffffffff8122989f>] remove_proc_entry+0x19f/0x1b0
[ 458.240434] [<ffffffffa0fdb4e2>] nfs_fs_proc_net_exit+0x62/0x70 [nfs]
[ 458.240443] [<ffffffffa0fe15b2>] nfs_net_exit+0x12/0x20 [nfs]
[ 458.240445] [<ffffffff8161b409>] ops_exit_list.isra.1+0x39/0x60
[ 458.240447] [<ffffffff8161bc90>] cleanup_net+0x110/0x250
[ 458.240450] [<ffffffff810839c2>] process_one_work+0x182/0x450
[ 458.240453] [<ffffffff810847b1>] worker_thread+0x121/0x410
[ 458.240455] [<ffffffff81084690>] ? rescuer_thread+0x430/0x430
[ 458.240458] [<ffffffff8108b4...

Read more...

gadLinux (gad-aguilardelgado) wrote :

You can see several kernel panics here. Lots of them until massive down and hangs.

gadLinux (gad-aguilardelgado) wrote :

I must add that I'm using kvm-qemu for virtualization. Maybe this also triggers the bug. Just in case you need to know.

Tony Abbott (t-abbott) wrote :

Picked up these two panics from a few days ago in my logs so unfortunately can't say what was happening at the time. Machine has continued to be usable since. No NFS mounts but possibly related to NFS exports on the box. I am running VMware Workstation on this box. Haven't noticed it with older kernels, but then I wasn't looking for it. I have tried exporting/unexporting NFS and mounting/unmounting those exports but not able to reproduce so far.

Paul Crawford (psc-sat) wrote :

We are seeing the same a dozen or so timers per day, and it happened after we updated from the -36 to the -37 kernel build.

Kyle O'Donnell (kyleo-t) wrote :

We have this same problem. Just upgraded to -38 , no change:

[184207.109594] ------------[ cut here ]------------
[184207.109604] WARNING: CPU: 15 PID: 10701 at /build/buildd/linux-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x139/0x1b0()
[184207.109606] name 'fs/nfsfs'
[184207.109608] Modules linked in: nfsv3 ipmi_devintf veth xt_conntrack ipt_REJECT ip6table_filter ip6_tables ebtable_nat ebtables xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables autofs4 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache bridge gpio_ich dcdbas intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw joydev lpc_ich i7core_edac edac_core ipmi_si bonding wmi 8021q acpi_power_meter mac_hid garp stp mrp llc ses enclosure hid_generic usbhid psmouse hid megaraid_sas bnx2
[184207.109664] CPU: 15 PID: 10701 Comm: kworker/u66:1 Tainted: G W I 3.13.0-38-generic #65-Ubuntu
[184207.109666] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.4.0 07/23/2013
[184207.109671] Workqueue: netns cleanup_net
[184207.109673] 0000000000000009 ffff880bfaefbc80 ffffffff8171ece7 ffff880bfaefbcc8
[184207.109679] ffff880bfaefbcb8 ffffffff8106773d 0000000000000000 0000000000000005
[184207.109683] ffffffffa02f58a8 ffff880c0f802db0 0000000000000840 ffff880bfaefbd18
[184207.109687] Call Trace:
[184207.109693] [<ffffffff8171ece7>] dump_stack+0x45/0x56
[184207.109698] [<ffffffff8106773d>] warn_slowpath_common+0x7d/0xa0
[184207.109702] [<ffffffff810677ac>] warn_slowpath_fmt+0x4c/0x50
[184207.109707] [<ffffffff81229889>] remove_proc_entry+0x139/0x1b0
[184207.109721] [<ffffffffa02d54e2>] nfs_fs_proc_net_exit+0x62/0x70 [nfs]
[184207.109732] [<ffffffffa02db5b2>] nfs_net_exit+0x12/0x20 [nfs]
[184207.109735] [<ffffffff8161b3a9>] ops_exit_list.isra.1+0x39/0x60
[184207.109739] [<ffffffff8161bc30>] cleanup_net+0x110/0x250
[184207.109745] [<ffffffff810839c2>] process_one_work+0x182/0x450
[184207.109749] [<ffffffff810847b1>] worker_thread+0x121/0x410
[184207.109753] [<ffffffff81084690>] ? rescuer_thread+0x430/0x430
[184207.109757] [<ffffffff8108b492>] kthread+0xd2/0xf0
[184207.109761] [<ffffffff8108b3c0>] ? kthread_create_on_node+0x1c0/0x1c0
[184207.109765] [<ffffffff8172f73c>] ret_from_fork+0x7c/0xb0
[184207.109769] [<ffffffff8108b3c0>] ? kthread_create_on_node+0x1c0/0x1c0
[184207.109771] ---[ end trace bad0608c86108035 ]---

Stefan Bader (smb) wrote :

There is an additional patch that hit 3.17 upstream recently. It is marked for stable but had not enough time to come through:

commit 2f3169fb18f4643ac9a6a097a6a6c71f0b2cef75
Author: Fabian Frederick <email address hidden>
Date: Wed Sep 24 18:56:11 2014 +0200

    nfs: fix duplicate proc entries

Reinhold Kainhofer (reinhold) wrote :

If it helps: The very first of those kernel stack traces has a slightly different message:

[49237.227779] ------------[ cut here ]------------
[49237.227787] WARNING: CPU: 2 PID: 24871 at /build/buildd/linux-3.13.0/fs/proc/generic.c:522 remove_proc_entry+0x19f/0x1b0()
[49237.227789] remove_proc_entry: removing non-empty directory 'fs/nfsfs', leaking at least 'volumes'
[49237.227790] Modules linked in: [...]

The rest is the same as already posted here.

Stefan Bader (smb) wrote :

Although it should come sooner or later via stable, if people want to test ahead of that I put kernel images with the anticipated fix at http://people.canonical.com/~smb/lp1376245/

Mark Wilkinson (mhw) wrote :

I was getting this error regularly in my system log yesterday. Since installing the kernel images from #18 I'm not seeing the error any more, so it looks like that's the fix.

Tim Gardner (timg-tpi) on 2014-10-21
Changed in linux (Ubuntu Trusty):
assignee: nobody → Stefan Bader (smb)
status: New → Fix Committed
Changed in linux (Ubuntu Utopic):
status: Confirmed → Fix Released

Note that I can reproduce the WARNING consistently with:

   ip netns add test; ip netns delete test

James McVey (w-ubuntuonq-2) wrote :

When will this fix be released for trusty?

I am experiencing this situation about every 3rd day.

Have 3 Ubuntu systems tied together - with the system that crashes as the NFS server - along with the primary MythTV service. The other 2 systems read/write to the file system as MythTV services and one is the primary PLEX service.

Looking forward to having this annoying situation resolved.

Seems to occur when the file system is being used by 2 or more of the secondary servers - however - not been able to pin point the exact usage pattern at the time of the fault.

James McVey (w-ubuntuonq-2) wrote :

System crashed again last night due to this bug.

How can this issue be escalated and placed into the next Trusty release?

James McVey (w-ubuntuonq-2) wrote :
Download full text (7.0 KiB)

Snippet from syslog.1

Last night:
=========================================

Nov 2 23:10:06 ubuntu-1 kernel: [140230.568863] ------------[ cut here ]------------
Nov 2 23:10:06 ubuntu-1 kernel: [140230.568877] WARNING: CPU: 1 PID: 11663 at /build/buildd/linux-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x139/0x1b0()
Nov 2 23:10:06 ubuntu-1 kernel: [140230.568880] name 'fs/nfsfs'
Nov 2 23:10:06 ubuntu-1 kernel: [140230.568883] Modules linked in: snd_hda_codec_hdmi bnep rfcomm kvm_amd bluetooth kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_seq_midi snd_seq_midi_event aesni_intel snd_hda_codec_realtek aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw fam15h_power edac_core edac_mce_amd k10temp snd_hda_intel snd_hda_codec snd_hwdep nouveau snd_pcm snd_page_alloc snd_rawmidi sp5100_tco i2c_piix4 snd_seq mxm_wmi wmi video snd_seq_device ttm snd_timer drm_kms_helper parport_pc drm binfmt_misc snd ppdev i2c_algo_bit soundcore mac_hid lp parport nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 firewire_ohci raid0 r8169 firewire_core mii crc_itu_t ahci multipath libahci linear
Nov 2 23:10:06 ubuntu-1 kernel: [140230.568977] CPU: 1 PID: 11663 Comm: kworker/u16:1 Tainted: G W 3.13.0-39-generic #66-Ubuntu
Nov 2 23:10:06 ubuntu-1 kernel: [140230.568981] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./990FXA-UD5, BIOS FB 01/23/2013
Nov 2 23:10:06 ubuntu-1 kernel: [140230.568988] Workqueue: netns cleanup_net
Nov 2 23:10:06 ubuntu-1 kernel: [140230.568992] 0000000000000009 ffff8803f8aa1c80 ffffffff8171ece7 ffff8803f8aa1cc8
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569000] ffff8803f8aa1cb8 ffffffff8106773d 0000000000000000 0000000000000005
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569007] ffffffffa01cd8a8 ffff88041e005ab0 0000000000000200 ffff8803f8aa1d18
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569014] Call Trace:
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569023] [<ffffffff8171ece7>] dump_stack+0x45/0x56
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569030] [<ffffffff8106773d>] warn_slowpath_common+0x7d/0xa0
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569039] [<ffffffff810677ac>] warn_slowpath_fmt+0x4c/0x50
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569050] [<ffffffff81229889>] remove_proc_entry+0x139/0x1b0
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569078] [<ffffffffa01ad4e2>] nfs_fs_proc_net_exit+0x62/0x70 [nfs]
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569098] [<ffffffffa01b35b2>] nfs_net_exit+0x12/0x20 [nfs]
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569104] [<ffffffff8161b3a9>] ops_exit_list.isra.1+0x39/0x60
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569112] [<ffffffff8161bc30>] cleanup_net+0x110/0x250
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569120] [<ffffffff810839c2>] process_one_work+0x182/0x450
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569126] [<ffffffff810847b1>] worker_thread+0x121/0x410
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569132] [<ffffffff81084690>] ? rescuer_thread+0x430/0x430
Nov 2 23:10:06 ubuntu-1 kernel: [140230.569138] [<ffffffff8108b492>] kthread+0xd2/0xf...

Read more...

James McVey (w-ubuntuonq-2) wrote :

System crashed again - soon after my last post.

Can we please get this fix promoted into the Trusty stream for release?

Thanks !

James McVey (w-ubuntuonq-2) wrote :

After re-booting the system - 4 failures have already appeared in the syslog within 20 minutes .....

Running a handbrakecli process directly on the machine.

This really frustrating - having to deal with this - when the solution exists - yet not promoted.

Please - Please push this fix into the next available Trusty release .....

berend (berenddeboer) wrote :
Download full text (3.8 KiB)

My system doesn't crash, but I get something similar (may be unrelated), not sure, just found this bug report via Google. This is what I get:

Nov 10 13:41:58 quadrio kernel: [ 1324.267970] WARNING: CPU: 0 PID: 14440 at /build/buildd/linux-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x139/0x1b0()
Nov 10 13:41:58 quadrio kernel: [ 1324.267972] name 'fs/nfsfs'
Nov 10 13:41:58 quadrio kernel: [ 1324.267974] Modules linked in: vmnet(OX) parport_pc vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OX) nfsv3 xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc ctr ccm aufs iptable_filter ip_tables x_tables pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) autofs4 uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core videodev btusb dm_crypt rfcomm bnep bluetooth mxm_wmi nfsd auth_rpcgss nfs_acl nfs lockd dm_multipath sunrpc scsi_dh fscache coretemp kvm_intel kvm nvidia(POX) joydev serio_raw pcmcia arc4 snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep snd_pcm iwldvm thinkpad_acpi nvram mac80211 snd_page_alloc snd_seq_midi snd_seq_midi_event r852 sm_common snd_rawmidi nand nand_ecc nand_bch snd_seq bch nand_ids snd_seq_device mtd snd_timer lpc_ich iwlwifi r592 snd memstick yenta_socket cfg80211 pcmcia_rsrc pcmcia_core soundcore drm wmi mei_me mei mac_hid ppdev lp parport xfs libcrc32c hid_generic usbhid hid psmouse ahci sdhci_pci libahci firewire_ohci firewire_core sdhci crc_itu_t e1000e ptp pps_core video [last unloaded: vmnet]
Nov 10 13:41:58 quadrio kernel: [ 1324.268106] CPU: 0 PID: 14440 Comm: kworker/u8:1 Tainted: P W OX 3.13.0-39-generic #66-Ubuntu
Nov 10 13:41:58 quadrio kernel: [ 1324.268109] Hardware name: LENOVO 2757CTO/2757CTO, BIOS 7ZET71WW (1.54 ) 11/27/2008
Nov 10 13:41:58 quadrio kernel: [ 1324.268111] Workqueue: netns cleanup_net
Nov 10 13:41:58 quadrio kernel: [ 1324.268113] 0000000000000009 ffff8801cb863c80 ffffffff8171ece7 ffff8801cb863cc8
Nov 10 13:41:58 quadrio kernel: [ 1324.268117] ffff8801cb863cb8 ffffffff8106773d 0000000000000000 0000000000000005
Nov 10 13:41:58 quadrio kernel: [ 1324.268120] ffffffffa10e08a8 ffff880253018ab0 0000000000000100 ffff8801cb863d18
Nov 10 13:41:58 quadrio kernel: [ 1324.268123] Call Trace:
Nov 10 13:41:58 quadrio kernel: [ 1324.268127] [<ffffffff8171ece7>] dump_stack+0x45/0x56
Nov 10 13:41:58 quadrio kernel: [ 1324.268130] [<ffffffff8106773d>] warn_slowpath_common+0x7d/0xa0
Nov 10 13:41:58 quadrio kernel: [ 1324.268134] [<ffffffff810677ac>] warn_slowpath_fmt+0x4c/0x50
Nov 10 13:41:58 quadrio kernel: [ 1324.268139] [<ffffffff81229889>] remove_proc_entry+0x139/0x1b0
Nov 10 13:41:58 quadrio kernel: [ 1324.268153] [<ffffffffa10c04e2>] nfs_fs_proc_net_exit+0x62/0x70 [nfs]
Nov 10 13:41:58 quadrio kernel: [ 1324.268167] [<ffffffffa10c65b2>] nfs_net_exit+0x12/0x20 [nfs]
Nov 10 13:41:58 quadrio kernel: [ 1324.268171] [<ffffffff8161b3a9>] ops_exit_list.isra.1+0x39/0x60
Nov 10 13:41:58 quadrio kernel: [ 1324.268175] [<ffffffff8161bc30>] cleanup_net+0x110/0x250
Nov 10 13:41:58 quadrio kernel: [ 1324.268179] [<ffffffff810839c2>] process_one_work+0x182/0x450
Nov 10 13:41:58 quadrio kern...

Read more...

berend (berenddeboer) wrote :
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-trusty
Onestone (onestone) on 2014-11-11
tags: added: verification-done-trusty
removed: verification-needed-trusty
Paul Crawford (psc-sat) wrote :

Updated a 12.04 LTS box today with the kernel "3.13.0-39-generic #66~precise1-Ubuntu SMP Wed Oct 29 09:56:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux" and problem still there.

James McVey (w-ubuntuonq-2) wrote :

System crashed again last night.

it appears this has been approved for promotion into next Trusty update.

Not able to obtain the timing of the next update for Trusty.

Is there a scheduled release date?

James McVey (w-ubuntuonq-2) wrote :

An FYI - my system crashed again last night.

Since this is the host for the NFS system - and gets used extensively for managing the DVR recordings - this system gets several large write operations a night - and more reads as the programs are streamed to Roku devices.

Marcus Furlong (furlongm) wrote :

Also seeing this on Trusty, with kernel 3.13.0-39-generic.

Bob (bgcladd) wrote :

Any eta on when this fix will hit trusty?

first-leon (first-leon) wrote :

uname -a
Linux amber 3.13.0-39-generic #66~precise1-Ubuntu SMP Wed Oct 29 09:56:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

In dmesg i see this message every time (but system is not crashes)

[56757.172197] ------------[ cut here ]------------
[56757.172222] WARNING: CPU: 1 PID: 12934 at /build/buildd/linux-lts-trusty-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x13a/0x1b0()
[56757.172226] name 'fs/nfsfs'
[56757.172230] Modules linked in: vhost_net macvtap macvlan vhost ip6table_filter ip6_tables ebtable_nat ebtables kvm_amd kvm xt_tcpudp xt_nat xt_conntrack iptable_mangle iptable_nat nf_nat_ipv4 nf_nat iptable_filter ip_tables x_tables nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bridge 8021q mrp garp stp radeon llc ttm drm_kms_helper drm ch341 usbserial psmouse serio_raw sp5100_tco edac_core i2c_piix4 i2c_algo_bit edac_mce_amd k8temp shpchp asus_atk0110 mac_hid nfsd nfs_acl auth_rpcgss nfs fscache lockd sunrpc lp parport hid_generic usbhid hid pata_acpi raid10 raid456 async_pq async_xor async_memcpy async_raid6_recov async_tx raid1 raid0 multipath linear btrfs pata_atiixp r8169 ahci mii raid6_pq libahci xor libcrc32c
[56757.172356] CPU: 1 PID: 12934 Comm: kworker/u8:2 Tainted: G W 3.13.0-39-generic #66~precise1-Ubuntu
[56757.172362] Hardware name: System manufacturer System Product Name/M2A-VM, BIOS ASUS M2A-VM ACPI BIOS Revision 1604 12/19/2007
[56757.172372] Workqueue: netns cleanup_net
[56757.172377] 00000000000001ff ffff880019c87c08 ffffffff81755f65 0000000000000007
[56757.172387] ffff880019c87c58 ffff880019c87c48 ffffffff8106afcc 0000000000000013
[56757.172394] 0000000000000000 0000000000000005 ffff8800754189b0 ffff880019c87db8
[56757.172401] Call Trace:
[56757.172416] [<ffffffff81755f65>] dump_stack+0x46/0x58
[56757.172427] [<ffffffff8106afcc>] warn_slowpath_common+0x8c/0xc0
[56757.172438] [<ffffffff8106b0b6>] warn_slowpath_fmt+0x46/0x50
[56757.172447] [<ffffffff81236faa>] remove_proc_entry+0x13a/0x1b0
[56757.172456] [<ffffffff811adb94>] ? kfree+0x134/0x170
[56757.172499] [<ffffffffa02695c6>] nfs_fs_proc_net_exit+0x66/0x80 [nfs]
[56757.172522] [<ffffffffa026f696>] nfs_net_exit+0x16/0x30 [nfs]
[56757.172531] [<ffffffff816467c8>] ops_exit_list.isra.1+0x38/0x60
[56757.172539] [<ffffffff816470c2>] cleanup_net+0x112/0x230
[56757.172548] [<ffffffff8108797f>] process_one_work+0x17f/0x4c0
[56757.172555] [<ffffffff81088b9b>] worker_thread+0x11b/0x3d0
[56757.172562] [<ffffffff81088a80>] ? manage_workers.isra.21+0x190/0x190
[56757.172569] [<ffffffff8108fb29>] kthread+0xc9/0xe0
[56757.172576] [<ffffffff8108fa60>] ? flush_kthread_worker+0xb0/0xb0
[56757.172583] [<ffffffff8176b5bc>] ret_from_fork+0x7c/0xb0
[56757.172589] [<ffffffff8108fa60>] ? flush_kthread_worker+0xb0/0xb0
[56757.172594] ---[ end trace a39f667ba5f1c01f ]---

Frantique (undernetangel) wrote :
Download full text (10.7 KiB)

No crash, but dmesg full with:

[ 105.544050] ------------[ cut here ]------------
[ 105.544057] WARNING: CPU: 1 PID: 59 at /build/buildd/linux-lts-trusty-3.13.0/fs/proc/generic.c:522 remove_proc_entry+0x16d/0x180()
[ 105.544059] remove_proc_entry: removing non-empty directory 'fs/nfsfs', leaking at least 'volumes'
[ 105.544060] Modules linked in: des_generic rpcsec_gss_krb5 md4 snd_hrtimer pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) dm_crypt rfcomm bnep bluetooth decnet nfsd nfsv4 nfs_acl auth_rpcgss nfs qmi_wwan uvcvideo usbnet videobuf2_core videodev mii qcserial usb_wwan nls_utf8 binfmt_misc videobuf2_vmalloc usbserial videobuf2_memops cdc_wdm lockd cifs sunrpc fscache arc4 iwldvm snd_hda_codec_hdmi snd_hda_codec_realtek mac80211 joydev snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device iwlwifi snd soundcore psmouse cfg80211 snd_page_alloc serio_raw fujitsu_laptop mei_me tpm_infineon mei lpc_ich rtsx_pci_ms ppdev mac_hid memstick parport_pc i2c_i801 coretemp lp parport hid_generic usbhid hid mmc_block rtsx_pci_sdmmc i915 sdhci_pci ahci sdhci libahci rtsx_pci drm_kms_helper drm e1000e ptp i2c_algo_bit video pps_core
[ 105.544107] CPU: 1 PID: 59 Comm: kworker/u16:1 Tainted: G OX 3.13.0-39-generic #66~precise1-Ubuntu
[ 105.544108] Hardware name: FUJITSU LIFEBOOK E781/FJNB222, BIOS Version 1.22 04/19/2013
[ 105.544111] Workqueue: netns cleanup_net
[ 105.544112] 00000000 00000000 f2141e10 c1683110 c18974d4 f2141e40 c1059c54 c1897538
[ 105.544116] f2141e6c 0000003b c18974d4 0000020a c11deb6d c11deb6d f1741840 f06809e4
[ 105.544119] 00000005 f2141e58 c1059d13 00000009 f2141e50 c1897538 f2141e6c f2141e9c
[ 105.544122] Call Trace:
[ 105.544127] [<c1683110>] dump_stack+0x41/0x52
[ 105.544131] [<c1059c54>] warn_slowpath_common+0x84/0xa0
[ 105.544133] [<c11deb6d>] ? remove_proc_entry+0x16d/0x180
[ 105.544135] [<c11deb6d>] ? remove_proc_entry+0x16d/0x180
[ 105.544136] [<c1059d13>] warn_slowpath_fmt+0x33/0x40
[ 105.544138] [<c11deb6d>] remove_proc_entry+0x16d/0x180
[ 105.544146] [<f996b20b>] nfs_fs_proc_net_exit+0x4b/0x60 [nfs]
[ 105.544151] [<f9970980>] nfs_net_exit+0x10/0x20 [nfs]
[ 105.544153] [<c1594e7e>] ops_exit_list.isra.1+0x2e/0x50
[ 105.544155] [<c15955c5>] cleanup_net+0xd5/0x1e0
[ 105.544158] [<c1072739>] process_one_work+0x119/0x3c0
[ 105.544160] [<c10734ca>] worker_thread+0xfa/0x370
[ 105.544162] [<c10733d0>] ? manage_workers.isra.24+0x170/0x170
[ 105.544164] [<c107928b>] kthread+0x9b/0xb0
[ 105.544167] [<c1695af7>] ret_from_kernel_thread+0x1b/0x28
[ 105.544169] [<c10791f0>] ? flush_kthread_worker+0x90/0x90
[ 105.544171] ---[ end trace 7c197d278182ac8c ]---
[ 105.640016] ------------[ cut here ]------------
[ 105.640024] WARNING: CPU: 2 PID: 59 at /build/buildd/linux-lts-trusty-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x125/0x180()
[ 105.640025] name 'fs/nfsfs'
[ 105.640026] Modules linked in: des_generic rpcsec_gss_krb5 md4 snd_hrtimer pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) dm_crypt rfcomm bnep bluetooth decnet nfsd nfsv4 nfs_acl auth_rpcgss nfs qmi_wwan uvcvid...

Stefan Bader (smb) wrote :

[ 105.544107] CPU: 1 PID: 59 Comm: kworker/u16:1 Tainted: G OX 3.13.0-39-generic #66~precise1-Ubuntu

^Note that the kernel version in proposed is 3.13.0-40. Released likely by next Monday.

Launchpad Janitor (janitor) wrote :
Download full text (22.1 KiB)

This bug was fixed in the package linux - 3.13.0-40.69

---------------
linux (3.13.0-40.69) trusty; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - re-used previous tracking bug

  [ Upstream Kernel Changes ]

  * regmap: fix kernel hang on regmap_bulk_write with zero val_count.

linux (3.13.0-40.68) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1388943
  * SAUCE: DEP8 test to run our regression tests
    - LP: #1385330
  * SAUCE: The very first thing we should do when testing is make sure we
    are testing the correct kernel
    - LP: #1385330

  [ dann frazier ]

  * [Config] Disable CONFIG_IPMI_SI_PROBE_DEFAULTS on armhf and arm64
    - LP: #1388952

  [ Duc Dang ]

  * SAUCE: (no-up) [PCIE] APM X-Gene: Remove debug messages in MSI
    interrupt handler path.
    - LP: #1382244
  * SAUCE: (no-up) PCI: X-Gene: Fix max payload size and phantom function
    configuration
    - LP: #1386261

  [ McAulay, Alistair ]

  * SAUCE: drm/i915: Rework GPU reset sequence to match driver load & thaw
    - LP: #1384469

  [ Timo Aaltonen ]

  * SAUCE: i915_bdw: Fix cherry-pick typo
    - LP: #1384469

  [ Upstream Kernel Changes ]

  * Revert "mac80211: disable uAPSD if all ACs are under ACM"
    - LP: #1381234
  * Revert "iwlwifi: dvm: don't enable CTS to self"
    - LP: #1381234
  * Revert "lzo: properly check for overruns"
    - LP: #1387886
  * drm/i915: provide interface for audio driver to query cdclk
    - LP: #1381168
  * regulatory: add NUL to alpha2
    - LP: #1381234
  * percpu: fix pcpu_alloc_pages() failure path
    - LP: #1381234
  * percpu: perform tlb flush after pcpu_map_pages() failure
    - LP: #1381234
  * cgroup: reject cgroup names with '\n'
    - LP: #1381234
  * vfs: add d_is_dir()
    - LP: #1381234
  * CIFS: Fix directory rename error
    - LP: #1381234
  * usb: phy: twl4030-usb: Fix lost interrupts after ID pin goes down
    - LP: #1381234
  * rtlwifi: rtl8192cu: Add new ID
    - LP: #1381234
  * CIFS: Fix wrong restart readdir for SMB1
    - LP: #1381234
  * CIFS: Fix wrong filename length for SMB2
    - LP: #1381234
  * ahci: Add Device IDs for Intel 9 Series PCH
    - LP: #1381234
  * ata_piix: Add Device IDs for Intel 9 Series PCH
    - LP: #1381234
  * USB: zte_ev: fix removed PIDs
    - LP: #1381234
  * USB: ftdi_sio: add support for NOVITUS Bono E thermal printer
    - LP: #1381234
  * USB: sierra: avoid CDC class functions on "68A3" devices
    - LP: #1381234
  * USB: sierra: add 1199:68AA device ID
    - LP: #1381234
  * iommu/arm-smmu: fix programming of SMMU_CBn_TCR for stage 1
    - LP: #1381234
  * iommu/arm-smmu: remove pgtable_page_{c,d}tor()
    - LP: #1381234
  * usb: gadget: fusb300_udc.h: Fix typo in include guard
    - LP: #1381234
  * usb: phy: tegra: Avoid use of sizeof(void)
    - LP: #1381234
  * arm64: use irq_set_affinity with force=false when migrating irqs
    - LP: #1381234
  * block: Fix dev_t minor allocation lifetime
    - LP: #1381234
  * usb: dwc3: core: fix order of PM runtime calls
    - LP: #1381234
  * usb: dwc3: core: fix ordering for PHY suspend
    - LP: #1381234
  * usb: dwc3: omap: fix ordering for runtime pm calls
    - LP: #1381234
  * ...

Changed in linux (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers