Kernel Oops : Dentry still in use (1) [unmount of nfs4 0:1d]

[Matt Mossholder]

This is occurring under natty with kernel 2.6.38-8-generic on x86_64. Under normal "light use" the system performs fine. However, when accessing a NFS share fairly heavily (attempting to rebuild my Picasa DB), the system regularly crashes with the trace below.

Apr 24 10:02:43 mourneblade kernel: [ 334.354859] svc: 127.0.0.1, port=1022: unknown version (4 for prog 100003, nfsd)
Apr 24 10:53:40 mourneblade kernel: [ 3391.539431] BUG: Dentry ffff88022da53f00{i=20a9e,n=mythtv} still in use (1) [unmount of nfs4 0:1d]
Apr 24 10:53:40 mourneblade kernel: [ 3391.539472] ------------[ cut here ]------------
Apr 24 10:53:40 mourneblade kernel: [ 3391.539476] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
Apr 24 10:53:40 mourneblade kernel: [ 3391.539481] invalid opcode: 0000 [#1] SMP
Apr 24 10:53:40 mourneblade kernel: [ 3391.539487] last sysfs file: /sys/devices/virtual/bdi/0:28/uevent
Apr 24 10:53:40 mourneblade kernel: [ 3391.539491] CPU 2
Apr 24 10:53:40 mourneblade kernel: [ 3391.539493] Modules linked in: nls_utf8 udf crc_itu_t binfmt_misc ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_amd kvm nfsd parport_pc ppdev exportfs nfs lockd fscache nfs_acl auth_rpcgss autofs4 sunrpc snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul nvidia(P) snd_emu10k1 snd_ac97_codec ac97_bus snd_util_mem snd_usb_audio snd_pcm snd_page_alloc snd_hwdep snd_usbmidi_lib snd_seq_midi tuner_simple tuner_types snd_seq_midi_event psmouse wm8775 snd_rawmidi tuner snd_seq cx25840 uvcvideo snd_timer edac_core edac_mce_amd k10temp serio_raw snd_seq_device ivtv snd cx2341x i2c_algo_bit joydev v4l2_common videodev soundcore v4l2_compat_ioctl32 tveeprom emu10k1_gp gameport i2c_nforce2 lp parport usbhid hid uvesafb usb_storage uas r8169 pata_amd ahci libahci
Apr 24 10:53:40 mourneblade kernel: [ 3391.539590]
Apr 24 10:53:40 mourneblade kernel: [ 3391.539595] Pid: 5573, comm: umount.nfs Tainted: P 2.6.38-8-generic #42-Ubuntu BIOSTAR Group TF520 A2+/TF520 A2+
Apr 24 10:53:40 mourneblade kernel: [ 3391.539605] RIP: 0010:[<ffffffff81179945>] [<ffffffff81179945>] shrink_dcache_for_umount_subtree+0x285/0x290
Apr 24 10:53:40 mourneblade kernel: [ 3391.539620] RSP: 0018:ffff880215fe5d78 EFLAGS: 00010292
Apr 24 10:53:40 mourneblade kernel: [ 3391.539624] RAX: 000000000000006c RBX: ffff88022da53f00 RCX: 000000000000009f
Apr 24 10:53:40 mourneblade kernel: [ 3391.539628] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
Apr 24 10:53:40 mourneblade kernel: [ 3391.539633] RBP: ffff880215fe5db8 R08: 0000000000000033 R09: 000000000000f689
Apr 24 10:53:40 mourneblade kernel: [ 3391.539637] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88022da53f5c
Apr 24 10:53:40 mourneblade kernel: [ 3391.539641] R13: ffff88022da53f00 R14: ffff88022da53760 R15: ffff88022da53f5c
Apr 24 10:53:40 mourneblade kernel: [ 3391.539647] FS: 00007f1a682eb720(0000) GS:ffff8800bfd00000(0000) knlGS:00000000f74ed6c0
Apr 24 10:53:40 mourneblade kernel: [ 3391.539651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 10:53:40 mourneblade kernel: [ 3391.539655] CR2: 00007f1a67e480d0 CR3: 000000019dfd2000 CR4: 00000000000006e0
Apr 24 10:53:40 mourneblade kernel: [ 3391.539660] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 24 10:53:40 mourneblade kernel: [ 3391.539665] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 24 10:53:40 mourneblade kernel: [ 3391.539671] Process umount.nfs (pid: 5573, threadinfo ffff880215fe4000, task ffff880209c4adc0)
Apr 24 10:53:40 mourneblade kernel: [ 3391.539674] Stack:
Apr 24 10:53:40 mourneblade kernel: [ 3391.539677] ffff8802082d8258 ffff88023f80b2e8 0000000000000000 ffff8802082d8000
Apr 24 10:53:40 mourneblade kernel: [ 3391.539684] ffff88003796bf5c ffff88003796bf00 ffff88022da536c0 0000000000000000
Apr 24 10:53:40 mourneblade kernel: [ 3391.539690] ffff880215fe5de8 ffffffff8117c2b9 ffffffff8105f640 ffff8802082d8000
Apr 24 10:53:40 mourneblade kernel: [ 3391.539697] Call Trace:
Apr 24 10:53:40 mourneblade kernel: [ 3391.539704] [<ffffffff8117c2b9>] shrink_dcache_for_umount+0x69/0x90
Apr 24 10:53:40 mourneblade kernel: [ 3391.539711] [<ffffffff8105f640>] ? default_wake_function+0x0/0x20
Apr 24 10:53:40 mourneblade kernel: [ 3391.539720] [<ffffffff81166bfc>] generic_shutdown_super+0x2c/0x100
Apr 24 10:53:40 mourneblade kernel: [ 3391.539726] [<ffffffff81166d66>] kill_anon_super+0x16/0x60
Apr 24 10:53:40 mourneblade kernel: [ 3391.539750] [<ffffffffa0cfd2e9>] nfs4_kill_super+0x39/0x90 [nfs]
Apr 24 10:53:40 mourneblade kernel: [ 3391.539757] [<ffffffff811671d5>] deactivate_locked_super+0x45/0x70
Apr 24 10:53:40 mourneblade kernel: [ 3391.539763] [<ffffffff81167e5a>] deactivate_super+0x4a/0x70
Apr 24 10:53:40 mourneblade kernel: [ 3391.539770] [<ffffffff81183254>] mntput_no_expire+0xa4/0xf0
Apr 24 10:53:40 mourneblade kernel: [ 3391.539776] [<ffffffff811832bf>] mntput+0x1f/0x30
Apr 24 10:53:40 mourneblade kernel: [ 3391.539782] [<ffffffff81183b72>] release_mounts+0x72/0x90
Apr 24 10:53:40 mourneblade kernel: [ 3391.539787] [<ffffffff81184162>] do_umount+0xd2/0x1f0
Apr 24 10:53:40 mourneblade kernel: [ 3391.539793] [<ffffffff81184343>] sys_umount+0xc3/0xd0
Apr 24 10:53:40 mourneblade kernel: [ 3391.539799] [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b
Apr 24 10:53:40 mourneblade kernel: [ 3391.539803] Code: 8b 40 28 4c 8b 08 49 8b 45 30 48 85 c0 74 07 48 8b 90 a8 00 00 00 48 89 34 24 48 c7 c7 50 6c 7e 81 4c 89 ee 31 c0 e8 94 66 44 00 <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89
Apr 24 10:53:40 mourneblade kernel: [ 3391.539850] RIP [<ffffffff81179945>] shrink_dcache_for_umount_subtree+0x285/0x290
Apr 24 10:53:40 mourneblade kernel: [ 3391.539858] RSP <ffff880215fe5d78>
Apr 24 10:53:40 mourneblade kernel: [ 3391.539862] ---[ end trace f997c3602fafb327 ]---
Apr 24 10:53:40 mourneblade kernel: [ 3391.669327] BUG: Dentry ffff88020f9330c0{i=672b,n=/} still in use (5) [unmount of autofs autofs]
Apr 24 10:53:40 mourneblade kernel: [ 3391.669370] ------------[ cut here ]------------
Apr 24 10:53:40 mourneblade kernel: [ 3391.669375] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
Apr 24 10:53:40 mourneblade kernel: [ 3391.669380] invalid opcode: 0000 [#2] SMP
Apr 24 10:53:40 mourneblade kernel: [ 3391.669386] last sysfs file: /sys/devices/virtual/bdi/0:28/uevent
Apr 24 10:53:40 mourneblade kernel: [ 3391.669390] CPU 1
Apr 24 10:53:40 mourneblade kernel: [ 3391.669392] Modules linked in: nls_utf8 udf crc_itu_t binfmt_misc ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_amd kvm nfsd parport_pc ppdev exportfs nfs lockd fscache nfs_acl auth_rpcgss autofs4 sunrpc snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul nvidia(P) snd_emu10k1 snd_ac97_codec ac97_bus snd_util_mem snd_usb_audio snd_pcm snd_page_alloc snd_hwdep snd_usbmidi_lib snd_seq_midi tuner_simple tuner_types snd_seq_midi_event psmouse wm8775 snd_rawmidi tuner snd_seq cx25840 uvcvideo snd_timer edac_core edac_mce_amd k10temp serio_raw snd_seq_device ivtv snd cx2341x i2c_algo_bit joydev v4l2_common videodev soundcore v4l2_compat_ioctl32 tveeprom emu10k1_gp gameport i2c_nforce2 lp parport usbhid hid uvesafb usb_storage uas r8169 pata_amd ahci libahci
Apr 24 10:53:40 mourneblade kernel: [ 3391.669491]
Apr 24 10:53:40 mourneblade kernel: [ 3391.669496] Pid: 5582, comm: automount Tainted: P D 2.6.38-8-generic #42-Ubuntu BIOSTAR Group TF520 A2+/TF520 A2+
Apr 24 10:53:40 mourneblade kernel: [ 3391.669506] RIP: 0010:[<ffffffff81179945>] [<ffffffff81179945>] shrink_dcache_for_umount_subtree+0x285/0x290
Apr 24 10:53:40 mourneblade kernel: [ 3391.669521] RSP: 0018:ffff8801a3a27de8 EFLAGS: 00010296
Apr 24 10:53:40 mourneblade kernel: [ 3391.669525] RAX: 000000000000006a RBX: ffff88020f93311c RCX: 000000000000009f
Apr 24 10:53:40 mourneblade kernel: [ 3391.669529] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000246
Apr 24 10:53:40 mourneblade kernel: [ 3391.669533] RBP: ffff8801a3a27e28 R08: 0000000000000033 R09: 00000000000106f1
Apr 24 10:53:40 mourneblade kernel: [ 3391.669538] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88020f93311c
Apr 24 10:53:40 mourneblade kernel: [ 3391.669542] R13: ffff88020f9330c0 R14: ffff88020f933160 R15: 00007fd8bc001e40
Apr 24 10:53:40 mourneblade kernel: [ 3391.669547] FS: 00007fd8b315c700(0000) GS:ffff8800bfc80000(0000) knlGS:00000000f694f880
Apr 24 10:53:40 mourneblade kernel: [ 3391.669552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 10:53:40 mourneblade kernel: [ 3391.669555] CR2: 00007fd8b3155a08 CR3: 0000000199fd2000 CR4: 00000000000006e0
Apr 24 10:53:40 mourneblade kernel: [ 3391.669560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 24 10:53:40 mourneblade kernel: [ 3391.669564] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 24 10:53:40 mourneblade kernel: [ 3391.669569] Process automount (pid: 5582, threadinfo ffff8801a3a26000, task ffff8802136196e0)
Apr 24 10:53:40 mourneblade kernel: [ 3391.669572] Stack:
Apr 24 10:53:40 mourneblade kernel: [ 3391.669575] ffff88019c747e58 ffffffff81177156 0000000000000000 ffff88019c747c00
Apr 24 10:53:40 mourneblade kernel: [ 3391.669581] ffff88020f93311c ffff88020f9330c0 00007fd8bbf9ce30 00007fd8bc001e40
Apr 24 10:53:40 mourneblade kernel: [ 3391.669588] ffff8801a3a27e58 ffffffff8117c2a1 ffff8801a3a27e48 ffff88019c747c00
Apr 24 10:53:40 mourneblade kernel: [ 3391.669594] Call Trace:
Apr 24 10:53:40 mourneblade kernel: [ 3391.669603] [<ffffffff81177156>] ? pollwake+0x56/0x60
Apr 24 10:53:40 mourneblade kernel: [ 3391.669609] [<ffffffff8117c2a1>] shrink_dcache_for_umount+0x51/0x90
Apr 24 10:53:40 mourneblade kernel: [ 3391.669618] [<ffffffff81166bfc>] generic_shutdown_super+0x2c/0x100
Apr 24 10:53:40 mourneblade kernel: [ 3391.669624] [<ffffffff81166d66>] kill_anon_super+0x16/0x60
Apr 24 10:53:40 mourneblade kernel: [ 3391.669629] [<ffffffff81166dd7>] kill_litter_super+0x27/0x30
Apr 24 10:53:40 mourneblade kernel: [ 3391.669639] [<ffffffffa0ca64d8>] autofs4_kill_sb+0x48/0x60 [autofs4]
Apr 24 10:53:40 mourneblade kernel: [ 3391.669646] [<ffffffff811671d5>] deactivate_locked_super+0x45/0x70
Apr 24 10:53:40 mourneblade kernel: [ 3391.669652] [<ffffffff81167e5a>] deactivate_super+0x4a/0x70
Apr 24 10:53:40 mourneblade kernel: [ 3391.669659] [<ffffffff81183254>] mntput_no_expire+0xa4/0xf0
Apr 24 10:53:40 mourneblade kernel: [ 3391.669664] [<ffffffff811842e0>] sys_umount+0x60/0xd0
Apr 24 10:53:40 mourneblade kernel: [ 3391.669671] [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b
Apr 24 10:53:40 mourneblade kernel: [ 3391.669674] Code: 8b 40 28 4c 8b 08 49 8b 45 30 48 85 c0 74 07 48 8b 90 a8 00 00 00 48 89 34 24 48 c7 c7 50 6c 7e 81 4c 89 ee 31 c0 e8 94 66 44 00 <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89
Apr 24 10:53:40 mourneblade kernel: [ 3391.669721] RIP [<ffffffff81179945>] shrink_dcache_for_umount_subtree+0x285/0x290
Apr 24 10:53:40 mourneblade kernel: [ 3391.669729] RSP <ffff8801a3a27de8>
Apr 24 10:53:40 mourneblade kernel: [ 3391.669742] ---[ end trace f997c3602fafb328 ]---

uname -a : Linux mourneblade.mossholder.com 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
version signature: Ubuntu 2.6.38-8.42-generic 2.6.38.2

[Matt Mossholder (matt-mossholder)]

Also happens under 2.6.38-9 from kernel-ppa/pre-proposed

Linux mourneblade.mossholder.com 2.6.38-9-generic #43~pre201104230901-Ubuntu SMP Sat Apr 23 09:16:31 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

[Jens Maus (jens.maus)]

After having tried to switch all remote NFS share mounts to use NFS3 instead of NFS4 as I thought that might have been the reason for the crash. However, now that two of our servers showed up with the exact same problem again, I can report that this seems to even happen with nfs3 shares. The traceback then looks like:

-- cut here --
[537189.380178] BUG: Dentry ffff88084c5c5b00{i=babc05,n=/} still in use (1) [unmount of autofs autofs]
[537189.380256] ------------[ cut here ]------------
[537189.380281] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
[537189.380309] invalid opcode: 0000 [#1] SMP
[537189.380335] last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state
[537189.380378] CPU 8
[537189.380382] Modules linked in: binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs parport_pc ppdev autofs4 mptctl k8temp lm75 lm87 hwmon_vid ipmi_watchdog ipmi_poweroff nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc radeon psmouse serio_raw ghes hed ttm amd64_edac_mod drm_kms_helper edac_core k10temp edac_mce_amd joydev drm nv_tco i2c_algo_bit i2c_nforce2 ipmi_si ipmi_devintf ipmi_msghandler lp parport usbhid hid usb_storage mptsas mptscsih mptbase qla2xxx e1000e scsi_transport_fc scsi_transport_sas scsi_tgt
[537189.380665]
[537189.380685] Pid: 12686, comm: automount Not tainted 2.6.38-8-server #42-Ubuntu SUN MICROSYSTEMS SUN BLADE X8440 SERVER MODULE/Sun Blade X8440 Server Module
[537189.380747] RIP: 0010:[<ffffffff81179be5>] [<ffffffff81179be5>] shrink_dcache_for_umount_subtree+0x285/0x290
[537189.380800] RSP: 0018:ffff880676e63de8 EFLAGS: 00010296
[537189.380825] RAX: 000000000000006d RBX: ffff88084c5c5b5c RCX: 00000000ffffffff
[537189.380867] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000246
[537189.380909] RBP: ffff880676e63e28 R08: 0000000000000000 R09: ffffffff816423e0
[537189.380951] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88084c5c5b5c
[537189.380992] R13: ffff88084c5c5b00 R14: ffff88084c5c5ba0 R15: 00007f739c43eb60
[537189.381035] FS: 00007f7397e6c700(0000) GS:ffff88068fc00000(0000) knlGS:00000000f68ff700
[537189.381078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[537189.381104] CR2: 00007f2e33b827c0 CR3: 0000000676637000 CR4: 00000000000006e0
[537189.381146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[537189.381188] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[537189.381230] Process automount (pid: 12686, threadinfo ffff880676e62000, task ffff88047af944a0)
[537189.381275] Stack:
[537189.381294] ffff88087c758658 ffff88067cee6000 ffff880676e63e08 ffff88087c758400
[537189.381340] ffff88084c5c5b5c ffff88084c5c5b00 00007f739c43eea0 00007f739c43eb60
[537189.381386] ffff880676e63e58 ffffffff8117c521 ffff880676e63e48 ffff88087c758400
[537189.381432] Call Trace:
[537189.381456] [<ffffffff8117c521>] shrink_dcache_for_umount+0x51/0x90
[537189.381487] [<ffffffff811670ac>] generic_shutdown_super+0x2c/0x100
[537189.381515] [<ffffffff81167216>] kill_anon_super+0x16/0x60
[537189.381542] [<ffffffff81167287>] kill_litter_super+0x27/0x30
[537189.381572] [<ffffffffa046f4d8>] autofs4_kill_sb+...

[Michael Pacey (michael-wd21)]

I just got this on my laptop:

[917043.074390] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
[917043.074468] invalid opcode: 0000 [#1] SMP
[917043.074521] last sysfs file: /sys/devices/platform/dock.0/flags
[917043.074593] CPU 0
[917043.074618] Modules linked in: binfmt_misc vboxnetadp vboxnetflt vboxdrv nfs lockd fscache nfs_acl auth_rpcgss sunrpc autofs4 nls_iso8859_1 nls_cp437 vfat fat usb_storage uas ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp snd_seq_dummy rfcomm sco bnep l2cap zaurus cdc_wdm cdc_acm cdc_ether usbnet parport_pc ppdev btusb bluetooth snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi snd_seq_midi arc4 usblp snd_rawmidi snd_seq_midi_event iwlagn snd_seq snd_timer snd_seq_device iwlcore snd tpm_tis uvcvideo mac80211 videodev psmouse nvram serio_raw tpm lp tpm_bios soundcore snd_page_alloc v4l2_compat_ioctl32 parport cfg80211 joydev sha256_generic cryptd aes_x86_64 aes_generic dm_crypt usbhid hid i915 drm_kms_helper ahci drm e1000e libahci i2c_algo_bit video
[917043.075920]
[917043.075942] Pid: 30720, comm: umount.nfs Not tainted 2.6.38-8-generic #42-Ubuntu LENOVO 7459L88/7459L88
[917043.076067] RIP: 0010:[<ffffffff81179945>] [<ffffffff81179945>] shrink_dcache_for_umount_subtree+0x285/0x290
[917043.079565] RSP: 0018:ffff88000bcc3d78 EFLAGS: 00010292
[917043.083050] RAX: 0000000000000066 RBX: ffff88009ea0b540 RCX: 000000000000009f
[917043.083357] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
[917043.083357] RBP: ffff88000bcc3db8 R08: 0000000000000033 R09: 000000000005e51e
[917043.083357] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88009ea0b59c
[917043.083357] R13: ffff88009ea0b540 R14: ffff880015626a60 R15: ffff88009ea0b59c
[917043.083357] FS: 00007fd1c1d9a720(0000) GS:ffff8800bd200000(0000) knlGS:0000000000000000
[917043.083357] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[917043.083357] CR2: 00007fb7cc183000 CR3: 00000001002bf000 CR4: 00000000000426f0
[917043.083357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[917043.083357] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[917043.083357] Process umount.nfs (pid: 30720, threadinfo ffff88000bcc2000, task ffff88012ed35b80)
[917043.083357] Stack:
[917043.083357] ffff88012dac4658 ffffffff8100a777 ffff8801179ec4a0 ffff88012dac4400
[917043.083357] ffff88009ea0ba1c ffff88009ea0b9c0 ffff880106e9e180 0000000000000000
[917043.083357] ffff88000bcc3de8 ffffffff8117c2b9 ffff8800bd213d00 ffff88012dac4400
[917043.083357] Call Trace:
[917043.083357] [<ffffffff8100a777>] ? __switch_to+0x157/0x2f0
[917043.083357] [<ffffffff8117c2b9>] shrink_dcache_for_umount+0x69/0x90
[917043.083357] [<ffffffff81166bfc>] generic_shutdown_super+0x2c/0x100
[917043.083357] [<ffffffff81166d66>] kill_anon_super+0x16/0x60
[917043.083357] [<ffffffffa05ce365>] nfs_kill_super+0x25/0x40 [nfs]
[917043.083357] [<ffffffff811671d5>] deactivate_locked_super+0x45/0x70
[917043.083357] [<ffffffff81167e5a>] deactivate_super+0x4a/0x7...

[Jeff (jeff-redhat-bugs)]

I've also found that I can manually unmount the filesystem and trigger the bug, so I don't need to rely on autofs expiring the mount (that's good since that seemed to be broken in some early pre 2.6.38-rc1 versions:

Since I have a fairly reliable reproducer, I decided to try bisecting this down. The log so far:

$ git bisect log
git bisect start '--' 'fs/'
# bad: [1bae4ce27c9c90344f23c65ea6966c50ffeae2f5] Linux 2.6.38-rc2
git bisect bad 1bae4ce27c9c90344f23c65ea6966c50ffeae2f5
# good: [3c0eee3fe6a3a1c745379547c7e7c904aa64f6d5] Linux 2.6.37
git bisect good 3c0eee3fe6a3a1c745379547c7e7c904aa64f6d5
# good: [b9d919a4ac6cf031b8e065f82ad8f1b0c9ed74b1] Merge branch 'nfs-for-2.6.38' of git://git.linux-nfs.org/projects/trondmy/nfs-2.6
git bisect good b9d919a4ac6cf031b8e065f82ad8f1b0c9ed74b1
# good: [b9d919a4ac6cf031b8e065f82ad8f1b0c9ed74b1] Merge branch 'nfs-for-2.6.38' of git://git.linux-nfs.org/projects/trondmy/nfs-2.6
git bisect good b9d919a4ac6cf031b8e065f82ad8f1b0c9ed74b1
# good: [6ab82196492a0b6968a654a06aae923b28afef0d] Merge branch 'for-linus' of git://git.kernel.dk/linux-2.6-block
git bisect good 6ab82196492a0b6968a654a06aae923b28afef0d
# bad: [9e8a462a0141b12e22c4a2f0c12e0542770401f0] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ecryptfs/ecryptfs-2.6
git bisect bad 9e8a462a0141b12e22c4a2f0c12e0542770401f0

...looks like I have about 5 more iterations to go.

[Jeff (jeff-redhat-bugs)]

Created attachment 501391
git bisect log and slightly different oops

Getting closer. Here's the log so far.

Note that with the last revision I couldn't reproduce the exact same bug as autofs failed to mount the fs. Suspecting that automountd was hoses, I did a service autofs restart and then it panicked with a very similar backtrace. So, I'm marking this bad on the assumption that it's a related problem...

[Jeff (jeff-redhat-bugs)]

And the winner (loser?) of the bisect is...

commit 36d43a43761b004ad1879ac21471d8fc5f3157ec
Author: David Howells <email address hidden>
Date: Fri Jan 14 18:45:42 2011 +0000

    NFS: Use d_automount() rather than abusing follow_link()

    Make NFS use the new d_automount() dentry operation rather than abusing
    follow_link() on directories.

    Signed-off-by: David Howells <email address hidden>
    Acked-by: Trond Myklebust <email address hidden>
    Acked-by: Ian Kent <email address hidden>
    Signed-off-by: Al Viro <email address hidden>

...which unfortunately doesn't narrow things down that much :-/

[Jeff (jeff-redhat-bugs)]

Created attachment 501403
final git bisect log

Here's the final git bisect log in case anyone is interested. Anyone have thoughts as to other possible avenues for investigation?

[Ian (ian-redhat-bugs)]

(In reply to comment #11)
> Created attachment 501306 [details]
> info on root dentry, post-oops
>
> Poking around with crash gives me this info on the root dentry, post-oops. The
> d_flags are:
>
> d_flags = 0xc004 = DCACHE_DISCONNECTED|DCACHE_OP_REVALIDATE|DCACHE_OP_DELETE
>
> ...nothing terribly exciting there. d_parent points to itself (as expected),
> and all of the list_heads seem to be empty.

It's about time for a complete stab in the dark!

It's odd that the dentry above hasn't been __d_drop()ed ....

Which call of shrink_dcache_for_umount_subtree() does this oops on,
the first call or the second in shrink_dcache_for_umount()? The second
call is the one that walks the anonymous dentry list so that should be
the one. And the first thing shrink_dcache_for_umount_subtree() does
is __d_drop() the dentry. If it is the first call then how does the
anonymous dentry come to be under s_root?

And why does shrink_dcache_for_umount() decrement the count for the
s_root dentry but not for dentrys found on the sb->s_anon list, David?

[Jeff (jeff-redhat-bugs)]

FWIW, I did some more testing yesterday. I did the reproducer on a kernel both with and without commit 36d43a437, and used crash to look at the d_count after triggering the mount and before the unmount. Those counts were the same in both cases -- the mnt_root had a d_count of 2, and s_root a d_count of 1.

Perhaps I should do it again and have a look at the d_flags, or just get the entire dentry structs in both cases so we can compare them.

[Ian (ian-redhat-bugs)]

(In reply to comment #17)
> FWIW, I did some more testing yesterday. I did the reproducer on a kernel both
> with and without commit 36d43a437, and used crash to look at the d_count after
> triggering the mount and before the unmount. Those counts were the same in both
> cases -- the mnt_root had a d_count of 2, and s_root a d_count of 1.

That makes sense, since s_root won't be used in path lookups
and then mnt_root will have one for when it was d_alloc()ed and
one from the umount code (I think).

>
> Perhaps I should do it again and have a look at the d_flags, or just get the
> entire dentry structs in both cases so we can compare them.

I think we are most interested in the mnt_root dentry because
it is the one assigned to the vfsmount struct and so will be
the one involved in path walks. I think s_root is tucked away
in the super block and is basically inert in the NFS case.

What I don't get is whether the oops occurs from the fist call
to shrink_dcache_for_umount_subtree() or not. If it does then
I'd be wondering how the anonymous dentry got into the tree
under s_root at all (although that itself deserves more
looking around in the code).

The other thing that bugs me is in __d_drop() where we have:

if dentry is hashed then
  if dentry is disconnected
    remove from appropriate list
  else
    remove from other appropriate list
    maybe block on rcuwalk barrier
  done
done

So we don't boot active rcu-walk walks when dropping the
anonymous dentry, if it even gets to the drop that is.

[Jeff (jeff-redhat-bugs)]

Created attachment 502245
pre and post bad patch dentry info

Here is the dentry info for the vfsmount->mnt_root and super_block->s_root, both without the "bad" patch (pre) and with the patch (post). I forgot to "set hex" beforehand so a lot of the interesting fields (e.g. d_flags) are in decimal.

The d_flags seem to be identical in both cases as well. I don't see much in the way of discrepancies...

Also, I should note that the busy dentry in the oops message is the vfsmount->mnt_root, and *not* the super_block->s_root.

[David (david-redhat-bugs)]

I'd be very surprised if you saw the bug appearing in relation to the dentry attached to s_root. NFS's s_root dentry is a placeholder to stop the VFS blowing up, no more. It isn't actually used for anything.

As NFS may share superblocks between several disjoint mounts from the same remote filesystem, it keeps its roots in the s_anon list instead. Should the root of a mount be detected in the tree of another mount, then the two trees are spliced together at that point by d_materialise_unique(). The resulting tree will still have an anonymous root.

It is done this way as we may not have access to the actual root directory of the remote filesystem, and so may not be able to put that into our tree.

Thus the root of the mount is attached directly to the vfsmount struct without going anywhere near the core tree in the superblock.

[Jeff (jeff-redhat-bugs)]

I was also able to reproduce this using a direct automount map as well (Ian asked if it was possible), so direct vs. indirect map doesn't seem to make any difference.

[Jeff (jeff-redhat-bugs)]

Also, just for giggles I tried Ian's suggestion to make __d_drop always call dentry_rcuwalk_barrier() and it didn't prevent the oops on unmount.

[Jeff (jeff-redhat-bugs)]

Here's what I don't quite get...

I've bisected the problem down to the patch that switched NFS over to use d_automount. That doesn't really affect unmounting afaict and should really only be affecting things when the automount is triggered.

So, it seems likely that that change is causing some sort of difference in the mounted filesystem. The question is what? I'll be happy to collect info from kernels with and without that patch to try and discern where the difference is -- any thoughts about what I should be collecting info on?

[Ian (ian-redhat-bugs)]

(In reply to comment #23)
> Here's what I don't quite get...
>
> I've bisected the problem down to the patch that switched NFS over to use
> d_automount. That doesn't really affect unmounting afaict and should really
> only be affecting things when the automount is triggered.
>
> So, it seems likely that that change is causing some sort of difference in the
> mounted filesystem. The question is what? I'll be happy to collect info from
> kernels with and without that patch to try and discern where the difference is
> -- any thoughts about what I should be collecting info on?

We should have also checked /proc/mounts in each case.

[Jeff (jeff-redhat-bugs)]

(In reply to comment #24)
> We should have also checked /proc/mounts in each case.

I got some clarification from Ian and he was just concerned that it might have been mounted more than once. I verified that it wasn't.

[Jonathan Sambrook (jonathan-hmmn)]

[ 661.863116] BUG: Dentry f6bcb000{i=145066,n=} still in use (1) [unmount of nfs4 0:1c]
[ 661.863138] ------------[ cut here ]------------
[ 661.863173] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
[ 661.863209] invalid opcode: 0000 [#1] SMP
[ 661.863238] last sysfs file: /sys/devices/pci0000:00/0000:00:19.0/net/eth0/statistics/collisions
[ 661.863289] Modules linked in: nls_utf8 udf snd_hrtimer binfmt_misc microcode parport_pc ppdev autofs4 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_usb_audio snd_hwdep snd_usbmidi_lib snd_pcm coretemp snd_seq_midi snd_seq_midi_event snd_rawmidi lm85 hwmon_vid i915 uvcvideo drm_kms_helper snd_seq i2c_i801 videodev drm snd_timer lp snd_seq_device psmouse joydev snd i2c_algo_bit parport soundcore snd_page_alloc serio_raw video usbhid hid raid10 raid456 async_raid6_recov async_pq firewire_ohci usb_storage firewire_core uas e1000e crc_itu_t raid6_pq async_xor async_memcpy async_tx raid1 raid0 multipath linear dm_raid45 xor
[ 661.863787]
[ 661.863799] Pid: 3434, comm: umount.nfs Tainted: G W 2.6.38-8-generic #42-Ubuntu /DG45ID
[ 661.863868] EIP: 0060:[<c1139629>] EFLAGS: 00010282 CPU: 0
[ 661.863904] EIP is at shrink_dcache_for_umount_subtree+0x1c9/0x1d0
[ 661.863938] EAX: 0000005f EBX: f8e66a09 ECX: c1739ba8 EDX: 00000000
[ 661.863973] ESI: f6bcb000 EDI: f6c5375c EBP: e5a75eb0 ESP: e5a75e7c
[ 661.864007] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 661.864037] Process umount.nfs (pid: 3434, ti=e5a74000 task=e5a68ca0 task.ti=e5a74000)
[ 661.864080] Stack:
[ 661.864094] c16918d0 f6bcb000 00145066 f6bcb024 00000001 f8e66a09 f6c5375c f6d3e500
[ 661.864155] f6bcb4cc f6bcb024 f6c53600 f6bcb800 f6bcb84c e5a75ec4 c113b690 f6c53600
[ 661.864214] e6e11800 f8e64ae0 e5a75ee0 c1128be7 e5a75eec f8e60206 0000001c e6e11800
[ 661.864272] Call Trace:
[ 661.864290] [<c113b690>] shrink_dcache_for_umount+0x50/0x60
[ 661.864325] [<c1128be7>] generic_shutdown_super+0x27/0xe0
[ 661.864369] [<f8e60206>] ? nfs_super_return_all_delegations+0x66/0x80 [nfs]
[ 661.864409] [<c1128d31>] kill_anon_super+0x11/0x50
[ 661.864444] [<f8e41390>] nfs4_kill_super+0x30/0x80 [nfs]
[ 661.864476] [<c11290cd>] deactivate_locked_super+0x3d/0x60
[ 661.864508] [<c1129b98>] deactivate_super+0x48/0x70
[ 661.864537] [<c1141400>] mntput_no_expire+0x90/0xd0
[ 661.865815] [<c1141458>] mntput+0x18/0x30
[ 661.865815] [<c1141b9e>] release_mounts+0x5e/0x80
[ 661.865815] [<c1142053>] do_umount+0xb3/0x1b0
[ 661.865815] [<c121b398>] ? security_capable+0x28/0x30
[ 661.865815] [<c11421fe>] sys_umount+0xae/0xc0
[ 661.865815] [<c114222e>] sys_oldumount+0x1e/0x20
[ 661.865815] [<c1509bf4>] syscall_call+0x7/0xb
[ 661.865815] Code: 03 8b 51 60 89 44 24 10 8b 45 f0 89 7c 24 18 89 5c 24 14 89 54 24 08 89 74 24 04 89 44 24 0c c7 04 24 d0 18 69 c1 e8 26 db 3c 00 <0f> 0b 0f 0b 8d 76 00 55 89 e5 83 ec 08 89 1c 24 89 74 24 04 3e
[ 661.865815] EIP: [<c1139629>] shrink_dcache_for_umount_subtree+0x1c9/0x1d0 SS:ESP 0068:e5a75e7c
[ 661.902862] ---[ end trace 174f5c548ffb60b5 ]---

[Jeff (jeff-redhat-bugs)]

I decided to strace claws-mail. The first thing it does to trigger an automount is an open on the directory:

18672 13:17:53.843597 open("/misc/salusa/scratch", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC <unfinished ...>

...while that crunching away, a different thread does an lstat of a file in that directory (the one it intends to save):

18682 13:17:53.975947 lstat("/misc/salusa/scratch/[CapitalCityCigars] And so it begins!!", <unfinished ...>

...that comes back first:

18682 13:17:54.301130 <... lstat resumed> 0x7f7baea2a910) = -1 ENOENT (No such file or directory) <0.325145>

...then it does an lstat of the parent:

18682 13:17:54.303366 lstat("/misc/salusa/scratch", {st_dev=makedev(0, 42), st_ino=2, st_mode=S_IFDIR|S_ISVTX|0777, st_nlink=8, st_uid=99, st_gid=99, st_blksize=524288, st_blocks=8, st_size=4096,
 st_atime=2011/06/07-12:13:09, st_mtime=2011/06/07-12:13:09, st_ctime=2011/06/07-12:13:09}) = 0 <0.000390>

...a little later, the opendir returns:

18672 13:17:54.340699 <... open resumed> ) = 23 <0.497070>

...then it sets an inotify watch (probably via some gtk open dialog goop):

18668 13:17:54.341626 inotify_add_watch(21, "/misc/salusa/scratch", IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_UNMOUNT|IN_ONLYDIR) = 3 <0.000043>

...and then starts a readdir:

18682 13:17:54.342309 getdents(23, <unfinished ...>

...that comes back (I won't paste it as it's pretty big), and then it starts lstat'ing each dentry:

18682 13:17:54.347897 lstat("/misc/salusa/scratch/testfile", {st_dev=makedev(0, 42), st_ino=13, st_mode=S_IFREG|0644, st_nlink=1, st_uid=99, st_gid=99, st_blksize=524288, st_blocks=2097160, st_si
ze=1073741824, st_atime=2011/05/19-10:56:52, st_mtime=2011/05/19-10:59:30, st_ctime=2011/05/19-10:59:30}) = 0 <0.000033>

...most of this is pretty straightforward stuff, but I wonder if the inotify_add_watch is important somehow as that's somewhat unusual from command-line apps.

[Jeff (jeff-redhat-bugs)]

Created attachment 504121
debug patch -- instrument nfs_d_automount

Taking Ian's suggestion on IRC into account, I decided to instrument nfs_d_automount. I turned all of the dprintk's into printk's, had them print current->pid. I also added a printk of the string representation of the struct path.

It looks like we do have 2 concurrent nfs_d_automount requests going on here. They both return different vfsmounts, though. The second vfsmount does not show up in the mount table (as shown by crash's mount command). Here's the snippet of dmesg when I did the reproducer to trigger a mount. Unfortunately, the dump_stack() got a little muddled, but it gives you an idea of what's going on...

[ 214.444300] 1670: --> nfs_d_automount()
[ 214.445701] Pid: 1670, comm: claws-mail Not tainted 2.6.39-1.fc16.x86_64.debug #1
[ 214.448353] Call Trace:
[ 214.449301] [<ffffffffa01dc897>] nfs_d_automount+0x60/0x579 [nfs]
[ 214.451247] 1677: --> nfs_d_automount()
[ 214.451251] Pid: 1677, comm: claws-mail Not tainted 2.6.39-1.fc16.x86_64.debug #1
[ 214.451253] Call Trace:
[ 214.451279] [<ffffffffa01dc897>] nfs_d_automount+0x60/0x579 [nfs]
[ 214.451334] [<ffffffff814b9a15>] ? _raw_spin_unlock+0x28/0x2c
[ 214.451362] [<ffffffff811455b4>] ? dput+0xdc/0xfc
[ 214.451373] [<ffffffffa01cc347>] ? nfs_lookup_revalidate+0x227/0x366 [nfs]
[ 214.451377] [<ffffffff8113c463>] follow_managed+0x12a/0x1ce
[ 214.451381] [<ffffffff8113d7b8>] walk_component+0x22c/0x335
[ 214.451402] [<ffffffff81207715>] ? security_inode_exec_permission+0x25/0x27
[ 214.451406] [<ffffffff8113da60>] link_path_walk+0x19f/0x439
[ 214.451409] [<ffffffff8113de1e>] path_lookupat+0x5a/0x2e1
[ 214.451432] [<ffffffff81101170>] ? might_fault+0xa5/0xac
[ 214.451435] [<ffffffff81101127>] ? might_fault+0x5c/0xac
[ 214.451438] [<ffffffff8113c8ee>] ? getname_flags+0x31/0x1ca
[ 214.451441] [<ffffffff8113ef4f>] do_path_lookup+0x28/0x96
[ 214.451444] [<ffffffff8113f374>] user_path_at+0x59/0x96
[ 214.451462] [<ffffffff810774b6>] ? up_read+0x28/0x2c
[ 214.451471] [<ffffffff814bcf30>] ? do_page_fault+0x360/0x3b4
[ 214.451475] [<ffffffff8113703f>] vfs_fstatat+0x44/0x6e
[ 214.451478] [<ffffffff81101127>] ? might_fault+0x5c/0xac
[ 214.451481] [<ffffffff81137087>] vfs_lstat+0x1e/0x20
[ 214.451484] [<ffffffff811371d6>] sys_newlstat+0x1a/0x33
[ 214.451493] [<ffffffff810870f3>] ? trace_hardirqs_on_caller+0x10b/0x12f
[ 214.451502] [<ffffffff810a9e5f>] ? audit_syscall_entry+0x11c/0x148
[ 214.451523] [<ffffffff81256e1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 214.451526] [<ffffffff814c05c2>] system_call_fastpath+0x16/0x1b
[ 214.451532] nfs_d_automount(1677): enter
[ 214.451548] 1677: dentry=ffff88001005c540 path=/misc/salusa/scratch
[ 214.471100] SELinux: initialized (dev 0:2b, type nfs4), uses genfs_contexts
[ 214.471587] nfs_d_automount(1677): done, success
[ 214.471607] 1677: <-- nfs_d_automount() = ffff88001194d140
[ 214.511636] [<ffffffff81078a0d>] ? sched_clock_local+0x12/0x75
[ 214.514001] [<ffffffff81083764>] ? trace_hardirqs_off+0xd/0xf
[ 214.515589] [<ffffffff81078bda>] ? local_clock+0x36/0x4d
[ 214.517049] [<ffffffff81083e91>] ? lock_release_holdtime.part....

[Jeff (jeff-redhat-bugs)]

I've limited the reproducer down a little...

I can reproduce this as long as I use a gtk/gnome dialog to trigger a mount. I don't actually need to save anything in the directory. I can just pull up a "browse" dialog, type in "/misc/salusa/scratch/" and hit return. That opens the directory and triggers the mount.

[Jeff (jeff-redhat-bugs)]

I hacked up a quick patch to make nfs_d_automount return NULL after the first automount attempt. The first time nfs_d_automount is called, it works normally and the second time it just returns NULL (indicating that . This seems to work around the bug, lending some weight to the idea that the concurrent d_automount activity is somehow causing the issue.

[Jeff (jeff-redhat-bugs)]

A little more printk debugging:

[ 450.447517] 1507: --> nfs_d_automount()
[ 450.449513] nfs_d_automount(1507): enter
[ 450.452785] 1512: --> nfs_d_automount()
[ 450.455700] 1507: dentry=ffff880012520a80 path=/misc/salusa/scratch
[ 450.471407] SELinux: initialized (dev 0:2b, type nfs4), uses genfs_contexts
[ 450.471963] nfs_d_automount(1507): done, success
[ 450.474253] 1507: <-- nfs_d_automount() = ffff880022765640
[ 450.475581] nfs_d_automount(1512): enter
[ 450.476725] 1512: dentry=ffff880012520a80 path=/misc/salusa/scratch
[ 450.480416] follow_automount(1507): finish_automount returned 0
[ 450.490699] nfs_d_automount(1512): done, success
[ 450.493240] 1512: <-- nfs_d_automount() = ffff8800114c2000
[ 450.496763] follow_automount(1512): finish_automount returned -16
[ 472.333182] BUG: Dentry ffff880012520930{i=2,n=} still in use (1) [unmount of nfs4 0:2a]

...so follow_automount is returning -EBUSY. It seems somewhat likely that the cleanup of the second vfsmount is not completely correct in this situation.

[Jeff (jeff-redhat-bugs)]

...and a little more -- show the mnt_count. It's 2 for both mounts prior to calling finish_automount:

[ 486.108984] SELinux: initialized (dev 0:2a, type nfs4), uses genfs_contexts
[ 486.117197] mount.nfs4 used greatest stack depth: 2304 bytes left
[ 486.127215] 1494: --> nfs_d_automount()
[ 486.128641] nfs_d_automount(1494): enter
[ 486.130170] 1494: dentry=ffff8800125dc3f0 path=/misc/salusa/scratch
[ 486.139573] 1499: --> nfs_d_automount()
[ 486.143287] SELinux: initialized (dev 0:2b, type nfs4), uses genfs_contexts
[ 486.143841] nfs_d_automount(1494): done, success
[ 486.145655] 1494: <-- nfs_d_automount() = ffff880011941140
[ 486.147213] follow_automount(1494): mnt->mnt_count = 2
[ 486.148720] nfs_d_automount(1499): enter
[ 486.149828] 1499: dentry=ffff8800125dc3f0 path=/misc/salusa/scratch
[ 486.154517] follow_automount(1494): finish_automount returned 0
[ 486.168236] nfs_d_automount(1499): done, success
[ 486.170616] 1499: <-- nfs_d_automount() = ffff880011941280
[ 486.174350] follow_automount(1499): mnt->mnt_count = 2
[ 486.180294] follow_automount(1499): finish_automount returned -16

[Ian (ian-redhat-bugs)]

I can't for the life of me. find where the extra reference is being
picked up.

I agree that the fsnotify system probably has a lot to do with this
problem but I can't see why. The most obvious way to take fsnotify
out of this is to not go through the vfsmount setup for a mount
that essentially already exists. That avoids mntput() calling
fsnotify_unmount_inodes(). Alternately, we need to consult someone
with in-depth knowledge of the fsnotify system. Any ideas who that
might be?

Anyway, this amounts to using a mutex in ->d_automount() to
serialize calls to it and then calling d_mountpoint() on the
incoming path dentry after aquiring it and returning NULL if
it is true. This should be all that's needed since the mount
would be complete by the time we exit ->d_automount().

I also believe that the CIFS d_automount patch suffers the same
fault.

[Ian (ian-redhat-bugs)]

(In reply to comment #32)
>
> I also believe that the CIFS d_automount patch suffers the same
> fault.

And quite probably nfs referral mounts as well.

[Jeff (jeff-redhat-bugs)]

I rolled up a small test program that can reproduce this now without a gui:

-----------------[snip]------------------
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>

int
main(int argc, char **argv)
{
 struct stat buf;

 fork();

 stat(argv[1], &buf);

 return 0;
}
-----------------[snip]------------------

...when I run this with /misc/salusa/scratch/foo as the first argument, then unmounting /misc/salusa triggers the panic. Ian, were you trying to reproduce this on a single-cpu box before? If so, then that might explain why you couldn't get it to happen.

With this, I think we can rule out inotify as the problem. The issue is something with simultaneous d_automounts running.

[Ian (ian-redhat-bugs)]

(In reply to comment #34)
> I rolled up a small test program that can reproduce this now without a gui:
>
> -----------------[snip]------------------
> #include <stdio.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <unistd.h>
>
> int
> main(int argc, char **argv)
> {
> struct stat buf;
>
> fork();
>
> stat(argv[1], &buf);
>
> return 0;
> }
> -----------------[snip]------------------

OK, I'll try this in a VM.
Only have an F15, a 2.6.38 based kernel I think, but that shouldn't make a difference.

>
> ...when I run this with /misc/salusa/scratch/foo as the first argument, then
> unmounting /misc/salusa triggers the panic. Ian, were you trying to reproduce
> this on a single-cpu box before? If so, then that might explain why you
> couldn't get it to happen.

I use at least 2-cpus in vms, and I think I had 4 in the one I
used to try and reproduce this.

>
> With this, I think we can rule out inotify as the problem. The issue is
> something with simultaneous d_automounts running.

It does look that way, yes.
At least that will save me the time trying to write an inotify
based reproducer.

Ian

[Jeff (jeff-redhat-bugs)]

...and with the above reproducer, I can reproduce this without autofs in the mix now. Here's the details:

The server (a f15 box named salusa) has a separate filesystem mounted on /scratch that is exported with (rw,insecure,no_root_squash). It has a file in it called "foo". The server does not declare an explicit fsid=0 export, so it fakes up a pseudoroot.

On the client, I mount the filesystem:

    # mount -t nfs4 salusa:/scratch /mnt/salusa

...I'm then careful not to touch anything under /mnt/salusa. I then run the reproducer above with the following path:

    # ./forkstat /mnt/salusa/scratch/foo

...and then:

    # umount /mnt/salusa

...at that point, the BUG pops.

[Geert Janssens (gjanssens)]

*** Bug 708099 has been marked as a duplicate of this bug. ***

[Jeff (jeff-redhat-bugs)]

...sorry, I made a mistake in comment #36. The mount command should read as:

    # mount -t nfs4 salusa:/ /mnt/salusa

...everything else should be correct.

[Joseph Salisbury (jsalisbury)]

Is there a test case that can be used to easily reproduce this bug? Or does it only require heavy I/O against the NFS mount?

[Ian (ian-redhat-bugs)]

(In reply to comment #38)
> ...sorry, I made a mistake in comment #36. The mount command should read as:
>
> # mount -t nfs4 salusa:/ /mnt/salusa
>
> ...everything else should be correct.

Ha, I'm able to reproduce this now, and with only 2-cpus.

[molostoff (molostoff)]

I see regularly this bug happen on my natty without nfs heavy load. In my case autofs mounter tries to umount nfs share by timeout (usually 300 secs by default) and then this kernel stack dump printed. System becomes unusable and unable to reboot, only hard reset can help.

[Jens Maus (jens.maus)]

I can support that observation. It doesn't matter if you put heavy load on the nfs mounts. All that matters is that the nfs mounts are mounted via autofs and that you have a timeout to get the system automatically unmounting the shares. That's where the crash happens actually. Here I have in fact increased the timeout to 3600 seconds and the problem improves because the automounter is not frequently unmounting the share again. Indeed I have moved some of our automounted shares to use /etc/fstab instead and have them permanently mounted. Since then the crashes are very very rare and just happens for our /net/HOST shares. Please also note that it doesn't matter if the share is an nfs3 or nfs4 mount. All that is required is to have autofs mounting the NFS share and having it automatically unmounting it.

[Alvin (alvind)]

On Thursday 16 June 2011 09:11:01 Jens Langner wrote:
> All that matters is that the nfs mounts are mounted
> via autofs [...]

No, I don't use autofs. The crash happens on shutdown here.

[Jens Maus (jens.maus)]

Yes, you are right. It also happens when trying to shutdown the machine. So it seems to be essentially related to unmounting NFS volumes under certain circumstances. Increasing the autofs unmount timeout simply reduces the probability of seeing this problem during runtime of the system.

[David (david-redhat-bugs)]

I tracked this down to lock_mount() releasing the caller's ref on path->mnt when in finds it has to transit to a new mountpoint from lookup_mnt(). The problem is that pathwalk may not be holding a ref on path->mnt if it's the same as nd->path.mnt as it tries to save on mntget()/mntput() calls.

[Ian (ian-redhat-bugs)]

Created attachment 505077
Patch - VFS: Fix vfsmount overput on simultaneous automount

We believe this patch resolves the problem seen in this bug.

Since Jeff has put so much effort into this already I'll set
about creating a test kernel package.

[Ian (ian-redhat-bugs)]

Created attachment 505085
Patch - VFS: Fix vfsmount overput on simultaneous automount (F15)

Oops, original patch didn't apply cleanly.
All the more reason to test it yourselves, please.

[Ian (ian-redhat-bugs)]

A kernel with the above patch has been built.
Please give it a try and reprot back with results.

It can be found at:
http://people.redhat.com/~ikent/kernel-2.6.38.8-32.bz708039.1.fc15

[Andy (andy-redhat-bugs)]

I have a similar (maybe?) bug that started with FC15. I run CrashPlan client on a nfs auto mounted file system. The kernel above didn't fix it. I only have 1 CPU.

2.6.38.8-32.bz708039.1.fc15.i686 #1 SMP Thu Jun 16 17:51:40 UTC 2011 i686 i686 i386 GNU/Linux

[138413.271428] BUG: Dentry f02ad480{i=13f2d0,n=/} still in use (1) [unmount of autofs autofs]
[138413.274358] ------------[ cut here ]------------
[138413.275316] kernel BUG at fs/dcache.c:947!
[138413.275316] invalid opcode: 0000 [#1] SMP
[138413.275316] last sysfs file: /sys/devices/virtual/bdi/0:81/uevent
[138413.275316] Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc p4_clockmod vfat fat snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev iTCO_wdt e100 iTCO_vendor_support parport_pc mii snd_timer snd microcode soundcore parport i2c_i801 snd_page_alloc serio_raw i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: mperf]
[138413.275316]
[138413.275316] Pid: 1885, comm: automount Not tainted 2.6.38.8-32.bz708039.1.fc15.i686 #1 Compaq Evo D510 CMT/07E8h
[138413.275316] EIP: 0060:[<c04f382d>] EFLAGS: 00010296 CPU: 0
[138413.275316] EIP is at shrink_dcache_for_umount_subtree+0xd0/0x162
[138413.275316] EAX: 00000065 EBX: f02ad480 ECX: 00000046 EDX: 00000000
[138413.275316] ESI: c093985f EDI: f282115c EBP: f43a9edc ESP: f43a9eac
[138413.275316] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[138413.275316] Process automount (pid: 1885, ti=f43a8000 task=f2bbd7f0 task.ti=f43a8000)
[138413.275316] Stack:
[138413.275316] c092d11a f02ad480 0013f2d0 f02ad4a4 00000001 c093985f f282115c 00000000
[138413.275316] f02ad4a4 f02ad480 f2821000 c07f9e64 f43a9eec c04f401b f2821000 c67ba780
[138413.275316] f43a9f08 c04e5e3d f5b9a740 c67ba780 00000029 c67ba780 f409fa00 f43a9f14
[138413.275316] Call Trace:
[138413.275316] [<c04f401b>] shrink_dcache_for_umount+0x3e/0x4a
[138413.275316] [<c04e5e3d>] generic_shutdown_super+0x1e/0xb8
[138413.275316] [<c04e5f48>] kill_anon_super+0x11/0x44
[138413.275316] [<c04e5f99>] kill_litter_super+0x1e/0x21
[138413.275316] [<c05812fe>] autofs4_kill_sb+0x35/0x39
[138413.275316] [<c04e610a>] deactivate_locked_super+0x1f/0x40
[138413.275316] [<c04e6a9b>] deactivate_super+0x2e/0x31
[138413.275316] [<c04f8694>] mntput_no_expire+0xb5/0xb9
[138413.275316] [<c04f8f32>] sys_umount+0x270/0x297
[138413.275316] [<c04f8f77>] sys_oldumount+0x1e/0x20
[138413.275316] [<c07d691c>] syscall_call+0x7/0xb
[138413.275316] [<c07d0000>] ? audit_log_exit+0xbca/0xdc3
[138413.275316] Code: 03 8b 41 60 89 54 24 10 8b 55 f0 89 7c 24 18 89 74 24 14 89 5c 24 04 89 54 24 0c 89 44 24 08 c7 04 24 1a d1 92 c0 e8 2c b5 2d 00 <0f> 0b 8b 73 10 39 f3 75 0c 8d 43 68 31 f6 e8 24 2b 0e 00 eb 16
[138413.275316] EIP: [<c04f382d>] shrink_dcache_for_umount_subtree+0xd0/0x162 SS:ESP 0068:f43a9eac
[138413.418764] ---[ end trace 073c5e8debef6d67 ]---

[Ian (ian-redhat-bugs)]

(In reply to comment #44)
> I have a similar (maybe?) bug that started with FC15. I run CrashPlan client
> on a nfs auto mounted file system. The kernel above didn't fix it. I only
> have 1 CPU.
>

That not the same bug, see bug 698806.

[Joseph Salisbury (jsalisbury)]

@Jens @Matt

I am trying to reproduce this issue, but I've been unsucessful so far. I lowered the timeout in /etc/default/autofs to 30. I see the NFS mount get unmounted without any issue. If I cd to the /net/HOSTNAME/share directory, the mount is never unmounted.

Is it possible for either of you to post additional information about your environment? It would be great if you would post your /etc/auto.* files, and your /etc/exports file from the NFS server.

Also, is the NFS server an Ubuntu server as well?

[molostoff (molostoff)]

Both client and server are ubuntu/natty with all regular and security updates installed.

autofs config on client (distro comments removed):
##
/net /etc/auto.net
+auto.master
##

nfs-kernel-server on server /etc/exports
##
/var/lib/transmission-daemon/downloads *(ro,all_squash,anonuid=65534,anongid=65534,no_subtree_check)
##

So, as for me I see nothing special here... If only that autofs (in my case) try to flush cache of read only volume, while it goes (or gone) to umount??

Client is a wifi connected notebook, server is a directly wired to ZTE-W300 router with static addresses for all of them.
Is there additional extra info needed?

[Jens Maus (jens.maus)]

Here, the clients are all ubuntu/natty (7 servers with different hardware) and the NFS server is a solaris10 server (SPARC) with all regular and security updates installed.

autofs config on client:
-- cut here --
/net /etc/auto.net
-- cut here --

nfs server config on solaris10 (/etc/dfs/dfstab)
-- cut here --
share -F nfs -o root=XXXX,rw=XXXX -d "DESC" /mnt/test
-- cut here --

Here also nothing special. In fact, we have two servers which are still on ubuntu/lucy and they are not showing the problem reported here. Since we moved some mounts to direct mounts with /etc/fstab and increasing the timeout to several hours, the problem does not show up that often anymore.

[Joseph Salisbury (jsalisbury)]

@molostoff @Jens

Could you provide the contents of your /etc/auto.net file? I'm mostly interested in what opts you have set.

[molostoff (molostoff)]

@Jens
A /etc/auto.net file is standard (came from ubuntu package)

[molostoff (molostoff)]

@Jens
/etc/default/nfs-common

Only option I have changed in this file is a line with

NEED_IDMAPD=yes

[molostoff (molostoff)]

@Jens

1) A server host config file /etc/default/nfs-common is the same as on client

2) Server side mount point (in server /etc/fstab) which is shared via /etc/exports is lvm2 volume with ext4:
/dev/myhost/downloads /var/lib/transmission-daemon auto defaults,noatime 0 2

(tune2fs -l /dev/myhost/downloads: Default mount options: journal_data_writeback)

3) A server host config file /etc/default/nfs-kernel-server attached here.

[Jens Maus (jens.maus)]

@Joseph

here the /etc/auto.net file is also mostly unmodified and it doesn't really matter which options I specify there (also tried the default options). However, my current options are:

-- cut here --
opts="-fstype=nfs,vers=3,rsize=32768,wsize=32768,nolock,hard,intr,nodev,nosuid"
-- cut here --

Regarding the /etc/default/nfs-common. Here are the options that are set

-- cut here --
NEED_STATD=yes
STATDOPTS=
NEED_IDMAPD=no
NEED_GSSD=no
-- cut here --

[Jens Maus (jens.maus)]

Any progress on a bugfix for that one? It is really unbelieveable that it hasn't been fixed in the meantime since this bug exists so long and is so important for servers using NFS connections regularly. Here, we are considering downgrading our kernels to the ones use in 11.04 as we still get regular hangups of our systems due to the dentry kernel ops reported here.

[Alvin (alvind)]

I agree with the above comment. Why is the bug not assigned and has the importance not been marked as 'critical' yet? NFS file sharing is a basic function and should be treated with priority.

[Russell Phillips (ignissport)]

I just spotted a truckload of upstream NFS fixes that rolled in with the 2.6.38-10.46 kernel that just hit the repos. Surely ONE of these fixes will resolve this issue.

Here's the NFS changes alone:

  * nfsd4: fix struct file leak on delegation
    - LP: #775809
  * nfsd4: Fix filp leak
    - LP: #775809
  * nfs: don't lose MS_SYNCHRONOUS on remount of noac mount
    - LP: #775809
  * NFSv4.1: Ensure state manager thread dies on last umount
    - LP: #775809
  * nfsd: fix auth_domain reference leak on nlm operations
    - LP: #761134
  * nfsd4: fix oops on lock failure
    - LP: #761134
 * NFS: Fix panic after nfs_umount()
    - LP: #683938
  * NFS: Fix a reference leak in nfs_wb_cancel_page()
  * NFS: Try to commit unstable writes in nfs_release_page()
  * NFSv4: Don't allow posix locking against servers that don't support it
  * NFSv4: Ensure that the NFSv4 locking can recover from stateid errors
  * NFS: Fix an Oops when truncating a file
  * NFS: Fix a umount race
  * NFS: Fix a bug in nfs_fscache_release_page()
  * NFS: Fix the mapping of the NFSERR_SERVERFAULT error
  * NFS: Too many GETATTR and ACCESS calls after direct I/O
  * nfsd: Fix sort_pacl in fs/nfsd/nf4acl.c to actually sort groups
  * NFS: Revert default r/wsize behavior
  * nfsd: make sure data is on disk before calling ->fsync
  * NFS: Fix nfs_migrate_page()

[Jens Maus (jens.maus)]

After some testing of the latest released Ubuntu kernel version (2.6.38-10-server) I have to unfortunately report that the problem still persists. Thus, the kernel still seems to Oops at the very same location. See the attached dmesg output:

--- cut here ---
[662197.101766] BUG: Dentry ffff880052fdab40{i=9ce3e,n=/} still in use (1) [unmount of autofs autofs]
[662197.102634] ------------[ cut here ]------------
[662197.103007] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
[662197.103457] invalid opcode: 0000 [#1] SMP
[662197.103814] last sysfs file: /sys/devices/virtual/bdi/0:46/uevent
[662197.104284] CPU 1
[662197.104351] Modules linked in: autofs4 ppdev psmouse serio_raw vmw_balloon nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc parport_pc i2c_piix4 vmxnet3 shpchp lp parport floppy e1000 mptspi mptscsih mptbase vmw_pvscsi
[662197.106028]
[662197.106316] Pid: 25545, comm: automount Not tainted 2.6.38-10-server #46-Ubuntu VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[662197.107209] RIP: 0010:[<ffffffff81179c15>] [<ffffffff81179c15>] shrink_dcache_for_umount_subtree+0x285/0x290
[662197.108003] RSP: 0018:ffff880071913de8 EFLAGS: 00010296
[662197.108411] RAX: 000000000000006c RBX: ffff880052fdab9c RCX: 00000000ffffffff
[662197.109083] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000246
[662197.109760] RBP: ffff880071913e28 R08: 0000000000000000 R09: ffffffff816423e0
[662197.110375] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880052fdab9c
[662197.110820] R13: ffff880052fdab40 R14: ffff880052fdabe0 R15: 00007f7343c15c10
[662197.111240] FS: 00007f734339f700(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
[662197.111462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[662197.111462] CR2: 00007f7343399ab8 CR3: 0000000070a9b000 CR4: 00000000000006e0
[662197.111462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[662197.111462] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[662197.111462] Process automount (pid: 25545, threadinfo ffff880071912000, task ffff88001403db80)
[662197.111462] Stack:
[662197.111462] ffff8800782cd658 ffff880071840000 ffff880071913e08 ffff8800782cd400
[662197.111462] ffff880052fdab9c ffff880052fdab40 00007f7343c160f0 00007f7343c15c10
[662197.111462] ffff880071913e58 ffffffff8117c551 ffff880071913e48 ffff8800782cd400
[662197.111462] Call Trace:
[662197.111462] [<ffffffff8117c551>] shrink_dcache_for_umount+0x51/0x90
[662197.111462] [<ffffffff811670bc>] generic_shutdown_super+0x2c/0x100
[662197.111462] [<ffffffff81167226>] kill_anon_super+0x16/0x60
[662197.111462] [<ffffffff81167297>] kill_litter_super+0x27/0x30
[662197.111462] [<ffffffffa02054d8>] autofs4_kill_sb+0x48/0x60 [autofs4]
[662197.111462] [<ffffffff81167695>] deactivate_locked_super+0x45/0x70
[662197.111462] [<ffffffff8116831a>] deactivate_super+0x4a/0x70
[662197.111462] [<ffffffff811834d4>] mntput_no_expire+0xa4/0xf0
[662197.111462] [<ffffffff81184560>] sys_umount+0x60/0xd0
[662197.111462] [<ffffffff8100bfc2>] system_call_fastpath+0x16/0x1b
[662197.111462] Code: 8b 40 28 4c 8b 08 49 8b 45 30 48 85 c0 74 07 48 8b 90 a8 00 00 00 48 89...

[Hartmut Manz (hartmut-manz)]

I can confirm that the error is still present.

I have activated /net in /etc/auto.master and I have one server
which is exporting several filesystems.

[Tim Gardner (timg-tpi)]

There are a couple of stable commits in 2.6.38-11.47 that look promising:

NFSv4.1: Fix the handling of NFS4ERR_SEQ_MISORDERED errors
NFSv4: Handle expired stateids when the lease is still valid

Either subscribe to -proposed or pull a kernel directly from https://launchpad.net/ubuntu/+source/linux/2.6.38-11.47

[Tim Gardner (timg-tpi)]

You could also give the Oneiric kernel a try from https://launchpad.net/ubuntu/+source/linux/3.0.0-7.8

[Hartmut Manz (hartmut-manz)]

I was already using kernel 2.6.38-11.47 and still had that problem.

Now I am trying 3.0.0-7.8

[Richard Huddleston (rhuddusa)]

i am seeing this crash after every timeout and umount in autofs. changed my timeout from 60 to 3600 and i still see the problem

Linux box05 2.6.38-10-server #46~lucid1-Ubuntu SMP Wed Jul 6 20:19:32 UTC 2011 x86_64 GNU/Linux

installed
linux-image-server-lts-backport-natty 2.6.38.10.20
linux-image-2.6.38-10-server 2.6.38-10.46~lucid1

i have symlinks in my home directory to a /net/anotherHost/export , so they always trigger a mount when i log in

I am using these options in auto.net
opts="-fstype=nfs,hard,intr,nodev,nosuid"

and /etc/default/autofs
TIMEOUT=3600
UMOUNT_WAIT=15

once i get this bug, i can see in pstree, automount is hung on an umount.nfs and mount.nfs commands
and then i can't shutdown my computer cleanly, the shutdown screen hangs forever after
unexporting nfs directories. even if i try and shutdown before the bug is trigged, it gets triggered at shutdown

thus, i never get to umount my root filesystems, and all that joy. i can't figure out a way to get the system back in order, after the bug is triggered.

[Richard Huddleston (rhuddusa)]

the bug might also be in one of these reports

BUG: Dentry ffff880137e720c0{i=2d319,n=/} still in use (1) [unmount of autofs autofs]:
https://bugzilla.redhat.com/show_bug.cgi?id=698806

dentry still in use on umount of autofs mounted nfs4 filesystem
https://bugzilla.redhat.com/show_bug.cgi?id=708039

both bug threads are quite long with lots of info

[John Johansen (jjohansen)]

there is a test kernel using the redhat patch sequence available at

people.canonical.com/~jj/linux-image-2.6.38-11-generic_2.6.38-11.47~lp813748_amd64.deb
people.canonical.com/~jj/linux-headers-2.6.38-11-generic_2.6.38-11.47~lp813748_amd64.deb

[Hartmut Manz (hartmut-manz)]

With the Kernel 3.0.0 may system was running for 24 hours without NFS problems,
but I had problems with resolv.conf.

Now I will try linux-image-2.6.38-11-generic_2.6.38-11.47~lp813748

[Hartmut Manz (hartmut-manz)]

Good news!

The kernel linux-image-2.6.38-11-generic_2.6.38-11.47~lp813748 seems to have fixed the nfs unmount problem
in natty for me.

Thanks a lot

[molostoff (molostoff)]

At 2.6.38-11-server #48 still oops before every reboot (cant umount / due too this issue)

[molostoff (molostoff)]

I have found that scatter/gather NIC features on server can affect the behavor described above - just disabled them on server:

# my primary iface is eth1.
# here below is a stanza in a server host "interfaces" config
# that requires ethtool be installed
# the same as "ethtool -K eth1 sg on"
auto eth1
iface eth1 inet manual
           offload-sg off

I am not sure that this can fix a problem or simply hide, please test it on your servers and respond. But surely, fix does not affect client side, and excuse hard kernel oop reaction to error in packets from server connections.

[molostoff (molostoff)]

should be: "ethtool -K eth1 sg off" - in cli mode

[John (john-redhat-bugs)]

Same issue here.
Once the last open file under the mountpoint is closed, umount causes kernel ooops. It looks to me like something has been freed when it shouldn't have been. Note the fact that umount reports "/cluster/gslap01: not mounted", when it most certainly was mounted:

[root@gslap02 ~]# killall -TERM lock_cleaner
[root@gslap02 ~]# lsof | grep /cluster/gslap01
lock_clea 7592 survey1 txt REG 0,23 1332384 95191157 /cluster/gslap01/gs/bin/a.1.1.0/lock_cleaner
lock_clea 7592 survey1 1w REG 0,23 3886 95191163 /cluster/gslap01/gs/bin/a.1.1.0/lock_cleaner.errors
lock_clea 7592 survey1 2w REG 0,23 3886 95191163 /cluster/gslap01/gs/bin/a.1.1.0/lock_cleaner.errors
lock_clea 7592 survey1 3u REG 0,23 16018 94405062 /cluster/gslap01/gs/database/11_7med/mission_plan/db_lock_table.dat
lock_clea 7592 survey1 4u REG 0,23 260 95191135 /cluster/gslap01/gs/bin/a.1.1.0/common_lock_table.dat
[root@gslap02 ~]#
[root@gslap02 ~]# lsof | grep /cluster/gslap01
[root@gslap02 ~]# umount /cluster/gslap01
umount: /cluster/gslap01: not mounted
[root@gslap02 ~]#
Message from syslogd@ at Mon Sep 5 10:54:52 2011 ...
gslap02 kernel: ------------[ cut here ]------------

Message from syslogd@ at Mon Sep 5 10:54:52 2011 ...
gslap02 kernel: invalid opcode: 0000 [#1] PREEMPT SMP

Message from syslogd@ at Mon Sep 5 10:54:52 2011 ...
gslap02 kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1c.5/0000:09:00.0/net/eth0/statistics/collisions

Message from syslogd@ at Mon Sep 5 10:54:52 2011 ...
gslap02 kernel: Process umount.nfs4 (pid: 8895, ti=e0834000 task=e08b7540 task.ti=e0834000)

Message from syslogd@ at Mon Sep 5 10:54:52 2011 ...
gslap02 kernel: Stack:

Message from syslogd@ at Mon Sep 5 10:54:52 2011 ...
gslap02 kernel: Call Trace:

Message from syslogd@ at Mon Sep 5 10:54:52 2011 ...
gslap02 kernel: Code: 46 20 c7 45 ec 00 00 00 00 85 c0 74 06 8b 40 60 89 45 ec 8d 82 5c 01 00 00 50 51 53 57 ff 75 ec 56 68 18 26 7f c0 e8 fe d2 f3 ff <0f> 0b 83 c4 1c eb fe 8b 7e 10 8d 46 68 89 45 e4 39 fe 75 1f 8b

Message from syslogd@ at Mon Sep 5 10:54:52 2011 ...
gslap02 kernel: EIP: [<c050360a>] shrink_dcache_for_umount_subtree+0xd5/0x1a0 SS:ESP 0068:e0835e8c

[John (john-redhat-bugs)]

I should add the above bug occurs with kernel-ml 2.6.39-4.el5.elrepo.x86_64

[John (john-redhat-bugs)]

I'm also seeing another nfs4 issue in the logs, stuff like:
      bad fileid, expected foo got bar

But it didn't look too bad - the fileid/inode concerned was the root of the NFS4 mount, and I figured some non-critical code inside nfs somewhere was just seeing inode of the mountpoint instead of the directory mounted on the mountpoint, or something along those lines. It didn't seem to immediately cause any problems, but did puzzle me a bit.

Now however, I think its possible the bad fileid issue may be related to this more serious oops on umount bug.

I am just speculating, as I have no knowledge of the source code, but the bad fileid errors I see ALWAYS involve the mountpoint where the nfs share is being mounted. So [erhaps the code to handle bad fileids is a trigger for dentry leak and therefore cause oops on umount?

In any case, please fix these dentry cache errors soon. It is disappointing when the NFSv4 lottery crashes servers and clients year after year, kernel after kernel.

Do we really need to sacrifice stability and lose data just to add new features to nfsv4?

Can we not fix 1 bug without creating 10 more?

Is it time to declare nfs4 fatally flawed and impossible to implement reliably - and then move it out of the kernel and into fuse?

[John (john-redhat-bugs)]

its all a bit too hard isnt it.

[Jonathan Sambrook (jonathan-hmmn)]

Is there a patched kernel deb available anywhere or any plans to include the patch in the Natty updates?

[Josh (josh-redhat-bugs)]

The patch from comment #42 has been included in the 3.0 kernel release. F15 is using 2.6.40 now, which is 3.0 renamed. If this issue is still present with the latest F15 kernel update, please reopen the bug

[Josh (josh-redhat-bugs)]

*** Bug 715374 has been marked as a duplicate of this bug. ***

[John Johansen (jjohansen)]

This bug should be fixed by upstream commit 8aef18845266f5c05904c610088f2d1ed58f6be3

    VFS: Fix vfsmount overput on simultaneous automount

I have built a test kernel with the commit, its available at
  people.canonical.com/~jj/linux-image-2.6.38-12-generic_2.6.38-12.51~lp769927_amd64.deb
  people.canonical.com/~jj/linux-headers-2.6.38-12-generic_2.6.38-12.51~lp769927_amd64.deb

[Jonathan Sambrook (jonathan-hmmn)]

On 13/10/11 16:31, John Johansen wrote:
> This bug should be fixed by upstream commit
> 8aef18845266f5c05904c610088f2d1ed58f6be3
>
> VFS: Fix vfsmount overput on simultaneous automount
>
> I have built a test kernel with the commit, its available at
> people.canonical.com/~jj/linux-image-2.6.38-12-generic_2.6.38-12.51~lp769927_amd64.deb
> people.canonical.com/~jj/linux-headers-2.6.38-12-generic_2.6.38-12.51~lp769927_amd64.deb
>

Any chance of an i386 build?

Cheers,
Jonathan

[Jay Age (yanoshik)]

@molostoff
No idea how did you come up with this hack, but no matter: it works great here!

Server: lucid. Client: natty. No oops observed in what has been about a week since switching SG off .

Thanks!

[Chris J Arges (arges)]

This patch applies cleanly to ubuntu-natty, I can boot fine with this kernel.
This patch is already in ubuntu-oneiric.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "patch cherry picked from 8aef18845266f5c05904c610088f2d1ed58f6be3" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]