kernel crash in /build/buildd/linux-2.6.38/fs/dcache.c:947

[Jens Maus]

Binary package hint: linux-image-2.6.38-8-server

We are running several servers with Ubuntu 10.10 without any problem so far. After updating two of the servers to 11.04 and thus kernel 2.6.38 we are seeing the following kernel crash/report showing up:

-- cut here --
[77459.430193] BUG: Dentry ffff8804552403c0{i=167d27,n=/} still in use (1) [unmount of autofs autofs]
[77459.430271] ------------[ cut here ]------------
[77459.430296] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
[77459.430323] invalid opcode: 0000 [#1] SMP
[77459.430349] last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state
[77459.430392] CPU 8
[77459.430396] Modules linked in: binfmt_misc parport_pc ppdev ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs autofs4 mptctl k8temp nfsd exportfs nfs lockd fscache nfs_acl lm75 auth_rpcgss sunrpc lm87 hwmon_vid ipmi_watchdog ipmi_poweroff psmouse radeon serio_raw ttm ipmi_si drm_kms_helper ipmi_devintf ipmi_msghandler nv_tco drm i2c_nforce2 amd64_edac_mod lp edac_core i2c_algo_bit edac_mce_amd parport k10temp ghes joydev hed usbhid hid usb_storage qla2xxx mptsas mptscsih scsi_transport_fc mptbase e1000e scsi_transport_sas scsi_tgt
[77459.430679]
[77459.430699] Pid: 31606, comm: automount Not tainted 2.6.38-8-server #42-Ubuntu SUN MICROSYSTEMS SUN BLADE X8440 SERVER MODULE/Sun Blade X8440 Server Module
[77459.430761] RIP: 0010:[<ffffffff81179be5>] [<ffffffff81179be5>] shrink_dcache_for_umount_subtree+0x285/0x290
[77459.430814] RSP: 0018:ffff88047dd19de8 EFLAGS: 00010296
[77459.430839] RAX: 000000000000006c RBX: ffff88045524041c RCX: 00000000ffffffff
[77459.430868] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000246
[77459.430896] RBP: ffff88047dd19e28 R08: 0000000000000000 R09: ffffffff816423e0
[77459.430924] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88045524041c
[77459.430953] R13: ffff8804552403c0 R14: ffff880455240460 R15: 00007f72bf923330
[77459.430981] FS: 00007f72bd73a700(0000) GS:ffff88068fc00000(0000) knlGS:00000000f68b2700
[77459.431025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[77459.431050] CR2: 00007f7916fa1eb0 CR3: 000000047ed7e000 CR4: 00000000000006e0
[77459.431079] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[77459.431107] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[77459.431136] Process automount (pid: 31606, threadinfo ffff88047dd18000, task ffff88047ef92dc0)
[77459.431180] Stack:
[77459.431199] ffff88027df22258 ffff88067c5df000 ffff88047dd19e08 ffff88027df22000
[77459.431245] ffff88045524041c ffff8804552403c0 00007f72bf923960 00007f72bf923330
[77459.431291] ffff88047dd19e58 ffffffff8117c521 ffff88047dd19e48 ffff88027df22000
[77459.431337] Call Trace:
[77459.431361] [<ffffffff8117c521>] shrink_dcache_for_umount+0x51/0x90
[77459.431392] [<ffffffff811670ac>] generic_shutdown_super+0x2c/0x100
[77459.431420] [<ffffffff81167216>] kill_anon_super+0x16/0x60
[77459.431446] [<ffffffff81167287>] kill_litter_super+0x27/0x30
[77459.431477] [<ffffffffa047b4d8>] autofs4_kill_sb+0x48/0x60 [autofs4]
[77459.431505] [<ffffffff81167685>] deactivate_locked_super+0x45/0x70
[77459.431533] [<ffffffff8116830a>] deactivate_super+0x4a/0x70
[77459.431562] [<ffffffff811834a4>] mntput_no_expire+0xa4/0xf0
[77459.431588] [<ffffffff81184530>] sys_umount+0x60/0xd0
[77459.431615] [<ffffffff8100bfc2>] system_call_fastpath+0x16/0x1b
[77459.431641] Code: 8b 40 28 4c 8b 08 49 8b 45 30 48 85 c0 74 07 48 8b 90 a8 00 00 00 48 89 34 24 48 c7 c7 c0 a6 7e 81 4c 89 ee 31 c0 e8 30 af 45 00 <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89
[77459.431796] RIP [<ffffffff81179be5>] shrink_dcache_for_umount_subtree+0x285/0x290
[77459.431839] RSP <ffff88047dd19de8>
[77459.432152] ---[ end trace 6814107156e603e1 ]---
-- cut here --

The two servers have different hardware (this one is from a SunBlade X8440 module), but the crash log is always the same.
The action to trigger that problem is still unclear as the Process it crashes in is always "automount". We are using autofs for automatically mounting home and data directories. We see that in case we have a long process running over certain directory structures which trigger autofs this crash shows up.
Finally the machine ends up in an unstable state which requires a hard reboot (simply 'reboot' command does not work).

When switching back to kernel linux-image-2.6.35-28-server from Ubuntu 10.10 immediately solves the problem.

[molostoff (molostoff)]

I suspected automount service on my machine.

Here is my kernel crash/report message:

14185.881413] ------------[ cut here ]------------
[14185.881480] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
[14185.881544] invalid opcode: 0000 [#1] SMP
[14185.881600] last sysfs file: /sys/devices/virtual/hwmon/hwmon0/temp1_input
[14185.881673] CPU 0
[14185.881700] Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc ip6table_filter ip6_tables parport_pc autofs
4 dm_crypt ppdev cryptd aes_x86_64 aes_generic snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq_
midi snd_rawmidi snd_seq_midi_event snd_seq rfcomm snd_timer sco bnep snd_seq_device l2cap kvm_intel kvm ipt_MASQUERADE ipt
able_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ipta
ble_filter ip_tables x_tables bridge stp coretemp uvcvideo r852 sm_common nand arc4 snd ir_lirc_codec lirc_dev iwl3945 iwlc
ore ir_sony_decoder nand_ids rc_rc6_mce ir_jvc_decoder mac80211 ir_rc6_decoder psmouse nand_ecc ir_rc5_decoder acer_wmi spa
rse_keymap videodev ene_ir lp soundcore microcode joydev btusb bluetooth v4l2_compat_ioctl32 mtd serio_raw cfg80211 snd_pag
e_alloc ir_nec_decoder rc_core parport usb_storage uas usbhid hid radeon sdhci_pci ttm firewire_ohci drm_kms_helper drm ahc
i video sdhci firewire_core crc_itu_t i2c_algo_bit libahci tg3
[14185.883329]
[14185.883353] Pid: 8313, comm: automount Not tainted 2.6.38-9-server #43-Ubuntu /eMachine
[14185.883473] RIP: 0010:[<ffffffff81179ba5>] [<ffffffff81179ba5>] shrink_dcache_for_umount_subtree+0x285/0x290
[14185.883585] RSP: 0018:ffff880060c75de8 EFLAGS: 00010296
[14185.883641] RAX: 000000000000006c RBX: ffff88005fb04a1c RCX: 00000000000000d4
[14185.883708] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000246
[14185.883777] RBP: ffff880060c75e28 R08: 0000000000000033 R09: 00000000000131f0
[14185.883844] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88005fb04a1c
[14185.883910] R13: ffff88005fb049c0 R14: ffff88005fb04a60 R15: 00007f6aafd2e160
[14185.883978] FS: 00007f6aae00b700(0000) GS:ffff8800bf400000(0000) knlGS:0000000000000000
[14185.884056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14185.884112] CR2: 00007f863902b000 CR3: 000000011b7bd000 CR4: 00000000000026e0
[14185.884181] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14185.884249] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[14185.884318] Process automount (pid: 8313, threadinfo ffff880060c74000, task ffff88008793c4a0)
[14185.884396] Stack:
[14185.884422] ffff8800613f8258 ffffffff81177566 0000000000000000 ffff8800613f8000
[14185.884516] ffff88005fb04a1c ffff88005fb049c0 00007f6aafd2f0b0 00007f6aafd2e160
[14185.884608] ffff880060c75e58 ffffffff8117c4e1 ffff880060c75e48 ffff8800613f8000
[14185.890038] Call Trace:
[14185.891126] [<ffffffff81177566>] ? pollwake+0x56/0x60
[14185.891126] [<ffffffff8117c4e1>] shrink_dcache_for_umount+0x51/0x90
[14185.891126] [<ffffffff8116704c>] generic_shutdown_super+0x2c/0x100
[14185.891126] [<ffffffff811671b6>] kill_anon_super+0x16/0x60
[14185.891126] [<ffffffff8...

[molostoff (molostoff)]

$ uname -a
Linux aspire 2.6.38-9-server #43-Ubuntu SMP Thu Apr 28 15:40:34 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

[Jens Maus (jens.maus)]

same here. After replacing automount of remote NFS shares by direct mounts in /etc/fstab the issue seems to be gone. so somehow automount is going to crash the kernel in that way.

[Guillaume (guillaume-zin)]

Hello,

Same here, and I'm using autofs with kernel 2.6.38-11-generic #48-Ubuntu SMP Fri Jul 29 19:02:55 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

[ 4212.551980] BUG: Dentry ffff88009c1d6900{i=8572,n=/} still in use (1) [unmount of autofs autofs]
[ 4212.552006] ------------[ cut here ]------------
[ 4212.552009] kernel BUG at /build/buildd/linux-2.6.38/fs/dcache.c:947!
[ 4212.552012] invalid opcode: 0000 [#2] SMP
[ 4212.552016] last sysfs file: /sys/devices/virtual/bdi/cifs-84/uevent
[ 4212.552019] CPU 1
[ 4212.552021] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat mmc_block hidp hid binfmt_misc rfcomm sco bnep l2cap vboxnetadp vboxnetflt vboxdrv md4 nls_utf8 cifs cryptd aes_x86_64 aes_generic parport_pc dm_crypt ppdev autofs4 vesafb nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc joydev arc4 nvidia(P) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq_midi uvcvideo videodev v4l2_compat_ioctl32 btusb iwlagn snd_rawmidi bluetooth snd_seq_midi_event iwlcore snd_seq mac80211 snd_timer nvidiabl snd_seq_device snd cfg80211 psmouse soundcore snd_page_alloc serio_raw sony_laptop firewire_sbp2 lp parport firewire_ohci sky2 ahci sdhci_pci sdhci firewire_core crc_itu_t libahci
[ 4212.552089]
[ 4212.552092] Pid: 5301, comm: automount Tainted: P D 2.6.38-11-generic #48-Ubuntu Sony Corporation VGN-AW11M_H/VAIO
[ 4212.552100] RIP: 0010:[<ffffffff811799b5>] [<ffffffff811799b5>] shrink_dcache_for_umount_subtree+0x285/0x290
[ 4212.552110] RSP: 0018:ffff880067c7bde8 EFLAGS: 00010296
[ 4212.552112] RAX: 000000000000006a RBX: ffff88009c1d695c RCX: 000000000000002d
[ 4212.552115] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000246
[ 4212.552119] RBP: ffff880067c7be28 R08: 0000000000000033 R09: 0000000000012012
[ 4212.552121] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88009c1d695c
[ 4212.552124] R13: ffff88009c1d6900 R14: ffff88009c1d69a0 R15: 00007f84e9e05620
[ 4212.552128] FS: 00007f84e5243700(0000) GS:ffff8800bdc80000(0000) knlGS:0000000000000000
[ 4212.552131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4212.552134] CR2: 00007f673aa05000 CR3: 000000011c47b000 CR4: 00000000000006e0
[ 4212.552137] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4212.552140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4212.552144] Process automount (pid: 5301, threadinfo ffff880067c7a000, task ffff8801327444a0)
[ 4212.552146] Stack:
[ 4212.552148] ffff88007a1c2258 ffffffff811771c6 0000000000000000 ffff88007a1c2000
[ 4212.552154] ffff88009c1d695c ffff88009c1d6900 00007f84e9e05440 00007f84e9e05620
[ 4212.552159] ffff880067c7be58 ffffffff8117c321 ffff880067c7be48 ffff88007a1c2000
[ 4212.552164] Call Trace:
[ 4212.552170] [<ffffffff811771c6>] ? pollwake+0x56/0x60
[ 4212.552174] [<ffffffff8117c321>] shrink_dcache_for_umount+0x51/0x90
[ 4212.552180] [<ffffffff81166c4c>] generic_shutdown_super+0x2c/0x100
[ 4212.552184] [<ffffffff81166db6>] kill_anon_super+0x16/0x60
[ 4212.552188] [<ffffffff81166e27>] kill_litter_super+0x27/0x30
[ 4212.552195] [<ffffffffa0d4e4d8>] autofs4_kill_sb+0x48/0x60 [autofs4]
[ 4212.552199] [<ffffffff811672...