Kernel Panic due to null pointer dereference in do_add_mount()

[Tejasvi]

Binary package hint: linux-source-2.6.22

[ 1169.629030] mtrr: type mismatch for d0000000,8000000 old: write-back new: write-combining
[ 3078.911785] audit(1207762306.159:6): type=1503 operation="inode_permission" requested_mask="a" denied_mask="a" name="/dev/tty" pid=11028 profile="/usr/sbin/cupsd"
[14232.047183] BUG: unable to handle kernel NULL pointer dereference at virtual address 0000006a
[14232.047188] printing eip:
[14232.047190] c0196b69
[14232.047191] *pde = 00000000
[14232.047194] Oops: 0000 [#1]
[14232.047196] SMP
[14232.047199] Modules linked in: nfs lockd sunrpc ipv6 rfcomm l2cap bluetooth af_packet ppdev autofs4 cpufreq_userspace cpufreq_conservative cpufreq_powersave cpufreq_ondemand cpufreq_stats freq_table button video ac sbs container dock battery sbp2 lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event agpgart snd_seq snd_timer snd_seq_device snd parport_pc parport i2c_core serio_raw iTCO_wdt iTCO_vendor_support soundcore psmouse snd_page_alloc pcspkr shpchp pci_hotplug evdev ext3 jbd mbcache sg sd_mod sr_mod cdrom ata_generic ehci_hcd ohci1394 ieee1394 ata_piix libata scsi_mod floppy uhci_hcd tg3 usbcore thermal processor fan fuse apparmor commoncap
[14232.047261] CPU: 0
[14232.047261] EIP: 0060:[<c0196b69>] Not tainted VLI
[14232.047263] EFLAGS: 00010206 (2.6.22-14-generic #1)
[14232.047268] EIP is at graft_tree+0x39/0xf0
[14232.047271] eax: f4c6f7f8 ebx: ffffffec ecx: 00000000 edx: f4c65f2c
[14232.047274] esi: c20e7a80 edi: f4c65f2c ebp: 00000000 esp: f4c65dfc
[14232.047277] ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
[14232.047280] Process mount.nfs (pid: 18108, ti=f4c64000 task=f762ea60 task.ti=f4c64000)
[14232.047282] Stack: f4c65f2c f4c65f30 c20e7a80 c0196f16 f8838380 00000000 f4c74003 f4c74000
[14232.047290] c01980e1 00000000 f4c76000 f4c75000 c20e7a80 00000000 f4c65e68 00000000
[14232.047297] f51b2800 f4c65eb4 00000001 f7491900 c02c4552 00000001 f68df300 00000000
[14232.047304] Call Trace:
[14232.047310] [<c0196f16>] do_add_mount+0x76/0x120
[14232.047316] [<c01980e1>] do_mount+0x691/0x740
[14232.047324] [<c02c4552>] ip4_datagram_connect+0x1d2/0x370
[14232.047333] [<c011fc6e>] kunmap_atomic+0x5e/0xa0
[14232.047337] [<c011fc7a>] kunmap_atomic+0x6a/0xa0
[14232.047343] [<c016d237>] __handle_mm_fault+0x687/0xb00
[14232.047351] [<c01ff040>] copy_to_user+0x30/0x60
[14232.047363] [<c02f5ef9>] do_page_fault+0x389/0x690
[14232.047376] [<c02f43f2>] error_code+0x72/0x80
[14232.047394] [<c02f00d8>] clip_ioctl+0x478/0x510
[14232.047397] [<c019695b>] copy_mount_options+0xab/0x150
[14232.047404] [<c01986f7>] sys_mount+0x77/0xc0
[14232.047410] [<c01041d2>] sysenter_past_esp+0x6b/0xa9
[14232.047416] [<c02f0000>] clip_ioctl+0x3a0/0x510
[14232.047421] =======================
[14232.047422] Code: 04 89 c6 89 7c 24 08 8b 40 14 89 d7 8b 40 30 85 c0 79 11 89 d8 8b 74 24 04 8b 1c 24 8b 7c 24 08 83 c4 0c c3 8b 02 b3 ec 8b 48 0c <0f> b7 41 6a 25 00 f0 00 00 3d 00 40 00 00 8b 46 10 0f 94 c2 8b
[14232.047449] EIP: [<c0196b69>] graft_tree+0x39/0xf0 SS:ESP 0068:f4c65dfc

===========

I moved to Ubuntu 7.10 recently:

$ uname -a
Linux taswatha-devd 2.6.22-14-generic #1 SMP Tue Feb 12 07:42:25 UTC 2008 i686 GNU/Linux

[Gareth Fitzworthington (mapping-gp-deactivatedaccount)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Can you supply more detailed information on the problem you have reported (for example when does the problem occur: during boot or does some other event trigger this? Does it always occur or just occasionally or just once?).

Please include the following additional information, if you have not already done so (pay attention to lspci's additional options):
1. Please include the output of the command "uname -a" in your next response. It should be one, long line of text which includes the exact kernel version you're running, as well as the CPU architecture.
2. Please run the command "dmesg > dmesg.log" after a fresh boot and attach the resulting file "dmesg.log" to this bug report.
3. Please run the command "sudo lspci -vvnn > lspci-vvnn.log" and attach the resulting file "lspci-vvnn.log" to this bug report.
4. Please run the command "sudo dmidecode > dmidecode.txt" and attach the resulting file "dmidecode.txt" to this bug report.
Thanks.

[Jonathan Sambrook (jonathan-hmmn)]

I'm getting this Oops on 2.6.24-16 apparently when autofs is mounting some nfs mounts.

Linux maternoty 2.6.24-16-386 #1 Thu Apr 10 12:50:06 UTC 2008 i686 GNU/Linux

[ 870.450520] BUG: unable to handle kernel NULL pointer dereference at virtual address 00000066
[ 870.450538] printing eip: c018c2b9 *pde = 00000000
[ 870.450550] Oops: 0000 [#1]
[ 870.450557] Modules linked in: nvidia(P) i2c_core binfmt_misc rfcomm l2cap nfsd auth_rpcgss exportfs ipx p8023 ppdev lp autofs4 container sbs dock sbshc video output battery cpufreq_conservative cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand nfs lockd nfs_acl sunrpc iptable_filter ip_tables x_tables reiserfs aes_i586 dm_crypt p4_clockmod speedstep_lib freq_table quickcam videodev v4l2_common v4l1_compat yenta_socket rsrc_nonstatic pcmcia_core ipv6 af_packet hci_usb bluetooth prism2_usb evdev p80211 parport_pc parport serio_raw snd_intel8x0 psmouse rtc snd_ac97_codec ac97_bus snd_pcm_oss snd_pcm snd_mixer_oss snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq pcspkr snd_timer snd_seq_device snd soundcore snd_page_alloc button iTCO_wdt iTCO_vendor_support shpchp pci_hotplug intel_agp agpgart ac ext3 jbd mbcache sg sr_mod cdrom sd_mod ata_piix pata_acpi 8139cp floppy ata_generic ohci1394 ieee1394 8139too mii libata scsi_mod ehci_hcd uhci_hcd usbcore raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod dm_mirror dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse
[ 870.450697]
[ 870.450704] Pid: 8729, comm: mount.nfs Tainted: P (2.6.24-16-386 #1)
[ 870.450711] EIP: 0060:[<c018c2b9>] EFLAGS: 00010206 CPU: 0
[ 870.450726] EIP is at graft_tree+0x39/0xf0
[ 870.450730] EAX: eb10d580 EBX: ffffffec ECX: 00000000 EDX: f77f9f2c
[ 870.450737] ESI: df479e00 EDI: f77f9f2c EBP: 00000023 ESP: f77f9e04
[ 870.450741] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[ 870.450748] Process mount.nfs (pid: 8729, ti=f77f8000 task=e0c11640 task.ti=f77f8000)
[ 870.450752] Stack: f77f9f2c f77f9f30 df479e00 c018c641 00200006 df479e00 f6a6d000 f77e0000
[ 870.450766] c018d73b 00000000 f778e000 00000023 df979e00 00000000 00000001 f6939000
[ 870.450777] c02d28a3 00000001 f7f2fe00 f7aa8c00 00000025 f704e900 f77f9edc 00000000
[ 870.450788] Call Trace:
[ 870.450813] [<c018c641>] do_add_mount+0x71/0xf0
[ 870.450850] [<c018d73b>] do_mount+0x5db/0x6c0
[ 870.450908] [<c02d28a3>] ip4_datagram_connect+0x193/0x350
[ 870.451033] [<c01563fc>] find_lock_page+0x2c/0xa0
[ 870.451074] [<c0158765>] filemap_fault+0x1b5/0x3b0
[ 870.451202] [<c0117eed>] fixup_exception+0x1d/0x60
[ 870.451326] [<c015b61f>] __alloc_pages+0x5f/0x380
[ 870.451351] [<c0302ba0>] do_page_fault+0x0/0x720
[ 870.451468] [<c018c080>] copy_mount_options+0x40/0x140
[ 870.451526] [<c018dbc2>] sys_mount+0x72/0xb0
[ 870.451581] [<c0104132>] sysenter_past_esp+0x6b/0xa9
[ 870.451716] =======================
[ 870.451720] Code: 04 89 c6 89 7c 24 08 8b 40 14 89 d7 8b 40 30 85 c0 79 11 89 d8 8b 74 24 04 8b 1c 24 8b 7c 24 08 83 c4 0c c3 8b 02 b3 ec 8b 48 08 <0f> b7 41 66 25 00 f0 00 00 3d 00 40 00 00 8b 46 10 0f 94 c...

[Jonathan Sambrook (jonathan-hmmn)]

Ubuntu 8.04

[Gareth Fitzworthington (mapping-gp-deactivatedaccount)]

Jonathan & Tejasvi,
Does this issue occur always or just occasionally or just once only?.
Waiting on Tejasvi for Gutsy info.
Thanks.

[Tejasvi (tejasvia)]

I haven't hit this bug again.

Ubuntu 7.10

$ uname -a

Linux taswatha-devd 2.6.22-14-generic #1 SMP Tue Feb 12 07:42:25 UTC 2008 i686 GNU/Linux

I am attaching all the other info in subsequent posts.

[Jonathan Sambrook (jonathan-hmmn)]

Hi Gareth,

It's not every time, but it is pretty much every session.

Regards,
Jonathan

[Tejasvi (tejasvia)]

Hit the bug, again, this time after upgrading to 8.04.

$ uname -a
Linux taswatha-devd 2.6.24-16-generic #1 SMP Thu Apr 10 13:23:42 UTC 2008 i686 GNU/Linux

FWIW, here's the backtrace:

[ 2827.690900] Pid: 12860, comm: mount.nfs Tainted: P (2.6.24-16-generic #1)
[ 2827.690903] EIP: 0060:[<c01a31e9>] EFLAGS: 00010206 CPU: 0
[ 2827.690909] EIP is at graft_tree+0x39/0xf0
[ 2827.690911] EAX: f371b330 EBX: ffffffec ECX: 00000000 EDX: f4e11f2c
[ 2827.690914] ESI: f6879800 EDI: f4e11f2c EBP: 00000020 ESP: f4e11e04
[ 2827.690916] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 2827.690919] Process mount.nfs (pid: 12860, ti=f4e10000 task=f6926000 task.ti=f4e10000)
[ 2827.690921] Stack: f4e11f2c f4e11f30 f6879800 c01a3582 00200000 f6879800 f4e95000 f75e2000
[ 2827.690928] c01a469b 00000000 f4e7a000 00000020 c029ed64 00000000 00000001 00000000
[ 2827.690935] 00000007 f4faf830 f69f4d80 c029e538 000000db f88dcaaa 00000011 f88dc7f6
[ 2827.690942] Call Trace:
[ 2827.690947] [<c01a3582>] do_add_mount+0x72/0x100
[ 2827.690954] [<c01a469b>] do_mount+0x5fb/0x700
[ 2827.690959] [<c029ed64>] skb_release_all+0x54/0x130
[ 2827.690966] [<c029e538>] __kfree_skb+0x8/0x80
[ 2827.690971] [<f88dcaaa>] tg3_poll+0x2fa/0x9a0 [tg3]
[ 2827.690982] [<f88dc7f6>] tg3_poll+0x46/0x9a0 [tg3]
[ 2827.690996] [<c0179227>] handle_mm_fault+0x387/0x730
[ 2827.691004] [<c02a64fd>] net_rx_action+0x12d/0x210
[ 2827.691014] [<c0131a22>] __do_softirq+0x82/0x110
[ 2827.691022] [<c0131dd1>] irq_exit+0x51/0x80
[ 2827.691036] [<c0106f20>] do_IRQ+0x40/0x70
[ 2827.691040] [<c016e7b0>] __alloc_pages+0x60/0x390
[ 2827.691046] [<c0105413>] common_interrupt+0x23/0x30
[ 2827.691052] [<c01a300c>] copy_mount_options+0xac/0x140
[ 2827.691057] [<c01a4db7>] sys_mount+0x77/0xb0
[ 2827.691062] [<c01043c2>] sysenter_past_esp+0x6b/0xa9
[ 2827.691068] [<c0310000>] vcc_getsockopt+0x150/0x170
[ 2827.691075] =======================
[ 2827.691076] Code: 04 89 c6 89 7c 24 08 8b 40 14 89 d7 8b 40 30 85 c0 79 11 89 d8 8b 74 24 04 8b 1c 24 8b 7c 24 08 83 c4 0c c3 8b 02 b3 ec 8b 48 0c <0f> b7 41 6a 25 00 f0 00 00 3d 00 40 00 00 8b 46 10 0f 94 c2 8b
[ 2827.691102] EIP: [<c01a31e9>] graft_tree+0x39/0xf0 SS:ESP 0068:f4e11e04
[ 2827.691113] ---[ end trace b95ef32372b7907f ]---

[Manuel McLure (manuel-mclure)]

I have something similar in /var/log/messages:

May 14 17:45:06 ulthar kernel: [78489.312979] PGD a1358067 PUD 61f22067 PMD 0
May 14 17:45:06 ulthar kernel: [78489.312984] CPU 1
May 14 17:45:06 ulthar kernel: [78489.312986] Modules linked in: af_packet binfmt_misc rfcomm l2cap nfsd auth_rpcgss exportfs ppdev autofs4 ipv6 acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_ondemand cpufreq_conservative freq_table video output container dock sbs sbshc battery nfs lockd nfs_acl sunrpc iptable_filter ip_tables x_tables ext2 aes_x86_64 dm_crypt ac coretemp it87 hwmon_vid sbp2 lp loop hci_usb bluetooth snd_hda_intel snd_pcm_oss snd_mixer_oss sr_mod cdrom snd_pcm snd_page_alloc snd_hwdep nvidia(P) i2c_core snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device button snd iTCO_wdt iTCO_vendor_support shpchp parport_pc parport pci_hotplug intel_agp soundcore pcspkr evdev ext3 jbd mbcache usb_storage libusual sg usbhid hid sd_mod pata_jmicron pata_acpi ata_piix floppy ohci1394 ata_generic ieee1394 r8169 libata scsi_mod uhci_hcd ehci_hcd usbcore dm_mirror dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcur
May 14 17:45:06 ulthar kernel: or fuse
May 14 17:45:06 ulthar kernel: [78489.313046] Pid: 15881, comm: mount.nfs Tainted: P 2.6.24-16-generic #1
May 14 17:45:06 ulthar kernel: [78489.313048] RIP: 0010:[graft_tree+0x4c/0x140] [graft_tree+0x4c/0x140] graft_tree+0x4c/0x140
May 14 17:45:06 ulthar kernel: [78489.313051] RSP: 0018:ffff8100aff75c48 EFLAGS: 00010246
May 14 17:45:06 ulthar kernel: [78489.313053] RAX: ffff81005a5e3410 RBX: 00000000ffffffec RCX: 0000000000000000
May 14 17:45:06 ulthar kernel: [78489.313055] RDX: ffff810118967700 RSI: ffff8100aff75e58 RDI: ffff810118f63200
May 14 17:45:06 ulthar kernel: [78489.313057] RBP: ffff810118f63200 R08: 0000000000000000 R09: 0000000000000000
May 14 17:45:06 ulthar kernel: [78489.313059] R10: 0000000000000000 R11: ffffffff80310fb0 R12: ffff8100aff75e58
May 14 17:45:06 ulthar kernel: [78489.313060] R13: 0000000000000000 R14: 0000000000000023 R15: 0000000000000023
May 14 17:45:06 ulthar kernel: [78489.313063] FS: 00007f37fd1376e0(0000) GS:ffff81011bc01800(0000) knlGS:0000000000000000
May 14 17:45:06 ulthar kernel: [78489.313065] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 14 17:45:06 ulthar kernel: [78489.313066] CR2: 00000000000000b2 CR3: 000000007f805000 CR4: 00000000000006e0
May 14 17:45:06 ulthar kernel: [78489.313068] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 14 17:45:06 ulthar kernel: [78489.313070] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 14 17:45:06 ulthar kernel: [78489.313072] Process mount.nfs (pid: 15881, threadinfo ffff8100aff74000, task ffff8100ca1bc7c0)
May 14 17:45:06 ulthar kernel: [78489.313074] Stack: ffff8100aff75e58 ffff8100aff75e60 ffff810118f63200 ffffffff802c5c42
May 14 17:45:06 ulthar kernel: [78489.313078] 0000000000200006 0000000018f63200 ffff810037970000 ffff810059dc9000
May 14 17:45:06 ulthar kernel: [78489.313082] ffff8100ca0f4000 ffffffff802c6f2a 0000000000000400 0000000000000000
May 14 17:45:06 ulthar kernel...

[Jesper Krogh (jk-novozymes)]

I can reproduce it on a 7.10 system:
May 19 16:02:51 node15 kernel: [2059317.829560] PGD efe18067 PUD efe19067 PMD 0
May 19 16:02:51 node15 kernel: [2059317.922748] CPU 1
May 19 16:02:51 node15 kernel: [2059317.948928] Modules linked in: nfs lockd sunrpc autofs4 ipv6 usbhid hid uhci_hcd ehci_hcd usbk
bd fuse parport_pc lp parport af_packet i2c_amd756 i2c_core amd_rng serio_raw pcspkr shpchp psmouse pci_hotplug k8temp evdev ext3
jbd mbcache sg sd_mod ide_cd cdrom ata_generic libata floppy tg3 mptspi mptscsih mptbase scsi_transport_spi scsi_mod ohci_hcd usbc
ore amd74xx ide_core thermal processor fan capability commoncap
May 19 16:02:51 node15 kernel: [2059318.394824] Pid: 22862, comm: mount.nfs Not tainted 2.6.22-14-generic #1
May 19 16:02:51 node15 kernel: [2059318.477001] RIP: 0010:[graft_tree+76/304] [graft_tree+76/304] graft_tree+0x4c/0x130
May 19 16:02:51 node15 kernel: [2059318.573826] RSP: 0018:ffff8100ccf55c48 EFLAGS: 00010246
May 19 16:02:51 node15 kernel: [2059318.639381] RAX: ffff8100bb90f4e0 RBX: 00000000ffffffec RCX: 0000000000000000
May 19 16:02:51 node15 kernel: [2059318.726753] RDX: ffff8100bc60f100 RSI: ffff8100ccf55e58 RDI: ffff81001b04ae00
May 19 16:02:51 node15 kernel: [2059318.814124] RBP: ffff81001b04ae00 R08: 0000000000000000 R09: ffff8100ee6f6980
May 19 16:02:51 node15 kernel: [2059318.901493] R10: 000000001785c20a R11: 0000000000000007 R12: ffff8100ccf55e58
May 19 16:02:51 node15 kernel: [2059318.988864] R13: 0000000000000000 R14: 000000000000000b R15: 000000000000000b
May 19 16:02:51 node15 kernel: [2059319.076237] FS: 00002b2f50a1e6e0(0000) GS:ffff8100faab7180(0000) knlGS:00000000f7d616b0
May 19 16:02:51 node15 kernel: [2059319.175036] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 19 16:02:51 node15 kernel: [2059319.245784] CR2: 00000000000000b2 CR3: 00000000c45b0000 CR4: 00000000000006e0
May 19 16:02:51 node15 kernel: [2059319.333156] Process mount.nfs (pid: 22862, threadinfo ffff8100ccf54000, task ffff8100facdb4a0)
May 19 16:02:51 node15 kernel: [2059319.438187] Stack: ffff8100ccf55e58 ffff8100ccf55e60 ffff81001b04ae00 ffffffff802b1fa0
May 19 16:02:51 node15 kernel: [2059319.536468] 0000000000000006 000000001b04ae00 ffff8100e22d5000 ffff8100e22d4000
May 19 16:02:51 node15 kernel: [2059319.627371] ffff8100e00be000 ffffffff802b3271 0000000000000010 0000000000000000
May 19 16:02:51 node15 kernel: [2059319.716092] Call Trace:
May 19 16:02:51 node15 kernel: [2059319.749550] [do_add_mount+160/352] do_add_mount+0xa0/0x160
May 19 16:02:51 node15 kernel: [2059319.816142] [do_mount+1329/2000] do_mount+0x531/0x7d0
May 19 16:02:51 node15 kernel: [2059319.879618] [__handle_mm_fault+1985/2912] __handle_mm_fault+0x7c1/0xb60
May 19 16:02:51 node15 kernel: [2059319.952451] [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
May 19 16:02:51 node15 kernel: [2059320.029428] [__up_read+33/176] __up_read+0x21/0xb0
May 19 16:02:51 node15 kernel: [2059320.091866] [do_page_fault+971/2160] do_page_fault+0x3cb/0x870
May 19 16:02:51 node15 kernel: [2059320.160540] [zone_statistics+125/128] zone_statistics+0x7d/0x80
May 19 16:02:51 node15 kernel: [2059320.229212] [error_exit+0/132] error_exit+0x0/0x84
May...

[Jonathan Sambrook (jonathan-hmmn)]

This is the third client host I've now had this problem on (the second was a visitor's box, which I no longer have access to).

uname:
Linux llwarchhen 2.6.24-16-generic #1 SMP Thu Apr 10 13:23:42 UTC 2008 i686 GNU/Linux

dmesg:
[ 1106.749396] BUG: unable to handle kernel NULL pointer dereference at virtual address 0000006a
[ 1106.749402] printing eip: c01a31e9 *pde = 00000000
[ 1106.749407] Oops: 0000 [#1] SMP
[ 1106.749410] Modules linked in: iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack vmnet(P) vmblock vmmon(P) binfmt_misc af_packet nfsd auth_rpcgss exportfs ppdev autofs4 ipv6 acpi_cpufreq sbs dock cpufreq_powersave cpufreq_conservative cpufreq_userspace container cpufreq_ondemand sbshc cpufreq_stats freq_table nfs lockd nfs_acl sunrpc iptable_filter ip_tables x_tables sbp2 lp pcmcia joydev wacom quickcam videodev v4l2_common v4l1_compat snd_intel8x0 ipw2200 snd_ac97_codec ieee80211 ieee80211_crypt yenta_socket rsrc_nonstatic pcmcia_core ac97_bus nvidia(P) snd_pcm_oss snd_mixer_oss snd_pcm i2c_core snd_seq_dummy video output snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq serio_raw iTCO_wdt iTCO_vendor_support irda snd_timer snd_seq_device shpchp button intel_agp battery ac crc_ccitt pci_hotplug agpgart dcdbas parport_pc parport evdev snd soundcore snd_page_alloc psmouse usbhid hid ext3 jbd mbcache sg sr_mod cdrom sd_mod ata_piix ata_generic ohci1394 pata_acpi ieee1394 libata scsi_mod tg3 ehci_hcd uhci_hcd usbcore thermal processor fan fbcon tileblit font bitblit softcursor fuse
[ 1106.749466]
[ 1106.749469] Pid: 14407, comm: mount.nfs Tainted: P (2.6.24-16-generic #1)
[ 1106.749471] EIP: 0060:[<c01a31e9>] EFLAGS: 00010206 CPU: 0
[ 1106.749479] EIP is at graft_tree+0x39/0xf0
[ 1106.749481] EAX: f43d1a18 EBX: ffffffec ECX: 00000000 EDX: e0a1df2c
[ 1106.749483] ESI: f6e96e00 EDI: e0a1df2c EBP: 00000023 ESP: e0a1de04
[ 1106.749485] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 1106.749487] Process mount.nfs (pid: 14407, ti=e0a1c000 task=f5852b40 task.ti=e0a1c000)
[ 1106.749489] Stack: e0a1df2c e0a1df30 f6e96e00 c01a3582 00200006 f6e96e00 e0e90000 e0e29000
[ 1106.749495] c01a469b 00000000 e0c05000 00000023 e0a1deac 00000000 e0c6a240 c02e690c
[ 1106.749499] 00000001 df8cc7e0 df8cc780 f79da6e8 00000025 f42a3180 0000014e f784013c
[ 1106.749504] Call Trace:
[ 1106.749512] [<c01a3582>] do_add_mount+0x72/0x100
[ 1106.749522] [<c01a469b>] do_mount+0x5fb/0x700
[ 1106.749533] [<c02e690c>] ip4_datagram_connect+0x23c/0x370
[ 1106.749550] [<c016917f>] find_lock_page+0x2f/0xb0
[ 1106.749561] [<c016b6c6>] filemap_fault+0x216/0x420
[ 1106.749568] [<c0216110>] copy_to_user+0x30/0x60
[ 1106.749584] [<c012103d>] kunmap_atomic+0x3d/0xb0
[ 1106.749601] [<c01206ad>] fixup_exception+0x1d/0x60
[ 1106.749623] [<c016e7b0>] __alloc_pages+0x60/0x390
[ 1106.749630] [<c03190f0>] do_page_fault+0x0/0x730
[ 1106.749654] [<c01a2fa0>] copy_mount_options+0x40/0x140
[ 1106.749665] [<c01a4db7>] sys_mount+0x77/0xb0
[ 1106.749676] [<c01043c2>] sysenter_past_esp+0x6b/0xa9
[ 1106.749691] [<c0310000>] vcc_getsockopt+0x150/0x170
[ 1106.749706] =======================
[ 1106.749708] Code: 04 89 c6 89 7c 24 08 8b 40 14 89 d...

[Farzad FARID (farzy)]

Hi, This bug also occurs on Ubuntu 8.10. It's pretty serious for me, because the crash now happens almost twice a day. Hi had the very same bug with Ubuntu 8.06's kernel.

Linux karma 2.6.24-18-generic #1 SMP Wed May 28 20:27:26 UTC 2008 i686 GNU/Linux

I'm using autofs with a Thecus N2100 NFS file-server, it's a linux box.

[29871.351477] WARNING: at /build/buildd/linux-2.6.24/fs/inotify.c:172 set_dentry_child_flags()
[29871.351490] Pid: 7229, comm: kded Tainted: P 2.6.24-18-generic #1
[29871.351530] [<c01bc0ff>] set_dentry_child_flags+0xcf/0x160
[29871.351590] [<c01bc1e0>] remove_watch_no_event+0x50/0x60
[29871.351611] [<c01bc2f8>] inotify_remove_watch_locked+0x18/0x50
[29871.351627] [<c019238c>] vfs_read+0x11c/0x170
[29871.351654] [<c01bc65c>] inotify_rm_wd+0x6c/0xb0
[29871.351690] [<c01bcc78>] sys_inotify_rm_watch+0x38/0x60
[29871.351717] [<c01043c2>] sysenter_past_esp+0x6b/0xa9
[29871.351810] =======================
[31382.177527] BUG: unable to handle kernel NULL pointer dereference at virtual address 0000006a
[31382.177536] printing eip: c01a7c39 *pde = 00000000
[31382.177541] Oops: 0000 [#1] SMP
[31382.177544] Modules linked in: autofs4 af_packet vmnet(P) vmblock vmmon(P) binfmt_misc rfcomm l2cap nfsd auth_rpcgss exportfs irtty_sir sir_dev ppdev ipv6 acpi_cpufreq cpufreq_userspace cpufreq_conservative cpufreq_stats cpufreq_powersave cpufreq_ondemand freq_table container sbs bay dock sbshc nfs lockd nfs_acl sunrpc iptable_filter ip_tables x_tables nls_iso8859_1 nls_cp437 vfat fat aes_i586 dm_crypt parport_pc lp parport snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_hwdep snd_seq_dummy serial_cs snd_seq_oss snd_seq_midi arc4 snd_rawmidi ecb blkcipher snd_seq_midi_event joydev snd_seq pcmcia hci_usb iwl3945 usbhid nvidia(P) bluetooth iwlwifi_mac80211 hid agpgart snd_timer cfg80211 i2c_core snd_seq_device iTCO_wdt iTCO_vendor_support serio_raw yenta_socket rsrc_nonstatic pcmcia_core battery irda video output snd ac wmi_acer crc_ccitt button soundcore dcdbas shpchp pci_hotplug evdev psmouse pcspkr ext3 jbd mbcache sg sr_mod cdrom sd_mod ata_generic ata_piix pata_acpi libata scsi_mod ehci_hcd uhci_hcd usbcore tg3 dm_mirror dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse
[31382.177619]
[31382.177622] Pid: 18847, comm: mount.nfs Tainted: P (2.6.24-18-generic #1)
[31382.177625] EIP: 0060:[<c01a7c39>] EFLAGS: 00010206 CPU: 1
[31382.177632] EIP is at graft_tree+0x39/0xf0
[31382.177634] EAX: deb6f770 EBX: ffffffec ECX: 00000000 EDX: dec6ff2c
[31382.177636] ESI: d119ea00 EDI: dec6ff2c EBP: 0000002b ESP: dec6fe04
[31382.177638] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[31382.177641] Process mount.nfs (pid: 18847, ti=dec6e000 task=d2b44b80 task.ti=dec6e000)
[31382.177643] Stack: dec6ff2c dec6ff30 d119ea00 c01a7fd2 00200006 d119ea00 ded03000 d32f5000
[31382.177649] c01a90eb 00000000 e21bf000 0000002b 00000246 00000000 00000001 00000040
[31382.177654] 00000007 d2b44b80 001280d2 00000000 c01730d0 00000044 0000014e f7807ac4
[31382.177660] Call Trace:
[31382.177677] [<c01a7fd2>] do_add_mount+0x72/0x100
[31382.177699] [<c01a90eb>] do_mount+0x5fb/0x700
[3138...

[Tejasvi (tejasvia)]

I kept hitting this bug so frequently that I had to disable the automounter and resort to static mounts, which is also a pain in my environment. Anybody know if this is ubuntu only ?

[Colin Ian King (colin-king)]

Hi there,

I believe I have a patch that may resolve this issue. I have built a collection of Hardy kernels to try this with first and if it's successful I shall backport it to Gutsy.

Can you download the appropriate Hardy kernel deb from:

http://people.ubuntu.com/~cking/sru-214810/

and try it out to see if this fixes the automount problem. Let me know of your results.

Thanks, Colin

[Farzad FARID (farzy)]

Hi,

Le Wednesday 02 July 2008 10:50:36 Colin King, vous avez écrit :
> Hi there,
>
> I believe I have a patch that may resolve this issue. I have built a
> collection of Hardy kernels to try this with first and if it's
> successful I shall backport it to Gutsy.
>
> Can you download the appropriate Hardy kernel deb from:

I'm going on holidays without NFS access in 2 days and will be back around
july the 21st, I'll try your kernel at this time.

But I may have a small issue: without the corresponding
linux-ubuntu-modules-2.6.24-20-generic and
linux-restricted-modules-2.6.24-20-generic packages I won't be able to run my
computer correctly. Can I get those somewhere? Last time I tried to compile
my own kernel packages it was a long and horrible experience :)

 Regards,

--
Farzad FARID / Architecte Open Source - Associé
Pragmatic Source / http://www.pragmatic-source.com
Tel : [phone] / Mob : [phone]

[Colin Ian King (colin-king)]

For your convenience, I've also added the lrm and lum debs for generic i386 and amd64 builds.

Colin

[Farzad FARID (farzy)]

Le jeudi 3 juillet 2008 13:06, Colin King a écrit :
> For your convenience, I've also added the lrm and lum debs for generic
> i386 and amd64 builds.
>
> Colin

Thanks, I'll stress test it in 2 weeks.

 Regards,
--
Farzad FARID / Architecte Open Source - Associé
Pragmatic Source / http://www.pragmatic-source.com
Tel : [phone] / Mob : [phone]

[Chad Netzer (chad-netzer)]

We have been coming across this bug quite frequently since moving to 8.04. Happened again today on 8.04.1. My work colleague tracked down the oops, and it is related to a bug in mainline:

http://www.kerneloops.org/search.php?search=graft_tree&btnG=Function+Search

He then found this discussion on the devel lists that discusses it, and possibly fixes it in 2.6.26 (upcoming).:

http://kerneltrap.org/mailarchive/linux-kernel/2008/6/3/2013554

So the good news is that the developers may have found the bug, but it'll likely need to be backported for Ubuntu. In any case, it is killing us at work, since a few of our desktops are being hit multiple times a week (others with nearly the same config seem fine, though).

BTW, symptoms are a slow deterioration of the system, and an inability to switch consoles or ssh in. The machine isn't dead, but its not well. I started logging in on a virtual console, so that when it happened I could dmesg, and found the Oops.

[Colin Ian King (colin-king)]

Hi Chad,

As noted in comment in https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/214810/comments/19 above I have built a load of debs that can be tried out to see if the relevant patch fixes this problems. If it can be tested then I can commit it to the kernel.

Thanks, Colin

[Chad Netzer (chad-netzer)]

Thanks Colin,

  You don't have the "server" versions of linux-ubuntu-modules, and linux-restricted-modules, which we are using to access all our memory (4G). I can try the generic kernel in the meantime. Can you publish the actual patch you made (or at least email it to me while I test?) Thanks! -Chad

[Colin Ian King (colin-king)]

Chad,

A patch from the autofs git may be the one that does the trick, namely:
 http://git.kernel.org/?p=linux/kernel/git/raven/linux-2.6-autofs4.git;a=commitdiff;h=b939c8bbeb9e49072aaf6a5faf688cb2b3e1c39f

Hope this helps

[Chad Netzer (chad-netzer)]

Colin, sorry to bug you again, but can you tell me if this is the same patch as you applied in your kernel binaries mentioned earlier in the thread? I was running your kernel for a couple days and just had the same Oops occur again. My next step is to take the existing ubuntu kernel, apply just the autofs4 patch you referenced above, and see if that fixes the problem.

[Colin Ian King (colin-king)]

Hi Chad,

Just to clarify, the original patch I built with was a fix that I saw a discussion on from the Linux Kernel Mailing List Archive; subsequently a tidied up version of the patch made it into the autofs git repo which I referred to in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/214810/comments/26 above - however this patch won't apply cleanly and needs a little effort to be back-ported.
I have discussed this with Ian Kent the maintainer and he has told me that there are a few more patches pending that will be coming soon, so am waiting to see what changes in the next week or so to see if any more relevant commits come into fix this specific issue.

Hope this helps. Colin

[Leann Ogasawara (leannogasawara)]

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

[Chad Netzer (chad-netzer)]

I actually just today built the current Intrepid kernel with "prevu", but have to wait a bit to reboot. In the meantime, for my colleague who was experiencing this Oops 2-3 times a week, we compiled and installed 2.6.26.3, and now he hasn't had the problem occur again in over 4 weeks.

However, another user was running the same 2.6.26.3 (not Ubuntu's BTW, but our own config and installation), and that had the same Oops as well (different machine, both running Ubuntu 8.04.1). So, whatever the issue is, 2.6.26 seems to have made the problem rarer, but not eliminated it entirely.

In the meantime, I'll install Ubuntu's Intrepid 2.6.27 kernels on a few machines, especially the ones that have had this oops in the past (it seems to affect only a few users, but it hits them often, like its a usage pattern thing).

[Colin Ian King (colin-king)]

Hi Chad,

Thanks for investigating this with 2.6.27. Any success with the kernels I built in https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/214810/comments/19 ?

Colin

[guenthert (guenthert)]

I ran into this problem this morning for the first time, but now it is reliably reproducable on my work PC ;-|
For me the short-term solution to be able to work again is to disable the automounter (and to live without NFS on this host for the time being). Strangely, I was using Ubuntu 8.04 (rc to 8.04.1) on the same hardware in the same network environment for several months now (I don't reboot often though).

I'm more than willing to test new kernels, but upgrading the OS is not planned for this PC (I'm hoping that Long Term Support includes backports of bug fixes). Could there be a 2.6.27 kernel be made available for Ubuntu 8.04, please (there are two more suspected kernel bug fixes I'm asked to verify)?

[guenthert (guenthert)]

The issue seems to go away (at least in 3 out of 3 attempts ;-) when using Colin's 2.6.24-20.35 kernel (amd64/generic tested only). IOW, works for me. Thanks!

[Farzad FARID (farzy)]

Hi,

Upgrading to Ubuntu 8.10 / Kernel 2.6.27-7-generic apparently solved the problem for me: autofs never crashes again.

[Chad Netzer (chad-netzer)]

An update:

We are now using Intrepid's kernel-2.6.27-7-generic and -server, and have stopped having the autofs4 issues seen above (at least so far). For now, we are sticking with Hardy for LTS, but just upgrading to Intrepid kernel images. FYI.

As for backporting, when I tried Colin's kernel, we still had the same lockup, and I haven't tried it since. So, it is doesn't seem to be a universal fix, but we are one datapoint. The autofs4 changes/fixes made in 2.6.26 and 2.6.27 are apparently pretty extensive, so back porting them to the Hardy kernel may not be possible. :( It'd be nice to find the minimal change/workaround for Hardy's kernel.

[Leann Ogasawara (leannogasawara)]

Marking this Fix Released for Intrepid.

[Launchpad Janitor (janitor)]

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.

[Sergio Zanchetta (primes2h)]

The 18 month support period for Gutsy Gibbon 7.10 has reached its end of life -
http://www.ubuntu.com/news/ubuntu-7.10-eol . As a result, we are closing the
linux-source-2.6.22 kernel task.