BUG: unable to handle kernel NULL pointer dereference at 0000000000000228

[Loïc Minier]

Hi

My home is on ecryptfs; this crash occurs when viewing emails with Mutt's built-in viewer. Mutt's tmpdir points to my home, if I set it to e.g. /tmp, the bug doesn't occur as it's not on ecryptfs.

Bye

ProblemType: KernelOops
Annotation: Your system might become unstable now and might need to be restarted.
Architecture: amd64
Date: Sun Jun 14 23:24:31 2009
DistroRelease: Ubuntu 9.10
Failure: oops
MachineType: LENOVO 2777CTO
Package: linux-image-2.6.30-9-generic 2.6.30-9.10
ProcCmdLine: root=/dev/mapper/hostname--vg0-ubuntu--root ro quiet splash text
ProcVersionSignature: Ubuntu 2.6.30-9.10-generic
RelatedPackageVersions:

SourcePackage: linux
Tags: kernel-oops
Title: BUG: unable to handle kernel NULL pointer dereference at 0000000000000228
Uname: Linux 2.6.30-9-generic x86_64
dmi.bios.date: 03/12/2009
dmi.bios.vendor: LENOVO
dmi.bios.version: 6EET27WW (1.08 )
dmi.board.name: 2777CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr6EET27WW(1.08):bd03/12/2009:svnLENOVO:pn2777CTO:pvrThinkPadX301:rvnLENOVO:rn2777CTO:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2777CTO
dmi.product.version: ThinkPad X301
dmi.sys.vendor: LENOVO

[Loïc Minier (lool)]

Still an issue with 2.6.31-5-generic

[ 2598.361308] BUG: unable to handle kernel NULL pointer dereference at 0000000000000220
[ 2598.361322] IP: [<ffffffff8112d4c0>] touch_atime+0x20/0x150
[ 2598.361340] PGD 127da0067 PUD 11c8d3067 PMD 0
[ 2598.361350] Oops: 0000 [#1] SMP
[ 2598.361358] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/sda2/stat
[ 2598.361366] CPU 0
[ 2598.361371] Modules linked in: cbc cryptd aes_x86_64 aes_generic usb_storage binfmt_misc ppdev bridge stp bnep lp parport joydev snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer i2400m_usb uvcvideo videodev v4l1_compat v4l2_compat_ioctl32 snd_seq_device i2400m iwlagn btusb iwlcore wimax mac80211 snd psmouse serio_raw soundcore cfg80211 snd_page_alloc pcspkr heci(C) thinkpad_acpi led_class nvram e1000e fbcon tileblit font bitblit softcursor i915 drm i2c_algo_bit video output intel_agp
[ 2598.361486] Pid: 4927, comm: mutt Tainted: G C 2.6.31-5-generic #24-Ubuntu 2777CTO
[ 2598.361492] RIP: 0010:[<ffffffff8112d4c0>] [<ffffffff8112d4c0>] touch_atime+0x20/0x150
[ 2598.361503] RSP: 0018:ffff8801028edd58 EFLAGS: 00010246
[ 2598.361509] RAX: 0000000000000000 RBX: ffff8801345d0200 RCX: 000000004a7abb90
[ 2598.361515] RDX: ffff880134c89800 RSI: ffff880110dede40 RDI: ffff8801345d0200
[ 2598.361521] RBP: ffff8801028edd88 R08: 0000000000000002 R09: ffffea0003ac7fa8
[ 2598.361527] R10: 0000000000000022 R11: 0000000000000001 R12: 0000000000000000
[ 2598.361533] R13: 00000000000005b1 R14: ffff8801028edf48 R15: 00007fffe8e16900
[ 2598.361540] FS: 00007fc3d8d5a710(0000) GS:ffff880028023000(0000) knlGS:0000000000000000
[ 2598.361546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2598.361552] CR2: 0000000000000220 CR3: 0000000119c60000 CR4: 00000000000006b0
[ 2598.361558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2598.361564] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2598.361571] Process mutt (pid: 4927, threadinfo ffff8801028ec000, task ffff88011e274410)
[ 2598.361575] Stack:
[ 2598.361579] ffff880136094000 ffff8801028eddc8 ffff8800b56fa540 ffff8801028edeb8
[ 2598.361588] <0> ffff8801028eddc8 ffff8800b56fa540 ffff8801028eddb8 ffffffff811f0237
[ 2598.361599] <0> ffff8801028eddf8 ffff8801028eddc8 ffff8800b56fa540 ffff8801028edeb8
[ 2598.361610] Call Trace:
[ 2598.361623] [<ffffffff811f0237>] ecryptfs_read_update_atime+0x47/0x70
[ 2598.361633] [<ffffffff81118122>] do_sync_read+0xf2/0x130
[ 2598.361644] [<ffffffff81072da0>] ? autoremove_wake_function+0x0/0x40
[ 2598.361654] [<ffffffff812193a1>] ? security_file_permission+0x11/0x20
[ 2598.361662] [<ffffffff81118705>] vfs_read+0xb5/0x1a0
[ 2598.361670] [<ffffffff81118d0c>] sys_read+0x4c/0x80
[ 2598.361680] [<ffffffff81011fc2>] system_call_fastpath+0x16/0x1b
[ 2598.361685] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 30 48 89 5d f0 4c 89 65 f8 48 89 fb 4c 8b 66 10 e8 74 36 00 00 85 c0 75 13 <41> f6 84 24 20 02 00 00 02 74 15 48 89 df e8 8d 37 00 00 48 8b
[ 2598.361769] RIP [<ffffffff8112...

[Aymeric Mansoux (aymeric)]

Same here on 2.6.31-7-generic
with tmpdir pointing to a folder in encrypted home

[ 2601.283398] BUG: unable to handle kernel NULL pointer dereference at 00000148
[ 2601.283411] IP: [<c01f716d>] touch_atime+0x1d/0x130
[ 2601.283428] *pde = bed72067
[ 2601.283435] Oops: 0000 [#5] SMP
[ 2601.283443] last sysfs file: /sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/ieee80211/phy0/rfkill2/uevent
[ 2601.283451] Modules linked in: tun cbc aes_i586 aes_generic dm_crypt snd_hda_codec_analog arc4 snd_hda_intel snd_hda_codec ecb snd_pcm_oss snd_mixer_oss snd_pcm pcmcia snd_seq_dummy iwlagn snd_seq_oss iwlcore yenta_socket snd_seq_midi snd_rawmidi thinkpad_acpi rsrc_nonstatic sdhci_pci snd_seq_midi_event mac80211 sdhci snd_seq pcmcia_core led_class psmouse snd_timer snd_seq_device nvram cfg80211 serio_raw snd soundcore snd_page_alloc btusb ohci1394 ieee1394 e1000e fbcon tileblit font bitblit softcursor i915 drm i2c_algo_bit video output intel_agp agpgart
[ 2601.283558]
[ 2601.283566] Pid: 3395, comm: mutt Tainted: G D (2.6.31-7-generic #27-Ubuntu) 7675H7U
[ 2601.283572] EIP: 0060:[<c01f716d>] EFLAGS: 00010246 CPU: 0
[ 2601.283580] EIP is at touch_atime+0x1d/0x130
[ 2601.283586] EAX: 00000000 EBX: f6bfe380 ECX: f69d2800 EDX: 00000000
[ 2601.283592] ESI: 00000000 EDI: f5985280 EBP: f5953e9c ESP: f5953e8c
[ 2601.283597] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 2601.283604] Process mutt (pid: 3395, ti=f5952000 task=f65fcb60 task.ti=f5952000)
[ 2601.283609] Stack:
[ 2601.283613] 00001000 f5953ecc 0000040b f5985280 f5953ebc c02a08eb 00000000 00000000
[ 2601.283629] <0> f5985280 c05842a0 f5985280 f5953ecc f5953f64 c01e459c 00000000 00000000
[ 2601.283645] <0> f5953f08 c01ca881 00000000 00000001 ffffffff f5985280 00000000 00000000
[ 2601.283664] Call Trace:
[ 2601.283677] [<c02a08eb>] ? ecryptfs_read_update_atime+0x4b/0x70
[ 2601.283686] [<c01e459c>] ? do_sync_read+0xbc/0x100
[ 2601.283696] [<c01ca881>] ? vma_merge+0x1b1/0x2a0
[ 2601.283708] [<c0157920>] ? autoremove_wake_function+0x0/0x40
[ 2601.283718] [<c02c4c4f>] ? security_file_permission+0xf/0x20
[ 2601.283726] [<c01e463f>] ? rw_verify_area+0x5f/0xe0
[ 2601.283734] [<c01e4b97>] ? vfs_read+0x97/0x190
[ 2601.283742] [<c01e44e0>] ? do_sync_read+0x0/0x100
[ 2601.283750] [<c01e51ad>] ? sys_read+0x3d/0x70
[ 2601.283758] [<c010334c>] ? syscall_call+0x7/0xb
[ 2601.283763] Code: 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5 83 ec 10 89 5d f4 89 c3 89 75 f8 89 7d fc 8b 72 10 e8 97 2f 00 00 85 c0 75 10 <f6> 86 48 01 00 00 02 74 1a 89 d8 e8 c3 30 00 00 8b 5d f4 8b 75
[ 2601.283860] EIP: [<c01f716d>] touch_atime+0x1d/0x130 SS:ESP 0068:f5953e8c
[ 2601.283872] CR2: 0000000000000148
[ 2601.283879] ---[ end trace 5e5c2e659e401ce7 ]---

[Jeremy Kerr (jk-ozlabs)]

I get the same oops here, usually triggered by plasma-desktop (causes KDE4 desktop, panel & widgets to disappear). Workaround is to restart plasma desktop.

Linux pororo 2.6.31-9-generic #29-Ubuntu SMP Sun Aug 30 17:39:26 UTC 2009 x86_64 GNU/Linux

I have a dmesg log saved, happy to attach if it'll help.

[Jeremy Kerr (jk-ozlabs)]

Had a bit of time to look into this, hope this helps:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000220
IP: [<ffffffff8112efd0>] touch_atime+0x20/0x150

ffffffff8102efb0 <touch_atime>:
ffffffff8112efb0: 55 push %rbp
ffffffff8112efb1: 48 89 e5 mov %rsp,%rbp
ffffffff8112efb4: 48 83 ec 30 sub $0x30,%rsp
ffffffff8112efb8: 48 89 5d f0 mov %rbx,-0x10(%rbp)
ffffffff8112efbc: 4c 89 65 f8 mov %r12,-0x8(%rbp)
ffffffff8112efc0: 48 89 fb mov %rdi,%rbx
ffffffff8112efc3: 4c 8b 66 10 mov 0x10(%rsi),%r12
ffffffff8112efc7: e8 94 36 00 00 callq 0xffffffff81132660
ffffffff8112efcc: 85 c0 test %eax,%eax
ffffffff8112efce: 75 13 jne 0xffffffff8112efe3
ffffffff8112efd0: 41 f6 84 24 20 02 00 testb $0x2,0x220(%r12)

From looking at the oops log, r12 is indeed 0. Referring to source of touch_atime:

void touch_atime(struct vfsmount *mnt, struct dentry *dentry)
{
 struct inode *inode = dentry->d_inode;
 struct timespec now;

 if (mnt_want_write(mnt))
  return;
 if (inode->i_flags & S_NOATIME)
  goto out;

We're oopsing on the ionode->i_flags dereference, so dentry->d_inode is NULL. I think it's safe to assume that this function should not be called on a negative dentry.

From a brief look at ecryptfs_read_update_atime, it seems that there is no reference on the "lower" dentry when it's added to d_fsdata, so perhaps we need a check for lower_dentry != NULL around that call to touch_atime. I'm not entirely sure that this is the correct solution, but we shouldn't be updating dentries for files that have been removed.

[VinX (vincenx)]

Hi all,
I wrote a patch that solve (I suppose) this bug.
To grab the inode I have used 'file->f_mapping->host'.
It works on my machine.

Best regards,
   Vincenzo

[Dustin Kirkland  (kirkland)]

Hmm, we can carry that patch against mutt in the mean time.

But, Tyler, this is more likely a problem with the ecryptfs kernel code, right?

:-Dustin

[Dustin Kirkland  (kirkland)]

Doh, now I look at the patch ... It is against ecryptfs. /me eats his words.

Tyler, can you take a look at that, and ack it and apply it upstream, if it's okay by you? We could carry this in Karmic, assuming it'll go upstream.

:-Dustin

[Dustin Kirkland  (kirkland)]

Tim, if Tyler acks this patch against ecryptfs, can you pull it into karmic?

:-Dustin

[Tyler Hicks (tyhicks)]

While Vincenzo's patch does keep the oops from occurring, I don't believe it is the correct fix. I had started working on this bug and didn't feel like the dentry should be negative at this point. My gut feeling is an error in ecryptfs_link() or more likely, ecryptfs_unlink(). I should get some more time to look at this in a day or two.

[Dustin Kirkland  (kirkland)]

Okay, so Tim, don't pull the above patch yet. Tyler is still looking at this one. I'll unassign it to you, and CC you instead.

:-Dustin

[Tim Gardner (timg-tpi)]

@Dustin - lets leave this assigned to me or Tyler. Its in the OMG kitten killer class, and I don't want to miss getting it fixed for the Beta release. After all, if folks can't read their email with mutt then the whole might grind to a halt.

[Tim Gardner (timg-tpi)]

*the whole _world_ might grind to a halt.

[Tyler Hicks (tyhicks)]

It turned out to be a pretty simple fix. I've tested this pretty thoroughly in f11 with the latest mainline git tree.

[Dustin Kirkland  (kirkland)]

Tim-

Can we pull this fix for Karmic?

:-Dustin

[Tim Gardner (timg-tpi)]

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-karmic.git;a=commit;h=86235e226279668355ca426f75be12845f3c3933

[Tyler Hicks (tyhicks)]

Pulled by Linus:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b9b9df62e7fd6b5f099c24bc867100ab86e1da5a

[david (davidelizondo2006)]

 fix boot system

ffffffff8102efb0 <touch_atime>:
ffffffff8112efb0: 55 push %rbp
ffffffff8112efb1: 48 89 e5 mov %rsp,%rbp
ffffffff8112efb4: 48 83 ec 30 sub $0x30,%rsp
ffffffff8112efb8: 48 89 5d f0 mov %rbx,-0x10(%rbp)
ffffffff8112efbc: 4c 89 65 f8 mov %r12,-0x8(%rbp)
ffffffff8112efc0: 48 89 fb mov %rdi,%rbx
ffffffff8112efc3: 4c 8b 66 10 mov 0x10(%rsi),%r12
ffffffff8112efc7: e8 94 36 00 00 callq 0xffffffff81132660
ffffffff8112efcc: 85 c0 test %eax,%eax
ffffffff8112efce: 75 13 jne 0xffffffff8112efe3
ffffffff8112efd0: 41 f6 84 24 20 02 00 testb $0x2,0x220(%r12)

[david (davidelizondo2006)]

very slow boot not booting as it should be good grub 2 should still be green
ay to check kernel and grub 2 on the version 2.6.31 and do a review or if the grub2
we should move this very green and use the version 9.04

[david (davidelizondo2006)]

I think Linus Torvalds in the solution tine git testi code should be implemented and new

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b9b9df62e7fd6b5f099c24bc867100ab86e1da5a

[Loïc Minier (lool)]

I confirm that with the latest ubuntu kernel, I can use tmpdir under ecryptfs with Mutt, but I get this in dmesg:
[79493.311126] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry = [\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
[79497.979867] ecryptfs_lookup: lookup_one_len() returned [-36] on lower_dentry = [\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\

I don't see the output "david" mentions.

[Tyler Hicks (tyhicks)]

Hi Loïc - I'm having a hard time reproducing that dmesg with 2.6.32-rc1. I decided that this would be a good time to organize my inbox, but after about 30 minutes of working inside of mutt, I'm still not seeing this those messages. Is there any chance that you can pinpoint the sequence of actions needed to trigger the message?

[Loïc Minier (lool)]

Well I can't either :-)

Sometime after these, I got a system hang with black screen in screensaver or something; no capslock, no blinking lights.

I hard rebooted and couldn't reproduce the dmesg issue either.

Let's leave this bug to rest; I shall open another one if I consistently reproduce the issue.

(BTW fixing this bug already save some data for me: I was composing an email with tmpdir under my $HOME again when the hard crash occured, and I could recover it after reboot -- yeah!)

[Tyler Hicks (tyhicks)]

That sounds like a good idea to me. Glad to hear you were able to recover the email!

[Kees Cook (kees)]

http://www.ubuntu.com/usn/USN-852-1