BUG: unable to handle kernel NULL pointer dereference at (null); RIP: 0010:[<0000000000000000>] [< (null)>] (null)

[roelof van der kleij]

When entering a dfs referral in the DFS root tree, I cannot follow the referral to the next server. I get either a "Permission denied" error or an "Object is remote" error.

if I cd to a non-existent folder I get a kernel oops:

username@COMP12345:~$ cd /DOMAIN/folder/dfsfolder/
username@COMP12345:/DOMAIN/folder/dfsfolder/ ls
ls: cannot open directory .: Permission denied
username@COMP12345:/DOMAIN/folder/dfsfolder$ klist
Ticket cache: FILE:/tmp/krb5cc_1234567
Default principal: <email address hidden>

Valid starting Expires Service principal
09/10/12 17:18:34 10/10/12 03:18:35 <email address hidden>
 renew until 16/10/12 17:18:34
09/10/12 17:18:35 10/10/12 03:18:35 COMP12345$@DOMAIN.LEIDENUNIV.NL
 renew until 16/10/12 17:18:34
09/10/12 17:19:13 10/10/12 03:18:35 <email address hidden>
 renew until 16/10/12 17:18:34

username@COMP12345:/DOMAIN/folder/dfsfolder$ cd XXXXXXXXXX

[ 214.095043] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 214.095106] IP: [< (null)>] (null)
[ 214.095139] PGD 1061bc067 PUD 1061bb067 PMD 0
[ 214.095177] Oops: 0010 [#1] SMP
[ 214.095204] CPU 1
[ 214.095218] Modules linked in: arc4 nls_utf8 cifs autofs4 rfcomm bnep bluetooth snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq_midi dell_wmi sparse_keymap ppdev snd_rawmidi rpcsec_gss_krb5 nfsd nfs lockd fscache auth_rpcgss nfs_acl snd_seq_midi_event sunrpc dcdbas snd_seq snd_timer snd_seq_device psmouse i915 serio_raw snd parport_pc drm_kms_helper wmi drm mac_hid soundcore i2c_algo_bit video snd_page_alloc lp parport usbhid hid tg3
[ 214.095573]
[ 214.095586] Pid: 1969, comm: bash Not tainted 3.2.0-31-generic #50-Ubuntu Dell Inc. OptiPlex 380 /01TKCC
[ 214.095656] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 214.095702] RSP: 0018:ffff88010614fcd0 EFLAGS: 00010286
[ 214.095734] RAX: ffffffffa0409000 RBX: ffff880036b3b3c0 RCX: 0000000000000000
[ 214.095775] RDX: ffff88010614fe78 RSI: ffff880036b3b3c0 RDI: ffff880106e90070
[ 214.095816] RBP: ffff88010614fd08 R08: ffff88012bc56ec0 R09: 0000000000000005
[ 214.095857] R10: 0000000000000000 R11: 0000000000000005 R12: ffff880106e90070
[ 214.095899] R13: ffff88010614fe78 R14: 0000000000000001 R15: ffff88010614fdf0
[ 214.095941] FS: 00007f6da7fdc700(0000) GS:ffff88012bc40000(0000) knlGS:0000000000000000
[ 214.095988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 214.096022] CR2: 0000000000000000 CR3: 00000001061b9000 CR4: 00000000000406e0
[ 214.096063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 214.096105] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 214.096146] Process bash (pid: 1969, threadinfo ffff88010614e000, task ffff88011107ae00)
[ 214.096192] Stack:
[ 214.096206] ffffffff81182b15 ffff88010614fd08 ffffffff81190235 ffff88010614fd08
[ 214.096260] ffff88010614fe78 ffff88010614fdc8 0000000000000000 ffff88010614fd88
[ 214.096313] ffffffff81184ca2 ffff88010614fd28 ffffffff00000001 ffff880106e90110
[ 214.096366] Call Trace:
[ 214.096387] [<ffffffff81182b15>] ? d_alloc_and_lookup+0x45/0x90
[ 214.096426] [<ffffffff81190235>] ? d_lookup+0x35/0x60
[ 214.096458] [<ffffffff81184ca2>] do_lookup+0x202/0x310
[ 214.096490] [<ffffffff811869dc>] path_lookupat+0x11c/0x750
[ 214.096524] [<ffffffff81187b53>] ? user_path_at_empty+0x63/0xa0
[ 214.096563] [<ffffffff813192b7>] ? __strncpy_from_user+0x27/0x60
[ 214.096600] [<ffffffff81187041>] do_path_lookup+0x31/0xc0
[ 214.098792] [<ffffffff81187b49>] user_path_at_empty+0x59/0xa0
[ 214.099014] [<ffffffff8117cc4b>] ? vfs_stat+0x1b/0x20
[ 214.099014] [<ffffffff8117cd8a>] ? sys_newstat+0x1a/0x40
[ 214.099014] [<ffffffff81187ba1>] user_path_at+0x11/0x20
[ 214.099014] [<ffffffff81176bb3>] sys_chdir+0x23/0x90
[ 214.099014] [<ffffffff81662b02>] system_call_fastpath+0x16/0x1b
[ 214.099014] Code: Bad RIP value.
[ 214.099014] RIP [< (null)>] (null)
[ 214.099014] RSP <ffff88010614fcd0>
[ 214.099014] CR2: 0000000000000000
[ 214.140115] ---[ end trace 563b8616dd09caad ]---

I mount the DFS root from the fstab:

//domain.leidenuniv.nl/Public /DOMAIN/folder/ cifs sec=krb5,intr,multiuser,_netdev,iocharset=utf8,noperm,nosuid,nodev,noexec,mapchars,nocase 0 0

Note due issues in netapp, kerberos and samba, the oldest Ubuntu kernel regression tested to is 3.2.0-2-generic: mount works, but bug present.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-31-generic 3.2.0-31.50
ProcVersionSignature: Ubuntu 3.2.0-31.50-generic 3.2.28
Uname: Linux 3.2.0-31-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
AplayDevices:
 **** List of PLAYBACK Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: ALC269VB Analog [ALC269VB Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
ApportVersion: 2.0.1-0ubuntu13
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: kleijrgvander 3293 F.... pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xfe6fc000 irq 42'
   Mixer name : 'Realtek ALC269VB'
   Components : 'HDA:10ec0269,10280400,00100100'
   Controls : 26
   Simple ctrls : 14
Date: Tue Oct 9 17:37:30 2012
HibernationDevice: RESUME=UUID=6c46ca78-30f2-4bc2-8e33-c3682ac842cc
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
MachineType: Dell Inc. OptiPlex 380
ProcEnviron:
 LANGUAGE=en_IE:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_IE.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-31-generic root=UUID=075b4583-73a6-4a44-bf81-3ddd983436c8 ro LTS quiet splash vt.handoff=7
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-31-generic N/A
 linux-backports-modules-3.2.0-31-generic N/A
 linux-firmware 1.79.1
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 03/11/2010
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A01
dmi.board.name: 01TKCC
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.asset.tag: i.033.321
dmi.chassis.type: 15
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA01:bd03/11/2010:svnDellInc.:pnOptiPlex380:pvr:rvnDellInc.:rn01TKCC:rvrA00:cvnDellInc.:ct15:cvr:
dmi.product.name: OptiPlex 380
dmi.sys.vendor: Dell Inc.

[roelof van der kleij (r-g-van-der-kleij)]

FYI:
The DFS root is an Windows 2008R2 server
The DFS targets are Netapp Cifs servers

The Ubuntu machine is joined to an Windows 2008R2 domain (upgraded from 2003R2) using winbind with the idmap_ad backend.
We have a faily large domain containing about 60000 accounts.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.6 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. Please only remove that one tag and leave the other tags. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6-quantal/

[roelof van der kleij (r-g-van-der-kleij)]

Will do.

[roelof van der kleij (r-g-van-der-kleij)]

uname -a :
Linux XXXXX 3.6.0-030600-generic #201209302035 SMP Mon Oct 1 00:36:01 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

fixes the problem.

[roelof van der kleij (r-g-van-der-kleij)]

I spoke too soon. One of the other referrals still gives a kernel oops

[roelof van der kleij (r-g-van-der-kleij)]

[ 710.752373] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 710.752436] IP: [< (null)>] (null)
[ 710.752470] PGD 107bc3067 PUD 107bc2067 PMD 0
[ 710.752507] Oops: 0010 [#1] SMP
[ 710.752535] Modules linked in: arc4 nls_utf8 cifs nfsv4 autofs4 rfcomm bnep bluetooth snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm i915 snd_seq_midi rpcsec_gss_krb5 snd_rawmidi snd_seq_midi_event nfsd snd_seq coretemp snd_timer gpio_ich snd_seq_device drm_kms_helper nfs_acl drm auth_rpcgss snd nfs soundcore snd_page_alloc i2c_algo_bit kvm fscache psmouse lpc_ich lockd video mac_hid dell_wmi serio_raw sunrpc hid_generic sparse_keymap ppdev lp wmi microcode dcdbas parport_pc parport usbhid hid tg3
[ 710.752930] CPU 0
[ 710.752947] Pid: 2063, comm: bash Not tainted 3.6.0-030600-generic #201209302035 Dell Inc. OptiPlex 380 /01TKCC
[ 710.753013] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 710.753059] RSP: 0018:ffff880123d09d00 EFLAGS: 00010246
[ 710.753092] RAX: ffffffffa0487bc0 RBX: ffff88010802ac00 RCX: 0000000000000000
[ 710.753132] RDX: 0000000000000003 RSI: ffff88010802ac00 RDI: ffff880110349028
[ 710.753173] RBP: ffff880123d09d28 R08: ffff88012bc17280 R09: ffff880123d09e98
[ 710.753214] R10: ffff880115593000 R11: ffff880110349028 R12: ffff88010e221c00
[ 710.753254] R13: ffff88010e221c00 R14: ffff880123d09e98 R15: 0000000000000000
[ 710.753296] FS: 00007fa1340bb700(0000) GS:ffff88012bc00000(0000) knlGS:0000000000000000
[ 710.753343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 710.753377] CR2: 0000000000000000 CR3: 0000000107bc0000 CR4: 00000000000407f0
[ 710.753418] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 710.753459] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 710.753501] Process bash (pid: 2063, threadinfo ffff880123d08000, task ffff88010d655b40)
[ 710.753547] Stack:
[ 710.753561] ffffffff8119071d ffff88010e221c00 ffff88010e221c00 ffff880123d09e98
[ 710.753614] 0000000000000003 ffff880123d09d58 ffffffff81190b88 ffff880123d09d58
[ 710.753666] 01ffffff8169bdcd ffff880123d09e88 ffff880123d09dd8 ffff880123d09d98
[ 710.753718] Call Trace:
[ 710.753740] [<ffffffff8119071d>] ? lookup_real+0x1d/0x60
[ 710.753775] [<ffffffff81190b88>] __lookup_hash+0x38/0x50
[ 710.753810] [<ffffffff8168a397>] lookup_slow+0x47/0xad
[ 710.753843] [<ffffffff81193176>] path_lookupat+0x236/0x790
[ 710.753877] [<ffffffff81190ef5>] ? putname+0x35/0x50
[ 710.753909] [<ffffffff811960f3>] ? user_path_at_empty+0x63/0xa0
[ 710.753947] [<ffffffff81179bcd>] ? kmem_cache_alloc+0x12d/0x150
[ 710.753984] [<ffffffff81190dd7>] ? getname_flags+0x37/0x100
[ 710.754019] [<ffffffff81193701>] do_path_lookup+0x31/0xc0
[ 710.754054] [<ffffffff811960e9>] user_path_at_empty+0x59/0xa0
[ 710.754090] [<ffffffff8118abdb>] ? vfs_stat+0x1b/0x20
[ 710.754122] [<ffffffff8118ad1a>] ? sys_newstat+0x1a/0x40
[ 710.754156] [<ffffffff81196141>] user_path_at+0x11/0x20
[ 710.754190] [<ffffffff81184ae3>] sys_chdir+0x23/0x90
[ 710.754222] [<ffffffff816a6a69>] system_call_fastpath+0x16/0x1b
[ 710.75425...

[roelof van der kleij (r-g-van-der-kleij)]

requestkey.conf:
#OP TYPE DESCRIPTION CALLOUT INFO PROGRAM ARG1 ARG2 ARG3 ...
#====== ======= =============== =============== ===============================
create dns_resolver * * /sbin/key.dns_resolver %k
create user debug:* negate /bin/keyctl negate %k 30 %S
create user debug:* rejected /bin/keyctl reject %k 30 %c %S
create user debug:* expired /bin/keyctl reject %k 30 %c %S
create user debug:* revoked /bin/keyctl reject %k 30 %c %S
create user debug:loop:* * |/bin/cat
create user debug:* * /usr/share/keyutils/request-key-debug.sh %k %d %c %S
create dns_resolver * * /usr/sbin/cifs.upcall %k
negate * * * /bin/keyctl negate %k 30 %S
create cifs.spnego * * /usr/sbin/cifs.upcall -l -t -c %k
#create cifs.spnego * * /usr/sbin/cifs.upcall -t -c %k

[penalvch (penalvch)]

roelof van der kleij, did this issue not occur in a release prior to Precise?

[roelof van der kleij (r-g-van-der-kleij)]

I cannot say. I only recently managed to join precise to our production domain, so I could not really test DFS functionality before that.

If you have any specific kernel versions you would like me to test, just let me know.

[penalvch (penalvch)]

roelof van der kleij, for regression testing purposes, could you please test for this problem in Lucid via http://releases.ubuntu.com/lucid/ ?

[roelof van der kleij (r-g-van-der-kleij)]

I'll see what I can do. Lucid contains a few other netapp, kerberos and samba related bugs which may make testing difficult.

Roelof

[penalvch (penalvch)]

roelof van der kleij, If trying Lucid is too onerous, could you please try the Lucid kernel in Precise and comment on the results via https://launchpad.net/ubuntu/lucid/+source/linux ?

[roelof van der kleij (r-g-van-der-kleij)]

with 3.2.0-23, the bug is already there

with 2.6.32-44-generic, the multiuser mount option does not exist, so I cannot see the dfs referral for a user logon.

[penalvch (penalvch)]

roelof van der kleij, darn. Could you please try the oldest kernel in Precise via https://launchpad.net/ubuntu/precise/+source/linux ?

[roelof van der kleij (r-g-van-der-kleij)]

I tried a few versions, but when going back older bugs are starting to bite..

3.2.0-2-generic: mount works, but bug present

3.1.0-2-generic: cannot mount at all
[ 20.341214] CIFS VFS: Error connecting to socket. Aborting operation
[ 20.356165] CIFS VFS: cifs_mount failed w/return code = -111
[ 20.524062] CIFS VFS: cifs_mount failed w/return code = -6
[ 21.856172] CIFS VFS: cifs_mount failed w/return code = -6
[ 23.504084] CIFS VFS: cifs_mount failed w/return code = -6
[ 24.652065] CIFS VFS: cifs_mount failed w/return code = -6
[ 101.064066] CIFS VFS: cifs_mount failed w/return code = -6
[ 101.236092] CIFS VFS: cifs_mount failed w/return code = -6

3.0.0-12 : mount fails
[ 17.362126] CIFS VFS: Error connecting to socket. Aborting operation
[ 17.376098] CIFS VFS: cifs_mount failed w/return code = -111
[ 17.400355] CIFS VFS: Send error in SessSetup = -126
[ 17.420072] CIFS VFS: cifs_mount failed w/return code = -126
[ 19.652064] CIFS VFS: cifs_mount failed w/return code = -6
[ 21.860094] CIFS VFS: cifs_mount failed w/return code = -6
[ 22.548085] CIFS VFS: cifs_mount failed w/return code = -6
[ 23.732120] CIFS VFS: cifs_mount failed w/return code = -6

[penalvch (penalvch)]

roelof van der kleij, thank you for testing prior Ubuntu kernels. Could you please test http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.7-rc2-raring/ ?

[roelof van der kleij (r-g-van-der-kleij)]

[ 187.632058] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 187.632117] IP: [< (null)>] (null)
[ 187.632151] PGD 102735067 PUD 102734067 PMD 0
[ 187.632190] Oops: 0010 [#1] SMP
[ 187.632218] Modules linked in: autofs4 arc4 nls_utf8 cifs nfsv4 rfcomm bnep bluetooth snd_hda_codec_realtek rpcsec_gss_krb5 snd_hda_intel nfsd snd_hda_codec nfs_acl coretemp snd_hwdep auth_rpcgss snd_pcm i915 gpio_ich nfs snd_seq_midi fscache snd_rawmidi kvm lockd snd_seq_midi_event snd_seq drm_kms_helper snd_timer drm hid_generic snd_seq_device sunrpc lpc_ich snd psmouse i2c_algo_bit soundcore usbhid snd_page_alloc hid dell_wmi serio_raw mac_hid sparse_keymap ppdev lp video microcode dcdbas wmi parport_pc parport tg3
[ 187.632629] CPU 1
[ 187.632646] Pid: 2143, comm: bash Not tainted 3.7.0-030700rc2-generic #201210220428 Dell Inc. OptiPlex 380 /01TKCC
[ 187.632713] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 187.632759] RSP: 0018:ffff880122829d00 EFLAGS: 00010246
[ 187.632791] RAX: ffffffffa04a3bc0 RBX: ffff8801060c0480 RCX: 0000000000000000
[ 187.632832] RDX: 0000000000000003 RSI: ffff8801060c0480 RDI: ffff8801102d50d8
[ 187.632872] RBP: ffff880122829d28 R08: ffff88012bc57340 R09: ffffc90000002000
[ 187.632912] R10: 0000000000000000 R11: ffff8801102d50d8 R12: ffff8801060c0780
[ 187.632952] R13: ffff8801060c0780 R14: ffff880122829e98 R15: 0000000000000000
[ 187.632993] FS: 00007f61f6026700(0000) GS:ffff88012bc40000(0000) knlGS:0000000000000000
[ 187.633039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 187.633072] CR2: 0000000000000000 CR3: 0000000102732000 CR4: 00000000000407e0
[ 187.633112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 187.633153] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 187.633194] Process bash (pid: 2143, threadinfo ffff880122828000, task ffff880106be0000)
[ 187.633240] Stack:
[ 187.633253] ffffffff8119b55d ffff8801060c0780 ffff8801060c0780 ffff880122829e98
[ 187.633306] 0000000000000003 ffff880122829d58 ffffffff8119b9c8 ffff880122829d58
[ 187.633358] 01ffffff816c4c1d ffff880122829e88 ffff880122829dd8 ffff880122829d98
[ 187.633410] Call Trace:
[ 187.633431] [<ffffffff8119b55d>] ? lookup_real+0x1d/0x60
[ 187.633465] [<ffffffff8119b9c8>] __lookup_hash+0x38/0x50
[ 187.633499] [<ffffffff816b25e8>] lookup_slow+0x47/0xab
[ 187.633531] [<ffffffff8119e0e6>] path_lookupat+0x236/0x7a0
[ 187.633566] [<ffffffff8117edb1>] ? kmem_cache_alloc+0x31/0x140
[ 187.633602] [<ffffffff8119cdd0>] ? getname_flags.part.23+0x30/0x150
[ 187.633641] [<ffffffff8119e684>] filename_lookup+0x34/0xc0
[ 187.633675] [<ffffffff8119cf56>] ? getname_flags+0x66/0x80
[ 187.633709] [<ffffffff811a1149>] user_path_at_empty+0x59/0xa0
[ 187.633745] [<ffffffff811968fb>] ? vfs_stat+0x1b/0x20
[ 187.633776] [<ffffffff81196a4a>] ? sys_newstat+0x1a/0x40
[ 187.633810] [<ffffffff811a11a1>] user_path_at+0x11/0x20
[ 187.633843] [<ffffffff81190943>] sys_chdir+0x23/0x90
[ 187.633875] [<ffffffff816cff59>] system_call_fastpath+0x16/0x1b
[ 187.633909] Code: Bad RIP value.
[ 187.633938] RIP [< ...

[roelof van der kleij (r-g-van-der-kleij)]

Hi Christopher

The kernel was tested on a fresh install.

[roelof van der kleij (r-g-van-der-kleij)]

I submitted a bug at samba.org

https://bugzilla.samba.org/show_bug.cgi?id=9364

[penalvch (penalvch)]

roelof van der kleij, as per http://www.dell.com/support/drivers/us/en/19/Product/optiplex-380 an update is available for your BIOS (A07). If you update to this following https://help.ubuntu.com/community/BiosUpdate , does it change anything? If it doesn't, could you please both specify what happened, and just provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

Please note your current BIOS is already in the Bug Description, so posting this on the old BIOS would not be helpful.

For more on BIOS updates and linux, please see https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette .

Thank you for your understanding.