deep-mounting below symlinked directory in cifs drive causes oops

[Kjell Braden]

== Regression details ==
Discovered in version: 3.2.0
Last known good version: 3.0.0

This issue was introduced in commit fec11dd9a0109fe52fd631e5c510778d6cbff6cc.

Mounting a cifs path which is below a symlink on the server causes oops. Consider the following tree:

 /srv/symtest
 /srv/symtest/dir
 /srv/symtest/dir/subdir
 /srv/symtest/link -> dir

this works:
  # mount.cifs //smbsrv/symtest/dir/subdir/ /mnt/

this causes oops:
  # mount.cifs //smbsrv/symtest/link/subdir/ /mnt/

A example of the oops may be found in https://launchpadlibrarian.net/125982530/CurrentDmesg.txt .

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-34-generic 3.2.0-34.53
ProcVersionSignature: Ubuntu 3.2.0-34.53-generic 3.2.33
Uname: Linux 3.2.0-34-generic x86_64
NonfreeKernelModules: nvidia
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu15
Architecture: amd64
ArecordDevices: **** List of CAPTURE Hardware Devices ****
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/hwC0D1', '/dev/snd/hwC0D2', '/dev/snd/hwC0D3', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D7p', '/dev/snd/pcmC0D8p', '/dev/snd/pcmC0D9p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'NVidia'/'HDA NVidia at 0xfbdfc000 irq 58'
   Mixer name : 'Nvidia GPU 10 HDMI/DP'
   Components : 'HDA:10de0010,10de0101,00100100'
   Controls : 24
   Simple ctrls : 4
Date: Mon Dec 17 11:41:21 2012
HibernationDevice: RESUME=UUID=b4ff9451-106e-4de4-9ef3-cbb27784222b
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
MachineType: Dell Inc. Precision WorkStation T7400
MarkForUpload: True
ProcEnviron:
 LANGUAGE=en_US:en
 TERM=screen
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-34-generic root=UUID=73221f0f-80e8-40ff-bcae-ec4f29a697fd ro quiet splash
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-34-generic N/A
 linux-backports-modules-3.2.0-34-generic N/A
 linux-firmware 1.79.1
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/31/2008
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A01
dmi.board.name: 0RW199
dmi.board.vendor: Dell Inc.
dmi.chassis.type: 7
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA01:bd01/31/2008:svnDellInc.:pnPrecisionWorkStationT7400:pvr:rvnDellInc.:rn0RW199:rvr:cvnDellInc.:ct7:cvr:
dmi.product.name: Precision WorkStation T7400
dmi.sys.vendor: Dell Inc.

[Kjell Braden (afflux)]

subscribed as per bug 1018852

[Brad Figg (brad-figg)]

This change was made by a bot.

[penalvch (penalvch)]

Kjell Braden, could you please test the latest upstream kernel available following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Please do not test the kernel in the mainline kernels archive directory daily folder, but the one all the way at the bottom. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested. For example:
kernel-fixed-upstream-v3.7

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description. As well, please remove the tag:
needs-upstream-testing

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

As well, please remove the tag:
needs-upstream-testing

If you are unable to test the mainline kernel, please comment as to why specifically you were unable to test it and add the following tags:
kernel-unable-to-test-upstream
kernel-unable-to-test-upstream-VERSION-NUMBER

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

Helpful bug reporting tips:
https://help.ubuntu.com/community/ReportingBugs

[Kjell Braden (afflux)]

Hi Christopher,

did the following:

1. tested the newest kernel with -precise suffix (v.3.4-precise)
    => same oops trace
2. realized there was a newer kernel in precise-security (3.5.0-19-generic), tested it
    => same oops trace
3. tried v3.7-raring
    => did not bring up network, keyboard input not working either, could not test failing mount

Let me know if I should try anything else, I won't be able to provide further feedback until Jan 14, 2013 though.

[penalvch (penalvch)]

Kjell Braden, thank you for attempting to test the newest mainline kernel. Could you please test http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6.11-raring/ ?

If this test in inconclusive due to your network not being brought up, etc., could you please test http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5.7.2-quantal/ ?

[Kjell Braden (afflux)]

Will do when I get back on site - January.
Thanks for your help.

[Kjell Braden (afflux)]

Tried both kernels on my machines, on 3.6.11 my network and keyboard were dead, 3.5.7.2 didn't boot at all (kernel panic in initramfs).

I've downloaded kubuntu raring alpha 1 with linux-image-3.7.0-4-generic (3.7.0-4.12) and the problem persists, though with a slightly different call trace:

[ 253.542823] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 253.542846] IP: [< (null)>] (null)
[ 253.542854] PGD 231c40067 PUD 225df9067 PMD 0
[ 253.542863] Oops: 0010 [#1] SMP
[ 253.542870] Modules linked in: des_generic(F) md4(F) nls_utf8 cifs(F) fscache(F) dm_crypt(F) snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep(F) snd_pcm(F) coretemp snd_page_alloc(F) snd_seq_midi(F) snd_seq_midi_event(F) gpio_ich kvm snd_rawmidi(F) dm_multipath(F) scsi_dh lpc_ich snd_seq(F) dcdbas snd_seq_device(F) snd_timer(F) i5400_edac mac_hid edac_core serio_raw(F) parport_pc(F) bnep snd(F) rfcomm microcode(F) i5k_amb ppdev(F) shpchp soundcore(F) bluetooth lp(F) parport(F) squashfs(F) overlayfs(F) nls_iso8859_1(F) dm_mirror(F) dm_region_hash(F) dm_log(F) hid_generic usbhid hid usb_storage(F) uas nouveau mxm_wmi wmi video(F) i2c_algo_bit firewire_ohci ttm firewire_core drm_kms_helper crc_itu_t(F) drm mptsas(F) mptscsih(F) mptbase(F) scsi_transport_sas(F) tg3
[ 253.542995] CPU 3
[ 253.542999] Pid: 5407, comm: mount.cifs Tainted: GF 3.7.0-4-generic #12-Ubuntu Dell Inc. Precision WorkStation T7400 /0RW199
[ 253.543008] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 253.543015] RSP: 0018:ffff88022edd3cf8 EFLAGS: 00010246
[ 253.543019] RAX: ffffffffa04ce6c0 RBX: ffff8801dcfe7780 RCX: 0000000000000004
[ 253.543024] RDX: 0000000000000000 RSI: ffff8801dcfe7780 RDI: ffff8801dcfa0340
[ 253.543029] RBP: ffff88022edd3d10 R08: 00000000000172e0 R09: ffff88023fcd6ba0
[ 253.543034] R10: ffffea0008c66a40 R11: ffffffffa04a9738 R12: ffff8801dcfe70c0
[ 253.543040] R13: ffff8801dcfe70c0 R14: ffff880231920831 R15: ffff88023192082a
[ 253.543045] FS: 00007f6cc16c2740(0000) GS:ffff88023fcc0000(0000) knlGS:0000000000000000
[ 253.543051] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 253.543056] CR2: 0000000000000000 CR3: 000000022f450000 CR4: 00000000000007e0
[ 253.543061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 253.543066] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 253.543072] Process mount.cifs (pid: 5407, threadinfo ffff88022edd2000, task ffff880231ecdc00)
[ 253.543077] Stack:
[ 253.543080] ffffffff81195fdd ffff880231920831 0000000000000000 ffff88022edd3d38
[ 253.543089] ffffffff811961e8 01ff8801dcfe70c0 ffff880231920831 0000000000000006
[ 253.543097] ffff88022edd3d70 ffffffff81197046 0000000775c6ded2 ffff88023192082a
[ 253.543105] Call Trace:
[ 253.543113] [<ffffffff81195fdd>] ? lookup_real+0x1d/0x60
[ 253.543120] [<ffffffff811961e8>] __lookup_hash+0x38/0x50
[ 253.543125] [<ffffffff81197046>] lookup_one_len+0xd6/0x110
[ 253.543138] [<ffffffffa049600c>] cifs_do_mount+0x28c/0x4c0 [cifs]
[ 253.543146] [<ffffffff811902d3>] mount_fs+0x43/0x1b0
[ 253.543152] [<ffffffff811aa7f4>] vfs_kern_mount+0x74/0x110
[...

[Kjell Braden (afflux)]

(I know this wasn't a mainline kernel, but I can't run a clean install here and I don't think I can test them on a live cd)

[penalvch (penalvch)]

Kjell Braden, thank you for testing the requested mainline kernels and Raring. Could you please test http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc2-raring/ ?

[Kjell Braden (afflux)]

Please refer to comment #9.

[Kjell Braden (afflux)]

It took me some hours to set up a vm and working around a few bugs (eg. bug 935585), but finally I got the mainline kernel working.

[ 1096.823058] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 1096.823064] IP: [< (null)>] (null)
[ 1096.823067] PGD 17c37067 PUD 929e067 PMD 0
[ 1096.823069] Oops: 0010 [#1] SMP
[ 1096.823072] Modules linked in: arc4 md4 nls_utf8 cifs fscache nf_conntrack_ipv6 nf_defrag_ipv6 xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack vboxvideo(OF) drm ip6table_filter ip6_tables iptable_filter ip_tables x_tables bnep rfcomm bluetooth lp snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device joydev snd_timer hid_generic psmouse usbhid snd ppdev hid serio_raw soundcore parport_pc i2c_piix4 parport microcode mac_hid e1000
[ 1096.823092] CPU 0
[ 1096.823096] Pid: 7375, comm: mount.cifs Tainted: GF O 3.8.0-030800rc2-generic #201301022235 innotek GmbH VirtualBox/VirtualBox
[ 1096.823098] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 1096.823100] RSP: 0018:ffff88000929dc70 EFLAGS: 00010246
[ 1096.823101] RAX: ffffffffa02a56c0 RBX: ffff88000d559e40 RCX: 0000000000000000
[ 1096.823102] RDX: 0000000000000000 RSI: ffff88000d559e40 RDI: ffff880007142350
[ 1096.823103] RBP: ffff88000929dc98 R08: ffff88001fc17380 R09: ffffc90000000000
[ 1096.823104] R10: ffffffffa02818b5 R11: 0000000007e44712 R12: ffff88000d5599c0
[ 1096.823105] R13: ffff88001b6c2651 R14: ffff88001b6c2640 R15: ffff8800071423f8
[ 1096.823109] FS: 00007fd220cfe740(0000) GS:ffff88001fc00000(0000) knlGS:0000000000000000
[ 1096.823110] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1096.823111] CR2: 0000000000000000 CR3: 0000000003d19000 CR4: 00000000000006f0
[ 1096.823114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1096.823120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1096.823122] Process mount.cifs (pid: 7375, threadinfo ffff88000929c000, task ffff880000ff2e40)
[ 1096.823122] Stack:
[ 1096.823157] ffffffff811a266d ffff88000d5599c0 ffff88001b6c2651 ffff88001b6c2640
[ 1096.823160] 0000000000000000 ffff88000929dcc8 ffffffff811a2ad8 ffff88001b6c2651
[ 1096.823162] 01ff88000d5599c0 ffff88001b6c2651 ffff88000d5599c0 ffff88000929dd08
[ 1096.823164] Call Trace:
[ 1096.823169] [<ffffffff811a266d>] ? lookup_real+0x1d/0x60
[ 1096.823171] [<ffffffff811a2ad8>] __lookup_hash+0x38/0x50
[ 1096.823174] [<ffffffff811a6b7e>] lookup_one_len+0xce/0x120
[ 1096.823181] [<ffffffffa026d7e1>] cifs_get_root+0x111/0x190 [cifs]
[ 1096.823186] [<ffffffffa026d9e2>] cifs_do_mount+0x182/0x270 [cifs]
[ 1096.823189] [<ffffffff8119c623>] mount_fs+0x43/0x1b0
[ 1096.823191] [<ffffffff811b73d6>] vfs_kern_mount+0x76/0x120
[ 1096.823193] [<ffffffff811b8851>] do_new_mount+0xb1/0x1e0
[ 1096.823195] [<ffffffff811ba066>] do_mount+0x1b6/0x1f0
[ 1096.823197] [<ffffffff811ba130>] sys_mount+0x90/0xe0
[ 1096.823200] [<ffffffff816f14dd>] system_call_fastpath+0x1a/0x1f
[ 1096.823201] Code: Bad RIP value.
[ 1096.823205] RIP [< (null)>] (null)
[ 1096.823206] RSP...

[Kjell Braden (afflux)]

Some more information: this seems to be related to the mount path being below a symlink.

On smbsrv, /mount/deep is a symlink to /mount/other

this works:
  # mount.cifs //smbsrv/mount/other/path/ /mnt/ -o user=me,dom=mydomain
  Password:
  # umount /mnt

this causes oops:
  # mount.cifs //smbsrv/mount/deep/path/ /mnt/ -o user=me,dom=mydomain
  Password:
  Killed

[penalvch (penalvch)]

Khell Braden, thank you for testing the newest mainline kernel. Did this problem not occur in a release prior to Precise?

[Kjell Braden (afflux)]

My company uses Ubuntu in this configuration since 12.04 so I can't say. I assume it worked fine with an earlier version of OpenSUSE but I'm not certain about this.

I've constructed a test case though:

# apt-get install samba
# cat >>/etc/samba/smb.conf <<DONTCOPYTHIS
[symtest]
  path = /srv/symtest
  guest ok = yes
DONTCOPYTHIS
# mkdir -p /srv/symtest/dir/subdir
# ln -s dir /srv/symtest/link
# reload smbd

Now these mounts (obviously) work:
# mount -t cifs //localhost/symtest /mnt -o guest
# mount -t cifs //localhost/symtest/dir /mnt -o guest
# mount -t cifs //localhost/symtest/dir/subdir /mnt -o guest

This one errors (which is acceptable in my opinion):
# mount -t cifs //localhost/symtest/link/ /mnt -o guest
mount error(22): Invalid argument
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

And this one oopses:
# mount -t cifs //localhost/symtest/link/subdir /mnt -o guest

[penalvch (penalvch)]

Kjell Braden, thank you for your response. Could you please test for this in http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc3-raring/ ?

If reproducible, for regression testing purposes, could you please test for this problem in Lucid via http://releases.ubuntu.com/lucid/ ?

[Kjell Braden (afflux)]

Does not appear in lucid's 2.6.32-38-generic.
Does not appear in natty's 2.6.38-8-generic.
Does not appear in oneiric's 3.0.0-12-generic.

Does appear in precise's 3.2.0-34-generic and every newer version tested, both mainline and ubuntu's.

[penalvch (penalvch)]

Kjell Braden, thank you for testing the newest mainline kernel, and performing a kernel version bisect. As per http://www.dell.com/support/drivers/us/en/19/Product/precision-t7400 a BIOS update is available for your computer (A11). If you update to this, does it change anything?

[Kjell Braden (afflux)]

You got to be kidding me.

No, it doesn't change anything - I replicated every single test on 10 different machines and a virtualbox.

[Kjell Braden (afflux)]

Would you please take five minutes to run the testcase I provided yourself, so realize it's not my hardware and you stop asking me for every rc release if it magically fixed itself?

[penalvch (penalvch)]

Kjell Braden, thank you for now noting this occurs on disparate hardware. The next step would be to bisect this issue from 3.0.0 to 3.2.0, in order to identify the offending commit. Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection ?