NFS regression causes subsequent mounts from same superblock to silently use previous mount options

[Philip Walls]

Two or more NFS mount points with the same FSID end up silently sharing mount options.

You have an NFS server with the following mount:

  /dev/sda3 => /mnt/data

And then you export two folders from that block device separately via NFS (/etc/exports):

  /mnt/data/a 10.0.0.0/16
  /mnt/data/b 10.0.0.0/16

And on the client you mount them:

mount -t nfs server:/mnt/data/a -o rw /exports/a
mount -t nfs server:/mnt/data/b -o ro /exports/b

You will notice that silently, the second partition gets mounted as 'rw' instead of 'ro'. This is a regression from Edgy (also in Feisty) and poses a potential security risk to system administrators, who may not notice that the second device has been mounted 'rw'. In fact, /etc/mtab (and by extension '/bin/mount') reports that the filesystem was mounted 'ro' (only '/proc/mounts' will tip you otherwise).

Two fixes were released to mitigate this problem:

   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c98451bdb2f3e6d6cc1e03adad641e9497512b49
   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=e89a5a43b95cdc4305b7c8e8121a380f02476636

The first commit fixed the security aspect of the problem but actually introduced an even more serious regression which resulted in EBUSY when two devices were mounted from the same superblock without specifying the 'nosharecache' option. The second patch resolves the security issue, and automatically disables shared caching on exports with the same FSID.

A high level overview of this problem with accompanying LKML thread can be found here:
  http://kerneltrap.org/Linux/NFS_Regression

This bug is fixed as of 2.6.23-r5

[Leann Ogasawara (leannogasawara)]

Hi Philip,

Thank you for your report. Just to clarify, this is also a security issue for Feisty correct? Thanks.

[Philip Walls (ubuntu-rabidgeek)]

Leann,

Yes, the problem is also reproducible in Feisty.

[Philip Walls (ubuntu-rabidgeek)]

It should be obvious from my description above, but I wanted to specifically mention that this regression makes it impossible to mount two directories from the same block device with different options. An example of a case that worked in Edgy but no longer works in Feisty/Gutsy is using sec=sys for one mount point and sec=krb5 for another.

I've attached a tested patch which applies cleanly to linux-source-2.6.22.9-14 and resolves the issue on Gutsy. Note, however, that I've only tested the patch with an NFSv3 client.

[Philip Walls (ubuntu-rabidgeek)]

The patch above also applies cleanly to Feisty's linux-source-2.6.20-16.32 kernel and builds. The system boots successfully and this bug is not reproducible with the patched kernel. Other NFSv3 mounts continue to work based on my limited testing.

[Philip Walls (ubuntu-rabidgeek)]

I backported the patch above to the Gutsy kernel based on the patch at:
   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=e89a5a43b95cdc4305b7c8e8121a380f02476636

[Leann Ogasawara (leannogasawara)]

I just wanted to add a note that these patches were automatically merged into the Hardy Heron kernel when it was sync'd with mainline. Thanks!

[Philip Walls (ubuntu-rabidgeek)]

Leanne,

It has been a few weeks since I submitted this bug and I haven't heard anything from the teams responsible for it. I did my best to hand over the bug on a silver platter (at the cost of a fair deal of my own time), so a couple answers would set my head at ease:

- Has the bug been confirmed? As a security issue?
- Is someone working on or planning on getting an update out for Gutsy? Feisty?
- Are there any updates at all from the kernel team?

Thanks,
Philip

[Leann Ogasawara (leannogasawara)]

Hi Philip,

I apologize for any lack of communication/updates regarding your report. The effort and time you have invested in getting this issue reosolved has not been overlooked and has been extremely helpful to us. I hope I can answer some of your questions:

1) Yes this bug has been confirmed and is a security issue.
2) Yes, a member of the kernel team is actively backporting this fix to Gutsy. For some reason I thought they had already posted a comment and updated you on this report.
3) I'll nudge the kernel team again to post a comment here regarding the current status.

Again, thank you so much for your help and your continued intersted in getting this resolved.

[Philip Walls (ubuntu-rabidgeek)]

Leanne,

Thanks a lot for the update. It's good to know that a fix is in the pipeline. I hope my previous message didn't come across as too snarky.

[Phillip Lougher (phillip-lougher)]

From the kernel team. Thanks for your work on this bug. Your patch has been evaluated and integrated into the kernel. It should be released very soon in the next lum security updates.

[Ben Collins (ben-collins)]

2.6.22 wont be in hardy.

[Ben Collins (ben-collins)]

Phillip, can you cleanup the Affects/targets for this bug. It's hard to see where it's fixed already. Note, "Affects: linux" should only be for hardy at the moment.

[Kees Cook (kees)]

I've cleaned up the tasks. This will be published as part of USN-558-1 shortly.

[Claude Boucher (bouchecl)]

Was this fix applied in kernel 2.6.22-14.47?

Two people reported kernel oops this mornings with the new kernel released on Dec. 18 and nfs4 shares. See

https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/148600/comments/3

https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/148600/comments/4

[Philip Walls (ubuntu-rabidgeek)]

Claude,

I apologize for missing an important part of the patch in my original submission which caused the NFSv4 regression that you are experiencing. If I'd know that Canonical wasn't going to do quality assurance of kernel patches to their stable release branch (despite my warning that I'd only tested NFSv3 clients) I would have tested the NFSv4 portion of the code myself.

After seeing this report, I set up an NFSv4 server to test, reproduce and fix the problem with my patch. NFSv4 and NFSv4 + Kerberos mounts are now working with the revised patch (attached). Note: this patch applies to the 2.6.22-14.46 kernel. I will attach a patch to 2.6.22-14.47 that fixes this regression momentarily.

Thanks,
Philip

[Philip Walls (ubuntu-rabidgeek)]

Reopening due to introduction of NFSv4 problem in patch

[Claude Boucher (bouchecl)]

Thanks Philip,

Don't feel too bad... we all make mistakes :) I reverted to 21.6.22-14.46 temporarily.

[Kees Cook (kees)]

The regression is being investigated. Sorry about the trouble; we should have a solution published shortly. For anyone that needs to roll-back to the earlier kernel, you can use:

 IMGPKG=$(dpkg -S /boot/vmlinuz-2.6.22-14* | cut -d: -f1)
 sudo apt-get install $IMGPKG=2.6.22-14.46

(Basically, force a down-grade of the kernel-image-2.6.22 flavour that is installed)

[Ben Collins (ben-collins)]

Oops, ignore that attachment...wrong bug

[Philip Walls (ubuntu-rabidgeek)]

As promised, here is the fix to unbreak NFSv4 in the 2.6.22-14.47 kernel.

I independently tested:
- Mounting NFSv4 export (sec=sys and sec=krb5)
- Reading/writing from partition, including seeks (tail)

Claude,

If you are willing to test the patch (or a new nfs.ko or linux-image-2.6.22-14-generic.deb, disclaiming the potential danger of running untrusted kernel code) in your setup to ensure that it fixes the problem you are experiencing, please contact me on IRC (<email address hidden>:#ubuntu-bugs) or respond here and I'll upload the fixed module to this bug.

Kees,

Feel free to contact me on IRC if you have any questions.

Thanks,
Philip

[Claude Boucher (bouchecl)]

OK Philip,

I'm trying to reach you through IRC as we speak.

[Philip Walls (ubuntu-rabidgeek)]

Here's the nfs.ko compiled against 2.6.22-14.47. Make sure that if you use this you first back up the current one in /lib/modules. You can attempt to reinsert the module into your running kernel or reboot.

[Philip Walls (ubuntu-rabidgeek)]

Claude tested the above patched module and reported that everything worked well.

[Charles Elwood (charles-technomancer)]

Just tried the rebuilt nfs.ko linked above, it appears to work. However it seems to be causing issues with the nvidia propriety module, it won't load. I'll post the specifics when I have a second box running to post from.

[Charles Elwood (charles-technomancer)]

Ignore above problem, tracked it down to something else entirely

[Philip Walls (ubuntu-rabidgeek)]

Charles,

You may need to run 'dpkg-reconfigure linux-restricted-modules-2.6.22-14-generic' to rebuild your module list to include the volatile modules ('depmod -a' on a fully booted system should also work). You can check to see if this is the cause by comparing the output of the following command with your machine:

$ modprobe -l |grep nvidia.ko
/lib/modules/2.6.22-14-generic/volatile/nvidia.ko

[DavidP (dperson)]

The patched kernel module resolved the problem for me. Thank you.

-- David P.

[BB||negBB (bernhard+ubuntu)]

Is there already a plan to release the fix for NFSv4 problem? I only ask as there haven't been any changes for almost a month now. We would like to migrate a bunch of machines to Ubuntu, and having a working default kernel would save us quite some work.

Thanks in advance!

[Karel Gardas (karel-gardas)]

Philip,

is it possible to push your patch to the Ubuntu distro pool? I'm using linux-image-2.6.22-14-xen 2.6.22-14.47 and also would like to get NFSv4 working as I'm using some Solaris in domU and would like to share ZFS-based storage from it with dom0 Ubuntu.

Thanks!
Karel
PS: otherwise, is it hard to rebuild Ubuntu kernel? How is it different from Debian?

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-source-2.6.22 - 2.6.22-14.51

---------------
linux-source-2.6.22 (2.6.22-14.51) gutsy-security; urgency=low

  [Amit Kucheria]

  * Poulsbo: Mass update of all patches from moblin repo
  * Update config.lpia to reflect new patches
  * [sata_sil][sata->ide-bridg] failed to set xfermode
    - LP: #153096
  * Poulsbo: remove extra patch

  [Kees Cook]

  * fix NFSv4 client mount regression
    - LP: #164231

  [Tim Gardner]

  * Support of new AMD PowerNow! (family 0x11 and beyond)
    - LP: #185649

  [Upstream Kernel Changes]

  * minixfs: limit minixfs printks on corrupted dir i_size (CVE-2006-6058)
  * [JFFS2] Fix ACL vs. mode handling. (CVE-2007-4849)
  * [IEEE80211]: avoid integer underflow for runt rx frames (CVE-2007-4997)
  * [TCP]: Make sure write_queue_from does not begin with NULL ptr
    (CVE-2007-5501)
  * wait_task_stopped: Check p->exit_state instead of TASK_TRACED
    (CVE-2007-5500)
  * fix DLM regression
  * CVE-2008-0001: Use access mode instead of open flags to determine
    needed permissions
  * hrtimers: avoid overflow for large relative timeouts (CVE-2007-5966)
  * isdn: avoid copying overly-long strings (CVE-2007-6063)
  * I4L: fix isdn_ioctl memory overrun vulnerability (CVE-2007-6151)
  * vfs: coredumping fix (CVE-2007-6206)
  * tmpfs: restore missing clear_highpage (CVE-2007-6417)
  * [UBUNTU] fs/dlm: Fix regression introduced with last security fix.

 -- Tim Gardner <email address hidden> Mon, 28 Jan 2008 13:46:21 -0700

[Jamie Strandboge (jdstrand)]

This bug was fixed in 2.6.20-16.34

---
 linux-source-2.6.20 (2.6.20-16.34) feisty-security; urgency=low

   [Kees Cook]

   * fix NFSv4 client mount regression
     - GIT-SHA 7a9e181ce37e0a862261b1814ee3ef358036dd5e
     - Bug #164231
   * ppc64: fix corrupted sigcontext during FPU stress (CVE-2007-3107)
     - GIT-SHA 7117942afa58a6331cc2abc498541b2784cf79ac

   [Upstream Kernel Changes]

   * minixfs: limit minixfs printks on corrupted dir i_size (CVE-2006-6058)
   * [IPV6]: Do no rely on skb->dst before it is assigned. (CVE-2007-4567)
   * [JFFS2] Fix ACL vs. mode handling. (CVE-2007-4849)
   * [IEEE80211]: avoid integer underflow for runt rx frames (CVE-2007-4997)
   * USB: fix DoS in pwc USB video driver (CVE-2007-5093)
   * Fix debug regression in video/pwc
   * wait_task_stopped: Check p->exit_state instead of TASK_TRACED
     (CVE-2007-5500)
   * fix DLM regression
   * CVE-2008-0001: Use access mode instead of open flags to determine
     needed permissions
   * hrtimers: avoid overflow for large relative timeouts (CVE-2007-5966)
   * isdn: avoid copying overly-long strings (CVE-2007-6063)
   * I4L: fix isdn_ioctl memory overrun vulnerability (CVE-2007-6151)
   * vfs: coredumping fix (CVE-2007-6206)
   * tmpfs: restore missing clear_highpage (CVE-2007-6417)
   * [UBUNTU] fs/dlm: Fix regression introduced with last security fix.

[Philip Walls (ubuntu-rabidgeek)]

Unless I'm mistaken, this bug is already resolved in Hardy since Hardy received the upstream kernel patch automatically.