Mounting NFS Shares Causes System Freeze After Kernel Update

[Donal]

I just upgraded my Ubuntu Gutsy server and Gutsy Desktop with the latest kernel updates (dated 19-12-2007) and when I try to mount my NFS shares my desktop pc freezes causing me to do a hard reboot everytime.

Anyone else having this problem?

[Alexandre Payment (alp)]

Here a Oops from a computer at startup:

Dec 19 06:08:44 mars kernel: [ 20.086200] BUG: unable to handle kernel NULL pointer dereference at virtual address 00000038
Dec 19 06:08:44 mars kernel: [ 20.086205] printing eip:
Dec 19 06:08:44 mars kernel: [ 20.086207] f8d03bb6
Dec 19 06:08:44 mars kernel: [ 20.086208] *pde = 00000000
Dec 19 06:08:44 mars kernel: [ 20.086211] Oops: 0000 [#1]
Dec 19 06:08:44 mars kernel: [ 20.086212] SMP
Dec 19 06:08:44 mars kernel: [ 20.086215] Modules linked in: nfs lockd sunrpc nls_utf8 ntfs nls_iso8859_1 nls_cp437 vfat fat it87 hwmon_vid i2c_isa eeprom i2c_i801 parport_pc lp parport fuse snd_ice1724 snd_ice17xx_ak4xxx snd_ac97_codec ac97_bus af_packet snd_ak4114 snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_ak4xxx_adda snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer xpad sky2 nvidia(P) snd_seq_device snd i2c_core soundcore intel_agp iTCO_wdt iTCO_vendor_support agpgart shpchp pci_hotplug psmouse serio_raw pcspkr tsdev evdev ext3 jbd mbcache sg sr_mod sd_mod cdrom generic usbhid hid ata_piix pata_jmicron usb_storage libusual ahci ata_generic via_rhine mii libata floppy scsi_mod uhci_hcd ehci_hcd usbcore thermal processor fan fbcon tileblit font bitblit softcursor vesafb capability commoncap
Dec 19 06:08:44 mars kernel: [ 20.086280] CPU: 0
Dec 19 06:08:44 mars kernel: [ 20.086281] EIP: 0060:[<f8d03bb6>] Tainted: P VLI
Dec 19 06:08:44 mars kernel: [ 20.086282] EFLAGS: 00010206 (2.6.20-16-generic #2)
Dec 19 06:08:44 mars kernel: [ 20.086293] EIP is at nfs_fhget+0xc6/0x3d0 [nfs]
Dec 19 06:08:44 mars kernel: [ 20.086295] eax: 00000000 ebx: 00000000 ecx: dfc15a74 edx: 00004180
Dec 19 06:08:44 mars kernel: [ 20.086297] esi: dfc15a00 edi: f504fbd8 ebp: f6653da0 esp: f504fb90
Dec 19 06:08:44 mars kernel: [ 20.086299] ds: 007b es: 007b ss: 0068
Dec 19 06:08:44 mars kernel: [ 20.086301] Process mount (pid: 4823, ti=f504e000 task=df8a5030 task.ti=f504e000)
Dec 19 06:08:44 mars kernel: [ 20.086302] Stack: f8d03a90 f504fbb0 c200ce34 c200ce34 00000000 f6653c78 00000080 00000080
Dec 19 06:08:44 mars kernel: [ 20.086307] f504fc6a f504fbd8 f504fbd8 dfc15a00 f504fc68 f504fd22 f8d024b7 00000000
Dec 19 06:08:44 mars kernel: [ 20.086312] 00000000 dfc15800 00000002 00000000 00000000 00000000 00000000 00000000
Dec 19 06:08:44 mars kernel: [ 20.086316] Call Trace:
Dec 19 06:08:44 mars kernel: [ 20.086320] [<f8d03a90>] nfs_init_locked+0x0/0x60 [nfs]
Dec 19 06:08:44 mars kernel: [ 20.086339] [<f8d024b7>] nfs4_get_root+0x177/0x260 [nfs]
Dec 19 06:08:44 mars kernel: [ 20.086371] [<f8d059bd>] nfs4_get_sb+0x28d/0x350 [nfs]
Dec 19 06:08:44 mars kernel: [ 20.086398] [vfs_kern_mount+182/304] vfs_kern_mount+0xb6/0x130
Dec 19 06:08:44 mars kernel: [ 20.086405] [do_kern_mount+57/96] do_kern_mount+0x39/0x60
Dec 19 06:08:44 mars kernel: [ 20.086409] [do_mount+762/1808] do_mount+0x2fa/0x710
Dec 19 06:08:44 mars kernel: [ 20.086415] [ip_rcv+700/1280] ip_rcv+0x2bc/0x500
Dec 19 06:08:44 mars kernel: [ 20.086420] [<f8af42ca>] packet_rcv_spkt+0x10a/0x1b0 [af_packet]
Dec 19 06:08:44 mar...

[Alexandre Payment (alp)]

This is with Feisty not in Gusty like Donal (Bug reporter).

After a few Oops I got that:

Dec 19 06:09:02 mars kernel: [ 39.710293] BUG: soft lockup detected on CPU#2!
Dec 19 06:09:02 mars kernel: [ 39.710315] [softlockup_tick+156/240] softlockup_tick+0x9c/0xf0
Dec 19 06:09:02 mars kernel: [ 39.710333] [pdflush+0/512] pdflush+0x0/0x200
Dec 19 06:09:02 mars kernel: [ 39.710338] [update_process_times+51/128] update_process_times+0x33/0x80
Dec 19 06:09:02 mars kernel: [ 39.710349] [smp_apic_timer_interrupt+112/128] smp_apic_timer_interrupt+0x70/0x80
Dec 19 06:09:02 mars kernel: [ 39.710358] [apic_timer_interrupt+40/48] apic_timer_interrupt+0x28/0x30
Dec 19 06:09:02 mars kernel: [ 39.710369] [pdflush+0/512] pdflush+0x0/0x200
Dec 19 06:09:02 mars kernel: [ 39.710380] [_spin_lock+7/16] _spin_lock+0x7/0x10
Dec 19 06:09:02 mars kernel: [ 39.710390] [sync_supers+12/192] sync_supers+0xc/0xc0
Dec 19 06:09:02 mars kernel: [ 39.710398] [wb_kupdate+36/240] wb_kupdate+0x24/0xf0
Dec 19 06:09:02 mars kernel: [ 39.710412] [pdflush+294/512] pdflush+0x126/0x200
Dec 19 06:09:02 mars kernel: [ 39.710421] [wb_kupdate+0/240] wb_kupdate+0x0/0xf0
Dec 19 06:09:02 mars kernel: [ 39.710431] [kthread+186/240] kthread+0xba/0xf0
Dec 19 06:09:02 mars kernel: [ 39.710439] [kthread+0/240] kthread+0x0/0xf0
Dec 19 06:09:02 mars kernel: [ 39.710447] [kernel_thread_helper+7/16] kernel_thread_helper+0x7/0x10
Dec 19 06:09:02 mars kernel: [ 39.710457] =======================
Dec 19 06:09:03 mars kernel: [ 40.714061] BUG: soft lockup detected on CPU#0!
Dec 19 06:09:03 mars kernel: [ 40.714082] [softlockup_tick+156/240] softlockup_tick+0x9c/0xf0
Dec 19 06:09:03 mars kernel: [ 40.714100] [update_process_times+51/128] update_process_times+0x33/0x80
Dec 19 06:09:03 mars kernel: [ 40.714112] [smp_apic_timer_interrupt+112/128] smp_apic_timer_interrupt+0x70/0x80
Dec 19 06:09:03 mars kernel: [ 40.714122] [apic_timer_interrupt+40/48] apic_timer_interrupt+0x28/0x30
Dec 19 06:09:03 mars kernel: [ 40.714134] [_spin_lock+10/16] _spin_lock+0xa/0x10
Dec 19 06:09:03 mars kernel: [ 40.714141] [get_super+25/176] get_super+0x19/0xb0
Dec 19 06:09:03 mars kernel: [ 40.714149] [__invalidate_device+23/80] __invalidate_device+0x17/0x50
Dec 19 06:09:03 mars kernel: [ 40.714157] [check_disk_change+55/128] check_disk_change+0x37/0x80
Dec 19 06:09:03 mars kernel: [ 40.714159] [class_device_get+14/32] class_device_get+0xe/0x20
Dec 19 06:09:03 mars kernel: [ 40.714169] [<f891463f>] sd_open+0xaf/0x150 [sd_mod]
Dec 19 06:09:03 mars kernel: [ 40.714175] [exact_match+0/16] exact_match+0x0/0x10
Dec 19 06:09:03 mars kernel: [ 40.714181] [blkdev_open+0/112] blkdev_open+0x0/0x70
Dec 19 06:09:03 mars kernel: [ 40.714185] [do_open+123/656] do_open+0x7b/0x290
Dec 19 06:09:03 mars kernel: [ 40.714195] [blkdev_open+0/112] blkdev_open+0x0/0x70
Dec 19 06:09:03 mars kernel: [ 40.714199] [blkdev_open+48/112] blkdev_open+0x30/0x70
Dec 19 06:09:03 mars kernel: [ 40.714205] [__dentry_open+186/448] __dentry_open+0xba/0x1c0
Dec 19 06:09:03 mars kernel: [ 40.714215] [nameidata_to_filp+53/64] nameidata_to_filp+0x35/0x40
Dec 19 06:09:03 ma...

[Brian Buchanan (brianbuchanan)]

I'm seeing the same BUG's here on a desktop with an NFS mounted /home. The system ends up freezeing around the time that X starts. (Probably due to the Bug: soft lockup dtected

I was able to successfully boot into 2.6.20-15.

I'm looking @ rolling back to replace linux-headers-2.6.20-16 2.6.20-16.32, linux-image-2.6.20-16-generic 2.6.20-16.32, smbclient 3.0.24-2ubuntu1.4, samba-common 3.0.24-2ubuntu1.4, libsmbclient 3.0.24-2ubuntu1.4 as these are the packages that updated this morning.

Here's a ubuntuforums,org thread on the issue:

http://ubuntuforums.org/showthread.php?p=3979308

[erlguta (gonzalomarcote)]

Me the same.
After last kernel upgrade in gutsy linux-image-2.6.22-14-generic (2.6.22-14.47) my NFSv4 mounted partitions does not works and freezing the systems if i try to mount.
Please test things before release upgrades. I have now several PC's without his NFS partitions.

[Georg Pichler (georgpichler)]

The same behavior here, with Feisty.
Trying to mount nfs4 partitions results in a complete system lockup.

[Paul Natsuo Kishimoto (khaeru)]

Confirming on gutsy after a 2007-12-19 autoupgrade to linux-image-2.6.22-14-generic.

I don't have any critical directories mounted via nfs4, so I simply see kern.log contents as in Alexandre's first comment and no mounted nfs4 shares. I can login to GNOME, but when I run:

$ sudo mount -a -t nfs4

from the GNOME terminal, the system freezes within a few seconds.

[drakesoft (powerschorsch21)]

/me too ... please fix it

[gpredrag (gavrilovic)]

Same here, with feisty 7.04.
Attaching syslog entries.

[gpredrag (gavrilovic)]

Sorry forgot to say, using linux-image-2.6.20-generic (2.6.20-16.33)

[Donal (donaljoconnor)]

I managed to revert to Version 2.6.22-14.46 using Synaptic and all is well again

Oddly one of the fixes in the update is "NFS: Fix the mount regression"

[Claude Boucher (bouchecl)]

The problem is a regression bug introduced by a patch to fix a security issue. It seems to affect people using NFSv4. See https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/164231/comments/15 for an explanation.

Reverting to 2.6.22-14.46 (which can be done with Synaptic, as Donal said) or turning off NFS shares is the best you can do until a new version of the kernel is released.

[AlexLG (alex-alexlg)]

Same problem here, hangup when I mount nfsv4 shares :(

[Claude Boucher (bouchecl)]

AlexLG:

Philip Walls compiled a new version of /lib/modules/2.6.22-14-generic/kernel/fs/nfs/nfs.ko which I have successfully tested on my installation.

If you need nfsv4 right now and you can't wait for an updated kernel, you can either recompile apply the patch and recompile your kernel or use the nfs.ko for 2.6.22-14.47 he compiled and made available in the thread (disclaiming the potential danger of running untrusted kernel code, as Philip said).

https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/164231/

Good luck.

Claude

[AlexLG (alex-alexlg)]

Works with the patched nfs.ko, thanks Claude

[Drew Woodard (drew-woodard)]

I was hit by this as well, fortunately I only did the kernel update on my desktop machine and not other machines I am responsible for.

Please don't take this as a flame but Canonical really needs to catch a problem like this in QA. If someone breaks their machine screwing around with the betas or running dozens of Compiz plugins or using beta proprietary drivers that is one thing. But rendering a machine with common hardware unbootable because it installed a security related kernel update from an official repo for the current stable version of the distro is not good.

Fortunately NFS is probably not used by a huge percentage of Ubuntu home users, but it's pretty common fundamental software in organizations using Linux. If someone had trusted the update enough to push it out to a computer lab with NFS mounted home directories, or servers using NFS shares it would cause quite a mess.

[Michael Szlapa (michael-itsolv)]

I can confirm the problem on feisty($ uname -r == 2.6.20-16-386), the NFS 4 shares are not being mounted, segmentation fault occurs:

Segmentation fault mount nfshost:/var/blabla /var/blabla -t nfs4 -o proto=tcp,port=2049,rsize=8192,wsize=8192

system also does not shutdown correctly.

Can someone from Ubuntu please provide workaround (rollback instructions) for each supported release (feisty, gutsy), while the bug is being worked on.
In case of Ubuntu 'Feisty' I see only one version of the package linux-image-2.6.20-16-generic in synaptic and on packages.ubuntu.com (version shown is 2.6.20-16.33) and I am not able to rollback to older version.

My understanding is that the latest version included non-critical fixes and that rolling back the kernel is a better/safer option then booting in old kernel version (2.6.20-15).

[Paul Natsuo Kishimoto (khaeru)]

@Michael: Use the fix suggested by Donal. To do this:

1. Open System > Administration > Synaptic and click on the Search button in the toolbar. Choose "Version" from the drop-down, and search for "2.6.20-16"
2. Click on the leftmost column header in the package list to show installed packages. I see linux-headers-_, linux-headers-_-generic, linux-image-_-generic and linux-libc-dev.
3. Select each installed package in turn and choose "Package > Force Version...". Choose the most recent, non-buggy version. On gutsy this meant ".46" instead of ".47". As Brian Buchanan suggests, on feisty it would be ".32" instead of ".33".
4. Click "Apply".

You will have to turn off unattended updates (System > Administration > Software Sources > Updates tab) so that the old packages are not automatically overwritten by the buggy ones. AFAIK the only difference between, e.g. 2.6.20-16.32 and 2.6.20-16.33 is the patch causing this error, so this shouldn't be terribly insecure.

[Michael Szlapa (michael-itsolv)]

The steps described by Paul above are not reproducible on Feisty based on experience and experience of other users on the ubuntuforums.com. I do not see version .32 you have described available in the repository. (I use Kubuntu but I have tried the Synaptics as well )

No other version then .33 is also shown in the web interface to ubuntu feisty: http://packages.ubuntu.com/cgi-bin/search_packages.pl?searchon=names&version=all&exact=1&keywords=linux-image-2.6.20-16-386

Here are the results:
$ sudo apt-get -s install linux-image-2.6.20-16-386=2.6.20-16.32
Password:
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '2.6.20-16.32' for 'linux-image-2.6.20-16-386' was not found

[erlguta (gonzalomarcote)]

I fixed it by now, with synaptic, has Donal says. Anyway i thing it is One "Important" bug. It think exactly the same as Drew says. NFS mounted directories is a common thing in Enterprise environments, and this kind of errors are very annoying. Maybe it is not important for newbies home users, but it can be a headache for one Sys admin with several pc's with NFSv4 mounted directories.

[erlguta (gonzalomarcote)]

It is now fixed with the new kernel 2.6.22-14.51 in gutsy and 2.6.20-16.34 in feisty.

https://bugs.launchpad.net/ubuntu/hardy/+source/linux/+bug/164231

Fixed.