[quantal] overlayfs over r/o NFS mount triggers OOPS

Bug #1038075 reported by Reinhard Tartler
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Andy Whitcroft
Precise
Won't Fix
Undecided
Unassigned

Bug Description

Scenario: booting from a read-only nfsroot share by using a tmpfs overlay

This is used e.g. by the fai package, which uses the live-boot package to create such an initramfs.

Test Case:

modprobe overlayfs
mkdir /tmp/live /tmp/root
mount -t nfs -o ro 192.168.42.40:/srv/fai/nfsroot.quantal64 /mnt
mount -t overlayfs -o noatime,lowerdir=/mnt,upperdir=/tmp/live overlayfs /tmp/root
# the mount command succeeds! - However using the mount point breaks:
# find /tmp/root
/tmp/root
Killed

Dmesg shows this kernel trace:
Aug 17 14:58:04 faui49i kernel: [ 1071.305101] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
Aug 17 14:58:04 faui49i kernel: [ 1071.306733] IP: [<ffffffffa01a2a71>] nfs_lookup_revalidate+0x21/0x3a0 [nfs]
Aug 17 14:58:04 faui49i kernel: [ 1071.308449] PGD c5697067 PUD c570c067 PMD 0
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] Oops: 0000 [#1] SMP
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] CPU 1
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] Modules linked in: overlayfs autofs4 bnep rfcomm bluetooth lp snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec i915 snd_hwdep snd_pcm drm_kms_helper snd_seq_midi drm coretemp snd_rawmidi snd_seq_midi_event snd_seq snd_timer kvm_intel snd_seq_device i2c_algo_bit kvm snd hid_generic tpm_infineon soundcore snd_page_alloc mei lpc_ich video microcode tpm_tis mac_hid ppdev parport_pc parport serio_raw nfsd nfs lockd fscache auth_rpcgss nfs_acl binfmt_misc sunrpc psmouse usbhid hid usbkbd raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usb_storage e1000e floppy raid6_pq async_tx raid1 raid0 multipath linear
Aug 17 14:58:04 faui49i kernel: [ 1071.309061]
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] Pid: 3543, comm: find Not tainted 3.5.0-10-generic #10-Ubuntu FUJITSU ESPRIMO P7935 /D2812-A2
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] RIP: 0010:[<ffffffffa01a2a71>] [<ffffffffa01a2a71>] nfs_lookup_revalidate+0x21/0x3a0 [nfs]
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] RSP: 0018:ffff8800c5773b48 EFLAGS: 00010286
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] RAX: ffffffffa01cf580 RBX: ffff8800c5dc4b40 RCX: 0000000000000020
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] RDX: ffff8800c5dc4038 RSI: 0000000000000000 RDI: ffff8800c5dc4b40
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] RBP: ffff8800c5773b88 R08: 656c6f006576696c R09: 0000000000000000
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] R10: ffff8800c5dc4b40 R11: ffffffffa01b5aa0 R12: ffff8800cb6cc600
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] R13: ffff8800c5773bd8 R14: 0000000000000000 R15: 0000000000000000
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] FS: 00007f42c1f2d700(0000) GS:ffff88010dc80000(0000) knlGS:0000000000000000
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] CR2: 0000000000000038 CR3: 00000000cb26b000 CR4: 00000000000407e0
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] Process find (pid: 3543, threadinfo ffff8800c5772000, task ffff8800cb340000)
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] Stack:
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] 0000000000000000 0000000000000000 ffff8800c5773b88 ffff8800c5dc4b40
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] ffff8800cb6cc600 ffff8800c5773bd8 0000000000000000 0000000000000000
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] ffff8800c5773bc8 ffffffff8118c79c 0000000000000000 ffff8800c5dc403c
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] Call Trace:
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8118c79c>] __lookup_hash+0xac/0x120
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8118d606>] lookup_one_len+0xd6/0x110
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffffa0498b17>] ovl_lookup+0x187/0x3d0 [overlayfs]
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff816808ee>] ? _raw_spin_lock+0xe/0x20
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8118c751>] __lookup_hash+0x61/0x120
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8118eaf9>] ? lookup_fast+0x219/0x310
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8167678e>] lookup_slow+0x47/0xab
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff81190d48>] path_lookupat+0x6f8/0x720
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffffa01a2692>] ? nfs_readdir+0x322/0x510 [nfs]
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8116dd81>] ? kmem_cache_alloc+0x31/0x130
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff81190da1>] do_path_lookup+0x31/0xc0
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8118c9a3>] ? getname_flags+0x53/0xf0
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8119190d>] user_path_at_empty+0x5d/0xa0
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff811a035f>] ? mntput_no_expire+0x10f/0x160
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff811a03d4>] ? mntput+0x24/0x40
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8116e04c>] ? kfree+0x2c/0x110
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff81191961>] user_path_at+0x11/0x20
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff81186a75>] vfs_fstatat+0x35/0x70
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff811a03d4>] ? mntput+0x24/0x40
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff811832f2>] ? fput+0x1a2/0x260
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff81186d9a>] sys_newfstatat+0x1a/0x40
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8117f856>] ? filp_close+0x66/0xa0
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff8117fcfe>] ? sys_close+0x9e/0x100
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] [<ffffffff81688b69>] system_call_fastpath+0x16/0x1b
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] Code: ff ff 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 40 48 89 5d d8 4c 89 65 e0 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 66 66 66 66 90 <f6> 46 38 40 b8 f6 ff ff ff 49 89 fd 49 89 f7 0f 85 c4 00 00 00
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] RIP [<ffffffffa01a2a71>] nfs_lookup_revalidate+0x21/0x3a0 [nfs]
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] RSP <ffff8800c5773b48>
Aug 17 14:58:04 faui49i kernel: [ 1071.309061] CR2: 0000000000000038
Aug 17 14:58:04 faui49i kernel: [ 1071.524837] ---[ end trace ec832cdfc17d33d8 ]---
---
ApportVersion: 2.4-0ubuntu6
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D2', '/dev/snd/hwC0D3', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D2c', '/dev/snd/pcmC0D7p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDmesg:
 Error: command ['sh', '-c', 'dmesg | comm -13 --nocheck-order /var/log/dmesg -'] failed with exit code 1: comm: /var/log/dmesg: Permission denied
 dmesg: write failed: Broken pipe
DistroRelease: Ubuntu 12.10
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
MachineType: FUJITSU ESPRIMO P7935
Package: linux (not installed)
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: root=UUID=871b8166-1de3-42f9-845b-d3353e32beec ro quiet splash
ProcVersionSignature: Ubuntu 3.5.0-10.10-generic 3.5.1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RfKill:

Tags: quantal
Uname: Linux 3.5.0-10-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: netgrp netgrp sbuild
WifiSyslog:

dmi.bios.date: 01/12/2010
dmi.bios.vendor: FUJITSU // Phoenix Technologies Ltd.
dmi.bios.version: 6.00 R1.20.2812.A2
dmi.board.name: D2812-A2
dmi.board.vendor: FUJITSU
dmi.board.version: S26361-D2812-A2
dmi.chassis.type: 6
dmi.chassis.vendor: FUJITSU
dmi.modalias: dmi:bvnFUJITSU//PhoenixTechnologiesLtd.:bvr6.00R1.20.2812.A2:bd01/12/2010:svnFUJITSU:pnESPRIMOP7935:pvr:rvnFUJITSU:rnD2812-A2:rvrS26361-D2812-A2:cvnFUJITSU:ct6:cvr:
dmi.product.name: ESPRIMO P7935
dmi.sys.vendor: FUJITSU

Revision history for this message
Reinhard Tartler (siretart) wrote : AcpiTables.txt

apport information

tags: added: apport-collected quantal
description: updated
Revision history for this message
Reinhard Tartler (siretart) wrote : AlsaInfo.txt

apport information

Revision history for this message
Reinhard Tartler (siretart) wrote : Lspci.txt

apport information

Revision history for this message
Reinhard Tartler (siretart) wrote : Lsusb.txt

apport information

Revision history for this message
Reinhard Tartler (siretart) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Reinhard Tartler (siretart) wrote : ProcEnviron.txt

apport information

Revision history for this message
Reinhard Tartler (siretart) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Reinhard Tartler (siretart) wrote : ProcModules.txt

apport information

Revision history for this message
Reinhard Tartler (siretart) wrote : UdevDb.txt

apport information

Revision history for this message
Reinhard Tartler (siretart) wrote : UdevLog.txt

apport information

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1038075

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Andy Whitcroft (apw)
summary: - [Overlayfs] kernel OOPS with NFS
+ [quantal] overlayfs over r/o NFS mount triggers OOPS
Revision history for this message
Reinhard Tartler (siretart) wrote :

Brad, that's exactly what I did half an hour ago. I assume that your message (#11) was automated and did not really notice the information that I have posted. Therefore, I'm setting the status back to 'confirmed'. If there is any information still missing, please do tell me!

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
status: Confirmed → In Progress
assignee: nobody → Andy Whitcroft (apw)
Revision history for this message
Andy Whitcroft (apw) wrote :

Confirmed here as well in my test rig.

Revision history for this message
Andy Whitcroft (apw) wrote :

Ok I think I have found this, a lack of validation in the RCU dentry walk handling. Unfortuanatly upstream has refactored this code massivly so we cannot use their fix. Will look at fixing this.

Revision history for this message
Andy Whitcroft (apw) wrote :

Ok I think I a fix at least it seems to work in my test rig, could you test the kernel below and see if it resolves the issues you are seeing as well. Please report any testing on this bug, thanks:

    http://people.canonical.com/~apw/lp1038075-quantal/

Revision history for this message
Reinhard Tartler (siretart) wrote :

Andy, the new kernel seems to work better, but I've found a new bug. It seems that overwriting existing files with 'echo foo > /etc/resolv.conf' returns permission denied. However, unlinking it before with 'rm /etc/resolv.conf' makes that command work afterwards.

This used to work with aufs and breaks the fai package.

Do you want me to file a new bugreport for this?

Changed in linux (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Andy Whitcroft (apw) wrote :

@Reinhard -- i think a new bug yes, please mention it in here.

Revision history for this message
Reinhard Tartler (siretart) wrote :

@andy: Sure, filed as Bug #1039402

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 3.5.0-14.15

---------------
linux (3.5.0-14.15) quantal-proposed; urgency=low

  [ Andy Whitcroft ]

  * SAUCE: fs: d_revalidate methods may be passed a NULL nameidata
    - LP: #1038075

  [ Dave Airlie ]

  * SAUCE: drm/vmwgfx: add MODULE_DEVICE_TABLE so vmwgfx loads at boot
    - LP: #1039157

  [ Ike Panhc ]

  * [Config] Enable CONFIG_DEVPTS_MULTIPLE_INSTANCES for highbank
    - LP: #1038259

  [ Tim Gardner ]

  * SAUCE: wlcore: Declare MODULE_FIRMWARE usage
    - LP: #1042918

  [ Upstream Kernel Changes ]

  * asus-nb-wmi: add some video toggle keys
    - LP: #1022427
  * [media] uvcvideo: Fix frame drop in bulk video stream
  * [media] uvcvideo: Fix alternate setting selection
  * Input: wacom - add support to Cintiq 22HD
    - LP: #1043733
  * ALSA: HDA: Create phantom jacks for fixed inputs and outputs
  * ALSA: HDA: Support single 3-pin jack without VREF on the actual pin
    - LP: #1018262
  * ALSA: hda - give 3-pin jack the name "Headphone Mic Jack"
  * ALSA: hda - Do not set GPIOs for speakers on IDT if there are no
    speakers
    - LP: #1040077
  * ALSA: hda - Fix pop noise in headphones on S3 for Asus X55A, X55V
    - LP: #1034779
  * ALSA: hda - Always call standard unsolicited event for Realtek codecs
    - LP: #1021192
  * ALSA: hda - Add the inverted digital mic workaround to Realtek codecs
  * ALSA: hda - Add inverted mic quirks for Asus U41SV, Acer 1810TZ and
    AOD260
    - LP: #1006089, #996611, #997227
  * ALSA: hda - don't create dysfunctional mixer controls for ca0132
    - LP: #1038651
  * ALSA: hda - Don't send invalid volume knob command on IDT 92hd75bxx
 -- Leann Ogasawara <email address hidden> Thu, 06 Sep 2012 10:06:28 -0700

Changed in linux (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Dominic Gross (domgross) wrote :

This also affects precise. We are currently using aufs to create a tmpfs overlay when booting from a read-only nfs root fs.
We planned to migrate to overlayfs because of some quirks with aufs. Result in the same problem discussed here. Should I open a separate bug report?

Revision history for this message
Dominic Gross (domgross) wrote :

Just a quick note, installing the current quantal Kernel in precise fixes this and (as expected) breaks a couple of other things.

Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Steve Langasek (vorlon) wrote :

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release

Changed in linux (Ubuntu Precise):
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.