[quantal] overlayfs over r/o NFS mount fails to overwrite existing files

Bug #1039402 reported by Reinhard Tartler on 2012-08-21
44
This bug affects 8 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Medium
Andy Whitcroft

Bug Description

This is a followup to bug #1038075

Testcase: (Kernel taken from http://people.canonical.com/~apw/lp1038075-quantal/)

$ uname -a
Linux faui49i 3.5.0-11-generic #11lp1038075v201208181342 SMP Sat Aug 18 12:56:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ modprobe overlayfs
$ mkdir /live/tmpfs /tmp/root
$ mount -t nfs -o ro 192.168.42.40:/srv/fai/nfsroot.quantal64 /mnt
$ stat /mnt/etc/resolv.conf
  File: `/mnt/etc/resolv.conf'
  Size: 206 Blocks: 8 IO Block: 1048576 regular file
Device: 1ch/28d Inode: 36044830 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-08-20 10:53:27.177775447 +0200
Modify: 2012-08-08 12:50:21.000000000 +0200
Change: 2012-08-16 04:06:12.317826670 +0200
 Birth: -
$ mount -t overlayfs -o noatime,lowerdir=/mnt,upperdir=/live/overlayfs overlayfs /live/root
$ stat /live/root/etc/resolv.conf
  File: `/live/root/etc/resolv.conf'
  Size: 206 Blocks: 8 IO Block: 1048576 regular file
Device: 1ch/28d Inode: 36044830 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-08-20 10:53:27.177775447 +0200
Modify: 2012-08-08 12:50:21.000000000 +0200
Change: 2012-08-16 04:06:12.317826670 +0200
 Birth: -
$ echo foo > /live/root/etc/resolv.conf
bash: /live/root/etc/resolv.conf: Permission denied

This permission error does not happen with a local filesystem but only with NFS.

However this seems to work:

$ rm /live/root/etc/resolv.conf
$ echo foo > /live/root/etc/resolv.conf
$ cat /live/root/etc/resolv.conf
foo
---
ApportVersion: 2.4-0ubuntu8
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D2', '/dev/snd/hwC0D3', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D2c', '/dev/snd/pcmC0D7p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDmesg:
 Error: command ['sh', '-c', 'dmesg | comm -13 --nocheck-order /var/log/dmesg -'] failed with exit code 1: comm: /var/log/dmesg: Permission denied
 dmesg: write failed: Broken pipe
DistroRelease: Ubuntu 12.10
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
MachineType: FUJITSU ESPRIMO P7935
Package: linux (not installed)
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: root=UUID=871b8166-1de3-42f9-845b-d3353e32beec ro quiet splash
ProcVersionSignature: Ubuntu 3.5.0-11.11lp1038075v201208181342-generic 3.5.2
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RfKill:

Tags: quantal
Uname: Linux 3.5.0-11-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: netgrp netgrp sbuild
WifiSyslog:

dmi.bios.date: 01/12/2010
dmi.bios.vendor: FUJITSU // Phoenix Technologies Ltd.
dmi.bios.version: 6.00 R1.20.2812.A2
dmi.board.name: D2812-A2
dmi.board.vendor: FUJITSU
dmi.board.version: S26361-D2812-A2
dmi.chassis.type: 6
dmi.chassis.vendor: FUJITSU
dmi.modalias: dmi:bvnFUJITSU//PhoenixTechnologiesLtd.:bvr6.00R1.20.2812.A2:bd01/12/2010:svnFUJITSU:pnESPRIMOP7935:pvr:rvnFUJITSU:rnD2812-A2:rvrS26361-D2812-A2:cvnFUJITSU:ct6:cvr:
dmi.product.name: ESPRIMO P7935
dmi.sys.vendor: FUJITSU

apport information

tags: added: apport-collected quantal
description: updated

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

description: updated

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1039402

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Reinhard Tartler (siretart) wrote :

Brad, that's exactly what I did half an hour ago. I assume that your message (#11) was automated and did not really notice the information that I have posted. Therefore, I'm setting the status back to 'confirmed'. If there is any information still missing, please do tell me!

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in linux (Ubuntu):
importance: Undecided → Medium
tags: added: bot-stop-nagging
Andy Whitcroft (apw) wrote :

This seems to be related to some of the inode information being invalid in the overlayfs inodes:

    $ ls -l
    ls: cannot access yyy: No such file or directory
    total 4
    -rw-rw-r-- 1 mru mru 6 Aug 17 16:56 xxx
    -????????? ? ? ? ? ? yyy

Investigating.

Changed in linux (Ubuntu):
status: Confirmed → In Progress
assignee: nobody → Andy Whitcroft (apw)
Andy Whitcroft (apw) wrote :

Ok think I have found the underlying issue here. It seems its related to a lack of xattr support in the lower filesystem throwing out the copyup operations. I have hopefully fixed this up, could you test the kernels below (pick the newer dated one) and report back here:

    http://people.canonical.com/~apw/lp1038075-quantal/

Thanks!

Reinhard Tartler (siretart) wrote :

Unfortunately, the new kernel doesn't solve the symptoms I'm experiencing. See the attached screenshot.

Reinhard Tartler (siretart) wrote :

Summary from irc.

The proposed patch does fix the symptoms if the NFS mount is mounted with locking enabled (option 'lock', which is enabled by default and always with nfsv4).

However, live-boot mounts the nfs with "-o nolock" (and on our site with nfsv3, which can be forced with the mount option "-o vers=3") to not require a locking daemon. In this scenario, the symptom persists.

Jason Gerard DeRose (jderose) wrote :

I can confirm the same behavior with NFSv3 and overlayfs on Raring.

Michał Sawicz (saviq) wrote :

Can we please get an update on this? @apw, is this actually in progress?

Ryan Tandy (rtandy) wrote :

This still affects linux 3.10.0-2.9 in saucy. In September apw posted a pair of patches related to this bug [1] but they don't seem to have made it past the RFC stage and neither one has been integrated (AFAICT) in the current overlayfs tree [2]. I couldn't find any more recent conversations about this.

[1] https://lkml.org/lkml/2012/9/6/378
[2] https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers